AI8/10
Experiment Finds Market Coordination Beats Manager Models for AI Agents
Rohit reports an experiment comparing solo, hub-and-spoke and market-based organization of multiple AI models. He finds the manager-subagent setup cost four times more and performed worse than a simple bidding market, especially on reasoning and synthesis tasks, and links to a longer essay.
Original post · 2 min read
🚨 New Experiment: Everyone thinks AI firms will look like little companies. A manager model decomposes the task and worker models do subtasks. The manager red-teams, revises, and recombines. A seemingly simple org chart.
But when I ran the experiment, the current in-vogue org setup, manager-subagent, cost 4x more and performed worse than letting a rather simple market do the trick.
I tested 3 ways to organize multiple AI models:
1. Solo: Onefrontier model does everything itself
2. Hub-Spoke: A "manager" model splits tasks, delegates, red-teams, revises
3. Market: Models bid on tasks, winner gets the job, reputation updates
I also tested were 3 types of tasks - Coding, Reasoning and Synthesis.
- Coding required most "global state" management, which the solo model did best at. In future @a1zhang's RLM will probably do even better here
- Reasoning is the hardest to cleanly decompose, and the market worked the best here
- Synthesis too, the market beat hub-spoke as the framing could be ambiguous
The reason is, a hub isn't a "manager" as we know it. It's a model that must somehow know:
- What the subtasks are
- What good recomposition looks like
And if either fails, as it does for complex or not-easily-decomposable tasks, competent workers still produce garbage.
As we move from coding to letting multi-agent systems do work across the entire economy we'll end up with more not-easily-verifiable tasks with ambiguous settings and uncertain payoffs. In those, we won't be able to use the factory approach to get work done.
The Coasean argument is that firms will get smaller, and the smaller firms will transact more, since the organisational premium reduces with AI. But how? Through central hubs, or markets? The fact is, Coase here needs Hayek. Setting up markets is not trivial, as @AndreyFradkin and I looked in our recent paper.
Essay: strangeloopcanon.com/p/why-smart-planners-lose…
But when I ran the experiment, the current in-vogue org setup, manager-subagent, cost 4x more and performed worse than letting a rather simple market do the trick.
I tested 3 ways to organize multiple AI models:
1. Solo: Onefrontier model does everything itself
2. Hub-Spoke: A "manager" model splits tasks, delegates, red-teams, revises
3. Market: Models bid on tasks, winner gets the job, reputation updates
I also tested were 3 types of tasks - Coding, Reasoning and Synthesis.
- Coding required most "global state" management, which the solo model did best at. In future @a1zhang's RLM will probably do even better here
- Reasoning is the hardest to cleanly decompose, and the market worked the best here
- Synthesis too, the market beat hub-spoke as the framing could be ambiguous
The reason is, a hub isn't a "manager" as we know it. It's a model that must somehow know:
- What the subtasks are
- What good recomposition looks like
And if either fails, as it does for complex or not-easily-decomposable tasks, competent workers still produce garbage.
As we move from coding to letting multi-agent systems do work across the entire economy we'll end up with more not-easily-verifiable tasks with ambiguous settings and uncertain payoffs. In those, we won't be able to use the factory approach to get work done.
The Coasean argument is that firms will get smaller, and the smaller firms will transact more, since the organisational premium reduces with AI. But how? Through central hubs, or markets? The fact is, Coase here needs Hayek. Setting up markets is not trivial, as @AndreyFradkin and I looked in our recent paper.
Essay: strangeloopcanon.com/p/why-smart-planners-lose…
♥ 503 · ⟲ 55 · 👁 126.7KView on X ↗
