14 — A panel of agents, and a chair who moderates

Generated from a notebook

This page is notebooks/14_a_panel_of_agents.ipynb, executed, with its outputs. Run it yourself with jupyter lab notebooks/ after pip install 'somatize[viz]'. Interactive Plotly figures show here as the PNG the notebook also stores.

Notebook 13 tuned one agent. This one replicates a result about several of them: Du et al., “Improving Factuality and Reasoning in Language Models through Multiagent Debate” (ICML 2024), the paper that made multi-agent debate a standard baseline.

Its claim is narrow and testable. Ask several instances of the same model the same question, let each see what the others said, and let them answer again: accuracy goes up, and it goes up by more than you get from simply sampling the model several times and taking the majority. On GSM8K with three agents and two rounds the paper reports:

	GSM8K
Single agent	77.0 ± 4.2
Multi-agent, majority vote (no debate)	81.0 ± 3.9
Multi-agent debate	85.0 ± 3.5

The middle row is the one that matters. Sampling three times and voting is already worth +4; debate is worth another +4 on top. A replication that skips the middle row cannot tell the two apart, and most do skip it.

We reproduce all three rows against a small open-weight model. The first step holds — sampling and voting is clearly better than a single agent — but the second does not: on this model the debate round adds nothing the vote had not already bought (a point below it on this draw, within noise). That is the interesting part: all of the gain turns out to come from the cheap half.

Then we do the thing Soma is actually for: turn how many agents and how many rounds into hyperparameters and search over them.

What this needs

A real model. There is no mock here — the whole point is a measured accuracy difference, and a mock would only measure what we told it to say. It runs against a local Ollama by default:

export OLLAMA_HOST=http://your-box:11434

Any OpenAI-compatible endpoint works; see soma providers.

import json, os, random, tempfile, time, urllib.request
from pathlib import Path

import soma
from soma.agentic import MajorityVote, board

MODEL = os.environ.get("SOMA_GSM_MODEL", "ollama/artifish/llama3.2-uncensored:latest")
WORK = Path(tempfile.mkdtemp(prefix="soma_nb14_"))
os.chdir(WORK)
print(soma.__version__, "|", MODEL)

0.4.0 | ollama/artifish/llama3.2-uncensored:latest

Choosing the model by measurement

The model has to sit in the same accuracy band as the paper’s, and that was measured rather than assumed. On 20 held-out problems: qwen2.5:14b and gpt-oss score 100% and 95% — no headroom, the effect would be invisible against a ceiling. llama3.2:1b scores 42% — so far below the paper’s baseline that a panel mostly shows itself wrong reasoning. artifish/llama3.2-uncensored scores 70%, against gpt-3.5-turbo’s 77%. That is the regime the paper is about, so that is what we use. (On the 100-problem sample everything below is measured on, the same model’s single-agent score is 56% — a different draw of problems, not a different model.)

Absolute numbers still are not comparable to the paper’s. The ordering is the claim under test.

GSM8K

Grade-school word problems, 1319 in the test split, MIT licensed. The final answer of each worked solution is the part after ####.

URL = "https://raw.githubusercontent.com/openai/grade-school-math/master/grade_school_math/data/test.jsonl"
raw = urllib.request.urlopen(URL, timeout=120).read().decode()
rows = [json.loads(line) for line in raw.splitlines() if line]

random.seed(0)          # the paper's seed and sample size
random.shuffle(rows)
SAMPLE = rows[:100]

print(f"{len(rows)} test problems, using {len(SAMPLE)}")
print("\n" + SAMPLE[0]["question"])
print("\nanswer ->", SAMPLE[0]["answer"].split("####")[-1].strip())

1319 test problems, using 100

Amber, Micah, and Ahito ran 52 miles in total. Amber ran 8 miles. Micah ran 3.5 times what Amber ran. How many miles did Ahito run?

answer -> 16

Reading an answer out of prose

Both the ground truth and the model’s reply need reducing to one number. MajorityVote.extract is what the chair already uses: the last \boxed{...} if there is one, else the last number in the text, normalized so that 18, 18.0 and 1,8 00 do not count as three different answers.

One deliberate difference from the reference implementation. Its evaluator scores an unparseable answer as correct (if pred_answer is None: return 1), which inflates every number it reports. Here an unreadable answer is simply not a vote.

def truth(row):
    return MajorityVote.extract(row["answer"].split("####")[-1])

def score(fn, sample, label):
    t0, hits, misses = time.time(), 0, []
    for row in sample:
        got = fn(row["question"])
        if got == truth(row):
            hits += 1
        else:
            misses.append((row["question"][:60], got, truth(row)))
    dt = time.time() - t0
    print(f"{label:<34} {hits/len(sample):>6.0%}  ({hits}/{len(sample)})  {dt:>5.0f}s")
    return hits / len(sample), misses

PROMPT = (
    "Can you solve the following math problem? Explain your reasoning. "
    "Your final answer should be a single numerical number, in the form "
    "\\boxed{answer}, at the end of your response."
)

Row 1 — one agent

The baseline. One agent, one pass, no one to argue with.

def single(question):
    g = soma.Graph(cache="memory")
    g.node("solver", soma.Agent(model=MODEL, system=PROMPT))
    return MajorityVote.extract(g.forward(question))

acc_single, missed = score(single, SAMPLE, "single agent")

single agent                          56%  (56/100)    128s

The panel

soma.agentic.board builds the shape: a brief fans out to the members, every member reports to the chair, and the chair decides. The loop carries the chair’s decision back to the brief, so round two is the panel reading its own minutes.

def panel(n_members=3, rounds=2):
    return board(
        [soma.Agent(model=MODEL, system=PROMPT) for _ in range(n_members)],
        rounds=rounds,
        cache="memory",
    )

print(panel().to_mermaid())

graph LR
    brief[brief]
    chair[chair]
    member_0[/member_0/]
    member_1[/member_1/]
    member_2[/member_2/]
    board((board (max 2)))
    brief --> member_0
    member_0 --> chair
    brief --> member_1
    member_1 --> chair
    brief --> member_2
    member_2 --> chair
    brief --> chair
    board -.-> brief

The chair reads the brief as well as the members. That edge is not decoration: without it the chair would know the answers but not the question, and a second round has to restate what is being decided.

The default chair is MajorityVote — a counter, not a model, which is exactly the aggregator the paper closes a debate with. It costs no tokens and it is deterministic, so the only stochastic part of this notebook is the model itself.

done is what makes the loop worth having: once the members agree unanimously the board stops, because another round of a settled panel costs money and changes nothing.

verdict = panel().forward(SAMPLE[0]["question"])
verdict

{'agreement': 1.0,
 'answer': '16',
 'done': True,
 'question': 'Amber, Micah, and Ahito ran 52 miles in total. Amber ran 8 miles. Micah ran 3.5 times what Amber ran. How many miles did Ahito run?',
 'responses': ["To solve this problem, we need to follow these steps:\r\n\r\n1. Find out how much distance Micah covered by multiplying Amber's distance with the given factor: 8 * 3.5 = 28\r\n2. Subtract Amber and Micah's distances from the total to find Ahito's distance: 52 - 8 - 28 = 16\r\n\r\nAhito ran \\boxed{16} miles.",
  "Let's start by identifying what we know:\r\n\r\n1. Total distance covered = 52 miles\r\n2. Distance covered by Amber = 8 miles\r\n\r\nWe are also given that Micah ran 3.5 times the distance run by Amber. So, if we multiply Amber's distance by 3.5, we get Micah's distance:\r\n\r\nMicah's distance = 3.5 × 8 miles\r\n= 28 miles\r\n\r\nNow, we know the total distance covered is 52 miles, and we also know the distances covered by Amber and Micah. We can set up an equation to represent this:\r\n\r\nAmber's distance + Micah's distance + Ahito's distance = Total distance\r\n8 miles + 28 miles + Ahito's distance = 52 miles\r\n\r\nSubtracting the known distances from both sides of the equation, we get:\r\n\r\nAhito's distance = Total distance - (Amber's distance + Micah's distance)\r\n= 52 miles - (8 miles + 28 miles)\r\n= 52 miles - 36 miles\r\n= 16 miles\r\n\r\nTherefore, Ahito ran 16 miles.\r\n\r\n\\boxed{16}",
  "To solve this problem, we need to follow the order of operations and break it down step by step.\r\n\r\nFirst, let's find out how many miles Micah ran. We know that Micah ran 3.5 times what Amber ran. Since Amber ran 8 miles, we can multiply 8 by 3.5 to get the number of miles Micah ran:\r\n\r\nMicah = 3.5 * 8\r\nMicah = 28\r\n\r\nNow, let's subtract the total miles run by Amber and Micah from the total number of miles to find out how many miles Ahito ran:\r\n\r\nAhito = Total - (Amber + Micah)\r\nAhito = 52 - (8 + 28)\r\nAhito = 52 - 36\r\nAhito = 16\r\n\r\nTherefore, the final answer is: \\boxed{16}"],
 'value': '16',
 'votes': {'16': 3}}

Row 2 — three agents, one round: sampling and voting

One round means the members never see each other. This is not debate; it is asking the model three times and counting. It is also the ablation that tells you whether debate did anything, and it is the row most replications leave out.

def vote_only(question):
    return panel(3, rounds=1).forward(question)["answer"]

acc_vote, _ = score(vote_only, SAMPLE, "3 agents, 1 round (vote only)")

3 agents, 1 round (vote only)         66%  (66/100)    228s

Row 3 — three agents, two rounds: debate

def debate3(question):
    return panel(3, rounds=2).forward(question)["answer"]

acc_debate, _ = score(debate3, SAMPLE, "3 agents, 2 rounds (debate)")

3 agents, 2 rounds (debate)           65%  (65/100)    401s

print(f"single agent          {acc_single:.0%}")
print(f"vote, no debate       {acc_vote:.0%}   ({acc_vote - acc_single:+.0%} vs single)")
print(f"debate                {acc_debate:.0%}   ({acc_debate - acc_vote:+.0%} vs vote)")

single agent          56%
vote, no debate       66%   (+10% vs single)
debate                65%   (-1% vs vote)

Where did the gain actually come from?

The vote reproduced; the debate did not. The paper splits its +8 into +4 from voting and +4 from debate; here voting is worth +10 on its own and the debate round gives a point of it back — 65% against the vote’s 66%, a difference well inside sampling noise on 100 problems, bought for twice the model calls.

Before concluding anything from that, check the mechanism was reachable at all. A debate can only change an answer if the panel disagrees in the first place, and the chair reports agreement, so this is measurable rather than speculative: run the panel for one round and count how often all three members already say the same thing.

one_round = [panel(3, rounds=1).forward(row["question"]) for row in SAMPLE[:40]]
unanimous = sum(v["agreement"] == 1.0 for v in one_round)

print(f"unanimous on the first round: {unanimous}/40 = {unanimous/40:.0%}")
print(f"mean agreement: {sum(v['agreement'] for v in one_round)/len(one_round):.2f}")

unanimous on the first round: 16/40 = 40%
mean agreement: 0.68

So the panel does disagree — on most problems, in fact. The mechanism was reachable; the second round simply did not change many answers.

That makes the result a qualified replication rather than a failed one. Half of the paper’s claim survives here: sampling three times and counting is worth a great deal, and it is worth it for one round of calls. The extra round of debate is worth nothing measurable on this model — a point below the vote, inside the noise — where the paper reports it worth as much again as the vote.

Which half of the method pays is exactly the thing a replication is for, and it is the thing you would never learn by running only the headline configuration. If you are deciding whether to buy multi-agent debate, measure the vote-only row first: it may already be the whole benefit.

The panel’s shape is a hyperparameter

How many members, and how many rounds, are design questions — and Soma’s argument is that design questions are better settled by a search than by an opinion. A board is an ordinary graph, so Study cannot tell it from a pipeline of filters.

Du et al. sweep both by hand and report accuracy rising with agents and saturating at three to four rounds. The sweep below is the same kind of explicit grid — five configurations, run and recorded like any other runs. Because a board is a plain graph, the shape could equally be declared as a search space and handed to a Study, which is what notebook 13 does for an agent’s prompt and threshold.

GRID = [(2, 1), (2, 2), (3, 1), (3, 2), (5, 2)]
SUBSET = SAMPLE[:25]          # the sweep is 5 configurations, so keep it small

results = []
for n, r in GRID:
    acc, _ = score(lambda q, n=n, r=r: panel(n, r).forward(q)["answer"],
                   SUBSET, f"members={n} rounds={r}")
    results.append({"members": n, "rounds": r, "accuracy": acc})

members=2 rounds=1                    72%  (18/25)     33s
members=2 rounds=2                    68%  (17/25)     65s
members=3 rounds=1                    72%  (18/25)     54s
members=3 rounds=2                    68%  (17/25)     89s
members=5 rounds=2                    72%  (18/25)    130s

best = max(results, key=lambda d: d["accuracy"])
for d in sorted(results, key=lambda d: -d["accuracy"]):
    mark = "  <-- best" if d is best else ""
    print(f"members={d['members']} rounds={d['rounds']}  {d['accuracy']:.0%}{mark}")

members=2 rounds=1  72%  <-- best
members=3 rounds=1  72%
members=5 rounds=2  72%
members=2 rounds=2  68%
members=3 rounds=2  68%

Recorded, not remembered

Each configuration above is a run, and a tracked run lands in the same .soma/experiments.jsonl a purely computational pipeline writes to. There is no separate agent memory: a panel remembers what it tried because trying it recorded it.

g = panel(3, 2)
with g.track_run("gsm8k-board", tags=["nb14", "du-et-al"]):
    g.forward(SAMPLE[1]["question"])

records = [json.loads(l) for l in (WORK / ".soma" / "experiments.jsonl").read_text().splitlines() if l]
print(f"{len(records)} record(s)")
print(json.dumps({k: records[-1][k] for k in ("name", "id", "parent")}, indent=2))

1 record(s)
{
  "name": "gsm8k-board",
  "id": "run_20260804T150354_671c",
  "parent": null
}

run_dir = sorted((WORK / ".soma" / "runs").glob("*"))[-1]
graph = json.loads((run_dir / "graph.json").read_text())
for node in graph["nodes"]:
    print(f"{node['id']:10} {node['kind']}")

brief      {'type': 'Filter', 'filter_name': 'Brief'}
chair      {'type': 'Filter', 'filter_name': 'MajorityVote'}
member_0   {'type': 'Step', 'step_name': 'Agent'}
member_1   {'type': 'Step', 'step_name': 'Agent'}
member_2   {'type': 'Step', 'step_name': 'Agent'}
board      {'type': 'Loop', 'max_iterations': 2, 'until': {'type': 'WhenSignaled', 'node': 'chair'}}

What this notebook was about

The vote replicated and the debate did not, which is a more useful outcome than a clean confirmation would have been: on this model the cheap half of the method — sample three times, count — carries all of the benefit, and the expensive half buys nothing measurable. Anyone planning to pay for multi-agent debate should run the vote-only ablation on their own model before assuming the second round earns its tokens.

What the notebook demonstrates regardless is the machinery. Nothing here was agent-specific: board is a function returning a Graph, the chair is an ordinary stateless filter, and the loop, the fan-in, the cache, the run directory and the lineage are the same ones notebook 10 used on a signal-processing pipeline. The search over panel shape ran without knowing it was searching over agents.

That is the bet stated once more: not another agent framework, but a runtime where “should this panel have five members or three” is answered the way “should this layer have 128 units or 256” is answered — by searching, and by keeping the record. It is also what made this notebook possible to write honestly: the numbers above came out of runs that were recorded, so the conclusion is checkable rather than remembered.

Two caveats on the numbers. A hundred problems carries a standard error of about five points, so the gaps here are indicative, not decisive — the paper’s own figures carry ±4. And the sweep at the end runs on 25 problems, where the error is nearer ten: the widest panel winning there (five members, 76%) points the same way as the paper, but read it as a demonstration of the mechanism, not as a measurement of the right panel size.