Decentralized Multi-Agent Systems with Shared Context

Yuzhen Mao, Azalia MirhoseiniStanford Universityyuzhenm,[email protected]

Abstract

Multi-agent systems (MAS) can scale large language model reasoning at test time by decomposing complex problems into parallel subtasks. However, most existing MAS rely on centralized orchestration, where a main agent assigns work, collects outputs, and merges results. As the number of subtasks grows, this controller becomes a communication and integration bottleneck. We propose Decentralized Language Models ($\textsc{DeLM}$), a MAS framework that decentralizes coordination through parallel agents, a shared verified context, and a task queue. Agents asynchronously claim subtasks, read accumulated progress, perform local reasoning, and write back compact verified updates. The shared context acts as a common communication substrate, enabling agents to build on one another’s verified progress without routing every update through a central controller. Empirically, $\textsc{DeLM}$ improves both software-engineering test-time scaling and long-context reasoning. On SWE-bench Verified, $\textsc{DeLM}$ achieves the best performance across Avg.@1, Pass@2, and Pass@4, with gains of up to 10.5 percentage points over the strongest baseline, while reducing cost per task by roughly 50%. On LongBench-v2 Multi-Doc QA, $\textsc{DeLM}$ achieves the highest average accuracy across four frontier model families, improving over the strongest baseline by up to 5.7 percentage points. The code is available on our project website at https://yuzhenmao.github.io/DeLM/.

Executive Summary: DELM is a new multi-agent framework designed to overcome coordination bottlenecks that arise in centralized systems when many language-model agents work on complex tasks in parallel. Most current multi-agent setups rely on a single main agent to assign work, collect results, and merge progress; this central controller becomes slower and less reliable as the number of subtasks or documents grows. The authors argue that shifting coordination to a shared, verified context and an asynchronous task queue lets agents build directly on one another’s reliable progress without repeated routing through one overloaded controller.

The document evaluates whether this decentralized design improves both software-engineering test-time scaling and long-context reasoning. The authors test DELM on SWE-bench Verified (real GitHub issues), LongBench-v2 Multi-Doc QA (125 multi-document questions), and OOLONG (structured aggregation tasks). They compare it with strong centralized baselines and an earlier gist-based method, using four frontier models and reporting accuracy, pass rates, and per-task cost.

On SWE-bench Verified with Gemini-3-Flash, DELM reaches 65.7 % Avg@1, 72.9 % Pass@2, and 77.4 % Pass@4—gains of up to 10.5 points over the best baseline—while cutting cost per task by roughly half to $0.12. On LongBench-v2 it improves average accuracy by 3.6–5.7 points across four model families. Trace analysis shows three mechanisms: failed hypotheses become reusable constraints, binding facts stay explicit, and compact patch summaries let later agents reuse discoveries without reading full transcripts. On OOLONG the hybrid of DELM with Recursive Language Models yields the highest accuracy and lowest cost, indicating the two approaches are complementary.

These results matter because they demonstrate that scalable multi-agent performance depends not only on more parallel calls but on reliable, low-overhead sharing of intermediate state. Organizations using agents for coding, document analysis, or automated research can expect both higher success rates and lower token spend once coordination moves from prompt routing to verified shared context.

The authors recommend deploying decentralized shared-state designs for test-time scaling workloads and exploring them in automated research systems. They note that verification adds modest overhead, that decomposition quality still depends on the underlying agents, and that results are based on three specific benchmarks; further validation on larger or more open-ended tasks would increase confidence.

1. Introduction

Section Summary: Multi-agent systems let large language models tackle complex tasks by dividing work among several agents that operate in parallel, which is useful for both test-time scaling on problems like software engineering and for long-context reasoning such as answering questions over many documents. Most current approaches rely on a central controller that assigns subtasks and gathers results, but this creates communication bottlenecks and risks losing or distorting useful information as the number of agents grows. The paper introduces Decentralized Language Models (DeLM), which replaces the central controller with a shared, verified context and a task queue so that agents can asynchronously claim work, read progress, and post compact updates that become immediately visible to everyone.

Multi-agent systems (MAS) offer a natural way to scale large language model reasoning at test time. Instead of relying on a single model invocation to solve a complex task end-to-end, MAS decompose the problem into subtasks, dispatch multiple agents in parallel, and aggregate their intermediate progress ([1, 2, 3, 4, 5, 6, 7]). This paradigm is increasingly important in two representative settings. The first is test-time scaling, where multiple agents can explore different hypotheses or pursue alternative reasoning paths in parallel while still sharing intermediate progress. The second is long-context reasoning, such as multi-document question answering, where agents can process different evidence clusters concurrently. In both cases, the value of MAS is not simply to issue more model calls, but to convert additional test-time computation into useful parallel progress. These two settings also capture core challenges in emerging automated research workflows ([8, 9]), where agents must both scale exploration across hypotheses and reason over large collections of papers, code, experiments, and intermediate findings.

Realizing this potential, however, depends critically on how agents coordinate. Most existing MAS frameworks, including Claude Code Subagents ([6]), Kimi Agent Swarm ([7]), and AOrchestra ([3]), rely on centralized orchestration: a main agent decomposes the problem, assigns subtasks and corresponding subcontexts to sub-agents, waits for their outputs, and then integrates the results or launches another round of subtasks, as illustrated in Figure 2 (left). The limitation is that centralized MAS parallelize sub-agent execution, but not the coordination around it. In test-time scaling, the benefit of additional agents depends on whether useful progress can be shared across workers efficiently and faithfully. Centralized orchestration weakens this benefit in two ways. First, it scales poorly: every useful finding, failure, or partial solution must return to the main agent, which then decides how to merge and broadcast that information to other sub-agents. As the number of agents grows, progress sharing becomes a serialized communication bottleneck. Second, during this routing process, the main agent may dilute, omit, or distort useful details, causing important progress to be lost. We provide a more detailed analysis in Section 4.2.1. This bottleneck also arises in long-context reasoning: the main agent must pre-assign evidence clusters to sub-agents, often before knowing which evidence is relevant or how different pieces should be combined ([6, 4]). If a sub-agent receives insufficient context, it return control to the main agent, triggering additional retrieval or another delegation round. As subtasks or evidence clusters grow, this back-and-forth makes coordination slower, more iterative, and increasingly constrained by a single overloaded main agent.

To address these bottlenecks, we propose Decentralized Language Models ($\textsc{DeLM}$), a MAS framework that shifts coordination from a central controller to a shared problem state. As shown in Figure 2 (right), $\textsc{DeLM}$ is built around three core components: parallel agents, a shared context, and a task queue. Rather than routing every interaction through a main agent, agents asynchronously draw tasks from the queue, read accumulated progress from the shared context, perform local reasoning, and write back compact verified updates. This design also differs from peer-to-peer communication, where agents exchange messages directly. Instead, $\textsc{DeLM}$ uses the verified shared context as a common communication substrate: once an update is admitted, it becomes visible to all agents as reusable problem state. As a result, $\textsc{DeLM}$ supports both motivating settings: in test-time scaling, useful findings, failures, and partial solutions can propagate through shared context rather than being repeatedly merged and rebroadcast by a main agent. In long-context reasoning, agents can process different evidence clusters concurrently while maintaining a compact global view of the corpus and accumulated evidence.

Key idea. Unlike existing MAS, which rely on a central controller and synchronous scatter–gather coordination, $\textsc{DeLM}$ lets agents coordinate asynchronously through a shared, verified context. Agents claim tasks from a queue and write back compact, verified results as they finish, making progress visible to all workers without requiring a main agent to merge, filter, and rebroadcast it.

We evaluate $\textsc{DeLM}$ in three settings that stress different forms of multi-agent coordination: (1) software-engineering test-time scaling on SWE-bench Verified ([10]), which stresses parallel exploration across different reasoning trajectories; (2) long-context multi-document QA on LongBench-v2 ([11]), which stresses within-task parallelism as agents concurrently inspect different evidence clusters; and (3) aggregation-heavy long-context reasoning on OOLONG ([12]), which highlights the complementarity between $\textsc{DeLM}$ ’s decentralized verified context and the code-mediated execution of RLM ([4]). Our key findings are:

• $\textsc{DeLM}$ converts parallel software-engineering attempts into shared exploration. On SWE-bench Verified (§ 4.2), as shown in Table 1, $\textsc{DeLM}$ achieves the strongest performance across test-time scaling metrics, reaching 77.4% pass@4 while reducing cost to \0.12 per task which is only roughly half of the baselines. Section 4.2.1 further explains these gains through trace-level examples showing how compact shared context helps agents reuse discoveries.
• $\textsc{DeLM}$ enables iterative shared-state reasoning over long contexts. On LongBench-v2 (§ 4.3), as shown in Table 2, $\textsc{DeLM}$ achieves the highest average accuracy across four frontier models, improving over the best baseline by up to 5.7 percentage points. Section 4.3.1 further shows that both admission-time verification and hierarchical summarization contribute to these gains.
• $\textsc{DeLM}$ acts as a coordination layer for programmatic reasoning systems. On OOLONG (§ 5), as shown in Table 3, vanilla $\textsc{DeLM}$ underperforms RLM because the benchmark requires exact row-level aggregation, where code-mediated execution is especially useful; however, combining RLM with $\textsc{DeLM}$ yields the best accuracy and lowest cost, showing that $\textsc{DeLM}$ extends beyond conversational agents to code-based reasoning workflows.

2. Motivation and Design Principles

Section Summary: The core limitation of current multi-agent systems is that they route all progress through a central controller, which rewrites intermediate results into new prompts and thereby prevents agents from truly working in parallel on shared information. DeLM replaces this with a persistent shared context in which agents record compact, verified updates that others can read directly, avoiding repeated prompt rewriting while still keeping the overall state manageable. To prevent errors from spreading, every update is checked against its source evidence before admission, and agents can expand from brief summaries to full details only when needed.

2.1 From Prompt Routing to Shared State

Section 1 identifies centralized coordination as a bottleneck in existing MAS. Here, we refine this observation by focusing on its communication mechanism. Centralized MAS are largely prompt-routed: intermediate progress is passed through the main agent, rewritten into subsequent prompts, and selectively exposed to other agents. Thus, parallel execution does not by itself guarantee parallel progress sharing.

This perspective motivates a different communication substrate. Instead of repeatedly encoding coordination decisions into prompts, $\textsc{DeLM}$ makes intermediate progress persistent: agents write compact, verified updates into a shared context that later agents can read directly. Coordination therefore becomes state-based, with useful findings, failures, and constraints accumulating as shared problem state rather than passing through a central controller.

2.2 Compact, Global, Unfoldable Shared Context

State-based communication is useful only if the shared context remains usable as the problem grows. Sharing raw documents or full agent traces preserves maximal information, but quickly overwhelms each agent's context window and increases cost. Sharing only compact summaries is cheaper, but risks losing details, qualifications, or cross-document evidence needed for reliable reasoning.

$\textsc{DeLM}$ addresses this trade-off with an unfolding mechanism. Agents read highly compact gists by default and selectively expand them into detailed summaries or raw evidence when needed. This coarse-to-fine access pattern enables coordination over the full problem while incurring detailed inspection costs only for relevant evidence.

2.3 Verified Before Admission

Because the shared context serves as the communication substrate for all agents, errors in this state can propagate widely. Once admitted, an unsupported claim may become reusable problem state, misleading later agents and shaping downstream reasoning. Post-hoc answer checking is insufficient, because such errors may have already influenced intermediate decisions.

$\textsc{DeLM}$ mitigates this risk through admission-time verification. Before an update is added to the shared context, it is checked against its underlying evidence, including source context and reasoning trajectories. Unsupported or distorted updates are rejected or regenerated, turning the shared context from a raw message buffer into curated shared state.

3. Decentralized Language Models ($\textsc{DeLM}$)

Section Summary: DeLM is a framework that coordinates multiple language-model agents to solve complex tasks by breaking them into subtasks that run in parallel. It relies on two shared structures: a task queue that holds pending work and a compact shared context that stores verified summaries of completed work. Agents asynchronously pick up subtasks, read the current shared summaries, produce results, and submit compressed, verified updates that become visible to everyone else, with new subtasks generated as needed until a final answer can be produced.

**Require:** Task D; optional source context U
**Ensure:** Final answer Y
C ← ∅ // shared context
T ← CallGenerateSubtasksD, U stage{1}
**repeat**
  $\{r_i\} \gets \operatorname{RunAgents}\left(\mathcal{T}, \mathcal{C}\right)$ stage[// execute subtasks in parallel]{2}
  $\{G_i\} \gets \operatorname{CompressAndVerify}\left(\{r_i\}\right)$ stage{3}
  C ← C ∪ (G(i)) // admit verified gists
  **if** $\mathcal{T}$ is empty **then**
    $\mathcal{T} \gets \operatorname{GenerateMoreSubtasks}\left(D, \mathcal{C}\right)$ stage{4}
  **end if**
**until** $\mathcal{T}$ is empty
$Y \gets \operatorname{Finalize}\left(D, \mathcal{C}\right)$ stage{5}
**return** Y

$\textsc{DeLM}$ implements these design principles through two global structures: a shared context $\mathcal{C}$ and a task queue $\mathcal{T}$. Given an input task $D$ and optional source context $U$, $\mathcal{C}$ stores compact, verified gists of accumulated progress, while $\mathcal{T}$ stores pending subtasks for parallel execution.

As shown in Figure 3 and Algorithm 1, $\textsc{DeLM}$ proceeds in five stages: stage1 initialize the task queue from the input, stage2 execute ready subtasks in parallel, stage3 compress, verify, and admit updates into the shared context, stage4 generate additional subtasks when the current shared context is insufficient, and stage5 produce the final answer once no further subtasks are needed. We next describe the two core mechanisms underlying this pipeline: Section 3.1 introduces the shared context and task queue, and Section 3.2 describes how intermediate updates are compressed, verified, and admitted.

3.1 Shared Context and Task Queue

$\textsc{DeLM}$ maintains two global structures: a shared context $\mathcal{C}$ and a task queue $\mathcal{T}$. The shared context stores compact, reusable problem state, including source evidence, completed subtask results, failed hypotheses, and intermediate constraints. The task queue stores pending subtasks that available agents can claim in parallel. Together, $\mathcal{C}$ and $\mathcal{T}$ define the coordination interface: agents choose work from $\mathcal{T}$ and communicate progress through admitted entries in $\mathcal{C}$.

The key design choice is that $\mathcal{C}$ contains gist entries $G_i$ rather than raw traces. Each gist summarizes information useful for future agents and points to more detailed evidence that can be retrieved through selective unfolding. Thus, every worker can read a lightweight global view of the current problem state without losing access to the underlying details.

The task queue determines how agents act on the shared state. Agents claim ready subtasks from $\mathcal{T}$ asynchronously; after a gist is admitted into $\mathcal{C}$, later agents can build on the finding, avoid a falsified hypothesis, or reuse a partial solution without waiting for a central controller to redistribute it. When the queue is exhausted, the most recently completed agent uses the current subtask states and shared context to determine whether further subtasks are needed. If so, it generates and enqueues new subtasks conditioned on $\mathcal{T}$ and $\mathcal{C}$; otherwise, it finalizes the answer from the accumulated shared state. Appendix B.4 provides further details on dependency-aware queueing, parallel execution, and systems optimizations

3.2 Compression and Verified Admission

$\textsc{DeLM}$ treats each shared-context update as an admission problem. Rather than writing raw agent outputs or unverified summaries directly into $\mathcal{C}$, it first compresses each completed result into a reusable gist and verifies that gist against its supporting evidence. Only updates that pass this gate become visible to other agents.

Specifically, given a completed result $r_i$, $\textsc{DeLM}$ chooses the compression path based on the type of content being admitted. If $r_i$ is a reasoning trajectory, it may contain a successful finding, a falsified hypothesis, execution feedback, or a constraint for future agents. Since later agents usually need the distilled conclusion rather than the full trace, $\textsc{DeLM}$ directly compresses $r_i$ into a gist $G_i$. If $r_i$ is a long source unit, however, direct gist-to-raw unfolding can be unreliable: the gist may not preserve enough detail to identify the exact raw span to unfold, while unfolding many candidate chunks to compensate is expensive. $\textsc{DeLM}$ therefore uses a hierarchical path, $r_i \rightarrow S_i \rightarrow G_i$: it first constructs a reference-grounded summary $S_i$, then compresses $S_i$ into the compact gist $G_i$ admitted to $\mathcal{C}$. Both $S_i$ and the raw content remain in backing stores and can be recovered through selective unfolding. Appendix B.1 and Appendix B.2 give the full hierarchy and unfolding procedure, and Section 4.3.1 validates this design through the "No Hierarchical Summary" ablation.

In operating-system terms, the gist layer functions as the small, always-resident working set visible to every agent, while $S_i$ and the raw store serve as backing storage. Selective unfolding acts like demand paging, loading finer-grained evidence into an agent's context only when the current subtask requires it.

Verification follows the same two paths. For a reasoning trajectory, $\textsc{DeLM}$ checks whether the gist $G_i$ faithfully captures the relevant finding, failure, feedback, or constraint in $r_i$ using an LLM verifier. For a long source unit, verification is performed at both levels: the summary $S_i$ must be supported by the raw source, and the gist $G_i$ must preserve the supported claims and important qualifiers from $S_i$. Gists that pass are appended to $\mathcal{C}$ and become visible to later agents; gists that fail are rejected, regenerated with feedback. Appendix B.3 gives the full admission-time verification procedure.

4. Experiments

Section Summary: The experiments evaluate DeLM on two benchmarks that test multi-agent coordination: SWE-bench Verified, which involves realistic software engineering tasks like fixing code in large repositories, and LongBench-v2, which requires multi-hop reasoning across long, multi-document inputs. The system is compared against baselines ranging from single direct LLM calls to tool-using agents and other multi-agent setups like AOrchestra, using models such as Gemini and Claude variants. On SWE-bench, DeLM outperforms the baselines on success rates when scaling across multiple attempts, often at lower cost, by enabling parallel agents to share verified context rather than relying on a central controller.

4.1 Experimental Setup

We first evaluate $\textsc{DeLM}$ on two representative benchmarks that stress complementary aspects of multi-agent coordination: software-engineering test-time scaling and long-context multi-doc reasoning. Section 5 later studies how $\textsc{DeLM}$ can be combined with RLM ([4]).

(1) SWE-bench Verified ([10]). A software-engineering benchmark built from real-world GitHub issues with human-verified task specifications and tests. We use it to evaluate agentic coding in realistic repository environments, where agents must inspect codebases, identify relevant files, implement fixes, and validate them through tests. This benchmark is especially suitable for our setting because its tasks require iterative exploration and coordination over large codebases rather than retrieving a single local answer.

(2) LongBench-v2 ([11]). A long-context benchmark designed to evaluate deep understanding and reasoning over realistic tasks. We focus on its multi-document QA setting, which contains 125 samples across diverse domains: Financial (15 samples), Government (23 samples), Multi-News (23 samples), Legal (14 samples), and Academic (50 samples). This setting is also suitable for our evaluation because its questions require multi-hop reasoning over evidence distributed across long contexts, rather than extracting a single local span.

We compare $\textsc{DeLM}$ against different baselines. Base directly invokes the LLM once over the full input to produce an answer, without any decomposition or multi-agent orchestration. Claude Code ([6]) is a tool-augmented agentic baseline that performs programmatic inspection and context compression to handle long inputs. mini-SWE-agent ([13]) is a lightweight software-engineering agent that uses a simple linear agent loop with Bash as its primary interface. AOrchestra ([3]) is a centralized multi-agent system in which a main orchestrator dynamically creates specialized sub-agents on demand, instantiating each as a tuple of instruction, context, tools, and model to execute individual subtasks. ReadAgent ([14]) is an agentic long-context baseline that compresses the input into short gist memories and looks up the original passages when a subtask needs details the gists omit. For base models, we use Gemini 3 Flash ([15]) and Claude Opus 4.6 ([16]) for the SWE-bench experiments; and GPT-5.4 ([17]), Claude Sonnet 4.6 ([18]), Gemini 3 Flash, and DeepSeek-V4-Pro ([19]) for the LongBench-v2 experiments.

All base models are evaluated through OpenRouter with default reasoning and sampling parameters. For Claude Code, we route the CLI through OpenRouter to support alternative base models and allow access to tools such as Read, Grep, and Bash. For AOrchestra, following the original configuration, we set max-attempts = 10 for the orchestrator and max-steps = 50 for each sub-agent. For ReadAgent, we set min-lookup-pages = 1 and max-lookup-pages = 6, and use a 100-token gist budget, matching the gist budget used by $\textsc{DeLM}$.

4.2 SWE-bench Verified

SWE-bench Verified evaluates whether $\textsc{DeLM}$ can improve test-time scaling in agentic software-engineering tasks. These tasks are largely sequential: each action depends on the outcome of the previous one, leaving little room to parallelize work within a single trajectory. Therefore, we scale across trajectories: we run each task $X$ times (with $X\in{2, 4}$) and count it as solved if any one of the $X$ attempts yields a correct patch. We report three metrics: Avg.@1 is the mean per-trial success rate, i.e., the expected accuracy of a single attempt; while Pass@2 and Pass@4 measure whether at least one of 2 or 4 attempts solves the task.

Different methods use the same test-time scaling budget in different ways. For Base, mini-SWE-agent, Claude Code, and the original AOrchestra, we run $X$ independent attempts per task. Each trajectory starts from scratch and does not reuse intermediate discoveries from other attempts. We also introduce AOrchestra-Parallel, a variant built on AOrchestra. It starts each task once, but allows the main agent at each step to spawn up to $X$ subtasks and sub-agents in parallel, aggregate their outputs, and use the combined information to decide the next step. These parallel threads are therefore coupled through the main agent. $\textsc{DeLM}$ uses the same agent harness as AOrchestra and likewise allows parallel threads to inform one another, but agents communicate through a verified shared context rather than through a central controller.

\begin{tabular}{@l ccc | c@}
\toprule
\textbf{Method} & \textbf{Avg.@1} & \textbf{Pass@2} & \textbf{Pass@4} & \textbf{Cost/Task} \\
\midrule
\multicolumn{5}{@l}{\textit{Gemini 3 Flash}}\\
mini-SWE-agent & 54.7\% & 65.6\% & 75.1\% & \$0.26 \\
Claude Code & 49.3\% & 57.1\% & 66.3\% & --\, { [TM:a]} \\
AOrchestra & 55.2\% & 64.5\% & 73.2\% & \$0.24 \\
AOrchestra-Parallel & 56.4\% & 63.2\% & 71.8\% & \$0.25 \\
\rowcolor{rowgray}\textbf{\textsc{DeLM}} & \textbf{65.7\%} & \textbf{72.9\%} & \textbf{77.4\%} & \$0.12 \\
\midrule
\multicolumn{5}{@l}{\textit{Claude Opus 4.6}}\\
mini-SWE-agent & 76.9\% & 79.8\% & 81.7\% & \$0.61 \\
Claude Code & 76.1\% & 79.5\% & 81.3\% & \$0.69 \\
AOrchestra & 74.7\% & 78.3\% & 80.1\% & \$0.70 \\
AOrchestra-Parallel & 75.2\% & 78.1\% & 79.7\% & \$0.73 \\
\rowcolor{rowgray}\textbf{\textsc{DeLM}} & \textbf{78.0\%} & \textbf{80.7\%} & \textbf{82.5\%} & \$0.63 \\
\bottomrule
\end{tabular}

Table 1 shows that $\textsc{DeLM}$ achieves the best performance across both base models. With Gemini 3 Flash, $\textsc{DeLM}$ reaches 65.7% Avg.@1, 72.9% Pass@2, and 77.4% Pass@4, outperforming all baselines on every metric. The largest gain appears in Avg.@1, where $\textsc{DeLM}$ exceeds the strongest baseline, AOrchestra-Parallel, by 9.3 percentage points. At the same time, $\textsc{DeLM}$ reduces cost to \0.12 per task, roughly half the cost of the strongest agentic baselines. This shows that the improvement comes from using the same test-time budget more effectively, rather than from launching more trajectories or spending more tokens.

AOrchestra-Parallel improves Avg.@1 over the original AOrchestra, but performs worse on Pass@2 and Pass@4. This suggests that coupling parallel threads through a main agent can make attempts more consistent, but may also reduce exploration diversity.

The gains with Claude Opus 4.6 are smaller, likely because the base model is already strong and leaves less room for coordination to help. Even so, $\textsc{DeLM}$ still achieves the highest accuracy across Avg.@1, Pass@2, and Pass@4, while remaining close to the lowest-cost method, only \0.02 per task above mini-SWE-agent.

Overall, these results support our central claim: test-time scaling is more effective when parallel agents communicate through shared state. By making compact progress visible across trajectories, $\textsc{DeLM}$ lets later agents avoid redundant exploration, build on prior findings, and focus on unresolved parts of the task. Section 4.2.1 analyzes these mechanisms through trace-level examples.

4.2.1 Why $\textsc{DeLM}$ Is Both Accurate and Cost-Efficient on SWE-bench

Three trace-level mechanisms explain why $\textsc{DeLM}$ improves both accuracy and cost efficiency. In the $\textsc{DeLM}$ traces below, each shared entry is one thread's verified note, tagged with its thread id (t0, t1, …) and a type such as FACT, FAIL, or PATCH_SUMMARY.

(1) Parallel agents complement each other by sharing failures.

The first mechanism is that failed hypotheses become reusable state rather than private dead ends. In isolated forks, a negative result remains local to one trajectory, so other attempts may spend their own budget rediscovering the same failure. In $\textsc{DeLM}$, once such a failure is admitted to the shared context, later agents can treat it as a constraint and redirect their search.

This mechanism appears in the trace below. The task fixes lambdify for single-element tuples. The natural guess, the Python printer layer, is a red herring; the real bug is on a separate tuple-building path in sympy/allowbreak utilities/allowbreak lambdify.py:

[t0/FACT] AbstractPythonCodePrinter change did not affect lambdify output
[t1/FACT] sympy/utilities/lambdify.py:964 _recursive_to_string manual tuple join bypasses printer fix
[t0/FACT] sympy/utilities/lambdify.py:964 manual join for tuples lacks trailing comma logic

The load-bearing update is the negative result in the first line: thread t0 shows that changing the printer does not affect the output. After reading this failure, t1 avoids repeating the same detour and localizes the actual bypass in _recursive_to_string; t0 then records the concrete defect, the missing trailing comma. Thus, $\textsc{DeLM}$ turns a failed hypothesis into shared progress, improving later agents' search efficiency.

(2) Admitted constraints remain binding shared state.

The second mechanism is that $\textsc{DeLM}$ preserves important constraints as shared state rather than routing them through a central controller. In centralized coordination, a main agent may soften, omit, or reopen constraints discovered by sub-agents before they can guide later work.

This mechanism appears in the trace below. Folding multiple search filters into one .filter() call is unsafe for multi-valued relations, where .filter(A).allowbreak filter(B) and .filter(A, B) differ. In AOrchestra-Parallel, a sub-agent finds exactly this danger, but the constraint must pass through the main agent, which reopens the optimization and softens it into a tradeoff about reducing joins "for many use cases"; the run fails. The relevant fact was found, but not preserved as binding state. In contrast, $\textsc{DeLM}$ keeps the constraint explicit and reusable:

[t3/FAIL] django/contrib/admin/options.py:1041 single .filter() breaks M2M multi-term search
[t3/FACT] Django ORM .filter(Q1, Q2) vs .filter(Q1).filter(Q2) differs for multi-valued relations
[t3/FACT] lookup_spawns_duplicates determines if single .filter() is safe
[t1/FACT] lookup_spawns_duplicates preserves M2M search semantics while optimizing FKs

Thread t3 records the unsafe case (FAIL), the semantic reason, and the predicate lookup_spawns_duplicates, which determines when the optimization is valid. Thread t1 then builds on this state to preserve M2M semantics while still optimizing foreign-key cases. Thus, later threads inherit the binding constraint rather than reopening a globally invalid simplification.

(3) Compact patch summaries carry discoveries.

The third mechanism is compact sharing, which lowers cost by preventing peer communication from becoming another long raw transcript. Sharing full traces would preserve information, but would also expose every worker to command history, file dumps, failed edits, and intermediate reasoning. $\textsc{DeLM}$ instead shares a compressed version of the useful discovery, so later workers can reuse the result without reading the entire trajectory.

This mechanism appears in the trace below. Both the shared and no-share settings solve the task, but compact sharing reduces the cost from \0.399 to \0.125. The difference is not that $\textsc{DeLM}$ shares more context; rather, it shares a compressed version of the useful search result (labeled as PATCH_SUMMARY), which is generated by an LLM summarizer. One thread first rules out the ordinary tuple-printer path and localizes the actual bypass in lambdify.py:

[t1/FACT] StrPrinter._print_tuple at sympy/printing/str.py:868 already includes trailing comma logic
[t1/FACT] sympy/utilities/lambdify.py:964 lacks trailing comma for single-element tuples
[t1/PATCH_SUMMARY] files=sympy/utilities/lambdify.py | idea=Modify _recursive_to_string to add a trailing comma for single-element tuples | evidence=reproduce_issue.py PASSED

This patch summary turns a multi-step debugging trajectory into a short, evidence-backed handoff. Later workers do not need the full command history, file dumps, failed edits, or intermediate reasoning. Thus, compact sharing preserves the reusable discovery while avoiding the cost of exposing every peer to the raw trace, explaining why $\textsc{DeLM}$ solves the task at much lower cost.

Together, these examples explain the trend in Table 1: $\textsc{DeLM}$ keeps reusable facts, failures, constraints, and patch summaries visible to all peers, while avoiding both redundant isolated search and the high token cost of raw trace sharing. This improves per-thread success while reducing cost.

4.3 LongBench-v2 Multi-Doc QA

::: {caption="Table 2: Accuracy (%) comparison on LongBench-v2 Multi-Doc QA across four base models. Results are mean ± std over 3 independent runs. Numbers in parentheses denote the number of samples in each domain. Best values are bolded."}

:::

LongBench-v2 Multi-Doc QA evaluates whether $\textsc{DeLM}$ can improve evidence aggregation across long documents. Unlike SWE-bench Verified, where progress within one trajectory is largely sequential, this setting exposes substantial within-task parallelism: different agents can inspect different documents, identify complementary evidence, and contribute to a shared answer state.

As shown in Table 2, $\textsc{DeLM}$ achieves the highest average accuracy across all four model families: 60.1% with GPT-5.4, 59.8% with Claude Sonnet 4.6, 61.5% with Gemini 3 Flash, and 67.5% with DeepSeek-V4-Pro. Compared with the strongest baseline for each model family, these correspond to gains of 5.7, 5.3, 4.4, and 3.6 percentage points, respectively. $\textsc{DeLM}$ also achieves the best or tied-best result in most domain-model combinations, suggesting that the benefit comes from a general improvement in evidence selection and reuse rather than from a single model or domain.

These gains come from first building a verified hierarchical view of the document set, enabling more targeted detailed inspection. $\textsc{DeLM}$ first splits the input into chunks and uses a lightweight summarizer, DeepSeek-V4-Flash ([19]) by default, to produce hierarchical summaries. Verified gists are then admitted to the shared context, giving all agents a compact view of the corpus before they choose what to inspect in detail. This upfront global view helps agents identify cross-document connections, selectively unfold evidence, and avoid poorly targeted reading.

In contrast, centralized and programmatic-inspection baselines make more local inspection decisions, often from metadata, file names, or keyword matches before observing the relevant content. Early mistakes can therefore compound: a controller may inspect the wrong evidence, miss cross-document links, or require additional rounds of delegation. ReadAgent also uses gists, but its lookup process directly depends on lossy summaries, without hierarchical unfolding and admission-time verification. As a result, these baselines can cost less in long-context settings because they avoid the upfront cost of constructing and verifying a structured view of the full context. $\textsc{DeLM}$ makes a different trade-off: it spends additional computation on hierarchical summarization and verification, but gains a more reliable understanding of the corpus, leading to more targeted evidence selection and higher prediction accuracy.

4.3.1 Which Components Make $\textsc{DeLM}$ Effective on LongBench-v2

Modular ablation.

To isolate the contribution of $\textsc{DeLM}$ 's two core components, we remove admission-time verification and hierarchical summarization in turn, and report average accuracy over the five domains in Figure 4(a). Removing verification causes the largest drop, from 60.1% to 55.2%, showing that unsupported claims can corrupt downstream reasoning when they enter the shared context unchecked. Removing hierarchical summarization also hurts performance, lowering accuracy to 57.7%, because gist-only routing provides a coarser path from global navigation to raw evidence. Thus, both components matter, with verification contributing the larger gain.

The trajectory evidence explains why the "No Hierarchical Summary" ablation degrades performance. Without the intermediate $S_i$ layer, $\textsc{DeLM}$ must either route from a lossy gist directly to raw chunks or unfold many raw chunks to recover missing details. The PUMA EBIT query illustrates this failure mode. The document contains competing 2024 EBIT outlooks: an earlier range from the annual report and a later narrowed range from the H1 2024 interim report. The gist layer identifies candidate chunks, but the $S_i$ summary is what determines which source is later in time, localizes the relevant sub-chunk, and flags that the exact EBIT range must be recovered from raw evidence: (S72.2 denotes the second segment of chunk 72, retrieved by scanning summary S72.)

[t1] The later-in-time source is the H1 2024 interim report, which reiterates sales growth expectations and a narrowed EBIT outlook. The latest 2024 EBIT guidance is located in [S72.2].
[t1] However, the summaries do not provide the actual EBIT range values, so the precise latest target cannot be determined from summaries alone.

Only after this coarse-to-fine localization does the worker request raw evidence. The deep unfold then retrieves the precise anchored sub-chunk rather than the full report (RAW denotes the corresponding raw text recovered by unfolding):

[S72.2 RAW] ... we narrow our outlook for the operating result (EBIT) to a range of EUR 620 million to EUR 670 million ...

This example shows why the hierarchy $G_i \rightarrow S_i \rightarrow \text{raw}$ is load-bearing. The gist supports cheap global navigation, the $S_i$ summary localizes the relevant raw span and detects when summary-level evidence is insufficient, and raw unfolding recovers the exact values. Removing the $S_i$ layer therefore weakens both evidence localization and cost-controlled reasoning, matching the performance drop in Figure 4(a).

The trajectory evidence also explains why removing verification hurts performance. In $\textsc{DeLM}$, agent outputs are not admitted into the shared context immediately; they must first be checked against their cited evidence. This prevents plausible but unsupported statements from becoming reusable shared state. In a legal question comparing Lucy v. Zehmer and Texaco v. Pennzoil, one agent introduced a specific damages claim that was not supported by its cited summary:

[t2 OUTPUT] Pennzoil/Texaco is summarized as only conditionally affirmed, with punitive damages reduced from USD 3 billion to USD 1 billion [S5].

However, the cited gist only states that the case involved a conditional affirmance with remittitur; it does not contain the specific damages amounts. The verifier (an LLM) therefore rejects the update:

[t2 VERIFY] WRONG: the claim that punitive damages were reduced from USD 3 billion to USD 1 billion is unsupported;[S5] only says there was a conditional affirmance with remittitur and gives no specific amounts.

After retry, the unsupported numerical claim is removed before the result is committed to $C$. Without this admission-time gate, the false detail would become available to later workers and the finisher as if it were grounded evidence, explaining the drop under the "No Verification" ablation in Figure 4(a).

Gist length.

We vary the target gist length across 50, 100, and 150 tokens per source unit (Figure 4(b)). Accuracy improves from 58.3% at 50 tokens to 60.1% at 100 tokens, then plateaus at 60.3% with 150 tokens. This suggests a threshold effect: once the gist is long enough to capture a source unit's relevance, additional length provides little benefit. We therefore use 100 tokens in the main experiments. Selective unfolding likely contributes to this stability, since agents can recover finer details from $S_i$ or the raw source when the gist is insufficient.

Summarization model.

We also vary the gist summarizer ($\phi_2$, defined in § B.1), using DeepSeek-V4-Flash, DeepSeek-V4-Pro, and GPT-5.4 while keeping the rest of the pipeline fixed (Figure 4(c)). Accuracy remains nearly unchanged across the three summarizers (60.1%, 60.4%, and 59.9%, respectively), with the cheapest model, DeepSeek-V4-Flash (our default), matching the stronger alternatives. Thus, a lightweight summarizer is sufficient for constructing the shared context, allowing $\textsc{DeLM}$ to reserve stronger models for subtask reasoning without sacrificing accuracy.

5. Combining $\textsc{DeLM}$ with RLM

Section Summary: Recursive language models handle complex structured tasks well by breaking them down through code-based coordination, but they are less effective for natural-language reasoning across documents. DeLM shows the reverse pattern, performing reliably on conversational questions yet struggling with precise counting or aggregation. Combining the two—by letting RLM agents coordinate through DeLM’s verified shared context and task queue—produces the highest accuracy and lowest cost on both kinds of benchmarks.

Recursive Language Models (RLMs) ([4]) process long contexts by recursively selecting context segments, issuing sub-calls, and composing the resulting partial answers. Crucially, RLM performs this process through a code-mediated Read-Eval-Print Loop (REPL): the long input is stored as an external variable that the model can inspect, parse, and aggregate programmatically. This makes RLM a natural fit for OOLONG ([12]), an aggregation-heavy benchmark where each sample consists of timestamped, user-attributed entries and answering a question often requires classifying many entries before computing a distributional answer.

\begin{tabular}{lcc}
\toprule
\textbf{Method} & \textbf{Acc.} & \textbf{Cost/Task} \\
\midrule
\multicolumn{3}{l}{\textit{OOLONG}} \\
RLM & 56.0\% & \$0.43 \\
\textsc{DeLM} & 53.3\% & \$0.47 \\
\textsc{DeLM}+RLM & \textbf{64.0\%} & \$0.40 \\
\midrule
\multicolumn{3}{l}{\textit{LongBench-v2 Multi-Doc QA}} \\
RLM & 55.8\% & \$0.29 \\
\textsc{DeLM} & 57.9\% & \$0.30 \\
\textsc{DeLM}+RLM & \textbf{60.3\%} & \$0.24 \\
\bottomrule
\end{tabular}

We first evaluate $\textsc{DeLM}$ and RLM separately on both LongBench-v2 Multi-Doc QA and OOLONG using GPT-5 with medium reasoning (RLM uses GPT-5-mini for sub-calls and recursion depth=1). As shown in Table 3, the two methods have complementary strengths. On LongBench-v2, $\textsc{DeLM}$ achieves higher average accuracy than RLM at roughly the same cost per task. On OOLONG, however, $\textsc{DeLM}$ underperforms RLM in both average accuracy and cost. This is because OOLONG is closer to a structured data-processing benchmark than a conversational reasoning benchmark (such as LongBench-v2): many questions require exact counting, filtering, comparison, and tie handling. In this setting, the natural-language shared context used by $\textsc{DeLM}$ is not reliable. RLM performs better because its core interface is code-mediated: through a REPL environment, the model can inspect, parse, transform, and aggregate the input using executable programs and programmatic sub-calls. However, RLM still relies on centralized coordination, where recursive calls report back to a root process that must reconcile and aggregate their outputs. This centralized aggregation is less suitable for natural-language multi-document reasoning, helping explain RLM's weaker LongBench-v2 performance. These results motivate a hybrid design that combines RLM's precise REPL-based execution with $\textsc{DeLM}$ 's decentralized coordination.

We therefore combine the two methods by retaining RLM as the underlying reasoner while adding two $\textsc{DeLM}$ components: a verified shared context and a decentralized task queue. In this hybrid setting, there is no main-agent/sub-agent hierarchy; all agents are equally ranked RLM instances that coordinate through shared state. An initial RLM call produces a compact work plan, after which subtasks are placed into the task queue and claimed by parallel RLM workers. Each worker's output is admitted into the shared context only after passing the verification gate. The system then generates any additional subtasks from the current problem state, and computes the final answer from the verified shared context once no further subtasks are needed. This design preserves RLM's strength in precise code-mediated execution while adding $\textsc{DeLM}$ 's decentralized coordination. As shown in Table 3, the combined method achieves the best accuracy and lowest cost on both benchmarks, indicating that RLM and $\textsc{DeLM}$ are complementary rather than competing approaches.

6. Related Work

Section Summary: Recent work on LLM-based multi-agent systems has explored role specialization, dynamic task delegation, shared blackboards, and orchestration mechanisms to improve coordination on complex tasks, yet most approaches still pass unverified or lightly summarized information between agents. Programmatic agentic frameworks add tool use and code-mediated loops for inspecting environments and reusing results, but continue to rely on a central agent or local loops that can allow unsupported claims to propagate. Long-context methods compress or retrieve beyond fixed windows through gist tokens and external memory, but often discard detail irreversibly or retrieve flatly; DeLM instead maintains a decentralized, verified hierarchy of compact entries that agents can asynchronously read, expand, or check against evidence.

Multi-Agent Systems.

Recent LLM-based multi-agent systems improve task performance by decomposing problems across specialized or interacting agents. Early frameworks such as CAMEL ([20]) instantiate role-playing agents to study autonomous cooperation. MetaGPT ([1]) and ChatDev ([21]) further structure collaboration through software-engineering-inspired roles, communication protocols, and standardized workflows, demonstrating that specialization can reduce coordination complexity in multi-step tasks. AOrchestra ([3]) dynamically creates task-specific sub-agents by instantiating their instructions, context, tools, and models; ToolOrchestra ([22]) trains a lightweight orchestrator to coordinate models and tools under accuracy, efficiency, and user-preference rewards; Squeeze Evolve ([23]) routes different stages of evolutionary inference to models of different cost and capability; and RecursiveMAS ([24]) replaces text-level communication with recursive latent-state transfer across agents. Combee ([25]) also identifies centralized aggregation as a bottleneck under high parallelism, but focuses on scaling parallel prompt learning rather than decentralized agent coordination. Recent blackboard-style MAS [26, 27] use a shared board to let agents exchange messages or volunteer information, either for general reasoning or for data discovery.

These systems show that agent coordination can be scaled through dynamic delegation, learned orchestration, model routing, recursive collaboration, and shared blackboards. However, the orchestration-based methods primarily optimize how agents are created, routed, or composed, leaving the communication substrate itself unverified: intermediate outputs are passed, summarized, or aggregated without grounding each entry in its underlying evidence before downstream reuse; the blackboard-based methods write raw or lightly structured messages to the board and still route coordination through a selection step or a central poster. In contrast, $\textsc{DeLM}$ treats the shared context as curated, verified state: agents coordinate fully asynchronously through a task queue with no central controller, every entry is admitted only after verification against its supporting evidence, and entries are stored as compact gists that unfold to finer detail on demand, keeping the shared state both reliable and scalable as the number of agents grows.

Programmatic Agentic Systems.

A closely related class of systems equips language models with programmatic interfaces to external environments, enabling them to inspect state, invoke tools, manipulate artifacts, and recursively delegate computation. Recursive Language Models (RLMs) ([4]) apply this paradigm to long-context reasoning by treating the prompt as an external environment and recursively selecting snippets for inspection. It performs this process through a code-mediated Read-Eval-Print Loop (REPL). Agentic coding systems such as Claude Code ([6]) and Codex ([5]) extend this idea to software environments, allowing models to read repositories, edit files, execute commands, and incorporate tool feedback. These approaches demonstrate that coupling language models with executable environments substantially enhances capability beyond single-pass generation.

However, programmatic access alone does not solve the coordination problem. In many systems, planning is still routed through a main agent or local tool loop, and intermediate results are reused as raw observations or prose summaries without an admission-time grounding step. As a result, useful discoveries may remain local to one trajectory, while unsupported claims or softened constraints can propagate through later reasoning. $\textsc{DeLM}$ is complementary to these systems: it does not replace tools, code execution, or REPL-style inspection, but provides a decentralized coordination layer in which tool-derived findings, failures, and partial solutions can be compressed, verified, and shared across agents. This complementarity is reflected in our OOLONG experiments (§ 5), where combining $\textsc{DeLM}$ with RLM improves over either method alone.

Long-context LM systems.

Another line of work addresses long-context reasoning by storing, compressing, or retrieving information beyond the model's current context window. [28] introduce a training-based approach that compresses long prompts into compact "gist" representations. Context-Folding ([29]) keeps long interactions within the context window by periodically compressing the conversation history. These methods reduce context length, but once fine-grained information is discarded, it cannot be reliably recovered when later reasoning requires precise details. ReadAgent ([14]) instead pairs compression with retrieval: it summarizes pages of text into short gist memories and looks up the original passages when a task needs details the gists omit. However, because lookup is triggered from lossy gists, ReadAgent can miss evidence whose relevance is not preserved in the compressed representation.

Memory-augmented systems such as LongMem ([30]), MemGPT ([31]), Mem0 ([32]), and MemOS ([33]) introduce external memory mechanisms for LLMs to retain information across long interactions or task histories. For instance, MemGPT ([31]) proposes an operating-system-inspired virtual context mechanism to decide what to keep in the active context and what to move into external memory. These systems make memory persistent, but retrieval is usually flat: a memory item is either selected into context or omitted. $\textsc{DeLM}$ instead organizes shared information by abstraction level. Agents first reason over compact verified gists across the full problem, then selectively unfold relevant entries into detailed summaries and raw evidence. This hierarchy makes the shared context both global and recoverable.

7. Limitations and Future Work

Section Summary: The method works well overall but still carries some drawbacks, including extra computational cost from verification steps, dependence on how effectively agents break large tasks into smaller ones, and the fact that prompts may need adjustment for different AI models. Future refinements could include lighter forms of checking, agents that learn when to split or combine subtasks on their own, and automated ways to tailor prompts for each model. The same decentralized approach also appears well suited to automated research systems, where agents could share verified findings across long, complex investigations without repeatedly redoing the same work.

While our results demonstrate the effectiveness of $\textsc{DeLM}$, several directions remain open for future exploration. First, admission-time verification trades a modest amount of overhead for stronger reliability guarantees; lighter-weight verifiers, including learned models or rule-based checks for common claim types, could further improve efficiency while preserving the grounding benefits we observe. Second, $\textsc{DeLM}$ inherits the decomposition quality of agents. Overly coarse decompositions leave agents with under-specified subtasks, while overly aggressive decompositions can spawn unnecessary agents and overcomplicate reasoning. A promising direction is to train adaptive agents that decide when to split, merge, or terminate subtasks based on the shared context. Third, as noted in prior work ([4]), there is no universally optimal prompt across models; different model families may require tailored prompts to elicit the intended behavior. Combining $\textsc{DeLM}$ with prompt-evolution methods such as GEPA ([34]) could further adapt its decomposition, summarization, and verification prompts to each model family.

Beyond the benchmarks studied here, $\textsc{DeLM}$ suggests a promising direction for automated research systems ([9, 8]). Research workflows naturally combine test-time scaling with long-context reasoning: agents must explore alternative hypotheses, inspect large collections of papers or experimental logs, compare evidence across sources, and iteratively refine conclusions. A decentralized shared context could make such systems more efficient by preventing agents from repeatedly reading the same papers or rerunning the same failed analyses; more effective by allowing useful findings to propagate across parallel research threads; and more robust by admitting only verified claims into the shared state. We therefore view automated research as a natural application domain for future decentralized MAS.

8. Conclusion

Section Summary: The DeLM system enables multiple AI agents to collaborate without a central controller by sharing a common space for verified updates and tasks. This approach turns each agent's discoveries and mistakes into reusable building blocks, allowing others to build on successes, avoid dead ends, and recover details only when necessary. Experiments on software engineering and long-document reasoning benchmarks show that the method raises accuracy, cuts costs, and works well alongside other techniques, highlighting the need for reliable shared communication in large-scale agent teams.

We introduce $\textsc{DeLM}$, a decentralized multi-agent system in which agents coordinate through a shared context and task queue rather than a central controller. By admitting only compact, verified updates into shared state, $\textsc{DeLM}$ turns intermediate progress into reusable problem state: agents can build on prior findings, avoid repeated failures, preserve constraints, and recover detailed evidence only when needed. On SWE-bench Verified, $\textsc{DeLM}$ improves test-time scaling by making discoveries and failures reusable across parallel attempts, achieving higher pass rates while reducing cost. On LongBench-v2 Multi-Doc QA, $\textsc{DeLM}$ improves long-context reasoning by constructing a verified, hierarchical view of the corpus, leading to higher accuracy. We further show that $\textsc{DeLM}$ is complementary to RLM, where decentralized validated state improves programmatic aggregation. These results suggest that scalable multi-agent systems require not only more parallel agents, but also a reliable communication substrate for sharing progress across them.

9. Acknowledgments

Section Summary: The authors thank a long list of individual colleagues and collaborators, along with teams from Stanford’s AI labs, for their help with the project. They also recognize financial and institutional backing from agencies such as the NSF and DARPA, along with companies including IBM, Google, Lightspeed, and Google DeepMind.

We thank Shayan Talaei, Jon Saad-Falcon, Jacky Kwok, Hermann Kumbong, Ishan Khare, Miria Feng, Qizheng Zhang, Swapnil Gandhi, Michael Y. Li, Chun Deng, Chong Zeng, Rui Li, Ke Li, Marquita Ellis, Hangoo Kang, Adrian Gamarra Lafuente, Charles Ding, Tarun Suresh, Ziyu Chen, Zhuohan Gu, Yize Liu, and Ligeng Zhu. We would also like to thank our collaborators at the Stanford Artificial Intelligence Laboratory (SAIL) and Stanford HAI.

We gratefully acknowledge support from federal sources: NSF under No. 24-554 (AIMing) and DARPA under No. HR00112520038 (Fallingwater). We also gratefully acknowledge support from Stanford HAI, IBM (Code Generation), Lightspeed, Google, and Google DeepMind.

Appendix

Section Summary: The appendix describes a system called DeLM that handles long documents by turning them into a three-level hierarchy of raw text, reference-grounded summaries, and very short gists that are stored in a shared context for agents to consult. Agents normally work with the compact gists but can request to “unfold” specific entries in stages to reveal more detailed summaries or the original source material only when needed. The process also includes built-in verification steps that check each level of compression against the original content before anything is added to the shared record, keeping the overall context lightweight while preserving access to precise evidence.

A. Supplementary Material

B. Method Details

B.1 Hierarchical Summarization

For long source units $r_i$ with label $\ell_i$, $\textsc{DeLM}$ uses a three-level representation, $r_i \rightarrow S_i \rightarrow G_i$. The shared context $\mathcal{C}$ stores only compact gist entries $(\ell_i, G_i)$, giving every agent a lightweight global view of the corpus. The detailed backing content is stored outside $\mathcal{C}$: $\mathcal{L}[\ell_i]$ contains the reference-grounded summary $S_i$, and $\mathcal{R}[\ell_i]$ contains the corresponding raw source unit.

This hierarchy avoids relying on a lossy gist to directly select raw evidence while also avoiding broad raw-context unfolding. Agents first reason over gists, then unfold to $S_i$ or raw evidence only when finer-grained support is needed. For reasoning trajectories, $\textsc{DeLM}$ usually admits a direct gist $G_i$, since later agents typically need the distilled finding, failure, feedback, or constraint rather than the full trace; the original trajectory remains available in $\mathcal{R}[\ell_i]$ when needed.

We now describe the hierarchical summarization path for long source units.

Stage 1: $S_i$ (ref-grounded evidence map).

Given a source unit $u_i$, the first-stage summarizer $\phi_1$ produces an evidence map consisting of a set of short bullets, each expressing a single atomic claim. Every bullet includes a ref-tag that identifies its supporting span in the raw text:

$ \texttt{[ref: } h ;\ldots; t, \texttt{]},\tag{1} $

where $h$ and $t$ denote the first and last $N \geq 5$ words of the supporting span, respectively, copied verbatim. Formally,

$ S_i = \phi_1(u_i;, Q) = {(b_{i, k}, , \rho_{i, k})}{k=1}^{m_i}, \quad \rho{i, k} = (h_{i, k}, t_{i, k}),\tag{2} $

where $b_{i, k}$ is the $k$-th bullet and $\rho_{i, k}$ is its ref-tag. $\phi_1$ is conditioned on the question $Q$ so that question-relevant content is extracted at higher fidelity. Full coverage of the source unit is required, as any portion may contain the critical evidence needed to answer the question. The resulting $S_i$ is stored in $\mathcal{L}[\ell_i]$, with each bullet addressable by its local index $k$.

Stage 2: $G_i$ (compact shared-context entry).

A second-stage summarizer $\phi_2$ compresses $S_i$ into a short, highly compact gist:

$ G_i = \phi_2(S_i;, Q).\tag{3} $

This gist captures the relevance of $u_i$ to the query at a density that allows downstream agents to decide, at a glance, whether further inspection is needed. Only $G_i$ is admitted to $\mathcal{C}$, while $S_i$ and the raw content remain in $\mathcal{L}$ and $\mathcal{R}$ and are accessed only through selective unfolding (§ B.2).

B.2 Selective Unfolding

Agents reason over the shared context $\mathcal{C}$ by default, using the compact $G_i$ entries. When additional detail is required, they invoke selective unfolding, a coarse-to-fine, opt-in mechanism for retrieving more detailed information on demand. We implement unfolding in two stages: first, from $G_i$ to the reference-grounded summary $S_i$, and then, if necessary, from $S_i$ to the raw source content.

Stage 1: $G \rightarrow S$ (UNFOLD).

At any reasoning step, the agent may request a small set of labels via an UNFOLD: $\ell_1, \ell_2, \ldots$ directive. The orchestrator retrieves the corresponding $S_i$ summaries $\mathcal{L}[\ell]$ and inlines their bullet-level evidence maps into the agent’s prompt for that call. This enables the agent to reason over grounded evidence rather than compressed gists.

Stage 2: $S \rightarrow \text{raw}$ (DEEP_UNFOLD).

After reasoning over the $S_i$ summaries, the agent may determine that finer-grained detail is required—for example, to resolve subtle qualifiers or verify exact wording. It can then emit a DEEP_UNFOLD: $\ell_1, \ldots$ directive specifying the labels whose raw content is needed. The orchestrator issues a follow-up call that combines the original task, the shared context $\mathcal{C}$, the previously unfolded summaries $S_i$, and the newly requested raw content from $\mathcal{R}$.

Unfolding proceeds in a coarse-to-fine, on-demand manner and may span multiple rounds up to a fixed limit, with each round allowing the agent to request additional $S_i$ or raw content as needed. Additionally, raw retrieval is also neighborhood-aware: requesting sub-chunk $n$ returns $n{\pm}1$ as well, which helps absorb off-by-one citation errors. Importantly, unfolded content is local to the requesting call and is not written back to the shared context $\mathcal{C}$, so subsequent agents still observe only the gist layer. This design prevents detailed intermediate content from polluting the shared context while preserving access to fine-grained evidence when necessary. As a result, the cost of detailed inspection scales with the amount of information actually required, rather than the total volume of available context.

B.3 Admission-Time Verification

Verification follows the compression path. For a reasoning trajectory, $\textsc{DeLM}$ checks whether the gist $G_i$ faithfully preserves the relevant finding, failure, feedback, or constraint from the original result $r_i$. For a long source unit, $\textsc{DeLM}$ verifies both levels of the hierarchy, first grounding the Stage-1 summary $S_i$ in the raw source and then checking the Stage-2 gist $G_i$ against $S_i$.

Specifically, each Stage-1 bullet $b$ with a reference $(h, t)$ is accepted only if the head $h$ and tail $\mathcal{T}$ both appear, in order and verbatim, in the underlying source unit $u$. Bullets that fail this condition are routed to a targeted rewrite step, which either supplies a valid reference or drops the claim.

After this gate, an iterative coverage loop identifies large uncovered regions, defined as contiguous spans not referenced by any admitted bullet, and requests additional bullets for each gap. Newly generated bullets must pass the same reference and semantic-support checks before being added to $S_i$. The loop terminates when no large gaps remain or a fixed iteration limit is reached.

The Stage-2 gist $G_i$ is then produced from the verified $S_i$. Because $S_i$ contains only grounded claims, $G_i$ inherits this grounding at a coarse level. To guard against abstraction errors, $G_i$ undergoes a lightweight verification step using a cheaper LLM, such as DeepSeek-V4-Flash, which checks for hallucinations, semantic drift, and missing critical qualifiers relative to $S_i$. Only gists that pass this check are admitted to $\mathcal{C}$; otherwise, the update is rejected, regenerated, or returned to the task queue $\mathcal{T}$ with a rejection reason, up to a fixed retry limit.

B.4 Parallel Execution and Systems Optimizations

$\textsc{DeLM}$ uses three lightweight mechanisms to support safe and efficient parallel execution: concurrent admission with disciplined reads and writes, a dependency-aware task queue, and KV-cache reuse across calls.

Concurrent admission and read/write discipline.

Compression and verification are parallelized across completed results. Each finished agent independently proposes an update from its local result $r_i$, and the corresponding summarization and verification calls can run concurrently. The only synchronized step is admission. Once an update passes verification, $\textsc{DeLM}$ first writes its backing content to $\mathcal{L}$ and $\mathcal{R}$, and then appends the visible entry $(\ell_i, G_i)$ to $\mathcal{C}$ through an atomic write. Agents read lock-free snapshots of $\mathcal{C}$ at dispatch time; entries committed later become visible only on the next snapshot.This write-before-publish ordering ensures that every visible label can be unfolded without a race, that later agents observe only fully verified entries, and that slow summarization or verification calls do not impose a global synchronization barrier.

Dependency-aware task queue.

At the beginning, given an input task $D$, $\textsc{DeLM}$ first constructs a task topology: a numbered list in which each task is annotated with a $[deps: \ldots]$ tag specifying its upstream dependencies. Tasks with no dependencies are dispatched first; subsequent tasks become eligible as their dependencies complete. and subsequent tasks become eligible once their dependencies complete. When the queue is exhausted, a single agent acquires a queue lock and invokes $\textsc{GenerateMoreSubtasks}$ with the current shared context $\mathcal{C}$, the original plan, and the list of executed tasks. The function may return $[\textsc{Done}]$ if the original plan has been completed and no further unfolding is expected to change the state of any sub-claim. This step also prevents dependency deadlock in practice: when no pending task is eligible, the system does not wait indefinitely, but invokes $\textsc{GenerateMoreSubtasks}$ to create missing prerequisite subtasks, revise blocked dependencies, or terminate if the current shared context is sufficient.

KV-cache reuse from a stable shared context.

The shared-context design also improves KV-cache reuse. Because $\mathcal{C}$ is largely stable and accumulative, and consistently placed at the beginning of each model invocation, it forms a persistent prefix across calls. This is useful in multi-step reasoning, where many LLM calls share the same global state but differ only in the task-specific suffix. As a result, repeated computation over the shared prefix can be amortized, yielding larger efficiency gains as the number of reasoning steps grows.

References

Section Summary: The references section compiles dozens of academic papers, preprints, conference proceedings, and industry announcements centered on multi-agent AI systems, long-context language models, coding agents, and automated orchestration tools. It draws from sources such as arXiv, NeurIPS, ICLR, and company blogs by OpenAI, Anthropic, and Google DeepMind, with dates ranging from 2023 to 2026. The citations collectively document rapid progress in collaborative AI frameworks, memory-augmented models, and real-world benchmarks for software engineering tasks.

[1] Hong et al. (2023). MetaGPT: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations.

[2] Feng et al. (2026). AgentSwing: Adaptive Parallel Context Management Routing for Long-Horizon Web Agents. arXiv preprint arXiv:2603.27490.

[3] Ruan et al. (2026). AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration. arXiv preprint arXiv:2602.03786.

[4] Zhang et al. (2025). Recursive language models. arXiv preprint arXiv:2512.24601.

[5] OpenAI (2025). Codex. https://openai.com/codex/.

[6] Anthropic (2026). Claude Code. https://claude.ai/.

[7] Kimi et al. (2026). Kimi K2. 5: Visual Agentic Intelligence. arXiv preprint arXiv:2602.02276.

[8] Liu et al. (2026). SkyDiscover: A Flexible, Adaptive Framework for AI-Driven Scientific and Algorithmic Discovery. In Proceedings of the ACM Conference on AI and Agentic Systems. pp. 1223–1227. doi:10.1145/3786335.3813221. https://doi.org/10.1145/3786335.3813221.

[9] AlphaEvolve team (2025). AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. Google DeepMind Blog. Accessed: 2026-06-03. https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/.

[10] Carlos E Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66.

[11] Bai et al. (2025). Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3639–3664.

[12] Bertsch et al. (2025). Oolong: Evaluating long context reasoning and aggregation capabilities. arXiv preprint arXiv:2511.02817.

[13] John Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://arxiv.org/abs/2405.15793.

[14] Lee et al. (2024). A human-inspired reading agent with gist memory of very long contexts. arXiv preprint arXiv:2402.09727.

[15] Tulsee Doshi (2025). Gemini 3 Flash: Frontier Intelligence Built for Speed. Google — The Keyword. Accessed: 2026-06-01. https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/.

[16] Anthropic (2026). Introducing Claude Opus 4.6. Anthropic. Accessed: 2026-06-01. https://www.anthropic.com/news/claude-opus-4-6.

[17] OpenAI (2026). Introducing GPT-5.4. https://openai.com/index/introducing-gpt-5-4/. Accessed: 2026-05-06.

[18] Anthropic (2025). Claude Sonnet 4.6. https://www.anthropic.com/news/claude-sonnet-4-6. Accessed: 2026-05-06.

[19] DeepSeek-AI (2026). DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence.

[20] Li et al. (2023). Camel: Communicative agents for" mind" exploration of large language model society. Advances in neural information processing systems. 36. pp. 51991–52008.

[21] Qian et al. (2024). Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers). pp. 15174–15186.

[22] Su et al. (2025). Toolorchestra: Elevating intelligence via efficient model and tool orchestration. arXiv preprint arXiv:2511.21689.

[23] Maheswaran et al. (2026). Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolution. arXiv preprint arXiv:2604.07725.

[24] Yang et al. (2026). Recursive Multi-Agent Systems. arXiv preprint arXiv:2604.25917.

[25] Li et al. (2026). Combee: Scaling Prompt Learning for Self-Improving Language Model Agents. arXiv preprint arXiv:2604.04247.

[26] Han, Bochen and Zhang, Songmao (2025). Exploring advanced llm multi-agent systems based on blackboard architecture. arXiv preprint arXiv:2507.01701.

[27] Salemi et al. (2025). Llm-based multi-agent blackboard system for information discovery in data science. arXiv preprint arXiv:2510.01285.

[28] Mu et al. (2023). Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems. 36. pp. 19327–19352.

[29] Sun et al. (2025). Scaling long-horizon llm agent via context-folding. arXiv preprint arXiv:2510.11967.

[30] Wang et al. (2023). Augmenting language models with long-term memory. Advances in Neural Information Processing Systems. 36. pp. 74530–74543.

[31] Packer et al. (2023). MemGPT: towards LLMs as operating systems..

[32] Chhikara et al. (2025). Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413.

[33] Li et al. (2025). Memos: An operating system for memory-augmented generation (mag) in large language models. arXiv preprint arXiv:2505.22101.

[34] Agrawal et al. (2025). Gepa: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457.