Reasoning Should Travel: How Relay Traces Help Local Models Go Beyond Repetition

A notebook-style experiment in executable reasoning, human knowledge transfer, and the discipline of carrying tested state forward.

Table of Contents

1. Reasoning Already Travels Between People

2. The Research Question

3. Completed Benchmark Results

4. Experimental Context and the Relay Runner

5. Two Prompt Contracts

6. MATH-500 Problem 474

7. One-Shot Failure and Relay Recovery

8. AIME Problem 6

9. Four Principles of Reliable Relay Reasoning

10. Relationship to Prior Work

11. Reasoning Should Travel

12. References

Reasoning Already Travels Between People

When a person solves a hard problem, the work rarely looks like a clean line from question to answer. A student reads the statement once, draws a diagram, notices a contradiction, rewrites the diagram, tries a small case, checks the arithmetic, and only then says the answer aloud. A good mathematician may appear fast, but the speed hides a discipline: do not confuse a guess with a derivation, do not confuse a derivation with a verified result, and do not confuse a verified result with a submitted answer.

Human reasoning is also rarely confined to one head. A scientist inherits an apparatus, a notation, and a stack of failed hypotheses. A junior engineer receives a bug report that contains the last reproducible state rather than the entire history of the codebase. A mathematician begins from lemmas proved by people who are no longer alive. The next person does not restart from the original question. They receive a compressed frontier: what has been established, what remains uncertain, and where effort should go next.

This pattern has biological and cultural precedents, although none is identical to an LLM relay. Research on cumulative culture shows that human groups can preserve improvements and build solutions that exceed what isolated individuals produce; in a comparative puzzle-box experiment, children combined observation, teaching, and prosocial behavior in ways that supported cumulative improvement (Dean et al., 2012). The interactionist account of reasoning similarly argues that producing and evaluating reasons are deeply social functions, often sharpened by dialogue rather than private reflection alone (Mercier & Sperber, 2011). At a wider biological scale, animal groups can integrate distributed information into collective decisions even though no individual has a global view (Couzin, 2009).

Social insects offer a second, more indirect analogy. In stigmergy, one animal changes an environment and that change guides later activity: a pheromone trail, a partly built nest wall, or another persistent local cue becomes shared external state (Theraulaz & Bonabeau, 1999). A relay trace behaves like an informational footprint, but with an important difference. It is explicit and typed. The next reasoner receives a representation, a candidate, a runtime error, a verifier-independent observation, or a confidence estimate—not merely a scalar attraction signal. Relay reasoning is therefore best understood as structured continuation, not biological imitation.

Relay reasoning advances an existing trace instead of repeatedly rebuilding its settled prefix.

That human discipline is the motivation behind this experiment. The benchmark is not only asking whether a Qwen language model can produce correct numbers. It is asking whether a compressed, locally hosted model can be placed inside a structure that resembles careful human work: a notebook, a calculator, an error log, and a rule that says the final answer is a commitment. The model is not rewarded for sounding confident. It is rewarded only when a parseable RESULT—the runner's explicit final-answer line—survives a hidden verifier that it cannot see during the run.

The public experiment suite evaluates that idea on two mathematical datasets. MATH-500 is a 500-problem evaluation set spanning several high-school mathematics domains and multiple answer formats. AIME 2026 is the thirty-problem American Invitational Mathematics Examination, whose answers are exact integers from 000 through 999. The repository stores their generated artifacts under results/qwen_math500_results/ and results/qwen_aime_results/. These names are output directories, not model variants: each contains aggregate summaries followed by one folder per benchmark problem, and each problem folder separates the one_shot_python and relay_python protocols.

The main MATH-500 example is problem 474, a Prealgebra Level 5 magic-square problem. Its relay_python trace records five bounded attempts, including the generated code, runtime feedback, progress handed to the next attempt, and final report. The example is interesting not because the answer is exotic, but because the model found the answer before it successfully submitted it. The AIME counterpart is AIME 2026 problem 6: its one-shot trace produced useful mathematical evidence but crashed before submission, whereas its relay trace recovered and submitted the correct integer 441.

The problem is a magic square. A diagram gives a 3 by 3 grid whose entries are expressions such as n - 3, 2n - 9, n + 1, and fixed numbers such as 1, 2, and 3. A magic square requires every row, every column, and both main diagonals to have the same sum. The task is simply to find n. On paper, this can be solved in a few lines once the diagram has been read correctly. In a tool-augmented model run, however, the hard part is not only algebra. The hard part is preserving a correct representation of the diagram, checking the equations, handling runtime failures, and emitting the final answer in exactly the format the evaluator expects.

That is why this example is a good small window into reasoning. It shows that reasoning is not a hidden private string of words. In a reliable system, reasoning becomes a loop of representation, execution, observation, repair, and final commitment. This is close in spirit to work on chain-of-thought prompting, where intermediate steps help models solve multi-step problems (Wei et al., 2022), but it is also closer to program-aided reasoning, where the model writes executable structure and delegates arithmetic or symbolic manipulation to a runtime (Gao et al., 2022). In the MATH-500 trace, the Python interpreter is not a decoration. It is the surface on which the model's assumptions become testable.

That human analogy becomes useful only when it is translated into a testable systems question. The next section therefore moves from motivation to the precise comparison implemented in the repository.

The Research Question: Can Tested State Replace Repetition?

The implementation, runners, benchmark preparation scripts, and experiment layout are available in the public Relay Reasoning repository. Its README explains how the project is installed, how the two datasets are prepared, how a run is launched, and what every output folder contains. The repository separates benchmark inputs, runner code, and generated results; that separation matters because a relay should be auditable as a system, not treated as a mysterious prompt incantation.

The experiment asks a deliberately narrow question: when a local language model is allowed to write and run Python, does it reason better if the interaction is organized as a relay of bounded attempts rather than as one large single-shot completion?

This is not only a leaderboard question. The goal is to make reasoning inspectable. A final answer tells us whether the model landed in the right place, but it usually hides how the system moved through the problem. A trace tells us more. It shows whether the model parsed the problem correctly, whether it formed equations correctly, whether it used computation as verification or as decoration, whether the runtime caught a bug, and whether later attempts reused useful information or started over.

The local benchmark runner therefore compares two named protocols. In one_shot_python, the model receives one large output budget and one Python execution chance; it must solve, verify, and submit in a single code block. In relay_python, the model receives smaller per-attempt budgets but may use several attempts. Early attempts can omit RESULT and print useful observations. The runner then feeds only operational feedback into the next prompt. That feedback can include stdout, stderr, exit code, selected code-block diagnostics, and compact summaries. It does not include hidden answer-key correctness. The two concrete directory names used later in the essay therefore mean “single execution” and “bounded sequence of executions,” respectively—not two different models.

The motivation is practical. If a local model is going to be used as a reasoning agent, we need to know whether extra inference-time structure can compensate for limited model size, quantization, and hardware constraints. We also need to know what kind of structure helps. Sometimes the answer is not "more tokens." Sometimes it is "a better loop."

Testing that loop requires benchmarks that expose different kinds of failure. MATH-500 provides variety, while AIME provides a strict commitment contract.

Two Benchmarks, Two Different Pressures

MATH-500 and AIME are useful together because they stress different parts of the same reasoning stack.

MATH-500 gives breadth. It contains problems across algebra, number theory, geometry, prealgebra, counting and probability, intermediate algebra, and precalculus. It also contains mixed answer formats. Some answers are integers, but others are fractions, radicals, symbolic expressions, intervals, equations, or short text. That makes it a strong test of whether the model and runner can preserve exact mathematical form instead of collapsing everything into decimals. It also tests whether the parser and verifier can normalize equivalent answers without being too permissive.

AIME gives strictness. AIME answers are integers, and the evaluation target is a single exact number. That makes it easier to audit leakage, formatting, and final-answer discipline. If the model prints RESULT: 7, there is little ambiguity about what was submitted. Because the completed AIME result tree contains both protocols for all thirty problems, AIME can play a second role in the essay: it is no longer only a design target; it is a compact pressure test of the relay protocol.

Together, the two benchmarks help separate several questions. MATH-500 asks whether the system can survive varied mathematical language and answer formats. AIME asks whether the system can do careful multi-step reasoning under an integer-answer contract. The shared research question is whether a relay with executable feedback can improve a local model's reasoning without giving it answer-key feedback during the run.

With those roles established, the aggregate results can be read as evidence about both accuracy and failure recovery—not as a single undifferentiated score.

What the Completed Runs Reveal

The aggregate numbers below come from the summary files at the top of each result tree. The MATH-500 results directory contains 500 paired problem traces together with benchmark_summary.csv. The AIME results directory applies the same two protocols to thirty exact-integer contest problems. Because every problem was run once in each mode, the comparison can be read two ways: as an aggregate accuracy difference and as a paired outcome for the same problem.

Relay raises verified accuracy on both benchmarks, while paired outcomes show recoveries, regressions, persistent successes, and persistent failures.

The Headline Gain Comes from Many More Recoveries Than Regressions

The aggregate comparison is substantial on MATH-500:

Mode	Problems	Verified correct	Accuracy	Submitted answers	Average attempts
`one_shot_python`	500	318	63.6 percent	349	1.00
`relay_python`	500	445	89.0 percent	495	1.45

On AIME, the absolute change is smaller but still visible:

Mode	Problems	Verified correct	Accuracy	Submitted answers	Average attempts
`one_shot_python`	30	18	60.0 percent	20	1.00
`relay_python`	30	21	70.0 percent	24	2.37

The paired view explains where those net changes came from. On MATH-500, 305 problems were correct in both modes, 140 were recovered by relay, 13 were correct only in one-shot mode, and 42 remained wrong in both. The difference between 140 recoveries and 13 regressions is the observed net gain of 127 correct answers. On AIME, 17 problems were correct in both modes, four were recovered, one regressed, and eight remained wrong, producing the net gain of three. Relay therefore improved the aggregate because recoveries outnumbered regressions, not because every individual trace improved.

This distinction matters methodologically. The results are a paired protocol comparison for one quantized model and one runner configuration; they are not a universal estimate of how much any model will gain from retries. The next plots ask a more diagnostic question: where did the recoveries occur?

The MATH-500 Gain Widens with Difficulty; AIME Is More Selective

MATH-500 relay gains increase with difficulty, while AIME gains concentrate in the middle and late problem bands and regress in the early band.

MATH-500 shows a clear descriptive gradient. At Level 1, verified accuracy rises from 74.4 to 95.3 percent, a gain of 20.9 percentage points. At Level 5, it rises from 48.5 to 80.6 percent, a gain of 32.1 points and 43 additional correct answers. The intermediate levels move in the same direction: +18.9 points at Level 2, +24.8 at Level 3, and +25.0 at Level 4. The runner appears to recover more value where the first attempt has more opportunities to misread, misformalize, or fail at the interface.

AIME prevents that pattern from becoming an easy slogan. Across its two fifteen-problem exams, the early positions P1–P5 fall from 8 of 10 correct to 7 of 10. The middle positions P6–P10 rise from 6 of 10 to 9 of 10, and the late positions P11–P15 rise from 4 of 10 to 5 of 10. Because each band contains only ten problems, these are descriptive counts rather than stable estimates. Even so, they reveal an important limit: additional attempts can repair difficult traces, but they can also disturb a correct early solution.

The single AIME regression is problem 16, where one-shot was correct and relay was wrong. The four recoveries are problem 6, problem 9, problem 12, and problem 22. These repository identifiers are stable problem folders, not separate model variants.

Subject-Area Gains Are Broad, Not Confined to One Kind of Mathematics

Relay improves verified accuracy in all seven MATH-500 subject areas, with the largest gains in intermediate algebra, geometry, and prealgebra.

Every MATH-500 subject area moves upward. Intermediate algebra has the largest increase, from 55.7 to 89.7 percent (+34.0 points), followed by geometry from 56.1 to 87.8 (+31.7) and prealgebra from 63.4 to 92.7 (+29.3). Counting and probability gains 28.9 points, precalculus 26.8, and number theory 24.2. Algebra begins with the strongest one-shot baseline at 78.2 percent and rises to 91.1 percent, so its smaller 12.9-point gain should not be read as a weakness of relay; it partly reflects less room to recover.

This breadth supports a protocol-level interpretation. The relay is not merely patching one answer format or one mathematical domain. It is giving the runner repeated opportunities to repair representations, programs, and submissions across several kinds of problem. The evidence remains descriptive, however: category sizes range from 38 counting-and-probability problems to 124 algebra problems, and no uncertainty intervals are reported here.

Relay Converts Interface Failures into More Informative Mathematical Failures

Relay sharply reduces rejected code and missing-result failures on MATH-500; most remaining relay errors are wrong submitted results.

The failure taxonomy makes the mechanism easier to see. In MATH-500 one-shot mode, 92 runs ended as CODE_REJECTED, 45 reached no parseable result line, 14 produced no usable code or result, and 31 submitted a wrong result. Relay reduced the total wrong-or-failed count from 182 to 55. Among those 55, only five were still interface failures—two with no usable code or result and three with no result line—while 50 were wrong submitted answers.

That shift is valuable even when it does not produce a correct score. A rejected program says little about whether the mathematics was close. A wrong submitted answer is a later-stage failure: the system at least produced executable work and made a commitment that can be inspected. Relay does not eliminate error; it moves many failures past syntax, safety, and submission barriers so the remaining error set is more directly about reasoning or final transformation.

AIME is smaller and less stable, but its categories tell the same story with a different residue. One-shot failures include seven missing-result lines, three rejected programs, and two wrong results. Relay failures include one rejected program, five attempts that produced no usable code or result, two wrong numeric results, and one wrong text result. The strict integer contract makes those labels unusually transparent: NO_RESULT_LINE is an interface failure, whereas a submitted integer that the offline verifier rejects is a mathematical or final-transformation failure.

The Compute Tradeoff Depends on the Benchmark and the Failure Path

Per-problem token pairs show that relay often generates fewer output tokens on MATH-500 but substantially more on difficult AIME failures.

The token plot corrects another tempting simplification: more attempts do not automatically mean more generated output. On MATH-500, relay used 1.45 attempts on average but produced 743.7 output tokens per problem, below the one-shot average of 810.5. The paired median tells the same story for problems correct in both modes: 188 relay output tokens versus 308 one-shot tokens. The protocol can stop a short successful attempt early, while the one-shot configuration permits one long generation. MATH-500 relay nevertheless consumed more prompt context—1,706.6 prompt tokens on average versus 756.3—because feedback and prior state must be carried into later attempts.

AIME is different. Relay increased average output from 2,012.4 to 4,086.7 tokens, average attempts from 1.00 to 2.37, and mean elapsed time from 7.4 to 21.4 seconds. Its eight problems that remained wrong in both modes are especially expensive: the median relay output was 12,175 tokens, compared with 3,207.5 in one-shot. By contrast, the four recovered AIME problems used a median of 2,551 relay output tokens versus 1,693.5 one-shot tokens. These observations suggest a practical stopping problem: the hardest unresolved traces can consume most of the available budget without producing a correct commitment.

Output tokens are only one component of inference cost, so they should not be treated as a complete efficiency measure. Prompt tokens, number of model calls, wall-clock time, and local hardware utilization all matter. The present data show a conditional tradeoff: relay can be economical when it terminates quickly or replaces a long one-shot failure, and expensive when a difficult case repeatedly fails to converge.

Use the slider above to compare verified accuracy, submission rate, and average attempts from the same completed runs. The charts and slider use observed aggregate values only; they do not interpolate unmeasured conditions.

The most important change is not only the additional correct answers. It is the change in observability. When one-shot fails, the trace usually stops at a broken artifact. When relay fails, the trace often shows a sequence of attempts, each with a verdict, safety diagnosis, stdout, stderr, parsed result status, and final submission state. This makes the experiment feel less like grading a sealed exam and more like reading a lab notebook.

Those gains are meaningful only inside the hardware and model budget that produced them. The next section defines that experimental boundary before turning to the runner itself.

Experimental Context: Model, Hardware, and Budgets

The repository's environment template and setup guide configure an Ollama-compatible generation endpoint—an HTTP service that runs the language model locally—and identify the model as hf.co/unsloth/Qwen3.8-27B-GGUF:UD-Q3_K_XL. That long tag names a compressed GGUF package of a 27-billion-parameter Qwen model; Q3 indicates an aggressively quantized representation intended to reduce memory use. The recorded experiment used a single 24 GB RTX 4090-class GPU. Those details define the envelope for the reported measurements: a quantized local model, finite context, finite output budgets, and real wall-clock cost for every additional attempt.

This constraint is part of the research question. A relay that always consumes five full generations would be difficult to justify on local hardware. In the completed MATH-500 run, however, relay mode had five attempts available but averaged 1.45 attempts. Many problems ended immediately; additional inference was concentrated on cases where the first representation, code block, or submission protocol failed. The AIME relay used more work—2.37 attempts on average—which is consistent with its smaller but stricter contest set.

The comparison is between execution protocols, not merely token counts. One-shot mode receives one uninterrupted generation and one execution chance. Relay mode receives smaller bounded attempts plus a controlled feedback channel. The experiment therefore asks whether structure at inference time can recover value that would otherwise be lost to an invalid code block, a runtime crash, or an unsubmitted answer.

Because the protocol shapes what can be observed and retried, the runner is part of the method rather than a neutral wrapper around the model.

Inside the Relay Runner

The relay runner separates bounded generation, code safety, sandboxed execution, result parsing, and hidden scoring.

The relay is not just a prompt trick. It is a small execution system. The core loop lives in src/relay_reasoning/qwen_math500_tool_relay_experiment.py, the current batch runner for tool-using MATH-500 and AIME experiments. It does the following work:

1. It builds a prompt for either one_shot or relay.

2. It sends the prompt to the local Ollama generation endpoint.

3. It extracts fenced Python code from the model response.

4. It runs static safety checks over the Python abstract syntax tree.

5. It executes the selected code in an isolated temporary directory.

6. It parses stdout for RESULT and CONFIDENCE.

7. It compares the submitted answer to the hidden verifier only after a result is submitted.

8. It writes trace artifacts such as model responses, extracted code, execution feedback, graph reports, and final reports. The README's output-file guide defines these filenames before any individual trace is discussed below.

9. In relay mode, if no final answer was submitted, it packages operational feedback for the next attempt.

The important design feature is that the model never receives the hidden verifier judgment while it can still act. The feedback channel is deliberately narrower than the scoring channel. The model can see that its code crashed. It can see that stdout contained Solutions: {n: 7}. It can see that no RESULT was parsed. It cannot see "correct" or "incorrect" from the answer key until after the run is over, and even then that is written only to offline reports.

The runner itself uses ordinary Python modules for orchestration and reporting. It uses argparse for command-line configuration, ast for static code inspection, html for escaping report content, json for structured reports, math and re for utility parsing, subprocess for execution, sys to select the current Python interpreter, tempfile to create isolated execution directories, time for timing, collections.defaultdict for internal counting examples, pathlib.Path for filesystem paths, and typing.Any for annotations. It also uses requests to call the Ollama API and SymPy, a Python library for exact symbolic algebra, to normalize and compare mathematical answers. The SymPy side includes exact objects and functions such as Integer, Rational, sqrt, pi, trigonometric functions, simplify, and parse_expr.

There are therefore two different kinds of Python in the system. The first is runner Python, which is trusted code written by us. It manages prompts, parsing, sandboxing, reporting, and hidden verification. The second is model-written Python, which is untrusted code emitted by the language model. The runner treats that code cautiously. It extracts it, checks it, executes it with limits, and records only bounded output.

That distinction leads directly to the execution boundary: useful feedback must be preserved without allowing model-written code to inspect the benchmark or the host environment.

A Controlled Python Execution Loop

The sandbox begins before execution. The runner first looks for fenced Python blocks using extract_python_blocks. It can handle normal Python fences, and it can also handle an open fence continuation when the model starts a Python block but forgets to close it. If several blocks exist, select_executable_code_block scans them from the end and prefers the last safe block. This detail matters because local models often produce a toy block, then a corrected block, then a final block.

After extraction, validate_python_code parses the code with ast.parse. If parsing fails, the code is rejected before execution. If parsing succeeds, the runner walks the syntax tree and rejects dangerous or out-of-contract constructs. In the current runner, the model-facing import allowlist is:

collections
functools
itertools
math
operator

That safety gate blocks names such as:

__import__, compile, delattr, dir, eval, exec, getattr, globals,
input, locals, open, setattr, vars

It also blocks import roots such as:

builtins, ctypes, ftplib, glob, http, importlib, inspect, multiprocessing,
os, pathlib, pickle, requests, shutil, socket, subprocess, sys,
threading, urllib

The purpose is not to make a perfect security boundary. The purpose is to prevent the model-written code from reading files, using the network, spawning processes, introspecting the environment, or escaping into Python internals during a benchmark run. The model is supposed to compute mathematics, not inspect the benchmark directory.

The execution step adds another layer. The runner writes the selected code to a temporary directory with a prefix like qwen_model_python_, then calls:

sys.executable -I model_code.py

The -I flag starts Python in isolated mode, which reduces ambient influence from user site packages and environment variables. The runner also sets the current working directory to the temporary directory, captures stdout and stderr, uses shell=False, and enforces a timeout. In the saved MATH-500 run, the timeout was 15 seconds. Captured stdout and stderr are truncated to bounded lengths before being written into feedback files.

There is one subtle artifact detail. The saved problem 474 relay trace contains model-written sympy imports and marks them safe in its code diagnostics. The current runner on the main branch now shows a stricter allowlist that does not include sympy for model-written code, while trusted runner code still imports SymPy for answer normalization. The saved directory should therefore be read as a historical experimental artifact, not as a claim that it was regenerated under the present allowlist. The conceptual point is unchanged: trusted runner code may use richer libraries, while model-written code passes through a narrower safety gate.

Once the execution boundary is clear, the remaining protocol difference lives in the prompts: one mode demands an immediate commitment, while the other permits bounded continuation.

Two Prompt Contracts

One-shot and relay modes share a hidden verifier but allocate execution opportunities differently.

The one-shot and relay prompts share a common philosophy: do not solve in free prose, write one executable Python block, preserve exact values, print a final answer in a parseable RESULT line, and print CONFIDENCE when submitting. The difference is how they treat time.

Prompt feature	`one_shot_python`	`relay_python`
Execution chances	One Python execution chance.	Up to `max_attempts`, which was 5 in this run.
Output budget in the saved run	`num_predict: 12500`.	`num_predict: 2500` per attempt.
Context in the saved example	`num_ctx: 16384`.	`num_ctx: 8192`.
Submission rule	The code must print `RESULT: <answer>` in the single response.	Exploratory attempts may omit `RESULT`; the first `RESULT` is final.
Feedback after failure	No follow-up attempt.	Later prompts receive stdout, stderr, safety diagnostics, and compact progress.
Hidden verifier	Offline only.	Offline only.

The one-shot prompt says, in effect: this is your only chance, so solve and submit now. The relay prompt says something more procedural: you may explore before final submission, but once you print RESULT, the run ends. This creates an explicit distinction between a scratchpad phase and a commitment phase.

For MATH-500 problem 474, this distinction is the whole difference. The one-shot run discovered Solutions for n: {n: 7}, but crashed before printing a parseable RESULT. The relay run made similar mistakes, but those mistakes became future context. By attempt 5, the model no longer tried to index the SymPy solution object. It printed the answer directly.

Problem 474 makes that distinction concrete because its mathematics is easy to solve by hand while its execution trace exposes several separate system failures.

MATH-500 Problem 474: From Magic Square to Verified Answer

MATH-500 problem 474 presents a 3 × 3 magic square whose cells contain constants and expressions in n. In the benchmark input, the picture is encoded with Asymptote, a vector-graphics language commonly used in mathematical documents; the model must recover the visual grid from that drawing code. A magic square has one rule: every row, every column, and both main diagonals must have the same total. The task is to determine n. The grid is:

n + 1       1       n - 1
3         2n - 9      n
n - 3     n + 2       2

How a person would solve it

A person does not need to derive all eight line sums. The top and middle rows are enough because both must equal the same magic sum. The top row is (n + 1) + 1 + (n - 1) = 2n + 1. The middle row is 3 + (2n - 9) + n = 3n - 6. Setting them equal gives 2n + 1 = 3n - 6, and therefore n = 7.

A short human solution compares two required row sums, obtains n = 7, and verifies the complete square.

The final human step is verification. Substituting n = 7 produces the square 8, 1, 6 / 3, 5, 7 / 4, 9, 2. Every row, column, and diagonal sums to 15. At that point, the mathematics is finished. The only remaining action is to submit 7.

What the relay had to solve beyond the algebra

The model faced three additional interfaces that a person solving on paper can often take for granted. It had to reconstruct the grid from diagram coordinates, express the constraints in executable Python, and print the answer in the runner's exact RESULT format. The trace is therefore more than a record of algebra. It shows where representation, execution, verification, and submission separated from one another.

The answer appears on attempt 2; later attempts verify it and repair the submission path.

The saved final report records the run's control settings. prompt_style: relay selects the multi-attempt contract; max_attempts: 5 caps the number of model calls; num_predict: 2500 caps generated tokens per attempt; and python_timeout: 15 limits each program execution to fifteen seconds. The answer key is hidden from the model, and verifier_feedback_to_model: false explicitly records that the correctness judgment was withheld during the run. The report says the answer was submitted on attempt 5 as execution_candidate: "7" with confidence: 95. Only then did the offline verifier mark it correct.

The five attempts unfolded like this:

Attempt	Runner verdict	What happened
1	`NO_RESULT_LINE`	The model wrote safe Python, but it misread the grid orientation and derived inconsistent line sums. The solver found no solution.
2	`NO_RESULT_LINE`	The model reread the diagram coordinates correctly and found `Solutions: {n: 7}`, but it did not emit a final `RESULT`.
3	`NO_RESULT_LINE`	The model verified that all rows, columns, and diagonals evaluate to `15` when `n = 7`, but the code crashed before printing the final answer because it treated a SymPy dictionary like a list.
4	`NO_RESULT_LINE`	The same mathematical result was preserved, and the same submission bug remained.
5	`VERIFIED_CORRECT_RESULT`	The model printed `RESULT: 7` and `CONFIDENCE: 95`; the offline verifier marked it correct.

The central mathematical representation, once fixed, was:

Top row:     n + 1,     1,       n - 1
Middle row:  3,         2n - 9,  n
Bottom row:  n - 3,     n + 2,   2

From that representation, the sums are:

r0 = 2n + 1
r1 = 3n - 6
r2 = 2n + 1

c0 = 2n + 1
c1 = 3n - 6
c2 = 2n + 1

d1 = 3n - 6
d2 = 4n - 13

The equality 2n + 1 = 3n - 6 gives n = 7. Substituting n = 7, the common sum is 15:

r0 = r1 = r2 = 15
c0 = c1 = c2 = 15
d1 = d2 = 15

The final code did not need a grand derivation. It needed to submit the already-verified answer:

RESULT: 7
CONFIDENCE: 95

This is exactly why the example is useful. The model did not fail because it could not do the math. It failed twice after doing the math because the result was trapped behind a small interface mistake. In attempt 3 and attempt 4, the stdout already contained the successful verification, but the program then crashed at sols[0] because solve(eqs, n) had produced a dictionary-like object, {n: 7}, rather than a list whose first item could be indexed.

One-Shot Failure and Relay Recovery

One shot discovers n = 7 but crashes; relay preserves the candidate and repairs the blocking interface error.

The one-shot trace is almost painful because it is so close to success. The model response correctly recognizes that the Asymptote drawing—the vector description of the problem's diagram—creates a 3 by 3 grid, not a 4 by 4 grid. It maps the nine labels into rows and columns, builds row sums, column sums, and diagonal sums, then calls SymPy's solver. The execution feedback contains:

Solutions for n: {n: 7}

But then the program crashes:

KeyError: 0

The crash happens because the model tries to read solutions[0]. The returned object behaves like a mapping from the symbol n to the value 7, not like a list. Because the crash occurs before print(f"RESULT: {n_val}"), the runner cannot parse a submitted answer. The final one-shot report therefore says NO_RESULT_LINE, answer_submitted: false, and solved: false.

The relay trace has a different rhythm. Attempt 1 makes a representation error. It reads the grid in a way that produces incompatible equations and Solutions: []. This is useful negative evidence. It says the model should not trust that coordinate interpretation. Attempt 2 repairs the coordinate interpretation and prints a coherent system with Solutions: {n: 7}. That is useful positive evidence. Attempts 3 and 4 preserve the candidate and verify the magic sum, but keep crashing on the same sols[0] issue. Attempt 5 finally stops trying to be clever about the solver return value and prints:

RESULT: 7
CONFIDENCE: 95

The relay is therefore not doing mystical reflection. It is carrying concrete artifacts:

Carried artifact	Example from the trace	Why it matters
Previous verdict	`NO_RESULT_LINE`	Tells the next attempt that no answer was accepted.
Code safety result	`safe: true`	Separates unsafe-code failures from math failures.
stdout	`Solutions: {n: 7}`	Preserves useful mathematical evidence.
stderr	`KeyError: 0`	Identifies the programming bug blocking submission.
Parsed result	`UNKNOWN` until attempt 5	Makes clear that the runner did not see a final answer.
Parsed confidence	`UNKNOWN` until attempt 5	Makes clear that no final confidence was submitted.
Hidden verifier	not included in progress	Prevents answer-key leakage.

This is the core insight that the relay carries out slowly. The model first needs a faithful representation of the diagram. Then it needs equations. Then it needs a candidate. Then it needs substitution checks. Then it needs a final protocol-compliant answer. In a single-shot run, all of those have to succeed in one uninterrupted completion. In a relay run, each layer can leave evidence for the next layer.

The AIME trace tests the same control idea on a different failure mode: numerical magnitude and formatting replace diagram interpretation as the main sources of friction.

AIME Problem 6: Recovering from a Numerical Dead End

Relay replaces an unstable numerical path with an exact Vieta-based shortcut and submits 441.

The AIME result that best complements the MATH-500 magic-square trace is AIME 2026 problem 6. The directory name aime26_06 encodes the competition year and problem number; beneath it, the repository stores separate one-shot and relay traces. The problem asks:

A real number x satisfies (x^(log_2026 x))^(1/20) = 26x.
What is the number of positive divisors of the product of all possible
positive values of x?

This is a compact contest problem, but it is not merely computational. It requires the solver to choose the right change of variables, preserve the domain of positive real solutions, use a root-sum identity, factor an integer, and submit a single integer. That makes it a good AIME companion to the MATH-500 diagram problem. The MATH-500 case tested whether the model could convert a picture into equations. This AIME case tests whether the model can convert an exponential-logarithmic equation into a finite arithmetic answer without getting lost in numerical scale.

The one-shot run is revealing because it produced useful evidence and still failed the benchmark contract. Its saved execution feedback included:

2026 factors: {2: 1, 1013: 1}
Number of divisors: 9261
t1 = -0.419... t2 = 20.419...
Check x1: True
OverflowError: Numerical result out of range
parsed_result: UNKNOWN

There are two separate issues hiding in that trace. First, the program crashed while trying to numerically check the larger value of x, so it never reached a parseable RESULT line. The answer was therefore not submitted, and the hidden verifier was not checked. Second, the printed divisor count 9261 was not the right final answer for the product. The code had correctly factored 2026 as {2: 1, 1013: 1}, but it still drifted into an inconsistent divisor calculation. This is exactly the kind of failure that is easy to miss if we only look at a model's prose. The trace shows both the mathematical direction and the point where the computation became unreliable.

The relay run failed differently on its first attempt. It did not provide a fenced Python block at all. In a one-shot benchmark, that kind of formatting failure would end the run. In relay mode, it became operational feedback saved in the problem 6 relay directory:

no fenced Python code block was found;
next response must start with a fenced Python code block

The second relay attempt then used the feedback channel correctly. It produced executable code, rederived the problem from first principles, and submitted:

2026 = 2026
Factorization of 2026: {2: 1, 1013: 1}
Is 1013 prime? True
Number of divisors of 2026^20: 441
RESULT: 441
CONFIDENCE: 95

The mathematical spine of the solution is short once the representation is right. Let:

t = log_2026(x), so x = 2026^t

The original equation becomes:

x^(t/20) = 26x
x^(t/20 - 1) = 26

Taking logarithms base 2026 gives:

(t/20 - 1)t = log_2026(26)
t^2 - 20t - 20 log_2026(26) = 0

The two possible positive values of x correspond to the two real roots of this quadratic in t. The product of the two x values is:

2026^(t1) * 2026^(t2) = 2026^(t1 + t2)

By Vieta's formula, t1 + t2 = 20, so the product is:

2026^20

Since:

2026 = 2 * 1013

the product factors as:

2026^20 = 2^20 * 1013^20

The number of positive divisors is therefore:

(20 + 1)(20 + 1) = 441

This AIME trace is a cleaner version of the human reasoning story. A person solving the problem might first try to evaluate the equation numerically, notice that the numbers become enormous, step back, introduce t = log_2026(x), and then recognize the root-sum shortcut. Relay Python creates a mechanical version of that rhythm. It does not ask the hidden answer key for help. It only carries forward what the system is allowed to know: the previous attempt did not submit, the code shape was invalid or valid, the stdout contained certain facts, the stderr contained certain failures, and the final answer was not yet parseable.

That is why the AIME result matters. The improvement from 18 correct to 21 correct is not just a scoreboard bump. It is evidence that a small local reasoning system can sometimes recover from the ordinary frictions of mathematical work: a missing code fence, a crashed numerical check, a wrong intermediate divisor count, or a submission line that never arrives. The regression on AIME 2026 problem 16 is equally important. It shows that the relay is a procedure, not a guarantee. A procedure can make better use of evidence, but it can also commit to a worse path if the evidence is interpreted poorly.

Together, the two examples reveal four recurring design principles. They begin with representation, continue through execution and feedback, and end with an explicit commitment.

Representation Comes Before Computation

Mathematical reasoning begins before any equation is solved. The first act is translation. Here, the diagram was written in Asymptote, which places labels at coordinates like (.5, .5), (1.5, 2.5), and (2.5, 1.5). The model had to infer that y = 2.5 is the top row, y = 1.5 is the middle row, and y = .5 is the bottom row.

Attempt 1 got this wrong. It assembled a grid that did not correspond to the picture, so the equations contradicted one another. That is an important kind of failure because the solver behaved correctly on a bad model of the world. The Python code computed line sums, compared them, and found no solution. The failure was not arithmetic. It was representational.

This is a recurring pattern in LLM reasoning. A model can be fluent at symbolic manipulation while still losing the problem at the boundary between text, diagram, notation, and code. Benchmarks like MATH were designed partly because high school competition problems stress this kind of flexible translation, not merely computation (Hendrycks et al., 2021). In this trace, the diagram-to-grid conversion was the real bottleneck.

Execution Turns Assumptions into Evidence

Once the model corrected the grid in attempt 2, the interpreter became a microscope. It printed each row sum, each column sum, each diagonal sum, and the candidate solution:

r0 = 2*n + 1
r1 = 3*n - 6
r2 = 2*n + 1
c0 = 2*n + 1
c1 = 3*n - 6
c2 = 2*n + 1
d1 = 3*n - 6
d2 = 4*n - 13
Solutions: {n: 7}

This is the Program-aided Language Models idea in miniature. PAL argues that the language model should handle decomposition while a symbolic runtime handles execution (Gao et al., 2022). The split is powerful because the runtime has no patience for vibes. Either the equations line up or they do not. Either the code runs or it crashes. Either the output contains a parseable answer or it does not.

Toolformer approaches tool use from a different direction, showing that models can learn when and how to call external tools through self-supervised training signals (Schick et al., 2023). The MATH-500 relay is simpler than Toolformer as a learning method, but it lives in the same design space: the model's tokens are not the whole computation. Some of the work is moved into a tool, and the tool's output becomes evidence for the next step.

Feedback Must Lead to a Different Action

Attempt 2 had the answer, but the runner did not accept it because there was no final RESULT line. Attempts 3 and 4 then showed a second class of failure. The model verified the answer but tried to print it through the wrong data access pattern:

KeyError: 0

This is not a math failure. It is a protocol and programming failure. The code already showed that n = 7 makes every row, column, and diagonal equal to 15. But because the runner treats the first parseable RESULT as the final answer submission, stdout that merely contains useful evidence is not enough. The system needs a clean final line.

This is where iterative feedback matters. ReAct frames reasoning and acting as interleaved: reasoning updates the plan, actions interact with an environment, and observations shape later moves (Yao et al., 2022). Self-Refine similarly studies how models can improve outputs over multiple rounds of feedback and revision without new training (Madaan et al., 2023). In this MATH-500 trace, the feedback is not a natural-language critique from the model itself. It is operational feedback from the runtime. The model sees the stdout and stderr. It can then repair the next attempt.

The distinction matters because not all feedback should be allowed. If the hidden verifier says "correct" or "incorrect" and that signal reaches the model during the same benchmark run, the evaluation leaks. This run avoids that. The progress handed to later attempts contains safety and execution information, such as exit_code, stdout, and stderr, but the hidden correctness check remains offline. The model can learn from its own program's behavior, but it cannot ask the answer key whether it is right.

Submission Is a Commitment

A reasoning trace can contain many candidates, but an evaluation needs one answer. This is why the runner's policy is useful:

answer_submission_policy: first_result_is_final
verifier_feedback_to_model: false

The policy separates exploration from commitment. Before RESULT, the model may run exploratory code and inspect operational feedback. After RESULT, the run ends. That makes the setup stricter than an interactive tutor and cleaner than a chat transcript where the model can revise an answer after seeing whether it was right.

This resembles verifier-based evaluation in spirit, although the verifier here is used only for offline scoring. Cobbe et al. (2021) studied trained verifiers for math word problems, where many candidate solutions are generated and a verifier ranks them. In this local benchmark, there is no trained verifier selecting among candidates for the model. Instead, the hidden answer key audits the final submitted result. The important methodological point is the same: generation and verification should be conceptually separated.

Five Layers of a Reliable Reasoning Trace

The trace can be decomposed into five smaller concepts.

First, the model must parse. It reads a problem statement, including diagram code, and maps it into objects. In this case, those objects are cells in a grid.

Second, it must formalize. It turns "magic square" into equality constraints across rows, columns, and diagonals.

Third, it must compute. It uses symbolic algebra to solve those constraints.

Fourth, it must verify. It substitutes the candidate answer back into the original constraints and checks that every required equality holds.

Fifth, it must submit. It emits the answer in the exact protocol expected by the runner.

The attempt history shows each layer separately. Attempt 1 failed at parsing. Attempt 2 succeeded at parsing, formalization, and computation, but failed at submission. Attempts 3 and 4 succeeded at verification, but failed at runtime robustness. Attempt 5 finally aligned all layers.

That is the practical lesson: "the model reasoned" is too vague. A better question is which layer succeeded and which layer failed. Did it misunderstand the input? Did it write the wrong equations? Did it compute incorrectly? Did it verify too weakly? Did it fail to present the answer? Once those layers are separated, debugging becomes much more humane.

What AIME Adds to the Story

AIME-style tasks raise the stakes because the final answer is an integer, and the benchmark should not reward vague symbolic residue or accidental formatting. The completed AIME run confirms that this strictness is useful. In one-shot mode, the model solved 18 of 30 problems, submitted only 20 answers, and left 7 runs without a parseable result line. In relay mode, the model solved 21 of 30, submitted 24 answers, and converted four one-shot non-submissions into verified correct results.

The important point is not that every retry is good. The important point is that the benchmark becomes more diagnostic. A one-shot failure can collapse together many causes: misunderstood problem, invalid code, runtime error, missing RESULT, wrong arithmetic, or bad final formatting. A relay trace separates those causes. It lets us say, "the model solved the equations but mishandled the answer object," or "the model found a product structure but crashed during a numerical check," instead of merely saying, "the model got it wrong."

That separation is especially valuable for AIME because the answer contract is simple. The final result should be an integer, so failures are less likely to be hidden inside symbolic equivalence debates. If a run prints no result, the interface failed. If a run prints an integer and the offline verifier marks it wrong, the mathematics or the final transformation failed. If relay improves a case, the trace shows what evidence carried forward. If relay regresses, the trace shows where the repair loop became a detour.

Tree of Thoughts makes a related argument at a larger scale: some problems require exploration, lookahead, and backtracking rather than a single left-to-right generation (Yao et al., 2023). The relay trace is not a full tree search, but it has a similar flavor. The model explores a representation, observes that it fails, repairs the representation, observes a candidate, then repairs the final output path. The path is linear, but the control idea is broader: reasoning improves when the system can externalize intermediate states and evaluate them before committing.

Relationship to Prior Work

Relay reasoning belongs to a family of methods that expose, test, or revise intermediate work, but its control contract is distinct.

Approach	What is carried forward	How relay reasoning is similar	How relay reasoning differs
Chain-of-thought prompting (Wei et al., 2022)	A single textual derivation	Both make intermediate structure available	A relay is multi-attempt and may carry executable observations rather than one uninterrupted prose trace
Program-aided language models (Gao et al., 2022)	Generated programs and their computed results	Both delegate exact operations to a runtime	Relay adds bounded retries, operational handoffs, and an explicit final-answer contract
ReAct (Yao et al., 2022)	Interleaved reasoning, action, and observation	Both allow environment feedback to change the next step	The tested relay narrows feedback to execution state and hides answer-key correctness during the run
Self-Refine (Madaan et al., 2023) and Reflexion (Shinn et al., 2023)	Critique or verbal memory derived from prior attempts	All use iterative improvement without changing model weights	Relay feedback can be non-linguistic and externally produced: exit codes, parser state, stdout, stderr, and safety diagnostics
Tree of Thoughts (Yao et al., 2023)	Multiple branches with evaluation and possible backtracking	Both reject the assumption that one left-to-right generation is enough	The current relay is a linear handoff, not a tree search; it prioritizes continuity and economy over broad branching
Multi-agent debate (Du et al., 2023)	Multiple independent proposals and critiques	Both distribute reasoning across calls or agents	Debate spends compute on simultaneous alternatives; a relay assigns the next agent the frontier created by the previous one
Human cumulative culture and stigmergy	Socially transmitted practices or persistent environmental cues	All let later actors begin from externally available state	Human and biological systems are open-ended and embodied; the relay here is engineered, typed, benchmarked, and governed by a hidden-verifier boundary

The last distinction is the one worth protecting. Calling every iterative loop a relay would make the term meaningless. In this experiment, a relay has four defining features: a bounded attempt has a clear local task; tested state is transferred forward; the next attempt is expected to extend rather than reenact the trace; and a separate commitment rule decides when exploration ends.

Reasoning Should Travel

The magic square example is small, and the AIME logarithm example is compact, but together they capture a larger idea. Reasoning is not just producing a beautiful explanation. Reasoning is maintaining a faithful representation, using tools to test that representation, listening to failures without leaking the answer, and committing only after the result has survived checks.

In MATH-500 problem 474, the final answer was 7. In AIME problem 6, the final answer was 441. The deeper result is the trace between the prompt and those integers. It shows that we should evaluate reasoning systems not only by whether they eventually land on the right number, but by whether their process gives us handles: where the problem was parsed, where the equations were formed, where the code was executed, where the answer was verified, and where the final submission became irreversible.

That is the shape of reasoning I want more benchmarks to expose: not a polished monologue, and not a pile of independent restarts, but a controlled sequence in which evidence survives, errors become useful, and new compute begins at the frontier. Reasoning should travel.

References

1. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training verifiers to solve math word problems. arXiv. https://doi.org/10.48550/arXiv.2110.14168

2. Couzin, I. D. (2009). Collective cognition in animal groups. Trends in Cognitive Sciences, 13(1), 36–43. https://doi.org/10.1016/j.tics.2008.10.002

3. Dean, L. G., Kendal, R. L., Schapiro, S. J., Thierry, B., & Laland, K. N. (2012). Identification of the social and cognitive processes underlying human cumulative culture. Science, 335(6072), 1114–1118. https://doi.org/10.1126/science.1213969

4. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023). Improving factuality and reasoning in language models through multiagent debate. arXiv. https://doi.org/10.48550/arXiv.2305.14325

5. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022). PAL: Program-aided language models. arXiv. https://doi.org/10.48550/arXiv.2211.10435

6. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. NeurIPS Datasets and Benchmarks. https://github.com/hendrycks/math

7. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K. M., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-Refine: Iterative refinement with self-feedback. arXiv. https://doi.org/10.48550/arXiv.2303.17651

8. Mercier, H., & Sperber, D. (2011). Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2), 57–74. https://doi.org/10.1017/S0140525X10000968

9. Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. arXiv. https://doi.org/10.48550/arXiv.2302.04761

10. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2303.11366

11. Theraulaz, G., & Bonabeau, E. (1999). A brief history of stigmergy. Artificial Life, 5(2), 97–116. https://doi.org/10.1162/106454699568700

12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv. https://doi.org/10.48550/arXiv.2201.11903

13. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. arXiv. https://doi.org/10.48550/arXiv.2305.10601

14. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). ReAct: Synergizing reasoning and acting in language models. arXiv. https://doi.org/10.48550/arXiv.2210.03629