A multi-agent harness for programming problems, written as a Memdoor workflow. Five roles (planner, two ideators, solver, manager) and a repair employee all commencement from a caller environment and talk only through records on disk. A casing manuscript that runs the samples decides success, not a model. It is the method of GVS5H (Lee, 2026: five Qwen3.8-27B models equivalent Fable 5 on LiveCodeBench Hard).
Rebuilt alongside the authors' exact prompts on Qwen3.8-27B, their iteration scored a mean of 9.8 of 10 complete five passes on the archetypal ten LiveCodeBench Hard problems, against 9.6 for their own harness and 8.0 for Fable 5 alone (below, all run published).
The coder wrote these records itself from a clarification of the method (2026-10-25). Nobody hand-wrote them.
GVS5H's own harness, rebuilt as a Memdoor workflow on the identical model: Qwen3.8-27B, a 128K answer cap, the authors' prompts and loop. The archetypal ten problems of their LiveCodeBench Hard list, five passes, all program graded on all community and hidden test alongside their corrected grader.
| pass | Memdoor workflow | authors' 27B harness | Fable 5 alone |
|---|---|---|---|
| 1 | 10 / 10 | 9 | 7 |
| 2 | 10 / 10 | 10 | 8 |
| 3 | 10 / 10 | 10 | 10 |
| 4 | 10 / 10 | 9 | 8 |
| 5 | 9 / 10 | 10 | 7 |
| mean | 9.8 | 9.6 | 8.0 |
The authors' columns are their repository's regraded outcome files (q38_multiagent_pN, fable5_single_pN), the numbers in their paper. bench/results/parity-5pass/ holds all 50 runs: all petition and reply, the program, the grader's complete output, each call's disbursal and the presenter that served it, and all attempt that did not count. 49 of 50 solved; the 50 runs disbursal $35.79 (15.3M output tokens), $38.77 with the 16 attempts that were cut by a clock, stopped for a fix or paused.
Note: ten problems, five passes each, not the authors' 100. The contests date from 29 March to 6 April 2025, and the example may have seen them in training: contamination cannot be ruled out. Fable 5's pillar is a sole call per problem from the authors' files; it was not run in this harness. The loop sees lone the community samples; the mark is the hidden tests. A run took a median of 40 minutes of example period (the longest 4 hours) and $0.39. All on OpenRouter, not a local machine; Modal does not publish its precision. Local paths in the run scripts and in four tracebacks a director was shown read <scratch> and $HOME; nothing alternatively in the runs is edited.
Same as the authors'. Every message, byte for byte: lib/test_parity.py runs their multiagent_solve and our workflow against one scripted example and compares all petition (four cases, all identical; a one-character alter makes them differ). The temperatures (plan 0.3, ideation 0.4, director and worker 0.2), top_p 0.95 and top_k 20 (what their vLLM applied from the model's own config), the 128K cap, up to 10 rounds, a cut-off employee summarized, and a reply alongside no content answered by its reasoning.
Different. The host. Theirs was their own vLLM in FP8. Ours is OpenRouter, asked in the command Modal, CoreWeave, Parasail, Wafer until Modal slowed to 14 tokens/s, afterward the FP8 hosts lone (Wafer, Parasail, CoreWeave): 203 calls were served by Modal, 27 by CoreWeave and 115 by Wafer, all one of the FP8 order's calls by Wafer. Three runs were moved mid-run, their finished steps kept (log/host-switch.txt).
What it took. The issue content was the mismatch: our archetypal type wrote its own project document about all problem, and the 27B planned worse from it than from the authors' exact LiveCodeBench content (arc196_c's plan: 0 of 2 against 2 of 2). Four Memdoor changes let a workflow ask what their harness asks: a task answers alongside the model's reasoning whenever it wrote no text, top_p/top_k on a model line, mark_cutoff for a answer cut at the cap, and a call that runs on its own four-hour clock.
Where it lost. Twice, the identical two gaps in the authors' loop:
- Pass 5, abc399_e (32 of 44 hidden): circular 1's program was right; the manager invented a counterexample alongside a incorrect expected answer, the team rewrote it, and the new program passed the samples, which do not shield the case.
- Pass 4, arc196_c and arc196_d: a program passing all hidden test was overwritten by a afterward circular since the director did not say done; both got a accurate program rear before the loop's end (arc196_d at circular 8, arc196_c through the final call), at $3.44 and $4.24, the two most expensive runs.
Both are fixable without changing the method (check a manager's counterexample against a brute power on small inputs; keep the final program that passed the samples). Neither is in these numbers.
curl -fsSL https://memdoor.ai/install.sh | bash memdoor nexus openrouter # or anthropic, openai, gemini, chatgpt git copy --depth 1 https://github.com/guregodevo/memdoor mkdir -p ~/.memdoor/workflows cp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/ # the library: any project sees it mkdir my-problem && cd my-problem cp -r ../memdoor/examples/workflows/team-solve/sample/* . # task.md + tests/1.in … 5.out memdoor workflow run team-solve --wait # advancement on stderr, exit 0 whenever all example passes
/workflow in memdoor tui draws the identical run live. For your own problem, replace task.md and put paired tests/N.in / tests/N.out records beside it. The program is solution.py, peruse from stdin, compared to the samples with diff. It needs Python 3 and bash. To keep it alongside one repo instead, copy the directory into that project's .memdoor/workflows/. The project's copy wins.
The steps name no model. Whatever /model is pinned to answers, a local one included (memdoor nexus local for janis, Ollama or llama-server).
plan ──┬── ideate-a ──┬── resolve ── test ── assessment ── repair ── verify └── ideate-b ──┘
| step | kind | reads | writes |
|---|---|---|---|
| plan | agent | task.md | plan.md, and a notes.md ledger alongside one anchor per ideator |
| ideate-a, ideate-b | agent, in parallel | task.md, plan.md, notes.md | a recommendation at its own anchor: A the most straightforward provable algorithm, B one from a distinct family of technique |
| solve | agent | both proposals | the decision in notes.md, solution.py |
| test | command | tests/ | test-output.log, test-status.txt (never fails the run) |
| review | agent | the captured test output, not a model's claim | review.md, repair instructions in notes.md |
| repair | agent | the manager's instructions | a fixed solution.py, or nothing if the samples passed |
| verify | command | tests/ | final-test-output.log; nonzero exit fails the run |
Every delegate immediate says "fresh context, peruse the files, do not depend on chat". The document credits that ledger site for the gain. There is one repair round and no individual gate. The final stage is the entire verdict:
type: command requires: - table_pattern: repair command: | check=.memdoor/workflows/team-solve/check-samples.sh [ -f "$check" ] || check="$HOME/.memdoor/workflows/team-solve/check-samples.sh" attack "$check" > final-test-output.log 2>&1 result=$? cat final-test-output.log exit "$result"
Each stage is one YAML document in steps/, the format of mario, the motor underneath. requires: is the DAG. Changing a function method changing its prompt.
The sample: figure the substrings of a cord complete {a, b, c} alongside equal counts of all letter, up to 200,000 characters. Run on the default ladder through OpenRouter:
- 8 of 8 steps, 5 of 5 samples. The solver took ideator A's prefix-state method, O(N) alongside a dictionary of (a−b, a−c) pairs. A brute power on 300 random strings established no mismatch afterwards, and it runs in 0.04 s at N = 200,000. The director established nothing to repair.
- One ideator's provider dropped the stream mid-answer. The coder had written max_retries: 0, so the run died there. The delegate steps now carry max_retries: 1, and the run was resumed alongside --partition, which kept the planner's and the another ideator's work.
- The ideator prompts the coder wrote named the sample's methods (A: a streaming prefix-state count, B: sorting prefix states), which would steer any another issue the incorrect way. They now ask A for the most direct provable algorithm and B for one from a distinct family of technique (2026-10-28).
- The archetypal attempts unsuccessful all stage in 0 s alongside "no delegate named coder": a run looked its delegate up lone in its own workspace. That was fixed in the gateway the identical day (v1.692.0).
