Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard
- ID
- 41684
- Status
- summarized
- Published
- 01 Nov 2026, 6:46 AM
- Fetched
- 01 Nov 2026, 8:26 PM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve
- Source URL
- https://hnrss.org/best
Summary
- Score
- 6.5
- Created
- 01 Nov 2026, 8:26 PM
- Tags
- Audience
- developersai_agent_usersai_ml_learnersvibe_coders
What happened
The memdoor 'team-solve' example implements GVS5H (Lee, 2026) as eight YAML files: five Qwen3.8-27B instances — planner, two ideators, solver, manager — plus a repair worker, each starting from a fresh context and communicating only through files on disk, with a script (not a model) deciding success. On the first ten LiveCodeBench Hard problems, five passes each, graded on hidden tests, the workflow scored 9.8 mean vs 9.6 for the original authors' harness and 8.0 for Fable 5 in a single call; a run took a median of 40 minutes and $0.39 on OpenRouter, with all 50 runs costing $35.79. The README concedes the sample is ten problems rather than 100, that 9.8 and 9.6 are 'the same number' at this size, that training contamination can't be ruled out since the contests date from spring 2025, and that Fable 5's column was copied from the authors' files rather than run in this harness.
Why it matters
It puts a real price on multi-agent orchestration: ~$0.39 and 40 minutes per problem to reach parity with a single-call baseline, so the decision is whether your task justifies five model calls plus a file-based coordination layer — not whether agents 'work'. The documented failure is the useful part: the manager invented a wrong counterexample and the team overwrote a correct program (twice more elsewhere, later recovered), meaning your orchestration layer needs a guard against the manager discarding passing output before you ship anything resembling this.
Discussion angle
The manager overwrote a correct program with a wrong-counterexample rewrite — is the fix a stricter manager prompt, a diff-gate that only accepts changes passing the public samples, or is five-model coordination just not worth $0.39 and 40 minutes over one call?