Synthesizing Long-Horizon Terminal Tasks At Low Costs
Long-horizon agents need executable experience, not just answers. With Ever-Evolving Tasks, a task that keeps GPT-5.6-Sol working for more than 100 agent steps costs just $0.05 to generate.
Measured recursive growth
Executable complexity increases across measured synthesis rounds.
Synthesis roundR1
Measured medians: 67 solution lines and 40 solution commands.
01 · The Data Problem
Long-horizon agent data is hard to produce and expensive to scale.
A terminal trajectory must hold together across inspection, commands, failures, recovery, and verification. That is much harder than writing a prompt and answer.
Each example bundles a runnable workspace, a public goal, a reference solution, a verifier, and the trajectory itself. Human experts can build this data, but expert time does not scale.
Ever-Evolving Tasks
$0.05average synthesis cost per task
Project-reported average to synthesize one executable task bundle. Accepted tasks then become environments for trajectory collection.
Human trajectory work
$70–$130 / hourMercor advertised software engineers to reconstruct and annotate model trajectories on real coding tasks.
Mercor listing ↗Production complexity
900+ tasksTuring describes 450+ parent-child pairs with screen recordings, event logs, screenshots, and structured failure modes.
Turing case study ↗Market demand
$300M revenueReuters reported $300M in annual revenue and a network of 4M+ human experts.
Reuters report ↗These numbers use different units. Mercor lists hourly labor; Turing and Reuters describe scale, not per-task pricing. The $0.05 figure is our project-reported synthesis cost.
The bottleneck is the cost of producing executable practice at scale.
02 · The Synthesis Loop
Generate a task. Extend the solution. Verify it.
Each round rewrites a full executable task, not just the prompt. Grow the solution first, then align the verifier and instruction. Only sandbox-validated tasks survive.
Task generation
Choose a realistic extension.
Pick from five rewrite families and forty operators while keeping the seed recognizable.
Solution expansion
Grow the executable work first.
Add more inspection, commands, artifacts, and state dependencies; then realign the verifier and instruction.
Verification
Prove the task works end to end.
Filter weak rewrites, run the oracle in a fresh Daytona sandbox, and repair only bounded failures.
Oracle-valid
The reference solution passes in a fresh sandbox.
Contract-valid
Every semantic check is public or discoverable.
03 · Task Evolution Results
Measured complexity grows while domain coverage remains stable.
Later measured rounds require more executable work, and DeepSeek-V4-Pro succeeds less often on matched subsets. Candidate validation pass rate and measured domain diversity remain stable.
Measured · R1–R15
Growth, yield, and fixed-solver comparisons through R15.
Executable complexity grows faster than prompt length.
Select a metric above to highlight its R15-to-R1 median ratio.
A fixed solver falls behind.
Fixed-solver completion
90% → 2.5%DeepSeek-V4-Pro pass@4: solved in at least one of four attemptsVerifier partial credit
0.970 → 0.170mean fraction of verifier checks passedCandidate validation pass rate
74.5%–81.5%candidates passing final validation divided by all generated candidatesFigure 4 · Updated paper
Harder does not mean narrower.
Figure 4 tracks domain composition and stability from the seed pool through R15.
04 · Training Results
Training on generated trajectories is associated with higher Qwen3.5 benchmark scores.
We collect Qwen3.5 trajectories on synthesized tasks, use trajectories from R1–R3 for staged SFT, and evaluate the resulting checkpoints on three held-out terminal benchmarks. The cards report checkpoint-level score points from the base model to the three-round checkpoint; repeat-run uncertainty is not shown for SFT.
37,484 verified tasks→Qwen3.5 rollouts→SFT / RL→Held-out benchmarks
Supervised fine-tuning
Qwen3.5-27B
Base checkpoint → SFT through R1–R3Terminal-Bench 2 score41.2 → 47.9+6.7
Terminal-Bench Hard score22.7 → 28.3+5.6
LHTB mean partial-credit score18.1 → 22.4+4.3
Supervised fine-tuning
Qwen3.5-122B-A10B
Base checkpoint → SFT through R1–R3Terminal-Bench 2 score43.8 → 49.4+5.6
Terminal-Bench Hard score20.0 → 30.0+10.0
LHTB mean partial-credit score18.9 → 23.6+4.7
Reinforcement learning
Qwen3.5-27B base checkpoint → PPO checkpoint
Verifier reward≈0.11 → >0.14
Smoothed reward rises during PPO training.
Interaction horizon≈19–20 → >30 turns
Longer trajectories co-occur with higher verifier reward.
The data is not just harder. Training on it is associated with higher held-out benchmark scores.
Audit layer · Inspect the evidence
Don’t just trust our metrics.
Synthetic data is easy to distrust. Inspect the task change, model trajectory, and rubric decision for the same case yourself.
Task change → trajectory evidence → rubric decision
Viewer source code · GitHub ↗
Evidence workspace
Loading viewer evidence…
Select a file
Failed trajectory
—— turns
Successful trajectory
—— turns
Selected trajectory
Successful Harbor trajectory
Final—/ 100
A
—