EarlyEval Early Stopping: Up to 44.1% Fewer Input Tokens in Agent Evals, With Caveats

EarlyEval, a framework from researchers at Shanghai Jiao Tong University, Singapore Management University, East China Normal University and Shanghai Innovation Institute, halts agent benchmark runs once a classifier is confident of the outcome. The authors report (arXiv:2609.02783v1, lab-reported) up to 44.1% fewer input tokens and 29.4% fewer output tokens, with 13% to 26% fewer agent steps.

That 44.1% is a best case. It is an input-token reduction on one benchmark (Toolathlon), in evaluation runs rather than production agents, and it is not a dollar figure. This post lays out what the number covers, the full trade-off curve, and the evidence against generalising it.

What does the 44% in EarlyEval actually measure?

It measures the reduction in input (prompt) tokens when agent runs on a benchmark are stopped early, relative to running every trajectory to completion. The paper defines ΔToken_in and ΔToken_out as relative changes in input and output tokens, and the 44.1% comes from Toolathlon at a 0.90 decision threshold.

At that same operating point, steps fell 23.0% and output tokens fell 29.4%. Input tokens drop faster than steps because early termination removes the late, long-context turns. The authors say the savings come from eliminating the compounding cost of expanding context windows.

The paper's motivation is that a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars (authors' claim). It does not convert the token savings into dollars in the sections reviewed, so any "44% cheaper" headline is an extrapolation. Input and output tokens are billed at different rates by most providers, so the dollar effect depends on the model.

How does EarlyEval decide when to stop a run?

It trains two LightGBM classifiers on prefixes of past trajectories, one predicting success and one predicting failure. A run stops at the first step where either calibrated probability meets its threshold.

The design has four parts:

• Features. Behavioral (pacing, error and test signals, stalling patterns), textual (TF-IDF over word n-grams, compressed with truncated SVD to 64 or 128 dimensions per block) and reference-solution features. The last family is used only where gold patches exist, as in SWE-bench Verified.
• Calibration. Platt scaling fitted on a held-out validation split, so a threshold of 0.90 means the same thing for both heads.
• Training data. Each trajectory is expanded into labelled prefixes, with trajectories under 10 steps discarded. One predictor pair is trained per benchmark and is agent-agnostic.
• Why trees. The authors say tree ensembles score a feature vector in well under a millisecond on one CPU core, which avoids paying an LLM judge call at every step.

Thresholds are the tuning knob: higher thresholds give higher prediction accuracy but later termination and smaller savings.

What did the authors measure, and on what data?

They evaluated three benchmarks under a leave-one-agent-out protocol, where each agent (a scaffold plus a base LLM) is held out and the predictors are trained only on the other agents. All figures below are lab-reported.

Source: arXiv:2609.02783v1, captured 15 October 2026. The paper names the benchmark "TerminalBench" and does not tie it to a version in the passages reviewed, so do not compare these numbers with results from any specific Terminal-Bench version.

For TerminalBench the authors ran two leakage-controlled splits: no same model in training, and no same scaffold in training. All experiments use a fixed random seed of 42, and the results tables give point estimates with no confidence intervals.

What are the savings at each operating point?

The authors chose, per benchmark, the lowest threshold where the score deviation stays within about 2 percentage points. Savings at those points range from 17.4% to 29.4% on output tokens and 29.2% to 44.1% on input tokens.

All lab-reported, dual-threshold mode, from Table III of the paper, captured 15 October 2026.

The threshold curve is steep. On SWE-bench Verified, moving from 0.95 to 0.75 raises the step reduction from 26.0% to 63.4% while the score distortion rises from 1.1% to 4.1%. At 0.75 input tokens fall 81.5%. A team that tunes for savings will pay for it in score fidelity.

Does early stopping preserve leaderboard rankings?

Mostly. The authors report Spearman's ρ of 0.991 on SWE-bench Verified, 0.994 on Toolathlon, 0.994 on TerminalBench (no same scaffold) and 0.959 on TerminalBench (no same model).

Exact rank agreement is weaker. The share of agents keeping their exact position is 81% on SWE-bench Verified, about 70% on Toolathlon and TerminalBench (no same scaffold), and 59% on TerminalBench (no same model). Individual agents move one or two places in the tables the paper shows.

Two details matter if you plan to use this for model selection:

• For TerminalBench and Toolathlon, the ranking experiment used only the failure predictor, because the success predictor was unreliable there.
• Signed Pass@1 changes for those two benchmarks are negative on average (-2.0 and -0.9 points in the paper's tables), so early stopping under-credits agents.

Our inference, not a claim from the paper: a failure-only policy can only turn would-be successes into recorded failures, which is consistent with the negative sign. If your decision turns on a gap of one or two points between two models, this bias is the same size as the gap.

Where does EarlyEval break down?

The success half of the system is the weak one outside SWE-bench Verified. Success-predictor precision drops to 61.4% to 69.0% on TerminalBench under the no-same-scaffold split, and coverage collapses toward zero on Toolathlon. The failure predictor holds up: 96.7% precision on SWE-bench Verified and 96.6% to 99.4% on Toolathlon at the chosen points.

Unseen scaffolds are harder than unseen models. Peak success precision falls from 82.7% to 69.0%, and the dual step reduction at the recommended point falls from 25.4% to 17.7%. The authors explain that a scaffold sets the structural rhythm of a trajectory, so a new one disturbs the behavioral features. This fits the wider point in our post on the harness effect in coding agent benchmarks: the scaffold is part of the system under test.

Other limits from the paper's own ablations on SWE-bench Verified:

• Removing the behavioral feature family cuts step savings from 26.0% to 16.4%.
• Removing the reference-solution family trims them only to 24.7%, which is why the method transfers to benchmarks without gold patches.
• LightGBM beats a LoRA-tuned Qwen-0.5B judge on this setup: 26.0% step savings versus 17.9%, and 95.0% accuracy versus 90.7%.

One further structural limit, also our inference: in leave-one-agent-out, the held-out agent runs tasks that other agents' trajectories in the training pool also cover. The predictor is tuned to a benchmark with a run history. The paper does not report a test on new tasks.

Has anyone independently checked it?

One preprint has. arXiv:2609.25647, by a single author, audited calibration transfer on public SWE-bench Verified trajectories with a frozen dual-head pipeline. According to the author's repository, the study imports EarlyEval's stopping-policy module unmodified.

Findings from the abstract:

• Broad pairwise heterogeneity was not supported. Median pairwise corrected-gap differences were 0.0180 (success head, 45 pairs) and 0.0385 (failure head, 35 pairs).
• Two combinations showed persistent calibration-transfer errors: gpt-5-mini on the success head and claude-opus-4.6 on the failure head, with median corrected gaps of 0.1377 and 0.1107.
• TerminalBench did not replicate. The success target produced zero decisions, and the failure target did not meet the pre-registered persistence criterion.

Read together: calibration mostly transfers on SWE-bench Verified, but a threshold of 0.95 does not guarantee 95% precision for every unseen agent. The audit is single-author and not known to be peer reviewed, as of 15 October 2026.

Should you use it, and for what?

Use it for internal iteration on a benchmark where you already hold many labelled runs, and keep full runs for numbers you publish. The method targets the repeated evaluation loop of tuning prompts, scaffolds and fine-tunes, where small deltas matter less than turnaround and cost.

This is an editorial scorecard, scored 1 to 5, based on the facts above.

For cost accounting, remember that cost per task depends on routing and token mix, not on step counts alone. Convert the token savings with your own provider's input and output prices before quoting a percentage.

What's still unclear?

Several questions remain open as of 15 October 2026.

• Dollar savings. The paper reports step and token reductions. The sections reviewed give no dollar figures, and the headline 44.1% is input tokens only.
• Version and generalisation. The paper does not tie "TerminalBench" to a version in the passages reviewed. Results on other benchmarks, newer models or tasks with no run history are not reported.
• Bias in rankings. Negative signed score changes on TerminalBench and Toolathlon suggest under-crediting. Whether that falls evenly across agents is not shown.
• Calibration transfer. The audit found two persistent failures on SWE-bench Verified and no replication on TerminalBench. Whether they reflect the method or those specific agents is not settled.
• Peer review. Both papers are arXiv preprints. The EarlyEval paper is listed as v1, submitted 2 September 2026.

The weakest point of the "cut agent costs" framing is scope. It applies to people who run agent benchmarks repeatedly, not to people who run agents in production.