A learned world model for ML research
See the outcome. Before the run.
Sample candidate programs, predict each outcome, then select, gate, and return feedback.
FIGURE 1(B)
Schematic from Figure 1(b): every candidate gets a predicted report before selection or combination. The returned program carries its MLE-World confidence into the gate; newly combined code receives a new prediction. Scores and the 0.80 threshold are illustrative. The full method also verifies new best predictions and uses outcome-specific thresholds.
Machine learning research agents repeatedly write, run, and revise programs. Each real execution can require training and evaluation before the agent receives useful feedback. MLE-World predicts that execution feedback directly from code and task context: terminal output, an exception class, and a validation metric.
The learned predictor is a fine-tuned Qwen3.5-4B model. Research agents use its reports to screen, compare, or improve candidate programs, while a confidence gate determines when to request real execution. Final submissions are verified through real execution. The experiments below separately examine prediction fidelity, search quality, and real-execution cost.
Turn candidate code, task data, environment, and runtime budget into a structured report. Compare those predictions with the outcomes of 1,700 held-out programs.
Explore prediction fidelityUse predicted reports across five candidate-search policies. Apply confidence gating after candidate selection, and evaluate final quality and real-execution cost separately.
Compare search resultsPredict a report for each candidate, select or combine programs, then decide when real execution is needed.
FIGURE 2
MLE-World is fine-tuned on agent trajectories pairing candidate programs with observed execution reports. It predicts one candidate's execution from its code and execution context.
The model also supplies an outcome-class confidence score. The fidelity evaluation below compares saved predictions with real execution.
The search policy returns a program. The confidence gate then decides whether its predicted report can supply feedback or whether the program needs to run.
Confidence is an outcome-class score, not calibrated accuracy. Candidate search and confidence-gated execution allocation are distinct parts of the method.
Watch the feedback loopPrediction accuracy across four output fields, and the GPU time needed to predict execution outcomes.
1,700 held-out programs · 17 tasks| Model / baseline | Terminal output | Exception class | Crash class | Validation metric |
|---|---|---|---|---|
| Prompted models | ||||
| DeepSeek | 41.1 ± 1.1 | 57.2 ± 1.2 | 48.2 ± 1.5 | 18.6 ± 1.7 |
| Coder | 17.8 ± 0.9 | 27.5 ± 1.1 | 13.2 ± 1.0 | 16.3 ± 1.7 |
| Corpus, no model | ||||
| Task majority | — | 34.2 ± 1.2 | — | 55.2 ± 2.3 |
| Nearest program | 28.0 ± 1.1 | 41.1 ± 1.2 | 14.0 ± 1.0 | 55.6 ± 2.2 |
| Parent record | 22.3 ± 1.0 | 34.2 ± 1.2 | 6.7 ± 0.7 | 48.8 ± 2.3 |
| MLE-World | 77.9 ± 1.0 | 86.4 ± 0.8 | 76.4 ± 1.3 | 78.1 ± 1.9 |
Scroll to compare all four prediction fields.
Compare measured prediction fidelity with individual examples of what the model predicts and what actually happens.
THE FULL EVALUATION
340 programs in every confidence group. Rates use all eligible references, including predictions that disagree with execution.
TWO ILLUSTRATIVE CASES
Key outcome lines from each prediction and its real execution.
These selected cases illustrate the contrast. The rates above summarize all 1,700 programs; confidence ranks saved predictions and does not guarantee a match.
BROWSE RECORDED EXAMPLES
Choose a group to read two saved predictions and their actual outcomes below.
Five groups of 340 programs, ranked by model confidence.
The selected examples below explain individual matches and misses. This rate uses the full group.
Curated examples: Q1–Q2 show incorrect predictions; Q3 shows a lower-confidence miss and a higher-confidence match; Q4–Q5 show correct predictions. Correctness follows all three terminal judges and can include correctly predicted program failures. Confidence percentiles rank the original scores across all 1,700 programs. Group accuracies use every program in each quantile. Changing levels switches saved examples.
FINAL RESULTS · FIGURE 1(C)
Final solution quality versus real execution time across five MLE-World search policies under a 12-hour budget.
Scroll to see the full chart, or open it full size.
Compare Base AIDE and five search policies on the same task. Inspect the strategies, saved programs, and independently graded outcomes.
Higher percentile is better. Each score belongs to the saved program shown below, for run seed 34; it is not an average across seeds. Base AIDE uses the historical reference run.
Loading selected programs…
These examples were selected for different solution strategies and higher UCB than BoN percentiles in the displayed runs. Every task includes all six policies with the same run seed. TTS runs are ungated; see the aggregate results above for the overall comparison.
For a closer look, choose any recorded task, seed, and pair of policies.
Selection used real validation scores. Private-test grades are shown only for evaluation.
Base AIDE is the same historical reference in both views. An unavailable or invalid final submission stays visible. Library labels are extracted from imports in the saved program.
Read the paper, download the model and dataset, and explore the recorded evidence.