{"holdoutCases":1700,"searchTrials":561,"tasks":17,"seeds":[34,35,36],"originalProgramsUnchanged":true,"allPaperConfidenceBinsMatched":true,"allPaperQualityCostEndpointsMatched":true,"finalSubmissionsHashVerified":509,"newModelCalls":0,"newExperimentRuns":0,"sourceDescription":"Frozen confidence analysis and independently graded final submissions used in the paper.","confidenceSource":"pmax_holdout_fidelity_20260919","policySource":"Paper main cohorts, including the updated β=0.5 UCB cohorts.","limitations":["Recorded examples; moving the slider filters predictions and does not regenerate them.","Confidence is an outcome-class score, not a calibrated probability of terminal or metric correctness.","Terminal agreement averages three frozen judge votes; the metric panel uses ≤5% relative error.","Offline coverage is not measured online execution savings.","Policy settings are separately run cohorts. Base AIDE is the historical reference.","Final solutions are chosen by real validation, never by private-test score.","Missing or invalid final submissions remain in the denominator with zero percentile.","Execution hours sum interpreter attempts and recovery cleanup, excluding model inference."]}