Could the Missing Tasks Reverse the Result?

A model comparison can finish with a positive result and unfinished tasks. Before treating the completed cases as the whole study, ask whether the missing outcomes could change its direction.

In our published coding comparison, GPT-6.1 Sol gained 15 successful fixes and lost seven against GPT-6 Sol across 200 fully graded task pairs. That is a raw lead of eight fixes. But the plan contained 228 tasks: eight more were attempted without complete paired grades, and 20 were not attempted.

A new calculation over the unchanged public records separates those gaps. Even the least favorable completion of the eight attempted tasks leaves a positive raw lead. The 20 unattempted tasks could reverse it: five more losses than gains among those tasks would make a reversal possible. That is a hypothetical completion of missing results, not five newly observed failures or an estimate of what would happen.

Here, “reverse” means a negative candidate-minus-baseline pass-count difference across the full planned set. It does not mean a changed policy verdict or a statistically established advantage for either model. The original study's accuracy difference remains uncertain.

Two gaps hidden behind one completed-pair count

The original evaluation used the same Python coding tasks and High reasoning effort for both models. A task enters the primary paired result only when both models have a valid grade from the required tests. A missing grade stays unknown; it is not silently turned into a failure.

Diagram

Coverage of 228 frozen coding tasks: 200 complete pairs, eight attempted without complete paired grades, and 20 not attempted.

228 frozen coding tasks, three coverage states

200 complete pairs

Both models have valid grades on these tasks, including passing and failing fixes.

8 incomplete pairs

Attempted tasks missing a valid grade for one or both models. Together with the 200 complete pairs, these make 208 attempted tasks.

20 tasks not attempted

Part of the frozen 228-task set, without primary model attempts.

Each task is counted once. The selected 30-task repeat set has three attempts per model, including the first attempt. Repeats add observations, not tasks, and do not expand primary task coverage.

The two gaps need different treatment. The eight incomplete tasks already contain some valid grades, which constrain what remains possible. The 20 unattempted tasks have no grades for either model. Two of those tasks were held at setup; they belong in the unattempted group.

The original methods report attributes missing grades to service availability, a timeout and incomplete test observations, including a capture-capacity limitation. An attempt could end normally without supplying enough evidence to grade it. These records cannot justify assuming that the missing outcomes resemble the completed ones.

The eight attempted tasks cannot erase the raw lead

For a paired task, a gain means only the candidate passed, and a loss means only the baseline passed. Both passing or both failing contributes zero to the net difference.

The eight incomplete tasks fall into three groups. Every already observed grade stays fixed:

Scroll horizontally to see every column.

For example, GPT-6.1 passed Cloudpathlib, but GPT-6's grade is unavailable. Resolving that missing grade could produce either a gain or a tie; it cannot produce a loss. On PINTS, the baseline passed and the candidate's grade is unavailable, so the possibilities run the other way. The calculation output lists all eight tasks and their possible contributions.

Together, those tasks can add −4 to +7 to the observed net of eight. The resulting raw lead across all 208 attempted tasks is therefore between four and 15 fixes, whatever values the 11 missing grades take.

This is a statement about the possible count on a fixed attempted set. It does not establish superiority, remove sampling uncertainty, or say anything about the tasks that never ran.

The 20 unattempted tasks can change the direction

Let the unattempted tasks contribute their own gains minus losses. Add that number to the attempted-task range of four to 15. The result is the possible net across all 228 planned tasks.

Scroll horizontally to see every column.

“Five more losses” does not mean any five failures. A loss requires the baseline to pass and the candidate to fail on the same task. If both fail, the task contributes zero. Gains elsewhere among the 20 offset losses.

The five-loss scenario also does not guarantee a reversal. It permits one if the incomplete attempted tasks resolve unfavorably enough. Sixteen more losses than gains would make the full-set net negative even under the most favorable resolution of those eight tasks. Neither scenario is a forecast.

Allowing every unknown grade to vary gives the original full-allocation range of −16 to +35 net fixes, or −7.02 to +15.35 percentage points. The new calculation explains where that range comes from and which missing results could change its sign.

Keep this separate from the uncertainty interval

The published estimate on the 200 complete pairs is +4.25 percentage points, after weighting task categories, with an exploratory 95% interval of −0.10 to +8.96 points. That interval concerns uncertainty conditional on the observed pairs. It already includes zero.

The missing-outcome range answers a different question: what values remain possible if unknown binary grades take their least or most favorable values? It is not a confidence interval, and its endpoints have no assigned probabilities.

The denominators differ too. The complete-pair estimate uses category-specific observed counts. A fully resolved 228-task allocation uses the original category weights—each category's planned task count divided by 228—so its weighted difference equals the raw net divided by 228. Adding hypothetical task counts directly to the published +4.25-point estimate would mix those calculations.

Close the gap that matters to the claim

If the claim concerns the direction across the full planned allocation, the unattempted tasks matter: resolving only the eight incomplete attempted tasks cannot settle it. If the claim concerns how consistently a model handles an already tested task, repeated attempts address that question instead.

The study already has three attempts per model on a selected 30-task subset. Those histories reveal variation on those tasks. They neither supply the first outcomes of the 20 unattempted tasks nor expand coverage to a different workload. Preserve the first outcomes when examining repeats.

Recovering a valid grade from an original retained execution, when its records suffice, is different from running the task again. A later run creates another observation under its recorded conditions; it cannot recover an unobserved historical result. Any follow-up should retain the original missingness, declare how new observations will be analyzed, and keep changed settings visible.

The practical review question is therefore specific: which unobserved outcomes could change the claim we want to make? Here the answer points to the unattempted tasks for the full-set direction. It does not prescribe rerunning favorable cases or treating all missing evidence as interchangeable.

Reproduce the calculation

The small reproduction package contains a standard-library Python script and its output. It checks the original allocation CSV's hash, excludes repeats, keeps observed grades fixed, enumerates the incomplete tasks' possible contributions, and calculates the table above. It also recovers the original full-allocation bounds.

This is a new descriptive analysis of existing public records. It adds no model calls, recovered grades or signed verification exercise. The public extracts support checking this arithmetic; they cannot authenticate their private source archive.

Limitations

• The scenarios vary missing binary grades without assigning probabilities. A possible reversal is not an observed reversal, and a positive descriptive count does not establish superiority or equivalence.
• The analysis preserves this study's 228-task allocation, fixed required tests and original model settings. It does not measure another workload, later model behavior, deployment costs or failure severity.
• The 30-task repeat subset was selected after primary collection from jointly evaluable tasks. Its results do not fill missing primary tasks or establish general reliability.
• The original capture limitations and unsigned-public-extract boundary remain. This analysis checks arithmetic, not original execution, private-source authenticity or recipient approval.