TASK 3
Development leaderboard
Every case receives exactly one label, and the first check that fails decides it. The four rates below are shares of the cases that received a valid label.
HOW SCORING WORKS
One label per case, decided in order
S — Structural error: the answer is not a valid two-key object. E — Extraction error: extracted_value does not match the filing. C — Calculation error: the extracted value is right but calculated_value is not. A — Accurate: both match.
The hierarchy short-circuits: a wrong extracted_value makes the case an extraction error even when the calculated value happens to be right. Getting the reported value right is the gate to everything else. Values are compared by numeric meaning, so -1,284 and -1284 are equal, and calculated_value is held to zero tolerance.
ACC is the headline. SER, EER and CER are the structural, extraction and calculation error rates.
Practice · 332 cases · Development · 680 cases
Organizer baselines
8 rows
| Baseline | Set | Judge | ACC | Structural error rate | Extraction error rate | Calculation error rate |
|---|---|---|---|---|---|---|
| Do nothingBaselineAnswers "0" for every field | Practice | Deterministic | 0.00% | 0.00% | 95.48% | 4.52% |
| Extraction onlyBaselineReads the reported figure, then asserts the filing agrees with itself | Practice | Deterministic | 0.00% | 0.00% | 48.49% | 51.51% |
| Extraction onlyBaselineReads the reported figure, then asserts the filing agrees with itself | Development | Deterministic | 0.00% | 0.00% | 23.97% | 76.03% |
| Rule-basedBaselineWeighted sum of the calculation children, else the taxonomy’s balance | Practice | Deterministic | 7.53% | 0.00% | 48.49% | 43.98% |
| Rule-basedBaselineWeighted sum of the calculation children, else the taxonomy’s balance | Development | Deterministic | 25.00% | 0.00% | 23.97% | 51.03% |
| Rule-basedBaselineWeighted sum of the calculation children, else the taxonomy’s balance | Development | Official | 25.29% | 0.29% | 16.91% | 57.50% |
| Sign flipShortcutNegates whatever it extracted; reasons about nothing | Practice | Deterministic | 9.64% | 0.00% | 48.49% | 41.87% |
| Sign flipShortcutNegates whatever it extracted; reasons about nothing | Development | Deterministic | 15.29% | 0.00% | 23.97% | 60.74% |
The board above ranks by the official judge, while most of these rows are measured with the deterministic one — which is why the rule-based baseline appears twice with different figures. The official judge accepts 7 points more of the same extractions, 16.91% extraction errors against 23.97%, because it reads values that mean the same number but are written differently. Compare like with like: a deterministic figure is a lower bound on the official one.
Reproducible from the starter kit: baselines/rule_baseline.py, with --mode extract and --mode negate for the other two. None of them is told which data-quality rule a case belongs to; the rule-based baseline picks its method from where the concept sits in the calculation linkbase.
Read the sign-flip row as a floor, not a method. It negates whatever it extracted and reasons about nothing, so a submission scoring near it has learned nothing whatever its rank says. On development the relationships earn their keep — 25.00% against the shortcut’s 15.29% — but on practice they do not, because a third of that set is the one rule whose answer is always the negation. Both numbers are published as measured.
The do-nothing row’s 4.52% calculation-error share is the 15 practice cases whose reported value genuinely is zero: answering "0" clears the extraction gate for those and fails at the calculation step. Every other case fails earlier.
Loading the leaderboard…
Practice results are never ranked and are excluded from this board: the practice answers are public, so a practice ranking would measure who read them. Test uploads return an acceptance receipt only, and test scores and ranks stay hidden until the final results are released.