TASK 3 / SUBMISSION

Submit Task 3 predictions on the web.

No pre-registration, approval, access code, or account is required. Practice is open now; the development and test phases open when their datasets are published.

Practice
Scores + receipt
Test
Receipt only

HOW TO SUBMIT

Five steps from data to receipt

Task 3 is method-agnostic. You are scored on your predictions, not on how you produced them.

  1. Step 1 of 5: Get the data.

    Clone the Task 3 starter kit and run its prepare script. It downloads the 332 public cases and writes both the answered and unanswered copies.

    For the two ranked phases, download development_inputs.jsonl and test_inputs.jsonl from YanAdjeNole/FinReason-Task3. Each holds 680 questions with the answers withheld, and no rule label.

    cd finreason_task3
pip install -r requirements.txt
python scripts/prepare_public_dev.py

  2. Step 2 of 5: Produce predictions.jsonl.

    One JSON object per line, one line per case. Only three fields are required, and only id is used to match your prediction to a case — predictions are matched by id, never by row order, so you may write the lines in any order.

    {"id": "DEV_000000", "extracted_value": "-1284", "calculated_value": "1284"}

    Number formatting is free: -1,284, -1284 and (1,284) are read as the same number. Emit "0" when your system cannot determine a value — never omit a line, because a missing id invalidates the whole submission rather than costing you one case.

    Both values must come from the filing materials shipped with the case. Each case is a real SEC filing and says so in its own identifiers, so you can work out which one it is — but reading either value out of a service that already publishes data-quality findings for that filing is not a solution to this task, and a submission produced that way is not a valid entry. See the Terms of Participation.

  3. Step 3 of 5: Validate before you upload.

    The validator catches missing, duplicate and unknown ids, missing fields, empty values and malformed JSON. Exit code 0 means valid.

    python scripts/validate_submission.py \
    --predictions predictions.jsonl \
    --reference data/public_dev_inputs.jsonl

    You can also score yourself locally against the practice answers, which are public. The rule-based judge is free, instant and offline:

    python scripts/score_submission.py \
    --predictions predictions.jsonl \
    --gold data/public_dev.jsonl \
    --judge deterministic

  4. Step 4 of 5: Upload.

    Upload predictions.jsonl directly, or a ZIP containing exactly one root-level file named predictions.jsonl. The file must be UTF-8. Practice asks only for a Team Name; the development and test phases also ask for a Contact Email, which is never published and is stored only as a salted hash.

    Practice and development upload (opens in a new tab)Test upload — receipt only, no score (opens in a new tab)

  5. Step 5 of 5: Read what comes back.

    Practice returns the four rates immediately, broken down by DQC rule, plus a receipt. Development returns a provisional score at once and the official score when the judge finishes. Test returns an acceptance receipt and nothing else — no score, rank, or diagnostic is shown before the final results are released.

    Each phase has a limit, and a submission counts only once it reaches scoring — a file rejected during validation costs nothing, so a format problem never spends an attempt.

    PhaseLimitCounted perResets
    Practice20 per hourUploaderHourly
    Development3 per dayTeam00:00 UTC
    TestUnlimited——

    Practice counts per uploader because it asks for no Contact Email and so has no team to count against. Test uploads are not counted at all — but only one of them decides your result: the last valid submission you make before the deadline. Earlier ones are kept privately for audit and are not ranked. Re-uploading a file you have already submitted returns the original receipt and does not become your latest.

    The last one counts rather than the best one. Test returns no score so that the held-out cases cannot be used to pick between your own candidates; taking the best of several uploads would hand that choice back. Decide which version to stand behind, then upload it.