TASK 3 / SUBMISSION

Submit Task 3 predictions on the web.

No pre-registration, approval, access code, or account is required. Practice is open now; the development and test phases open when their datasets are published.

Practice
Scores + receipt
Test
Receipt only

HOW TO SUBMIT

Five steps from data to receipt

Task 3 is method-agnostic. You are scored on your predictions, not on how you produced them.

  1. Step 1 of 5: Get the data.

    Clone the Task 3 starter kit and run its prepare script. It downloads the 332 public cases and writes both the answered and unanswered copies.

    For the two ranked phases, download development_inputs.jsonl and test_inputs.jsonl from YanAdjeNole/FinReason-Task3. Each holds 680 questions with the answers withheld, and no rule label.

    cd finreason_task3
pip install -r requirements.txt
python scripts/prepare_public_dev.py

  2. Step 2 of 5: Produce predictions.jsonl.

    One JSON object per line, one line per case. Only three fields are required, and only id is used to match your prediction to a case — predictions are matched by id, never by row order, so you may write the lines in any order.

    {"id": "DEV_000000", "extracted_value": "-1284", "calculated_value": "1284"}

    Number formatting is free: -1,284, -1284 and (1,284) are read as the same number. Emit "0" when your system cannot determine a value — never omit a line, because a missing id invalidates the whole submission rather than costing you one case.

    Both values must come from the filing materials shipped with the case. Each case is a real SEC filing and says so in its own identifiers, so you can work out which one it is — but reading either value out of a service that already publishes data-quality findings for that filing is not a solution to this task, and a submission produced that way is not a valid entry. See the Terms of Participation.

  3. Step 3 of 5: Validate before you upload.

    The validator catches missing, duplicate and unknown ids, missing fields, empty values and malformed JSON. Exit code 0 means valid.

    python scripts/validate_submission.py \
    --predictions predictions.jsonl \
    --reference data/public_dev_inputs.jsonl

    You can also score yourself locally against the practice answers, which are public. The rule-based judge is free, instant and offline:

    python scripts/score_submission.py \
    --predictions predictions.jsonl \
    --gold data/public_dev.jsonl \
    --judge deterministic

  4. Step 4 of 5: Upload.

    Upload predictions.jsonl directly, or a ZIP containing exactly one root-level file named predictions.jsonl. The file must be UTF-8. Practice asks only for a Team Name; the development and test phases also ask for a Contact Email, which is never published and is stored only as a salted hash.

    Practice and development upload (opens in a new tab)Test upload — receipt only, no score (opens in a new tab)

  5. Step 5 of 5: Read what comes back.

    Practice returns the four rates immediately, broken down by DQC rule, plus a receipt. Development returns a provisional score at once and the official score when the judge finishes. Test returns an acceptance receipt and nothing else — no score, rank, or diagnostic is shown before the final results are released.

    Each phase has a limit, and a submission counts only once it reaches scoring — a file rejected during validation costs nothing, so a format problem never spends an attempt.

    PhaseLimitCounted perResets
    Practice20 per hourUploaderHourly
    Development3 per dayTeam00:00 UTC
    Test3 in totalTeamNever

    Practice counts per uploader because it asks for no Contact Email and so has no team to count against. The test limit does not reset: three accepted test submissions is all a team gets.