Challenge API contract (language-agnostic)

This document describes what you must produce and which public endpoints the site provides for evaluation, submissions, and the leaderboard. You may use any programming language, framework, or model — Python and the bundled sicss2026 tools are convenience helpers, not requirements.

What you must deliver

A file with 38 rows — one per test scenario — containing:

Field	Type	Required
`record_id`	string	yes — must match test CSV exactly
`persona_id`	string	yes — one of the four segment ids
`purchase_probability`	number in [0, 1]	yes
`reason`	string	yes — one sentence grounded in page content

Accepted upload formats at https://sicss2026.derlem.com/submit: JSONL, JSON array, or CSV.

Example JSONL line:

{"record_id":"R0001","persona_id":"trust_first_verifier","purchase_probability":0.72,"reason":"Visible return policy and strong reviews increase trust."}

Test record_id values are listed in data/public/synthetic_shoppers_test.csv (included in the participant kit).

Data you may use

Resource	Location	Notes
Training labels	`synthetic_shoppers_train.csv` (90 rows)	Includes hidden probabilities for self-check
Test scenarios	`synthetic_shoppers_test.csv` (38 rows)	No labels — predict these
Product pages	`scenarios/R00xx.html` or live site	Ground decisions in visible page content
Screenshots	`screenshots/R00xx.png`	For vision models
Persona prompts	`exercises/prompts/*.txt`	Starter templates — edit these

Three ways to run inference

1. Your own Hugging Face or OpenAI account (BYOK)

Call an inference provider directly:

export HF_TOKEN=hf_...
export INFERENCE_BASE_URL=https://router.huggingface.co/v1

You may use any model your account can access (subject to provider support for vision vs text). Workshop quotas do not apply; you pay your own provider costs.

Same OpenAI client libraries work — point base_url and api_key at your provider.

2. No API at all

You may hand-craft prompts, use local models, rules, or spreadsheets — as long as the submission file matches the contract above. The leaderboard scores predictions only, not how you produced them.

Public site APIs (no inference)

Endpoint	Purpose
`POST /v1/evaluate/train`	Score predictions against public training labels
`POST /v1/submissions`	Upload final test predictions
`GET /v1/ranking`	Public leaderboard JSON

These endpoints are public and do not run inference.

Training self-check (curl example)

curl -s https://sicss2026.derlem.com/v1/evaluate/train \
  -H "Content-Type: application/json" \
  -d '{"predictions_text": "..."}'

Evaluation (what the server computes)

Submissions are scored against hidden test probabilities:

Overall MAE — primary leaderboard sort (lower is better)
Per-persona MAE — error within each shopper segment
Brier score — calibration
Design-effect MAE — A→B lift per segment×category

Minimal inference request (curl, BYOK)

Use the same OpenAI-compatible payload, but send it to your provider.

Example (Hugging Face router):

curl -s https://router.huggingface.co/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "max_tokens": 256,
    "messages": [
      {"role": "system", "content": "Estimate purchase probability from the screenshot."},
      {"role": "user", "content": [
        {"type": "text", "text": "Persona: trust-first shopper. Respond with JSON purchase_probability and reason."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
      ]}
    ]
  }'