# Challenge API contract (language-agnostic)

This document describes what you must **produce** and which **public endpoints** the site provides for evaluation, submissions, and the leaderboard. You may use **any programming language**, framework, or model — Python and the bundled `sicss2026` tools are convenience helpers, not requirements.

## What you must deliver

A file with **38 rows** — one per test scenario — containing:

| Field | Type | Required |
|-------|------|----------|
| `record_id` | string | yes — must match test CSV exactly |
| `persona_id` | string | yes — one of the four segment ids |
| `purchase_probability` | number in [0, 1] | yes |
| `reason` | string | yes — one sentence grounded in page content |

Accepted upload formats at https://sicss2026.derlem.com/submit: **JSONL**, **JSON array**, or **CSV**.

Example JSONL line:

```json
{"record_id":"R0001","persona_id":"trust_first_verifier","purchase_probability":0.72,"reason":"Visible return policy and strong reviews increase trust."}
```

Test `record_id` values are listed in `data/public/synthetic_shoppers_test.csv` (included in the [participant kit](https://sicss2026.derlem.com/downloads/latest)).

## Data you may use

| Resource | Location | Notes |
|----------|----------|-------|
| Training labels | `synthetic_shoppers_train.csv` (90 rows) | Includes hidden probabilities for self-check |
| Test scenarios | `synthetic_shoppers_test.csv` (38 rows) | No labels — predict these |
| Product pages | `scenarios/R00xx.html` or live site | Ground decisions in visible page content |
| Screenshots | `screenshots/R00xx.png` | For vision models |
| Persona prompts | `exercises/prompts/*.txt` | Starter templates — edit these |

## Three ways to run inference

### 1. Your own Hugging Face or OpenAI account (BYOK)

Call an inference provider **directly**:

```bash
export HF_TOKEN=hf_...
export INFERENCE_BASE_URL=https://router.huggingface.co/v1
```

You may use **any model** your account can access (subject to provider support for vision vs text). **Workshop quotas do not apply**; you pay your own provider costs.

Same OpenAI client libraries work — point `base_url` and `api_key` at your provider.

### 2. No API at all

You may hand-craft prompts, use local models, rules, or spreadsheets — as long as the **submission file** matches the contract above. The leaderboard scores predictions only, not how you produced them.

## Public site APIs (no inference)

| Endpoint | Purpose |
|----------|---------|
| `POST /v1/evaluate/train` | Score predictions against public training labels |
| `POST /v1/submissions` | Upload final test predictions |
| `GET /v1/ranking` | Public leaderboard JSON |

These endpoints are **public** and do **not** run inference.

### Training self-check (curl example)

```bash
curl -s https://sicss2026.derlem.com/v1/evaluate/train \
  -H "Content-Type: application/json" \
  -d '{"predictions_text": "..."}'
```

## Evaluation (what the server computes)

Submissions are scored against hidden test probabilities:

- **Overall MAE** — primary leaderboard sort (lower is better)
- **Per-persona MAE** — error within each shopper segment
- **Brier score** — calibration
- **Design-effect MAE** — A→B lift per segment×category

## Minimal inference request (curl, BYOK)

Use the same OpenAI-compatible payload, but send it to **your provider**.

Example (Hugging Face router):

```bash
curl -s https://router.huggingface.co/v1/chat/completions \
  -H "Authorization: Bearer $HF_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "max_tokens": 256,
    "messages": [
      {"role": "system", "content": "Estimate purchase probability from the screenshot."},
      {"role": "user", "content": [
        {"type": "text", "text": "Persona: trust-first shopper. Respond with JSON purchase_probability and reason."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
      ]}
    ]
  }'
```

## Further reading

- [Participant code guide](participant-code-guide.html) — Python CLIs (`shop-vl-predict`, `shop-evaluate`)
- [Task description](sicss2026-task-description.md) — challenge goals (also on the [homepage](/))
