SICSS 2026 Synthetic Shoppers Challenge · Public research contest

# Challenge API contract (language-agnostic)

This document describes what you must produce and which public endpoints the site provides for evaluation, submissions, and the leaderboard. You may use any programming language, framework, or model — Python and the bundled `sicss2026` tools are convenience helpers, not requirements.

## What you must deliver

A file with 38 rows — one per test scenario — containing:

Field | Type | Required
--- | --- | ---
`record_id` | string | yes — must match test CSV exactly
`persona_id` | string | yes — one of the four segment ids
`purchase_probability` | number in [0, 1] | yes
`reason` | string | yes — one sentence grounded in page content

Accepted upload formats at https://sicss2026.derlem.com/submit: JSONL, JSON array, or CSV.

Example JSONL line:

    {"record_id":"R0001","persona_id":"trust_first_verifier","purchase_probability":0.72,"reason":"Visible return policy and strong reviews increase trust."}

Test `record_id` values are listed in `data/public/synthetic_shoppers_test.csv` (included in the participant kit).

## Data you may use

Resource | Location | Notes
--- | --- | ---
Training labels | `synthetic_shoppers_train.csv` (90 rows) | Includes hidden probabilities for self-check
Test scenarios | `synthetic_shoppers_test.csv` (38 rows) | No labels — predict these
Product pages | `scenarios/R00xx.html` or live site | Ground decisions in visible page content
Screenshots | `screenshots/R00xx.png` | For vision models
Persona prompts | `exercises/prompts/*.txt` | Starter templates — edit these

## Three ways to run inference

### 1. Your own Hugging Face or OpenAI account (BYOK)

Call an inference provider directly:

    export HF_TOKEN=hf_...
    export INFERENCE_BASE_URL=https://router.huggingface.co/v1

You may use any model your account can access (subject to provider support for vision vs text). Workshop quotas do not apply; you pay your own provider costs.

Same OpenAI client libraries work — point `base_url` and `api_key` at your provider.

### 2. No API at all

You may hand-craft prompts, use local models, rules, or spreadsheets — as long as the submission file matches the contract above. The leaderboard scores predictions only, not how you produced them.

## Public site APIs (no inference)

Endpoint | Purpose
--- | ---
`POST /v1/evaluate/train` | Score predictions against public training labels
`POST /v1/submissions` | Upload final test predictions
`GET /v1/ranking` | Public leaderboard JSON

These endpoints are public and do not run inference.

### Training self-check (curl example)

    curl -s https://sicss2026.derlem.com/v1/evaluate/train \
      -H "Content-Type: application/json" \
      -d '{"predictions_text": "..."}'

## Evaluation (what the server computes)

Submissions are scored against hidden test probabilities:

- Overall MAE — primary leaderboard sort (lower is better)
- Per-persona MAE — error within each shopper segment
- Brier score — calibration
- Design-effect MAE — A→B lift per segment×category

## Minimal inference request (curl, BYOK)

Use the same OpenAI-compatible payload, but send it to your provider.

Example (Hugging Face router):

    curl -s https://router.huggingface.co/v1/chat/completions \
      -H "Authorization: Bearer $HF_TOKEN" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
        "max_tokens": 256,
        "messages": [
          {"role": "system", "content": "Estimate purchase probability from the screenshot."},
          {"role": "user", "content": [
            {"type": "text", "text": "Persona: trust-first shopper. Respond with JSON purchase_probability and reason."},
            {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
          ]}
        ]
      }'

## Further reading

- Participant code guide — Python CLIs (`shop-vl-predict`, `shop-evaluate`)
- Task description — challenge goals (also on the homepage)