SICSS 2026 Synthetic Shoppers Challenge · Public research contest

Background & contact

This site was originally built for a 2-hour SICSS Istanbul 2026 guest session held at Boğaziçi University on 11 July 2026. It is now left open for researchers who want to work on this problem.

SICSS Istanbul 2026: https://sicss.io/2026/istanbul/

Contact: [email protected] · [email protected]

SICSS 2026 Synthetic Shoppers Challenge

Overview

You are working with data from a simulated e-commerce interface experiment.

The site contains one representative product from each of the following categories:

Each product scenario is shown in two product-detail-page designs:

The company has segmented its users into four behavioral groups. However, you are given only the short segment labels and labeled training examples. You are not given the full behavioral definitions of the segments.

Your goal is to build prompt-based shopping agents that mimic these segments as well as possible.

Dataset: 128 scenarios total — 90 labeled training rows and 38 unlabeled test rows you must predict and submit.

Your Task

Build one prompt-based agent for each user segment.

Each agent should inspect the product page (HTML in the browser, or the provided screenshot) as if it were a shopper from that segment and estimate the probability that the shopper would purchase the product.

For each test scenario, your agent should output:

{
  "record_id": "R0001",
  "persona_id": "trust_first_verifier",
  "purchase_probability": 0.72,
  "reason": "The product has strong review and return-policy signals, which make the purchase feel relatively safe for this segment."
}

The output should include:

1. record_id

2. persona_id (stable segment key, e.g. trust_first_verifier)

3. purchase_probability, a number between 0 and 1

4. reason, a short explanation grounded in the page content and the inferred segment behavior

Data Provided

You will receive:

1. A labeled training file (synthetic_shoppers_train.csv, 90 rows)

This includes examples of how each segment behaved in previous product-page scenarios.

2. An unlabeled test file (synthetic_shoppers_test.csv, 38 rows)

This includes the scenarios for which you must submit purchase-probability predictions.

3. HTML pages and PNG screenshots for each scenario

Each row has html_path and screenshot_path on the live mock shop. The recommended VL tool sends the screenshot plus your persona prompt to the model.

The training and test files include only high-level identifiers such as:

record_id
persona_id
persona_name
category
design_variant
html_path
screenshot_path

Use persona_id in submissions. The persona_name column is a human-readable label only (e.g. Trust-First Verifier).

Live mock shop: https://sicss2026.derlem.com — open html_path for each row (e.g. /scenarios/R0001.html).

Participant kit (data + code): download zip — includes CSVs, scenario HTML, screenshots, the sicss2026/ Python tools with pyproject.toml, the code guide, and the language-agnostic API contract.

You may use any tools or models you prefer. The bundled Python CLIs are optional conveniences; see the API contract for curl examples and BYOK setup.

Background & contact

This site was originally built for a 2-hour SICSS Istanbul 2026 guest session held at Boğaziçi University on 11 July 2026. It is now left open for researchers who want to work on this problem.

SICSS Istanbul 2026: https://sicss.io/2026/istanbul/

Contact: [email protected] · [email protected]

Tools (optional)

Python helpers (in the kit):

# Vision — default
poetry run shop-vl-predict batch \
  --mode vision \
  --input data/public/synthetic_shoppers_test.csv \
  --output test_predictions.jsonl

# Text-only — reads scenario HTML as plain text
poetry run shop-vl-predict batch \
  --mode text \
  --input data/public/synthetic_shoppers_test.csv \
  --output test_predictions.jsonl

Inference (BYOK): set HF_TOKEN or OPENAI_API_KEY, and point INFERENCE_BASE_URL at your provider (e.g. https://router.huggingface.co/v1). Any model your account supports works. Details: API contract.

Starter persona prompts: exercises/prompts/*.txt.

Optional (browser agent UI): https://huggingface.co/spaces/smolagents/computer-use-agent

Important Constraint

Your agent must ground its decision in information visible on the product page (HTML or screenshot).

It should not invent product facts, assume missing discounts, or rely only on generic category stereotypes. The best agents will infer the segment behavior from the labeled training examples and then apply that inferred behavior consistently to new pages.

Goal

Your goal is not to produce a binary buy / do-not-buy decision.

Your goal is to estimate a calibrated purchase probability.

For example:

0.10 = very unlikely to purchase
0.50 = uncertain / could go either way
0.90 = very likely to purchase

Evaluation

Your submissions will be evaluated against hidden purchase probabilities generated by the experiment.

The main evaluation criterion is how close your predicted probabilities are to the hidden probabilities.

We will use metrics such as:

The most important question is not only whether your predictions are accurate, but whether your agents reproduce meaningful behavioral differences between segments and between interface designs.

The public leaderboard (https://sicss2026.derlem.com/ranking) ranks submissions by overall MAE (lower is better) and shows per-persona MAE (Value, Trust, Convenience, Inspiration), plus Brier score and design-effect MAE.

Suggested Workflow

1. Explore the training examples

Look at how each segment behaves across:

Try to infer what each short segment label means behaviorally.

2. Write one prompt per segment

For each segment, write a prompt that describes how that shopper should think and decide.

The prompt should be specific enough to guide behavior, but general enough to work across categories and page designs.

3. Test your agents on training examples

Run your agents on the training scenarios and compare their predicted probabilities with the provided labels.

Revise your prompts based on the errors.

4. Predict and submit the test scenarios

Run your final agents on all 38 unlabeled test scenarios.

Submit one purchase probability and one short reason for every test record_id.

1. Go to https://sicss2026.derlem.com/submit

2. Choose your predictions file (.jsonl, .json, or .csv)

3. Enter your display name (shown on the leaderboard)

4. Click Submit file

Your file must include all 38 test rows. Submitting again replaces your previous entry (only your latest submission counts).

Self-check on training rows locally before submitting:

poetry run shop-evaluate --predictions my_train_preds.jsonl

After submitting, scores appear on the leaderboard: https://sicss2026.derlem.com/ranking

5. Reflect

After submitting your predictions, consider:

Submission Format

Submit a JSONL, JSON (array of objects), or CSV file with one row per test scenario (38 rows).

Required fields:

record_id
persona_id
purchase_probability
reason

Example CSV:

record_id,persona_id,purchase_probability,reason
R0001,trust_first_verifier,0.72,"Strong reviews and a visible return policy increase confidence for this segment."
R0002,value_optimizer,0.34,"The discount is modest and the price remains relatively high, so this segment is unlikely to purchase."