Artifact

What the Reviewer Saw: A Coded Study of the Non-Visual Interventions One Developer Made in AI Coding Agents' Work
# What the reviewer saw: artifact

Coded dataset, scripts and figures for the technical report
"What the Reviewer Saw" (`/research/what-the-reviewer-saw/`).

The data are one developer's (Yousof Selim's) typed messages to AI coding agents
(Claude Code, with Codex in some repositories) between 13 June and 1 November 2026.
**No transcript text is published.** Message rows carry only an id, date, project
hint, length, keyword hits and a 12-character SHA-256 prefix of the text.
Intervention rows carry *paraphrases* written by the coder. No credentials,
emails, phone numbers, private URLs or third parties' personal data are included.

Coding was done by an AI assistant (Claude) under the owner's direction and review.

## Files

| File | What it is |
|---|---|
| `data/messages_index.csv` | All 1,879 unique owner messages in the snapshot (no text). |
| `data/sample.csv` | The seeded coding sample: 360 pre-filter matches + 80 non-matches. |
| `data/session_projects.csv` | Hashed session id -> inferred project (majority hint). |
| `data/codes_pass1.psv` | First-pass codes, one line per intervention (or NONE/EXCL per message). Paraphrases only. |
| `data/codes_pass2.psv` | Second-pass codes on a 60-message subsample (seed 99). |
| `data/coding_pass1.csv` | Status and categories for all 440 sampled messages. |
| `data/interventions.csv` | 220 coded intervention rows (171 non-visual, 49 visual). |
| `data/consistency_check.csv` | Pass 1 vs pass 2 on the subsample. |
| `data/chains.csv` | 11 hand-traced recurrence chains across the whole corpus (purposive, not random). |
| `data/memory_rules.csv` | The 32 `feedback-*` memory rules, coded by origin, category and kind (two file names withheld because they contain a third person's name). |
| `data/codebook.json` | Unit, inclusion rule and category definitions. |
| `data/stats.json`, `data/tables.md` | Every number in the paper. |
| `scripts/mine_transcripts.py` | Read-only extraction of owner messages from `~/.claude/projects/*/*.jsonl`. |
| `scripts/draw_sample.py` | Seeded sampling (seed 20261102). |
| `scripts/build_dataset.py` | Joins codes with message metadata; writes the public CSVs. |
| `scripts/analyse.py` | Statistics, tables and the four SVG figures (no plotting library). |

## Reproduce

```bash
# 1. mine (private output stays outside the artifact). ALIAS_FILE is optional:
#    a private, unshipped TSV of extra keyword->project aliases.
python3 -I scripts/mine_transcripts.py --exclude <commissioning-session-prefix> \
  --until 2026-11-01T17:01:00Z --out-private /tmp/msgs.jsonl --out-public data/messages_index.csv
# 2. sample
python3 -I scripts/draw_sample.py /tmp/msgs.jsonl data
# 3. join codes
python3 -I scripts/build_dataset.py --codes data/codes_pass1.psv --second data/codes_pass2.psv \
  --msgs /tmp/msgs.jsonl --sample data/sample.csv --out data
# 4. analyse
python3 -I scripts/analyse.py --data data --figdir ../../../images/research/what-the-reviewer-saw
```

Steps 1 to 3 need the owner's local transcripts. Step 4 runs on the published CSVs alone.

## Limits

One reviewer, his own products, coded by an AI assistant with one second pass by the
same assistant. Chains and memory rules are purposive. See section 7 and 8 of the paper.