Artifact

Beyond the Default: A Coded Study of 242 Design Corrections One Designer Gave AI Coding Agents Across 15 Products
# Beyond the Default: artifact

Dataset, coding scheme and scripts for the technical report *Beyond the
Default: A Coded Study of 242 Design Corrections One Designer Gave AI Coding
Agents Across 15 Products* (Yousof Selim, November 2026).

The study codes the owner's own design feedback to AI coding agents (Claude
Code, Codex). The raw message text is **not** published: his messages mention
other people, private links and unreleased work. What is here is a hashed
index of every message the miner selected, the screening outcome for each
candidate, and the coded corrections, whose pattern and replacement fields are
paraphrases. Twenty rows carry a short quotation of his own words. Nothing
here contains another person's personal data, credentials or private URLs.

## Contents

| Path | What it is |
|---|---|
| `data/corrections.csv` | The 242 coded corrections (one row each). |
| `data/laws.csv` | The 27 written rules (memory notes, design documents, executable checks) with date, scope and the tells each covers. |
| `data/reversals.csv` | Nine cases where the owner had asked for a pattern, or agreed a ban, and later reversed it. |
| `data/codebook.json` | Tell taxonomy, the two poles, and the strategy codes. |
| `data/messages_index.csv` | All 537 messages the miner selected (id, date, project hint, length, keyword hits, attachment flag, SHA-256 prefix of the text). No text. |
| `data/screening.csv` | The 250 candidate messages read by hand, with the outcome: 155 coded, 9 excluded as agent-written or pasted briefs, 86 excluded as not about a design pattern. |
| `data/consistency_check.csv` | Hand codes against an independent keyword coder on a seeded sample (n = 60, seed 20261102). |
| `data/stats.json`, `data/tables.md` | Every count and cross-tab reported in the paper. |
| `scripts/mine_transcripts.py` | Read-only transcript miner (inclusion rule in its docstring). |
| `scripts/build_dataset.py` | The hand-coded dataset; writes the CSV and JSON files above. |
| `scripts/analyse.py` | Statistics, consistency check and the four SVG figures. |

## Reproduce

```sh
# 1. Mining (needs the owner's ~/.claude/projects; writes text to a private file)
python3 -I scripts/mine_transcripts.py --exclude d86e3a59 \
  --out-private /tmp/msgs_private.jsonl --out-public data/messages_index.csv
# 2. Dataset and statistics
python3 -I scripts/build_dataset.py
python3 -m venv .venv && .venv/bin/pip install matplotlib
.venv/bin/python -I scripts/analyse.py --figdir ../../../images/research/beyond-the-default
```

The transcript snapshot was taken on 2 November 2026 (87 sessions, 2,194
owner messages). Sessions are still live, so a later run will select more
messages; message ids are assigned in time order and were stable across two
runs on that day. The session that commissioned this study is excluded.
Claude Code keeps transcripts for a limited time, so nothing before 13 June
2026 survives; earlier feedback is visible only through memory notes and
commits.

## Codebook

**Unit.** One correction = the owner rejecting one kind of pattern in work an
agent produced (or telling an agent in advance to avoid it), in one project,
on one date. A message that rejects three kinds of pattern yields three rows.
A correction recorded in several places (his message, the memory note written
from it, the commit that reverted it) is one row whose `source` lists all of
them. Memory-only rows (34) were added only where no surviving message carries
the same correction.

**Inclusion.** The pattern must be visual, interaction, motion or copy design.
Functional bugs (a button that does nothing, a sync failure) are excluded even
when phrased with the same words.

**Tells** (`tell`): COL default colour; GLO decorative effects (glow, glass,
blur, sheen, textures); TYP generic or robotic type; ICO generic icons and
emoji; IDN generic marks; DOT status dots, pills, badges, indicators; SRF
surface defaults; TPL template composition; CLU clutter, over-explaining and
copy tells; MOT generic or decorative motion; DED dead or under-designed; NAV
duplicate or misplaced entry points; CMP stock components; GIM gimmicks; GEN
an unspecified "AI feel". Poles: *too much* = COL, GLO, DOT, MOT, GIM, SRF;
*too safe* = TYP, ICO, IDN, TPL, DED, CMP.

**Strategies** (`strategies`, multi-valued): BEN named-product benchmark; REF
own reference component or project; BAN named banned patterns; RES research
mandate; PRO rendered options or prototype first; SHOT screenshot or device
review (also added automatically when an evidence message carried an image);
RESET full visual reset; AUD audit of another agent's work.

**Rule status** (derived in `analyse.py`): *in force* if a written rule
covering that tell applied to that project (or to all his products) before the
correction date; *elsewhere* if such a rule existed only in another project;
*none* otherwise.

**owner_requested** = 1 when the rejected pattern was one he had himself asked
for earlier.

## Who coded

The coding, scripts and this README were produced by an AI assistant (Claude)
from the owner's records, under his direction. There is no second human coder.
The consistency check compares the hand codes with a deliberately simple
keyword coder; it measures how predictable the codes are from the pattern
text, not inter-rater reliability.