Artifact

Green but Wrong: An Experience Report on Defects in AI-Coding-Agent Output Across Eight Products, and the Revert-the-Fix Protocol
# Green but Wrong: artifact

Dataset and scripts for the technical report *Green but Wrong: An Empirical
Experience Report on Defects in AI-Coding-Agent Output Across Eight Products,
and the Revert-the-Fix Protocol* (Yousof Selim, November 2026).

Several of the source repositories are private. This artifact therefore holds
commit hashes, file paths and short paraphrased descriptions only: no source
code is reproduced, and nothing here can be used to rebuild a product.

## Contents

| Path | What it is |
|---|---|
| `data/defects.csv`, `data/defects.json` | The 183 coded defects (one row each). |
| `data/census.csv` | Per-repository commit census and author-kind attribution (1,721 commits). |
| `data/candidates.csv` | Every commit whose message matched the fix pattern (531), with attribution. |
| `data/bupples_sample.csv` | The seeded random sample (n = 40, seed 20261101) of Bupples fix candidates, with the screening outcome (`included`, `defect_id`, `reason`) added by hand; rerunning the miner rewrites the sample without those three columns. |
| `data/consistency_check.csv` | Second-pass category coding of a seeded sample (n = 40, seed 20261102). |
| `data/tables.md`, `data/stats.json` | Every cross-tab and count reported in the paper. |
| `scripts/mine_commits.py` | Read-only git mining: census, candidates, Bupples sample. |
| `scripts/build_dataset.py` | The hand-coded dataset (source of `defects.csv`). |
| `scripts/analyse.py` | Tables, statistics, consistency check and the four SVG figures. |

## Reproduce

```sh
python3 -I scripts/mine_commits.py --root ~/Documents --out data   # needs the eight repos
python3 -I scripts/build_dataset.py
python3 -m venv .venv && .venv/bin/pip install matplotlib
.venv/bin/python -I scripts/analyse.py --figdir ../../../images/research/green-but-wrong
```

`mine_commits.py` runs only `git log`, `git rev-list`, `git for-each-ref` and
`git show -s`. The census was taken on 1 to 2 November 2026; the repositories
are live, so a later run will count more commits (Planket gained one during
the study). The defect dataset does not depend on the census.

## Codebook

Columns of `defects.csv`:

- `id`: B = Bupples, C = CareerOS, W = WayClub, P = Photoshoot, G = CodeAtlas,
  A = Adelante, K = Planket, H = HACKFAM.
- `category`: AUTHZ authorisation and tenant isolation; PRIV privacy and data
  exposure; LOGIC business-rule and workflow logic (incl. fail-open rules and
  evidence-integrity rules); CONC concurrency, locking, idempotency; STALE
  stale state and caching; UI UI state, visibility, layout; ABUSE abuse,
  injection, resource limits (rate limits, ReDoS, SSRF, denial of wallet);
  PERF performance and resource lifecycle; CONTRACT mismatch between two
  components that each look right alone; TEST the test or verification
  itself was wrong or missing; MONEY money arithmetic and ledger; PLAT
  platform-specific runtime behaviour; ERR error handling and honest states;
  UNITS units, coordinate frames, precision, idempotence of transforms;
  BUILD build, configuration, operations; DOCS overclaiming or stale docs.
- `severity`: H = security or privacy breach, data loss, wrong money, or the
  product's core action unusable; M = a feature wrong or unusable on a path or
  platform, or user-visible availability or performance loss; L = cosmetic,
  minor or latent. A severity stated by the source audit is used when present.
- `detection`: REVIEW dedicated adversarial or audit review (read-only, by an
  agent or several); SELF the implementing agent's own verification (re-runs,
  mutation testing, writing proof tests, driving the product); OWNER the
  owner using the product or testing on a device; GATE an automated gate
  (CI, an existing test, axe, eval); UNK not recorded in the source.
- `red` (strength of evidence that the fix's test fails without the fix):
  RV revert or mutation of the fix observed to turn the test red;
  RF test or exploit observed failing before the fix; RO a test commit that
  precedes the fix commit (inferred, not observed); RM failure reproduced by
  hand before and after, no committed test; T test added, no RED recorded;
  U no test recorded; N no test, stated or evident; X not fixed.
- `green_over`: Y when the source documents that a passing test, suite, gate
  or written claim covered the defect while it was present.
- `author`, `attr_tier`: A = introducing commit identified (`git log -S` or
  `--diff-filter=A`) and attributed by its Co-Authored-By trailer;
  B = provenance documented in the repository (for example "inherited Codex
  code") or a codex/* branch without trailers; C = introducing commit not
  identified, attributed from the repository period (`claude` when every
  commit in the period carries a Claude trailer, otherwise `agent`);
  D = unknown.
- `path`: D = defect recorded in a committed document (audit, handoff,
  progress, security, testing, ADR); C = found by screening fix-pattern
  commits; S = found in the Bupples random sample.

## Inclusion rule

A defect is included when a primary source in the repository (a commit
message or a document; five CareerOS rows come from the uncommitted working
tree of `docs/PROGRESS.md` and `docs/DECISIONS.md`, read but not modified)
(a) states that previously committed code, tests or documents behaved wrongly,
(b) describes the wrong behaviour concretely enough to classify, and (c)
records a fix or an explicit decision. Excluded: design or taste feedback,
missing features, commits whose message gives no behaviour (for example
"fix: serialize launch prompts after setup" with no body), research text that
was never code, and test-only fixes for flaky selectors. Sources screened:
every fix-pattern commit in the seven smaller repositories, a seeded sample of
40 of Bupples' 433, and the documents named in `source`.