CodeAtlas

Finds the tests that reach a code change, runs both revisions and signs the evidence.

GitHub (opens in a new tab)

Role: Sole developer: product design, analysis pipeline, CLI, evidence model and workspace
Dates: Jul 2026 – present
Status: Local CLI · merged 6 Oct
Stack: TypeScript, Node.js, Vitest

In short

Observed behaviour
It runs the tests on both revisions, three times each, and confirms a regression only when the repeated runs agree.
Tamper-checked replay
Replay checks the SHA-256 artifact digests and the Ed25519 manifest signature before it starts a test.
Stated limits
It runs trusted local code only, and unsupported or incomplete evidence stays marked unverified.

Engineering challenges

A generated test cannot certify itself
The authentication fixture shows an observed HTTP 401 on the base revision and HTTP 500 on the head, with a command to replay it.

The confirmed regression
Bind replay to the recorded evidence
Replay reports REPRODUCED, NOT_REPRODUCED or ENVIRONMENT_MISMATCH and states which evidence it checked.

How each one was solved

Technical decisions

Decision	What I chose	Instead of
Local execution has an explicit trust boundary	Run trusted local snapshots with bounded subprocesses	Calling temporary copies and process limits a sandbox for hostile code

Why, and what it cost

242 tests passed in CI

tests in the 6 October CI run: 242
runs of each test per revision: 3
regression reproduced on the fixture: 401 → 500

That CI run also found a temporary-file cleanup defect, since fixed and covered by a regression test.

Working with AI tools

Define: Test-impact evidence must remain independently verifiable and tamper-proof. Decide: Run separate execution passes for both Git revisions and cryptographically sign the resulting evidence. Investigate: Agents suggested letting generated test-selection outputs certify their own coverage without independent verification. Verify: Enforced dual-revision execution where evidence artifacts cannot attest to themselves.

Full engineering notes

9 sections, about 4 minutes to read

Summary

A code review shows what changed in the text, but a passing test suite can still miss a changed branch. CodeAtlas maps TypeScript and JavaScript snapshots, selects the tests that reach changed symbols, and runs both revisions to compare their behaviour. For its supported authentication fixture, it generates a narrow test for an uncovered expired-session condition and records the observed 401-to-500 regression. I built the local analysis, execution, evidence and replay workflow, plus a web workspace for reading the result. It is a working local tool for trusted repositories; there is no hosted service, sign-in, GitHub App or hostile-code sandbox.

The problem

A diff shows which lines changed. Showing which behaviour changed takes running both revisions.

Existing tests can pass on both revisions while an uncovered condition regresses.
A finding needs a traceable connection between the changed source, the test, the observed result and the environment.
A replay command must reject missing signatures or altered evidence before it executes code.

My role

What I owned

The snapshot analysis, explainable test selection and differential comparison pipeline.
The evidence schemas, canonical artifact digests, signed manifest and Change Passport.
The local CLI, replay validation and bounded process execution for trusted repositories.
The workspace that presents evidence, uncertainty and replay commands.

What I did not own

TypeScript parsing and compiler infrastructure, Node.js, Vitest and Playwright.
Next.js and React for the workspace.
No hosted authentication or repository sandbox is claimed as implemented.

Product flow

Parse base and head snapshots without executing repository code; identify changed symbols and branches.
Select tests through the call graph and record why each test was chosen.
Derive a supported test objective from uncovered changed behaviour and generate a narrow test.
Run existing and generated tests on both revisions, three times, with time and output limits.
Confirm a regression only when compilation, execution, environment and repeated observations agree; otherwise retain the uncertainty.
Canonicalise and sign the evidence, then verify the bundle before replay launches a test process.

From two local snapshots to a finding that can be rerun.

How each challenge was solved

A generated test cannot certify itself

Why it was hard: A generator can produce plausible code or forged-looking output. Neither proves a behavioural difference.
What I built: Keep objective derivation, test generation, execution observations and differential classification separate. Generated tests must compile and execute on both snapshots; missing or contradictory evidence remains UNVERIFIED.
Trade-off: The supported generated objective is deliberately narrow. Unsupported objectives fail closed instead of promising broad automatic coverage.
Validation: Runner and pipeline tests reject substituted suites, forged assertion data, incomplete execution and changed generated artifacts.

Bind replay to the recorded evidence

Why it was hard: A replay that accepts a stale or modified bundle may run a different claim from the one displayed on the Proof Card.
What I built: Use canonical SHA-256 artifact digests and an Ed25519 manifest signature. Verify artifacts, manifest and signature before resolving snapshots or starting tests; publish the current bundle alongside retained immutable artifacts.
Trade-off: The local signing key establishes integrity within this workflow, not independent third-party attestation.
Validation: CLI tests reject missing public keys, signatures and tampered artifact digests; repeated analyses use the matching current reproduction bundle.

Why I made each decision

Local execution has an explicit trust boundary

Because: A working local evidence workflow can be checked without pretending that process cleanup isolates malicious code. Repository tests still execute as the current user.
Trade-off: Untrusted repositories and hosted execution remain out of scope. Windows execution fails closed until process-tree containment is implemented.

Testing

Tests exercise schemas, analysis, selection, generated tests, process limits, tamper rejection, replay and the web workspace. A full CI run on 6 October passed 242 tests and exposed one temporary-file cleanup defect; the fix has its own regression test, and the Linux CI run on the merge to main passed.

Artifact-store regressions passed after correction: 17

Representative cases

The authentication fixture executes the generated expired-session test on base and head and reproduces the observed regression.
Malformed, missing or forged runtime evidence cannot be promoted into a confirmed finding.
Replay rejects altered artifact bytes or missing signature material before execution.
The artifact store keeps a temporary inode pinned through ownership-checked cleanup, preventing a replacement file from being mistaken for the original.

Results in detail

An executable local evidence workflow
The CLI produces a signed Change Passport and a replay command for the supported fixture. The workspace presents the same findings and distinguishes inferred relationships from runtime observations.
Hosted capabilities remain separate milestones
There is no public service or sign-in, no GitHub App and no sandbox for untrusted repositories. The current result is local analysis and replay for trusted snapshots.

Stack and release record

Platforms
Local CLI on macOS and Linux, Local web workspace
Built with
TypeScript 5.9, Node.js, pnpm workspaces, TypeScript compiler API, Vitest, Next.js · React, Playwright, SHA-256 · Ed25519
Testing and delivery
Local implementation on main: the CLI, browser acceptance tests and a production workspace build run in the verification gate, and Linux CI passed on 6 October 2026. No hosted deployment or public release is claimed.
Status in full
Local CLI and workspace implementation merged to main on 6 October 2026. Signed replay and artifact-cleanup fixes are tested; dependency audits report zero vulnerabilities. Focused checks, real base/head execution, type checking, production build and the Linux CI verification passed.