Purpose: define the two-layer verification practice that backs the Definition of Done’s
Test: passrequirement. Automated tests catch one kind of failure cheaply; actual-UI verification catches the rest. Both are required for user-observable changes.
This doc explains how to do testing and verification well. The DoD says it must be done; this doc says how.
A change can pass an automated test suite and still be broken for the user. A change can also be visually verified and still regress under an unhappy code path. Each kind of test catches what the other misses.
The actual-UI verification layer exists because these are real failure modes that the test suite cannot reach. Tests are necessary; they are not sufficient.
Without the split:
pnpm test, sees green, marks the item done, and ships a broken UI.With the split:
Test: pass.Every behavior change has at least one test that exercises the new behavior. Every bug fix has a regression test that fails on the broken code and passes after the fix.
A good test asks “does this function/route/page produce the right outcome for this input?” A bad test asks “does this function call its internal helper three times?”
The behavior test survives refactors. The implementation test breaks every time you rearrange internals — even when the behavior is unchanged. Implementation-coupled tests are noise.
A useful check: if you rewrote the implementation completely while preserving the externally-visible behavior, would the test still pass? If yes, it’s testing behavior. If no, it’s testing implementation.
A test that has to wait for something — a background job, a queue, a UI update, a second process — never waits a fixed time. A sleep(2) is a guess about how fast the machine is: it passes on the author’s machine, flakes in CI, and invites whoever sees the flake (often an agent) to raise the number, which makes the suite slower and the flake rarer without fixing it.
A change can pass its own new tests while breaking three pre-existing ones. Running only the new tests would miss the regression.
Run the full suite locally before marking the change ready for review. If the suite is slow, find ways to make it faster (parallelism, test selection by changed file, smarter mocking) — don’t skip running it.
And count only the tests that ran (the gate form is DoD Gate 2). it.skip, an accidentally-narrowed --grep, a suite excluded from the runner config — each produces a green run that silently verifies less than it claims.
This doc does not prescribe a test framework. Pick what fits the stack:
The test types are not exclusive. A healthy codebase has all of them.
Two common conventions:
Foo.ts and Foo.test.ts in the same directory. Easy to find, easy to keep in sync.src/Foo.ts and test/Foo.test.ts. Useful when test infrastructure differs significantly from production code (e.g., requires different tooling).Either is fine. Pick one and stay consistent across the project.
Tests should describe behavior in plain language:
describe('exportActivityReport', () => {
it('returns CSV with the right columns for the given filter', ...)
it('streams large result sets without buffering', ...)
it('returns 400 when the date range is inverted', ...)
})
A reader looking at the test list should be able to tell what the function is supposed to do without reading the function.
For any non-trivial unit:
For a bug fix:
Coverage percentages measure lines; the useful question measures behavior. For each behavior the change adds or alters, ask: “if this behavior broke, would any test fail?” Every “no” is a verification gap — an untested behavior change — and the gap list, not the green run, is what tells you whether the suite protects the change.
The gate form of this check, with its sharpening rules, is the DoD’s Gate 2. Worth adding here: it makes a strong standing lens for a cross-AI validation pass — a reviewer hunting only for untested behavior changes finds what a bug-hunting reviewer skips past, because the two are looking for different things (absence versus error).
This is the most important and most-skipped gate. Take it slowly the first few times; the discipline pays off.
1. Start the dev server(s).
- For a multi-server stack (e.g., frontend + backend + worker), ALL
relevant servers must be running.
- Check that the wiring is correct: frontend can reach backend,
backend can reach the database, etc.
2. Open the app at the user-facing entry point.
- Not the raw API port. Not the test harness. The real URL.
3. Navigate to every surface the change touches.
- Not just the headline screen. Every page, modal, and state path
the change can affect, directly or indirectly.
4. Execute the user flow end to end.
- Click the buttons. Submit the forms. Trigger the error states.
- Watch the result of each interaction.
5. Watch the browser console and network panel.
- A 500 on a background fetch counts. A console warning about an
unmounted component counts. A missing asset counts.
6. If you find an issue, fix it.
- Then return to step 3 — re-navigate, re-execute.
- DO NOT skip back to just the broken screen. Your fix may have
broken something earlier in the flow.
7. Repeat until you complete a full pass with zero issues across every
surface the change touches.
A fix at step 4 can introduce a regression at step 3. If you skip back to step 4 alone, you don’t see it. The discipline is: when you find an issue and fix it, restart the verification from a point upstream of the fix.
For small fixes (a typo, a CSS tweak), this might mean re-navigating the affected screen. For larger fixes (a change to shared state, a route refactor), it means restarting from the beginning of the user flow.
Before declaring the verification complete, cover the dimensions below. Only those that could be affected by your change are required, but err on the side of checking — the cost of one extra check is small.
Common theme failures:
Common viewport failures:
For each user role the product supports, verify the change in that role:
Common auth failures:
A user with no data:
Common empty-state failures:
What happens when the API fails, the network drops, the user input is invalid?
Common error-state failures:
If the product is meant to work offline:
Skip this dimension only if the product genuinely does not support offline.
The six dimensions above are all states of the surface you changed. This one asks a different question: what else consumes what I changed, and did I check one of each?
Common blast-radius failures:
Its honest limit: this dimension is only as good as your ability to enumerate. When the consumers are listable — and in most codebases a grep for the symbol lists them — list them and check one of each. When they are not, this degrades into “run the full suite,” which the automated layer already requires, and the real protection is contract tests at the boundary rather than manual sweeps.
The discipline is: fix what you find, then re-verify from upstream.
Specifically:
The loop terminates when a full pass completes with zero new issues.
Verification can be manual or automated. Both have a place.
A human clicking through the app. Necessary for:
The downside: manual verification is slow, expensive, and prone to fatigue. Use it for what only humans can do.
Scripts that drive a browser to navigate and assert. Useful for:
Common categories:
The choice is project-specific. Adopt what fits the stack and the team’s tolerance for tooling.
Built into some development environments: a tool that lets an AI agent or contributor inspect a running preview of the app — read DOM snapshots, click elements, capture screenshots — without a full browser automation framework.
When available, preview tools are the right path for AI agents doing UI verification. They are faster than full browser automation and tailored for the AI-driven verification loop.
| Situation | Suggested tool |
|---|---|
| Manual verification of a small change. | Click through manually. |
| Repeated verification of a critical user flow. | Scripted E2E test (Playwright et al.). |
| AI agent verifying its own UI change. | Preview tool / headless browser MCP if available; manual click-through if user is present. |
| Visual quality / pixel-level verification. | Visual regression tool, then human eye for actual assessment. |
| Accessibility check. | Dedicated tool (axe, Lighthouse, etc.) + manual keyboard navigation. |
This is worth stating in detail because the failure mode is common.
A clean npm test (or equivalent), a successful compile, a green CI run, a dev server that started without errors — none of these prove the application works for the user.
What they prove:
What they do NOT prove:
The Actual UI Verification loop exists because of this gap. Tests are necessary; they are not sufficient.
The UI verification loop is required for any user-observable change. It is not required for:
If you skip verification, state that you skipped it in the PR description with the reason. The reviewer can challenge the skip if it looks wrong.
Do not skip verification for “tiny” user-observable changes. A one-line CSS change can break a layout. A “trivial” copy update can change the meaning of an error message. The size of the change is not the right criterion; whether the user can see the change is.
When fixing a bug, write a test that reproduces the bug first.
1. Identify the failing behavior.
2. Write a test that asserts the correct behavior.
3. Run the test. It MUST FAIL on the broken code.
4. Implement the fix.
5. Run the test again. It MUST PASS.
6. Run the full suite. It MUST STILL PASS.
A test that passes on broken code does not actually test what you think it tests. It might be testing a different code path, asserting against the wrong value, or skipping silently.
The “must fail first” step proves the test is real. If the test passes on broken code, the test is wrong — find the discrepancy before continuing.
Alongside the existing tests for the affected code. The test name should reference the original symptom or the item:
it('does not throw when the trips array is empty (regression for BL-0517)', ...)
A future reader who breaks the same code will see the test fail and read the comment / item reference to understand what they broke.
Bugs return when fixes are not pinned by regression tests. Six weeks after the fix, a different change touches the same area; if no test pinned the original behavior, the bug returns silently. Regression tests prevent this.
The base testing/verification practice covers what every project needs. Some projects need additional checks. Layer them on top, do not replace.
| Check | When to add | Tool category |
|---|---|---|
| Accessibility | Any user-facing product. Catches contrast, keyboard navigation, screen-reader issues. | Lighthouse, axe-core, manual keyboard test. |
| Performance budget | Any product with hard latency or bundle-size constraints. | Lighthouse, custom bundle analyzers, load tests. |
| Visual regression | Any product with a strict design system or pixel-level expectations. | Percy, Chromatic, or local snapshot diffs. |
| Security scan | Any product handling sensitive data, auth, or payments. | OWASP ZAP, Snyk, project-specific scanners. |
| Cross-browser smoke | Any product targeting more than one browser engine. | Playwright with multiple browser configs. |
| API contract tests | Any product with public or internal API consumers depending on stable contracts. | Pact, JSON-schema based tooling, OpenAPI lint. |
| Type coverage threshold | Any project enforcing typed code with a coverage floor. | TypeScript strict mode, tsc –noUnusedLocals, etc. |
| Snapshot diffs of derived artifacts | Any project that generates files (PDFs, reports, configs) where the output should be stable. | Project-specific. |
For each addition:
The short form. Adapt by removing dimensions that genuinely don’t apply.
## Test plan
- [ ] Full automated test suite passes locally.
- [ ] New behavior covered by at least one test.
- [ ] If a bug fix: regression test added and verified to fail on the
broken code.
- [ ] Manually verified the change in the running app:
- [ ] Affected screens navigated end to end.
- [ ] Console clean (no errors, no concerning warnings).
- [ ] Network panel clean (no unexpected 4xx/5xx).
- [ ] Theme verified:
- [ ] Light.
- [ ] Dark.
- [ ] Viewport verified:
- [ ] Primary target (<viewport>).
- [ ] Secondary target (<viewport>).
- [ ] Auth states verified (as applicable):
- [ ] Logged out.
- [ ] Logged in (free / default tier).
- [ ] Premium / paid tier.
- [ ] Admin / operator.
- [ ] Empty state verified.
- [ ] Error state verified (API failure path).
- [ ] Offline state verified (if applicable).
- [ ] Blast radius checked: other consumers of the changed code
enumerated (profiles, flags, platform builds, tenants, locales)
and one representative of each verified.
- [ ] Documentation updated:
- [ ] CHANGELOG.
- [ ] README (if commands or setup changed).
- [ ] Status / project-readiness doc (if scope shifted).
- [ ] Project instruction file (if a new convention/rule landed).
- [ ] Backlog state correct:
- [ ] Status: done.
- [ ] Test: pass.
- [ ] Lock: —.
- [ ] Item moved to ARCHIVE.md.
- [ ] EPICS.md rollup counts updated.
A pseudocode view of the loop. Useful as a mental model for both human and AI agent contributors.
function verify(change):
surfaces = identify_affected_surfaces(change)
dimensions = required_dimensions(change)
loop:
issues = []
for surface in surfaces:
for dimension in dimensions:
result = check(surface, dimension)
if result.failed:
issues.append(result)
if issues is empty:
return PASS
for issue in issues:
fix = implement_fix(issue)
apply(fix)
# Loop again. The fix may have broken something upstream.
# Do not narrow the verification scope on subsequent passes —
# re-verify all surfaces and dimensions.
Key properties:
When an AI agent writes both the implementation AND the tests for the same change, a failure mode appears: the AI may write a test that validates the broken code it wrote. The test passes; the code is wrong; the loop is self-consistent. No human in the loop catches it because the green test suite looks like done.
This is the AI-coding equivalent of marking your own homework.
The test suite is the durable artifact — when it accurately captures intent, it can survive a language rewrite or framework migration. Implementation is replaceable; the test suite (assuming it tests behavior, not implementation) is not. That makes test quality MORE important in an AI-assisted workflow, not less.
See 11_human_roles.md “The cheating agent” for the human-role framing and “Specification as the primary artifact” for the related shift in what the team should be best at producing.
A separate AI system, periodically, audits the work that the implementing AI did. The audit catches blind spots the implementer missed — pattern-recognition gaps, security oversights, performance regressions, accessibility issues.
AI models have characteristic blind spots:
Cross-AI validation is not a substitute for code review by a human or for actual user testing. It is a cheap “fresh eyes” pass that catches what would otherwise reach review or production.
Cross-AI validation has three distinct modes, each appropriate at a different step:
Spec-verification (before implementation starts). The planning session produces an item body — goal, approach, and at Effort M+ a Code Map making factual claims about the codebase. A fresh session checks those claims against the actual code, before anyone implements:
Output: PASS / FAIL per claim, with citations.
Why this mode earns its cost. Failure-layer routing says an intent- or plan-layer finding cancels every code-layer finding beneath it — the code gets re-derived, so the review effort spent on it was wasted. That is an argument for catching those layers earliest, not just routing them best. Spec-verification is that argument applied: it moves the plan-layer check to before the code exists, where the fix is editing a paragraph rather than reverting a branch.
Use it where a wrong plan is expensive: Effort M+, anything whose Code Map asserts things about unfamiliar code, and anything where the implementer will be a cold session that cannot sanity-check the premises. For an XS typo fix it is pure ceremony.
Findings-verification (the usual mode). The implementing session produces work; the fresh session verifies the work meets a checklist (e.g., “this BL-#### item’s **Done means:** checkboxes — see 04_backlog_items.md item skeleton — are all satisfied,” or “this PR’s described changes match what the diff actually does”). The validator is checking completeness and correctness of claims. Output: PASS / FAIL per checklist item, with grounded citations.
Diff-verification (when a loop or agent proposes an autonomous patch to authoritative content). The implementing session produces a proposed patch — typically a branch with an edit, a CHANGELOG entry, and a finding the patch addresses. The fresh session reads the original cited content, the proposed edit, and the finding, then verifies three things:
Diff-verification is the cross-AI gate for the patch-branch convention in 09_git_workflow.md “Patch-branch convention for authoritative artifacts”. Without it, an autonomous loop’s “I fixed the typo” claim is unverified. With it, the maintainer reviews a verified diff — yes/no on a concrete change, not translate-finding-into-fix (matching the 09 framing).
All three modes use the same “fresh session, different model where possible” setup. What differs is when they run and what they take as input: spec-verification runs before implementation against the plan’s claims; findings-verification runs after, against a checklist; diff-verification runs against an original + proposed-edit + finding triple. Their outputs share a shape — PASS/FAIL per claim, with citations.
The previous sections describe what each layer of verification does. This section is about which layers a given change requires. Test: pass is binary in the DoD, but the work to earn it should match the change’s risk profile. Verifying a typo fix with cross-AI validation is theatre; verifying a payment-flow refactor with only unit tests is negligence.
A0–A3 assurance profiles set accountability and additional evidence requirements; these L0–L4 levels describe verification methods/depth. Satisfy both. A fresh AI review counts toward A2 independence only when it follows the independent-evidence requirements; L3 agreement alone is not assurance, and L4 user testing does not replace A3 specialist verification. Pending required assurance or human acceptance keeps Test at partial/pending, not pass with a caveat.
| Level | Name | What it covers | Time cost |
|---|---|---|---|
| L0 | Type / compile | Static type check; build passes; no syntax or import errors. | Seconds |
| L1 | Automated tests | Full automated suite passes locally — unit, integration, contract tests. | Seconds to minutes |
| L2 | Actual-UI fix-test loop | The fix-test loop above, with required dimensions (theme, viewport, auth, empty, error, offline). | Minutes to an hour |
| L3 | Cross-AI validation | A separate AI audits the diff for performance, security, accessibility, missed edge cases. | Minutes |
| L4 | User testing | A real user (or representative) tries the change and accepts it. | Hours to days |
Levels are cumulative — L3 assumes L0–L2 have been done. Skipping a lower level to “save time” usually costs more time when a defect surfaces later.
A pragmatic mapping — adjust for your project’s risk tolerance:
| Change class | Required | Optional |
|---|---|---|
| Typo, comment, doc edit | L0 | — |
| Pure refactor, behavior unchanged | L0, L1 | L2 (smoke check of affected surfaces) |
| Internal-only change (no user-observable effect) | L0, L1 | L3 if architecturally significant |
| User-facing fix or small feature | L0, L1, L2 | L3 if non-trivial |
| New feature | L0, L1, L2, L4 | L3 |
| Schema migration / breaking API change | L0, L1, L2, L3, L4 | — |
| Security-sensitive change (auth, authz, payment, PII) | L0, L1, L2, L3, L4 | — |
| Performance-critical change | L0, L1, L2 + perf measurement | L3, L4 |
| Production incident hotfix | L0, L1, L2 minimum (verified on staging) | L3 deferred to follow-up |
These are not absolute. A trivial typo in a security-critical place still warrants L2. A feature behind a flag with no users may not need L4. Use judgment; document the choice.
The level annotation lives on the backlog item’s Test: field — not on the PR description, not on the commit message:
Test: pass (L2) — user-facing fix; L2 is this class's required level
Test: pass (L4) — new feature, user-accepted
Test: pass — assumed L2 minimum (default for user-observable)
Test: pass (L1, refactor) — explicit lower bar with reason
Test: partial (L3; L4 pending) — cross-AI done, user gate outstanding
Read the last two lines together. pass records that the item reached its required level, not the
highest level anyone ran. A new feature requires L4, so cross-AI validation at L3 with the user gate
still outstanding is partial, never pass (L3). The level in parentheses says which bar was cleared;
the value before it says whether the bar was the right one.
Storing the level on the item (not the PR) keeps the audit trail in the backlog where the rest of the item’s history lives. A reviewer looking at a PR can glance at the linked BL-#### item to see which level was reached; the level travels with the item, not the (eventually-closed) PR. The level annotation is optional but useful for risk-tracking. A PR that ships a security-sensitive change at L2 is a red flag the reviewer can catch.
Without graduation:
Naming the levels makes the choice explicit. When a contributor says “L3 wasn’t required for this class of change,” the reviewer can challenge the classification rather than re-derive what was checked.
The Definition of Done’s hard rule remains: Status: done requires Test: pass. The taxonomy doesn’t soften it — it clarifies what pass means for the specific change.
pass is reserved for the highest level the item’s change class requires. A change requiring level n cannot be marked Test: pass after only level n−1 — that holds at every step, not just L2→L3. Record intermediate progress as partial (L<reached>; L<required> pending), or leave Test: pending; both keep the item out of done, which is the point.
The trap this closes is specific. Test: is a machine-readable token and the DoD closes an item from any value beginning with pass, so a prose suffix — “awaiting user gate”, “pending sign-off” — does not hold the gate. An agent that writes pass (L3) on a change requiring L4 has already opened the door to done, and the words after the dash will not stop it. If the required level has not been reached, the value is not pass.
The DoD overlay for the project should name which change classes require which minimum level.
No matter how thorough the automated tests, the UI verification loop, or the cross-AI validation — a real user trying the feature is the only test that produces the truth.
Automated tests → UI verification loop → Cross-AI validation → User testing → Ship
Each gate is cheaper to run than the next. Each one catches what the previous doesn’t. Skipping the last one is the most common methodology violation; it’s also the most expensive when it bites.
When the user accepts a feature (“yes, this works”), the loop terminates. Until then, you are still iterating. “I ran the tests and they pass” is not user acceptance; only the user can give that.
Test: pass value on an item requires both the automated suite and the UI verification loop to have passed.Test: field on the item reflects the verification status. not-tested is the default; pass requires the full loop; fail: <detail> records what specifically failed.| Mistake | Fix |
|---|---|
| Ran only the new tests, not the full suite. | Run the full suite. A change can pass its own tests and break others. |
| Test asserts implementation details, breaks on every refactor. | Rewrite to assert behavior — input/output, not internal calls. |
| Bug fix shipped without a regression test. | Add the regression test. The bug will return otherwise. |
| Regression test passes on the broken code (i.e., didn’t actually reproduce the bug). | Test is wrong. Re-run on broken code to verify it fails; fix until it does. |
| Single UI pass declared “verified.” | The single-pass approach is not the loop. Iterate fix-and-verify until a full pass is clean. |
| Verified only on desktop; product is mobile-first. | Verify primary viewport first. Mobile-first products: mobile before desktop. |
| Verified only in light theme. | Verify both themes. Many regressions appear in only one. |
| Skipped UI verification because “the change is small.” | Size is not the criterion; observability is. Small changes break things. |
| “Tests pass” claimed as proof for a user-observable change. | Not sufficient. UI verification is required. Tests cover what tests cover. |
| Verification dimensions check-listed in the PR but never actually run. | The checklist is a memory aid, not a substitute. The checks must be performed. |
| Headless browser ran the checks but a real human never looked. | Automated checks miss visual quality issues. A human glance is part of the loop for visible changes. |
| Empty state never tested; page crashes for a new user. | Empty state is its own dimension; verify it. |
| Error state never tested; user gets a stack trace on a transient API failure. | Error state is its own dimension; verify the recovery path. |
| Console errors ignored because “they were there before.” | Triage them. Pre-existing console errors are technical debt; new ones are regressions. Don’t conflate. |
| A skipped or filtered-out test counted as coverage. | A test that didn’t run is a missing test. Re-enable it (or fix the selection pattern) and re-run; only executed tests back Test: pass. |
| Review showed the plan was wrong; the code was patched to compensate. | Fix the item body first, then re-derive the code. See 07 — Routing findings by failure layer. |
Testing and verification rules outrank speed. They do not outrank explicit user direction, but they do outrank “we’ll add the test later” and “it’s fine, I checked it once.” The lesson “checked it once” produces silently broken changes is too widespread to allow exceptions.
The user can explicitly accept an unverified ship (e.g., “I need this on staging in 30 minutes for a demo; ship without full verification”) — and that risk is then theirs, recorded in the item.
What the contributor never does on their own:
Test: pass without running the loop.These rules bind humans and AI agents identically. Their purpose is not bureaucratic; it is to prevent the failure modes that have happened before.
Next: 00 — README