Purpose: define how a project measures whether it is ready for the next milestone — beyond per-item DoD. Per-item gates (doc 07) catch whether a single change is correct; this doc catches whether the accumulated changes have moved the project closer to its goals. Pairs with the autonomous-loop pattern in
templates/AUTONOMOUS_LOOP.md.
Per-item testing alone is insufficient at scale. A hundred items can each pass their DoD while the overall product is unfit to ship. The milestone-evaluation loop catches the gap.
This doc operationalizes four ideas:
The whole structure exists to keep loops honest: per-item green ≠ milestone-ready.
The per-item DoD (doc 07) asks: did this change pass its own gates? It catches single-change defects.
It does not catch:
The milestone-evaluation loop is the corrective. Every Nth iteration of the autonomous loop, the project scores itself against the next milestone’s readiness criteria — not against per-item DoD. The aggregate matters; per-item green is necessary but not sufficient.
A project moves through a sequence of named readiness states. The names matter because they communicate expected quality to internal contributors and external observers.
| Milestone | Audience | Bar |
|---|---|---|
| Pre-alpha | Maintainer only | Code compiles, primary happy path works, no public exposure. Internal-only validation. |
| Alpha | Trusted internal users (≤ 5) | Core feature set demonstrably works; obvious bugs absent; UI may be rough; no SLA. |
| Closed beta — wave 1 (friends) | Maintainer’s network (≤ 10) | Stable enough that friends will trust it for real (low-stakes) use. Acquisition signals start being measurable. |
| Closed beta — wave 2 (strangers) | Curated external users (≤ 50) | Onboarding works for users without context. Failure modes have explanations. Feedback flow is operational. |
| Open beta | Public, opt-in | Anyone can sign up. Free tier is meaningful. Quality bar is “trustworthy enough that bad PR is unlikely.” |
| First public final (v1.0) | Public, advertised | The project’s headline promise is delivered. Pricing (if any) is committed. Removal of “beta” label means the maintainer is willing to be quoted on its readiness. |
| General availability | Public, scale-tested | Has handled meaningful load; ops on-call exists; SLA published. |
Projects adapt the sequence. A solo CLI tool may collapse closed beta and open beta into one. A regulated-industry product may insert compliance review between open beta and v1.0. A research artifact may skip the betas entirely and ship to “public preview” directly. Pick the sequence that fits the project; name it explicitly in the strategy doc.
A milestone is reached or it isn’t. “We’re sort of in beta” is a smell — it means the team is calling itself one thing while operating as another. The readiness criteria for the milestone are the binary check.
The criteria live in the strategy master plan (doc 01) or in a dedicated milestones.md referenced from there. Format suggestion:
## Milestone: Closed beta (wave 2)
**Audience:** ~50 curated strangers (acquisition via 3 channels).
**Readiness criteria (binary):**
- [ ] Onboarding flow tested by ≥ 3 strangers without maintainer intervention; ≥ 2 succeeded.
- [ ] Crash rate < 1% over 7-day window.
- [ ] All P0 + P1 items closed.
- [ ] Pricing page published if monetization is a beta concern; deferred otherwise.
- [ ] Feedback inbox has a defined triage cadence (e.g., within 48h).
- [ ] Scoring rubric (see below) averages ≥ 9.0/10 across all areas with no area below 8.0.
**Exit criterion:** maintainer reviews the rubric and signs off explicitly.
The criteria are the eval target. Per-item DoD continues to gate individual changes; readiness criteria gate milestones.
The rubric is the operational tool the milestone-evaluation loop uses. It scores the project on the dimensions that matter for this project, on a 0–10 scale.
Adapt per project. The starting set below covers most product-shaped projects. Pick the areas that match your project; ignore the rest; add new ones that fit your domain. A score is only meaningful if “10/10” has a project-specific definition.
| Area | What 10/10 means | What 0/10 means |
|---|---|---|
| UX / UI design | Every flow lands; no broken states; navigation is unambiguous; visual quality is at-or-above adopter expectations for this product class. | Users get stuck; key flows broken; visual quality embarrassing. |
| Frontend | No console errors; assets cached correctly; bundle size under budget; works in target browsers + viewports. | Console errors per page; load times exceed budget; broken in target browsers. |
| Backend | All APIs respond correctly; no N+1 queries on hot paths; error responses are structured; uptime > target. | APIs throw 500s on common inputs; slow queries; opaque errors. |
| Database | Schema is normalized appropriately; migrations are reversible; indexes match query patterns; slow-query log is clean. | Schema rot; migrations break prod data; sequential scans on hot tables. |
| Authentication (authn) | Login flows work for all supported methods; session lifetime + refresh sensible; no auth bypass paths. | Login broken on common inputs; tokens leak in URLs; session fixation possible. |
| Authorization (authz) | All data access is owner-derived from server-trusted session, never from request body. RLS or equivalent boundary present at every privacy-sensitive read/write. | IDOR present; client-supplied IDs accepted as authorization input; missing checks on privileged routes. |
| Security | Input validation present at boundaries; no exposed secrets; OWASP-relevant categories audited; dependencies have no known critical CVEs. | Secrets in commits; XSS/SQLi possible; outdated deps with known exploits. |
| Performance | p50 / p95 latencies under budget; bundle sizes under budget; perf budget enforced in CI. | Pages > 5s on target hardware; perf budget unenforced. |
| Test coverage | Behavior tests cover all critical paths; regression tests for every fixed bug; suite runs in CI. | Critical paths untested; bugs recur because regression tests missing. |
| Accessibility — design | Keyboard navigation works; focus states visible; contrast meets WCAG AA; screen reader can use core flows. | Keyboard navigation broken; focus invisible; failed contrast; screen reader blocked. |
| Accessibility — testing | Automated a11y checks in CI (e.g., axe-core); manual screen-reader QA on critical flows; user testing with assistive-tech users where feasible. | No a11y in CI; no manual screen-reader pass; a11y “checked once at launch and never since.” |
| Content quality | Copy is consistent in voice; no typos in user-visible text; placeholder content removed; translations (if any) parity-checked. | Inconsistent voice; placeholder text in production; broken translations. |
| Documentation | Adopter can set up the project from README alone; key concepts have docs; CHANGELOG honest; on-call runbooks exist where relevant. | Setup requires asking; docs contradict code; CHANGELOG outdated; tribal knowledge dominates. |
| CI / CD | Every push runs the full pipeline; deploys are scripted + reversible; staging mirrors production; failed deploys roll back automatically. | Manual deploys with scp; no staging; CI flaky enough to be ignored. |
| Production / operational readiness | Monitoring + alerting in place; on-call rotation defined; incident playbook exists; SLO + error-budget tracked. | No monitoring; bugs surface from user reports; no playbook; SLOs aspirational not measured. |
| Paywall / monetization | Pricing visible + correct; checkout works on all supported payment methods; entitlements + grace periods documented; refund flow tested. | Pricing inconsistent across pages; checkout failures unhandled; entitlement bugs lock paid users out. |
| Administration / operator tools | Operators have UI for the common ops (review queue, ban, override, refund); CLI alternatives exist for emergencies; all ops are audit-logged. | Operators ssh into prod to fix things; no audit log; one-off SQL queries in chat. |
| Internationalization | Strings externalized; right-to-left support if relevant; date/number/currency formatting matches locale. | Hardcoded English strings; broken layout in RTL; date format assumes US. |
| Privacy + data handling | Per-user data scoped correctly; deletion works; export works; data-retention policy enforced. | User data leaks across tenants; no delete path; retention unset. |
| Brand + voice | Consistent visual identity; tone matches strategy doc; no off-brand assets in production. | Inconsistent visual; tone shifts mid-flow; placeholder logos. |
| Onboarding | A new user reaches first-success in under N minutes (target per project); failure modes recoverable; no required-but-unexplained step. | Users abandon at signup; first-success requires support contact. |
Add or drop areas based on the project’s domain. A regulated-industry product adds Compliance. A research artifact adds Reproducibility. A consumer mobile app adds Mobile UX (iOS vs Android parity). A B2B platform adds Tenant isolation. Document the chosen areas + 10/10 definition per area in the strategy doc or milestones.md.
When a strategy is drafted, the maintainer can ask a fresh AI session: “For a project of shape X with users Y and milestone Z, what are the right scoring areas, and what does 10/10 mean for each?” The AI’s first-pass list, refined by the maintainer, is the project’s rubric. This is a one-time setup activity per milestone; the rubric stabilizes across the milestone’s eval cycles.
The rubric can apply at different levels of granularity. Pick the level(s) that fit your project’s structure:
| Scope | What it scores | When to use |
|---|---|---|
| Project-wide | The entire product, scored once on the area list. Single aggregate score. | Default for small / mid-sized projects. The whole product as one unit. |
| Per-pillar | Each pillar from 02_pillars.md gets its own rubric pass — different areas per pillar where appropriate. | Projects where pillars are decoupled enough that a strong backend pillar shouldn’t paper over a weak content pillar. |
| Per-epic | Each active epic gets its own area list + scores. The aggregate is the epic-weighted average. | Useful when an epic represents a significant scope (e.g., a whole new product surface) that deserves its own quality check. |
| Per-item (user-story scoped) | Each major user story / item gets a quality score. Aggregates roll up to epic → pillar → project. | Heavy ceremony; reserve for items where the user-story bar is the whole point (e.g., one critical user flow that gates a milestone). |
Most projects start at project-wide and add per-pillar at the second eval pass if the project-wide average is hiding under-served pillars. Per-epic and per-item are powerful but expensive; use deliberately, not by default.
Mixed scope is allowed. A project may score project-wide for most areas + per-pillar for security (because security drift in any one pillar matters disproportionately) + per-item for a specific user-flow item the maintainer is watching closely. The rubric is yours to shape.
A rubric becomes theater the moment scores stop reflecting reality. Guardrails:
docs/perf.md” beats “Backend: 7 because performance has issues.”For a project to declare it has reached a milestone:
Each periodic deep-eval produces a single rubric pass. Scores are integers 0–10 with a one-paragraph justification per area (cite evidence — specific items, bug reports, test results, log samples, user feedback). The justification is more important than the number; the number is the summary.
Use cross-AI for scoring, not the implementing session. A session that wrote the code is biased toward charity. A fresh session with read-only access to the codebase, the strategy doc, the rubric, and recent test/log/feedback data scores more honestly.
The per-item DoD runs every item. The periodic deep-eval runs every Nth loop iteration.
Adopters tune per project. The point is predictable cadence, not a specific number. A team that “evaluates whenever it feels right” doesn’t evaluate.
evaluations/YYYY-MM-DD-milestone-eval-NN.md with: scores per area, items filed, milestone-readiness verdict.Loop 1: items → DoD → ship.
Loop 2: items → DoD → ship.
Loop 3: items → DoD → ship → DEEP EVAL → fix-and-adjust based on scores → file new items.
Loop 4: items (including new ones from Loop-3 eval) → DoD → ship.
Loop 5: items → DoD → ship.
Loop 6: items → DoD → ship → DEEP EVAL → ...
The deep-eval is not a separate loop; it’s an extra step at the end of every Nth loop. The next loop picks up the items the deep-eval filed.
Some issues do not yield to additional loop iterations. The methodology is honest about this.
| Disposition | When to use | What it means |
|---|---|---|
| Handled | A workaround exists that’s good enough for the current milestone. | Item closes with Status: done; a Limitation: note in the item body documents the workaround. |
| Postponed | The issue’s correct fix is real but its priority doesn’t justify blocking the current milestone. | Item moves to FUTURE.md with a postponement reason; re-surface at the next milestone planning cycle. |
| Marked (known-broken) | The issue can’t be fixed cleanly within the project’s constraints, and forcing a fix would make things worse. | Item closes with Status: rejected + a Known issue: documentation entry in CHANGELOG / README. User-facing if it affects users. |
“Multiple attempts” is judgment-dependent unless a number makes it executable — an autonomous loop with no cap will happily attempt the same issue a tenth time, re-spending its full context on every retry. The default cap: three failed fix-test attempts at the same issue — three iterations that each ended with verification still red, or that shipped a regression — makes choosing a disposition mandatory. The loop does not attempt a fourth fix; it picks handle, postpone, or mark.
Three clarifications keep the cap honest:
Status: rejected; the third failure’s disposition can still be “handled” via an acceptable workaround. The cap only forbids silent retry number four.When an issue is marked:
Status: rejected (per doc 04 lifecycle).Known issue: entry lands in the user-facing documentation (CHANGELOG.md “Known issues” section; README.md “Limitations” section).The discipline matters because the alternative is worse. A team that refuses to mark issues will either ship broken work or stall indefinitely. The methodology’s stance: marking is honest; forced progression is dishonest.
Even when the rubric scores 9+ across the board and the milestone criteria are met, the maintainer’s human review is the final gate before milestone declaration.
The rubric measures what it knows how to measure. A 10/10 UX score from a cross-AI session and an unbiased automated test suite still cannot replace a human’s intuition about whether the product feels right. The methodology defers to human judgment on intuition-grade questions (per doc 11 “Decision-ownership matrix” → human-only column).
After the rubric passes and items are closed:
Once a milestone is reached (especially closed beta or later), real users start submitting feedback. The backlog must support fast triage so feedback doesn’t pile up unprocessed.
Inbound feedback (email / GitHub Discussion / form / app-internal report)
↓
Single inbox: backlog/FEEDBACK.md (append-only NDJSON or single markdown file)
↓
Triage cadence (every N hours / daily / weekly — pick per project)
↓
For each feedback item:
├─ Classify: bug / feature request / question / praise / spam.
├─ If bug: file as BL-#### in appropriate epic with `Reported-by:` field.
├─ If feature request: file in FUTURE.md or relevant pillar's backlog.
├─ If question: respond + consider whether the docs need an addition.
├─ If praise: log for sentiment tracking; consider testimonial use.
└─ If spam: drop.
↓
Reply to the user (where possible) with what was filed + ID.
↓
The next loop cycle picks up the new items per ROI heuristic.
Two formats work:
Format A — single markdown file, one section per feedback item:
# Feedback inbox
## FB-0001 — 2026-05-25 — Email — "Onboarding broken on Safari"
Reported by: jane@example.com
Triaged by: maintainer
Disposition: filed as BL-0428 in E03 (browser-compat epic)
Reply sent: 2026-05-26 12:00 UTC
> Original message:
> "I tried signing up and the page just stays blank in Safari…"
Format B — append-only NDJSON (for projects with feedback volume that justifies tooling):
{"id":"FB-0001","ts":"2026-05-25T14:00Z","src":"email","reporter":"jane@example.com","subject":"Onboarding broken on Safari","status":"triaged","filed_as":"BL-0428"}
Either is fine. Pick one and stay consistent.
The cadence is a project decision, but a cadence must exist. “We’ll get to it when we get to it” is the failure mode.
templates/AUTONOMOUS_LOOP.md) runs the per-item work; this doc defines the periodic deep-eval that runs every Nth iteration.milestones.md referenced from strategy.A fictional project at end of Loop 3 (the deep-eval loop). The project’s milestone target is “Closed beta wave 2”; rubric is the standard 10-area set; thresholds are minimum 8, average 9.
Loop 3 wraps:
Loops 4–5 close the new items.
Loop 6 wraps; deep-eval runs again:
The cycle moves to the next milestone (open beta) with its own readiness criteria and (possibly) a refined rubric.
When the milestone evaluation and per-item DoD conflict — e.g., an item passes its DoD but moves a rubric area further from threshold — the rubric wins for milestone progression. The item still ships (its DoD is valid); the deep-eval simply files compensating items to bring the area back.
When the maintainer’s human review and the rubric conflict — e.g., rubric says READY but the maintainer’s intuition says NOT READY — the maintainer wins. The rubric measures what it knows; the maintainer measures the unmeasured (per doc 11).
When the unsolvable-issue heuristic says “mark” and a contributor’s energy says “let’s try one more time” — the heuristic wins by default, overridden only by explicit maintainer authorization. The cost of forcing the issue exceeds the cost of marking it.
Next: with the milestone evaluation pattern in place, the methodology has its full vertical: strategy → pillars → epics → items → DoD per item → milestone eval across items → human review at milestone declaration. Adopters can run a project end-to-end through this stack with high autonomy and high quality.