AI Development Methodology

Loop notes — 2026-05-25

First entry in self-development/loop-notes/. Per AUTONOMOUS_LOOP.md Constraint 1 and Constraint 2a: the loop never auto-edits abstract methodology / templates / brief / strategy / pillars. Insights surfaced here are surfaced for the maintainer’s review cycle, not auto-applied.

Runs covered

• Run 1 (2026-05-25 earlier in the day) — BL-0006 (evaluations folder + skeleton). Notes appear retroactively here.
• Run 2 (2026-05-25) — BL-0007 (cold-read of methodology docs 00–05). 25 findings landed.
• Run 3 (2026-05-25, post v1.13.0 — first run under the tier matrix) — BL-0008 (cold-read of methodology docs 06–11). 29 findings + 5 cross-batch inconsistencies landed. Cross-AI validation: PASS on all four Done-means.
• Run 4 (2026-05-25, close E02) — BL-0009 (two-axis classification of 59 findings) + BL-0010 (eval-report finalization) + materialization of T0/T1 patches as v1.14.0 batch. 30 patches shipped to methodology, 5 T1 escalated to T2 by Sonnet tier-verifier, 28 T2 deferred to maintainer authorship. Cross-AI diff-verification: CONDITIONAL PASS → 3 issues fixed → PASS.

What worked

• Run 1 → Run 2 hand-off was clean. The skeleton’s _(empty)_ placeholders made it unambiguous where the Run-2 Opus agent should land output. Zero confusion at the Edit step.
• BL-0007 charter was specific enough for cold pickup. A fresh general-purpose agent with no prior context completed it correctly on the second attempt (first attempt hit the role-tool issue below). The charter’s “stale / unclear / inconsistent” three-bucket framing produced consistent output across all six docs.
• The methodology version (v1.12.0) held up well under cold-read. Of 25 findings, exactly one is a likely-real bug (stale grep examples in doc 04 created as a side effect of v1.12.0’s C1 fix). The other 24 are unclear-passage or cross-doc-drift findings — the kind of cleanup a six-month evaluation cadence is designed to catch.

What didn’t work — Run 2 operational issue

Agent role + write permission interaction. First Run-2 attempt spawned the Opus cold-read agent in the Explore role. Explore is read-only (no Edit/Write tools). The agent completed the cold-read correctly and returned ~43 findings as text, but couldn’t apply them to the file. Lost ~5 minutes plus the cost of one Opus agent run when the orchestrating session had to re-spawn in general-purpose role.

Root cause: `AUTONOMOUS_LOOP.md “fresh session” definition requires “no prior turns referencing this project” and “ideally a different model family,” but says nothing about agent role / tool availability. The loop’s prompt step 6 (“CROSS-AI VALIDATION”) implies fresh = read-only because cross-AI validation is non-destructive, but cold-reads need write permission to land findings. Cross-AI validation and cold-reads are not the same operation, but both use “fresh session” language.

Suggested methodology addition (surface for maintainer; not auto-applied per Constraint 1):

Add a sentence to AUTONOMOUS_LOOP.md “Operational definition of fresh session” or to methodology/10_testing_and_verification.md "Cross-AI validation":

“Fresh-session agents that must land output in a file (e.g., cold-read findings, classification tables) need a role with Write/Edit tools — use general-purpose or a project’s equivalent, not a read-only role like Explore. Read-only roles are correct for verification (cross-AI Done-means checks) but wrong for production work.”

This is a one-sentence clarification, not a methodology rewrite. Ship in the next patch release or fold into BL-0009/0010’s classification + summary work.

Methodology-change suggestions surfaced this run

These come from BL-0007’s cold-read findings + Run-2 operational observations. Each is a candidate for the next methodology release cycle (after E02 closes via BL-0010). None auto-applied.

From the cold-read itself (formal mechanism, see evaluations file)

The 25 findings sort into three rough tiers by maintainer-effort to fix:

Tier A — patch-release immediately (1 finding, mechanical):

• 04_backlog_items.md grep examples are stale (lines 813–839). Use \| \*\*Pillar\*\* +\| regex against **bold** field names; the canonical frontmatter template uses plain field names. The v1.12.0 C1 fix rewrote items to plain-table form without updating the grep examples. Returns zero matches against compliant items. Fix: drop the \*\* wrappers from all grep examples in that file. Estimated effort: XS.

Tier B — clarify in a patch release (~6 findings, cross-doc agreement + missing default callouts):

• 00_README reading-path table omits doc 11.
• 03_epics WIP-cap default presentation buries the solo recommendation.
• 04/05 Lock+Status coupling described as “automatic” vs “optional” — two docs need to agree.
• 04/05 “orchestrator” (doc 05) vs “lock-holder” (doc 04) naming — cross-link to clarify they’re the same role.
• 03_epics Epic charter template Started: field misses — form for planned epics.
• 05 TTL ceiling vs Common Mistakes table — clarify “24h is ceiling for rare cases, not default.”

Tier C — defer to a future eval pass or treat as known limitations (~18 findings):

• Vague-but-not-wrong passages (e.g., 03_epics parked → done "somehow" — explain or remove).
• Cross-doc rule restatement risk (e.g., “strategy outranks pillars” stated in 00/01/02 — drift risk if any one changes).
• “Illustrative” labeled examples that look canonical (e.g., 02_pillars P1-P9 example table).

BL-0009 (classification) is the formal disposition mechanism for all 25; the tiering above is a preview, not a substitute.

From Run-2 operational observation (not in eval file; logged here)

1. Fresh-session role guidance (described in “What didn’t work” above). Single-sentence addition to AUTONOMOUS_LOOP.md or methodology/10.

2. “No issues found” granularity in the cold-read pattern. BL-0007’s charter says “each doc gets at least one finding entry (even ‘no issues found’ is recorded)” but doesn’t specify per-category vs per-doc. The Run-2 cold-read used _(none)_ per-category (clearest signal: reviewer considered the category and found nothing). Worth pinning down for BL-0008 + future cold-read items so cross-AI validation has a deterministic “did the reviewer actually consider this?” check. Suggested charter addition for BL-0008 + future cold-read items: “For each category (stale/unclear/inconsistent) with no findings, write _(none)_ under that category heading. Omitting a category heading entirely fails the Done-means check.”

3. BL-0009 classification work scales linearly with findings count. 25 findings in BL-0007’s batch + ~25 expected in BL-0008’s batch = ~50 findings for BL-0009 to classify and disposition. Effort estimate (S) may be light. Worth re-estimating once BL-0008 lands. Not a methodology change — just a backlog-management observation.

4. Cold-read sessions consistently surface “cross-doc rule restatement” drift. Several findings flagged the same rule stated in 2–3 different docs with slightly different wording. This is a recurring pattern, not a one-off — could be worth adding to methodology/02_pillars.md or methodology/00_README.md as a “single-source-of-truth” guideline for canonical rules. Defer until BL-0008’s findings either confirm or contradict the pattern.

Run 3 — what worked

• Tier matrix in effect from the start. Run 3 was the first run under v1.13.0’s tiered-autonomy model. The Opus agent was given the charter as-is (no changes from BL-0008’s pre-v1.13.0 text); the new model didn’t disrupt the cold-read pattern. T0/T1 patches don’t appear in BL-0008 itself (per BL-0009’s revised plan, those spawn separate items after classification). The matrix’s existence didn’t change the cold-read mechanics — exactly the intended design.
• Per-category _(none)_ convention from Run 2 was applied throughout. Mechanical Done-means verification is now deterministic — every doc has all three category headings; absence is explicit, not implied. Recommend pinning the convention in the methodology’s cold-read template if/when one is added (currently lives only in BL-0007/0008 charters).
• Cross-batch inconsistencies surfaced as expected. 5 cross-batch findings landed (six-gates count framing between 00 and 07; em-dash vs hyphen-minus drift between 00 and 07; “two-occurrence” promotion trigger fragmented across 04 and 08; authority-chain ordering inconsistency between 00 and 06/07; lock-override rule omitted from the 00/09/11 hard-rules tables). All 5 are exactly the class of finding that single-batch reads cannot produce — the sequential two-batch structure is doing real work. (The Opus agent’s run summary said “4” but actually wrote 5; the validator’s recount confirmed 5.)

Run 3 — methodology-change suggestions surfaced

29 findings + 4 cross-batch from Run 3. The two new v1.13.0 sub-sections (in 09 + 10) had real gaps — a useful signal that the same-release cold-read does catch its own additions. Highlights for BL-0009’s two-axis classification:

New v1.13.0 sub-section gaps (likely T1, surgical):

• methodology/09_git_workflow.md patch-branch convention — area-prefix policy unspecified. The convention says <area>-patch/YYYY-MM-DD-NN with examples (methodology-patch, designsystem-patch, spec-patch) but doesn’t say how a project picks or documents its area prefixes. New adopter has no guidance on whether methodology-patch is canonical for this project or whether they’re free to pick their own. Recommend a single sentence: “Each project records its area prefixes in 09_git_workflow.md or equivalent project doc.”
• methodology/09_git_workflow.md — “never auto-merge” wording. Tier matrix says “loop never auto-merges a patch branch.” But the affirmative table at lines ~250-290 (the can/can’t list for AI agents in git) uses ⚠ for “Creates release tags only with explicit user OK” and “Opens GitHub Releases only with explicit user OK” — the same ⚠ that could plausibly read as “with permission, OK.” Cold reader could mistake patch-branch merging as ⚠ rather than ✗. Recommend explicit ✗ row: “Merges a patch branch into the trunk — always maintainer-led, even when cross-AI PASS on diff-verification.”
• methodology/10_testing_and_verification.md two-modes — ambiguous PASS/FAIL for partial diff-verifications. What happens when grounded=PASS, correct=PASS, scoped=CONCERN (e.g., diff touches the right region but also adds a stray whitespace change)? The doc doesn’t say. Recommend: “A PASS on all three axes is required; any FAIL or CONCERN blocks the patch branch from being offered for merge until resolved.”
• methodology/10_testing_and_verification.md two-modes — “Done-means” term collision. The two-modes section uses “Done means” loosely; the BL item template uses “Done means” as a load-bearing section header. Subtle but a cold reader could collapse the two. Recommend: rename the loose usage to “the validator confirms” or similar.

Cross-batch findings (00–05 vs 06–11) — likely T1, surgical except where noted:

• Em-dash vs hyphen-minus drift in Lock: — field. The DoD checklist in 07_definition_of_done.md:410 uses ASCII hyphen Lock: - while every other location uses em-dash Lock: —. Copy-paste from the checklist produces non-matching items. T0 candidate — pure cosmetic fix in one location.
• “2+ occurrences” promotion trigger. Doc 08 (08_lessons_and_memory.md) says lessons promote to durable rules when seen twice; doc 04 (04_backlog_items.md) doesn’t reference this in the item-pattern discussion where reused lessons would actually be invoked. Cross-link or briefly restate.
• Authority chain ordering inconsistency. Doc 00’s Authority section and doc 06/07’s Authority sections don’t enumerate the layers in the same order. Cold reader can’t tell which is canonical. T1.
• Lock-override rule omitted from hard-rules tables. Doc 05 says “AI agents never override locks.” But the hard-rules table in 00_README.md and the can/can’t table in 09_git_workflow.md and the human-roles ownership table in 11_human_roles.md all omit this. The rule is in one doc; three other docs that summarize “the inviolable rules” don’t surface it. T1 — add a row to each table.

Cold-read pattern observations (potentially methodology-level):

• Zero stale findings in Run 3. Run 2 had 3 stale findings (the grep regression among them); Run 3 had 0. The methodology’s actual content holds up — drift is happening at the clarity and consistency axes, not at the correctness axis. Worth telling adopters: “If your six-month eval surfaces zero stale findings, the methodology is healthy. If it surfaces many stale findings, the methodology has drifted from practice and needs surgery.” Useful as a calibration sentence for adopters running their own eval cycles.

Run 4 — what worked

• Escalate-on-doubt actually caught over-claims. The Sonnet tier-verifier sampled 10 T0/T1 findings and flagged 5 as T1 → T2 (F07 numbering convention, F13 parked→done state machine, F14 solo:1 default, F21 status-flip rule wording, F40 patch-branch ✗ row). All 5 were borderline cases the Opus classifier called T1 — and all 5 implied policy decisions or rule changes the maintainer should own. Without the verifier, those would have shipped in v1.14.0 without proper deliberation. The matrix is not theatre.
• Diff-verification caught real correctness bugs. The post-patch Sonnet diff-verifier found: 1 broken cross-doc anchor (#item-skeleton vs actual #item-heading--frontmatter-table-skeleton), 1 inaccurate inline paraphrase (XS = under one focused session vs canonical XS = under 2 hours), 1 typo (greppping). All three were mechanical mistakes Opus introduced but didn’t notice when authoring 30 patches. The diff-verification gate has now demonstrated it catches what the implementing agent misses.
• The two-axis classification scaled. ~60 findings classified in one Opus pass with the per-finding ID structure (F01–F59) gave the verifier and the maintainer a stable reference vocabulary. The classification table in the eval file is the canonical artifact other agents (and future maintainers) read.
• 30 patches in one batch was manageable. Single patch branch (methodology-patch/2026-05-25-01) + single diff-verification pass + single commit per logical chunk worked at this volume. For a much larger batch (100+ patches across more files), the pattern would benefit from per-doc branches or sub-batches to keep verifier scope tractable.

Run 4 — what to refine before next cycle

• The Opus classifier was over-confident. 5 escalations on a 10-finding sample = 50% over-claim rate. Two routes to reduce: (1) bake more specific T1-vs-T2 examples into the classification prompt so Opus self-rejects borderline calls; (2) make the verifier sample size scale with batch size rather than fixed at 10. For 60 findings, sampling 10 may miss issues in the 36 ship-now patches. Next cycle: 1-in-3 sampling minimum.
• The Tier-A grep-fix demo in v1.13.0 was a pre-flight. v1.14.0 demonstrates the cycle at production volume. The pre-flight vs production gap surfaced one ergonomic improvement: when the implementing agent makes inline paraphrases of content from another doc (e.g., paraphrasing doc 04’s effort definitions in doc 11), the agent should quote rather than paraphrase to avoid drift. Worth folding into the next AUTONOMOUS_LOOP.md revision.
• CHANGELOG updates pile up. Each loop run added an [Unreleased] entry; by Run 4 there were 4 stacked entries. The v1.14.0 release rolled them into the release narrative, which works but felt awkward. Possible improvement: per-run CHANGELOG entries should land under the upcoming release header rather than [Unreleased], so the release prep doesn’t need to re-organize.

Recommendation for next run (Run 5+)

Per AUTONOMOUS_LOOP.md "Subsequent-run targets":

Run 5+: work in the now-active second epic per ROI heuristic; primary target is “close one epic per run” once the cadence is established.

After E02 closes (maintainer signoff on v1.14.0 → ARCHIVE move → EPICS rollup):

1. WIP cap rises from 1 to 2. Maintainer promotes one of E01 / E03 / E04 / E05 to active.
2. Run 5 picks the highest-ROI item from the newly-active epic per the methodology’s ROI heuristic.
3. The 28 T2 findings deferred this cycle remain in loop-notes/2026-05-25.md. The maintainer may file them as items in an appropriate epic (some belong to E03 “Trim 09_git_workflow.md”; some are stand-alone clarifications) OR defer until the next semi-annual eval cycle and re-surface alongside fresh findings. Maintainer’s call.

Recommendation for next eval cycle (Nov 2026)

• Apply the per-category _(none)_ empty-marker convention from the start (now standard).
• Spawn all cold-read agents in general-purpose role (now standard).
• Use 1-in-3 sampling for tier-verification (not fixed at 10).
• Have the classifying agent quote rather than paraphrase when summarizing content from other docs.

Run 4 has more work than Runs 1–3 individually:

1. BL-0009 — two-axis classification (per v1.13.0). Classify ~54 findings (25 from Run 2 + 29 from Run 3) on the practice/docs axis AND on the patch-tier T0/T1/T2/T3 axis. Tier classifications cross-AI verified per escalate-on-doubt. Result: a table identifying which findings spawn patch-branch items (T0/T1 + docs-wrong) and which become loop-notes drafts (T2/T3) or memory entries (practice-wrong). The two-axis structure makes BL-0009 mechanically larger than the original S effort — the v1.13.0 charter update raised it to M.
2. Materialize T0/T1 patches as separate items (not in BL-0009 itself per the revised charter). Each patch gets its own methodology-patch/YYYY-MM-DD-NN branch + diff-verification before merge. The Tier-A grep fix shipped in v1.13.0 was a one-off; Run 4 is the first batch of multiple patches landing through the new mechanism.
3. BL-0010 — finalize eval report + close E02. Summary statistics, next eval date, maintainer signoff line, prepared patch releases. Brings E02 to maintainer-review-pending state.

The Run 4 prompt should also incorporate any T0/T1 patches the maintainer wants prioritized for immediate ship vs. deferred to next minor release.

Status

This file is loop-private — never read by methodology/ or templates/. It’s the surface where loop-detected methodology insights queue for maintainer review. Per AUTONOMOUS_LOOP.md reporting protocol, maintainer reads at next check-in, decides which suggestions to promote to actual methodology releases via normal human-reviewed cycle.

This site is open source. Improve this page.