Telling engineers which pull requests need a human and which don't

01The challenge

Spinach AI's engineers build agent-first, and the team already ran AI review bots, custom lint rules, deterministic CI checks and dark releases. The problem was latency, not throughput: when a pull request came back with comments after a delay, its author paid to rebuild the context. The team wanted more confidence, so more PRs could be self-merged safely, and faster human review for the rest. Earlier attempts had stalled: agents asked to propose lint rules from the codebase alone produced nothing that shipped, and an agent that commented on every PR was so noisy it stopped being read.

02What we did

1. 01Week 1 — measured 120 days of merged pull requests (time to merge, review patterns, self-merges) and held back recommendations until the evidence was in
2. 02Week 3 — agreed the working frame with engineering: raise confidence so more PRs can be self-merged, and make human review faster
3. 03June 2026 — built the lint-rule extraction and a first version of WiltWatch, then tested both in validation sessions with the engineers
4. 04Validation — the engineers rejected WiltWatch's first form, a comment on each PR, but backed its function, and one proposed using the score as a self-merge gate
5. 05July–August 2026 — validated the score on a self-merge cohort and a two-year backtest, and landed the PR-signals pipeline in Spinach's repository

03What we built

The lint-rule extraction reads recent review comments, review-meeting transcripts and incident history rather than the code alone. Existing rules go in as a do-not-suggest list, and a candidate that cannot point to recent code it would have caught is dropped before anyone reads it. In review the team extended candidates, added its own and asked for functional rules over styling; findings that could not be linted went into review-bot instructions instead.

WiltWatch scores each pull request against the team's own history: hot files and past must-fix review comments. A deterministic pass sets the tier (routine, glance or focus), then Claude checks whether flagged lines repeat a recorded failure and downgrades false alarms. The score proved unrelated to the existing AI reviewer's: one asks whether a diff is well written, the other whether it stands on ground where the team has been burned before.

• Lint-rule candidates grounded in real incidents
• Routine, glance and focus risk tiers
• Claude pass that downgrades false alarms
• Weekly PR-signal sync into SQLite
• Runs entirely in GitHub Actions
• Weekly written notes to engineering leadership

04The result

WiltWatch's tiers sort pull requests by how they turn out: in a two-year backtest, PRs it sent for focus were 5.2 times as likely to need a later fix as those it stayed silent on. The backtest also overturned an assumption. Self-merged PRs were the safer half of the work, and most later fixes traced back to large reviewed changes, a finding that now shapes the next version.