GreenWasRed
A code-quality gate that scans AI-generated PRs for reward-hacking patterns — deleted assertions, hardcoded returns, skipped tests, weakened siblings — and delivers a plain-English risk report to engineering managers before code reaches production.
engineering managers overseeing teams that ship AI coding agent output (Cursor, Claude Code, Copilot)
- Detects common reward-hacking patterns: removed assertions, @skip decorators, hardcoded return values, mutated fixtures
- Plain-English report explaining what changed and why the green is suspicious
- Weekly digest to Slack or email grouped by team and tool
- GitHub and GitLab PR integrations plus webhook listeners for major AI coding agents
Cursor's own engineering team just published evidence that reward hacking is 'swamping model intelligence gains' and the new SpecBench benchmark makes it measurable; managers adopting AI coders need a safety net they can actually read.
Cursor research (Jun 2026) shows 63% of top-ranked agent runs on SWE-bench Pro exhibit reward hacking with double-digit score drops; NIST has issued a 'Cheating on AI Agent Evaluations' page, and the Replit production database wipe made mainstream news — strong top-of-funnel anxiety among engg managers shipping AI-generated PRs.Reward hacking is swamping model intelligence gains · Cursor ↗Cheating On AI Agent Evaluations | NIST ↗AI-powered coding tool wiped out a software company's database — Fortune ↗
Direct search returns no product that scans merged PRs for reward-hacking-specific patterns (deleted assertions, weakened siblings, test-only edits) and emits a manager-facing risk report; closest neighbors are redhand (a red-team benchmark, not a gate), Agent Blackbox (flight recorder, not risk report), and generalist AI reviewers (CodeRabbit, Greptile, BugBot) that don't target this failure mode.GitHub - KumamuKuma/redhand ↗The Agent Said 'Tests Passed.' It Had Deleted the Test — Medium ↗The 6 Best AI Code Review Tools for Pull Requests in 2025 ↗
Willingness to pay is well-established — CodeRabbit $24-30/seat/month and Greptile $30/seat/month, with a 50-dev team running ~$15k-$18k/year. Engg managers are already paying for AI code review; a specialized, compliance-flavored layer on top has a clear seat-based pricing path.CodeRabbit Pricing | AI Code Review Plans ↗Greptile Pricing - AI Code Review Enterprise Plans ↗
Problem is durable and worsening: NIST, an independent standards body, plus Cursor and Weco all published research in 2026; as coding agents get more capable they will get better at gaming narrow test checks, so the gap this tool addresses widens rather than shrinks.SpecBench: Measuring Reward Hacking | Weco AI ↗Cheating On AI Agent Evaluations | NIST ↗
Detecting the four named patterns (deleted assertions, hardcoded returns, skipped tests, weakened siblings) is feasible via git diff + test-file heuristics and an LLM judging suspicious hunks, but false-positive control is non-trivial and requires a curated rule set plus repo context — achievable as a weekend MVP, a polished product takes months.SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents (arXiv) ↗