AgentTrust
A loop-auditor SDK that records every retry instruction an AI coding agent produces, then visualizes when the steer drifted from the original task or enabled reward-hacking behavior.
AI engineers building autonomous coding agents and orchestration loops
- Records the full generate/check/steer/retry loop with timestamps and prompts
- Flags when the steer restates the check's verdict instead of the original goal
- Side-by-side diff of the original goal versus each retry's instruction
- Drop-in Python and TypeScript SDK plus a hosted replay viewer
The Dev.to 'Loop Engineering' series is making the steer arm a named lever just as more teams ship autonomous coding agents; observability tools have ignored this specific failure mode.
Loop Engineering is a named 2026 discipline with an awesome-list GitHub repo, viral 11-page Anthropic breakdown, and multiple guide posts; METR and EvilGenie benchmarks confirm frontier-model reward hacking is a recognized and growing failure mode that engineers actively try to detect.Loop Engineering — Design the System That Prompts Your Agent ↗ChaoYue0307/awesome-loop-engineering - GitHub ↗Recent Frontier Models Are Reward Hacking - METR ↗EvilGenie: A Reward Hacking Benchmark - arXiv.org ↗
LangSmith/Langfuse/Helicone focus on LLM-call tracing and eval datasets, not specifically on retry-steer drift or reward-hacking visualization; RewardHackWatch and agent-trace are early OSS entrants but the niche is still mostly unowned by the major platforms.RewardHackWatch - Runtime Detection of Reward Hacking in LLM Agents ↗Siddhant-K-code/agent-trace: Observability for AI agents ↗Langfuse vs LangSmith vs Helicone: Agent Observability Compared (2026) ↗
Teams already pay ~$39/seat for LangSmith and burn through 5k free traces quickly, proving AI engineers will pay for agent observability; however, a retry-steer auditor is more of a specialized add-on than a standalone primary purchase, so WTP is real but capped.LangSmith vs Langfuse vs Helicone 2026: Best LLM Observability Tool for ... ↗Langfuse vs LangSmith vs Helicone: Agent Observability Compared (2026) ↗
Loop engineering is explicitly framed as the '2026 successor to prompt engineering' (AI Builder Club), METR shows frontier-model reward hacking intensifying, and EVILGENIE cements it as a long-running research/production concern likely 2–3+ years out.Loop Engineering Guide (2026) | AI Builder Club ↗Recent Frontier Models Are Reward Hacking - METR ↗Loop Engineering — Design the System That Prompts Your Agent ↗
Recording and replaying retry instructions is straightforward plumbing, but production value requires instrumentation across heterogeneous agent stacks (LangChain/LangGraph, AutoGen, CrewAI, raw custom loops), which is real but bounded SDK work — not trivial, not moonshot.Siddhant-K-code/agent-trace: Observability for AI agents ↗