DryRun Studio — Cheap QA for AI courses, tutors, and support bots
A no-code scenario harness for small AI product teams and course creators to script realistic user conversations, replay them with deterministic responses, and catch regressions before pushing updates — without paying the LLM bill on every test run.
Solo founders and course creators shipping AI tutors, support bots, or stakeholder-simulation agents
- Scripted scenario library that swaps only the model call with a stub, keeping UI and routing real
- Cost-guardrail alerts if any test accidentally hits a real provider and burns credit
- One-click regression run on every content update with a 'what changed' diff against last version
- Pre-built templates for common flows: tutor Q&A, rubric-based evaluator, stakeholder interview
AI course pilots (like Kent C. Dodds' Product Engineering Workshop) and SMB support bots are now layering multi-agent flows with rubric evaluators on top of LLM calls — naive testing costs more per run than the product earns.
The cited source trend (Dev.to post with 2 reactions) is a near-zero engagement signal, and the target buyers (solo course creators, non-technical SMBs) historically are not buyers of QA/dev-tools — that audience is engineers already served by Promptfoo, LangSmith, etc.LLM Evaluation Tools: The Complete Comparison Guide (2026) ↗
Bottest.ai is essentially this exact product — no-code regression testing for AI chatbots with LLM-based evaluators, consistency checks, multi-baseline answers, root-cause reports — and it is free and open source. Promptfoo, LangSmith, Braintrust, DeepEval, Cekura all occupy adjacent space.Regression Testing | bottest.ai ↗Promptfoo Alternatives (2026): Braintrust, LangSmith, DeepEval, PromptEval Compared ↗
Closest direct competitor (Bottest.ai) is free/open source, and most LLM-eval tools compete on generous free tiers (Promptfoo OSS, LangSmith free dev tier); course creators are notoriously price-sensitive, undermining willingness to pay.Regression Testing | bottest.ai ↗LangSmith vs Promptfoo (2026) | Respan ↗
LLM evaluation and regression testing is clearly durable as AI agents proliferate across courses and support; active academic work (NAACL 2025 tutor-eval taxonomy) confirms ongoing category growth.Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors ↗
Building a no-code scenario harness with deterministic replay, multi-baseline grading, and framework integrations is non-trivial (UI + eval logic + many bot SDKs); open-source Bottest.ai is a clear baseline to fork or compete with, raising the technical bar to differentiate.Regression Testing | bottest.ai ↗