AI-Failure Lab — a Teaching Pack on Multimodal Blind Spots
A downloadable lesson pack for high-school and undergraduate teachers that uses curated real AI failure cases (the 235-sample Blind-Spots-Bench dataset and selected submissions) to teach AI literacy, with discussion prompts, slide decks, and student exercises.
secondary and undergraduate educators teaching media literacy, AI ethics, or computer science
- Curated set of 30-50 'before/after' AI failure examples with discussion prompts and difficulty ratings
- Lesson slide decks (Canva/PowerPoint) ready for classroom use on AI literacy and bias
- Student worksheet where learners attempt to predict which model will fail a given task
- Discussion-guide on why current benchmarks under-measure these failures
The paper publishes an open, classroom-ready dataset of 'humans find this trivial, AI doesn't' failures at a moment when schools are scrambling for AI literacy materials that go beyond 'AI hallucinated a fact.'
Major institutional pushes confirm real K-12/undergrad demand: Stanford Digital Education launched an AI curriculum for high schools (Feb 2025), Quill.org + aiEDU released a free year-long AI literacy curriculum for 2025-26, and the OECD/EC published an AI Literacy Framework for primary/secondary education in May 2025.Stanford Digital Education designs AI curriculum for high schools ↗Building AI Knowledge: A New AI Literacy Curriculum from Quill + aiEDU ↗Empowering Learners for the Age of AI (OECD/EC AILit Framework) ↗
Most existing resources cover AI fundamentals or generic 'hallucination' warnings; a Substack explicitly argues schools lack failure-case pedagogy, and the 235-sample Blind-Spots-Bench (humans-trivial-AI-fails) has not yet been wrapped into a downloadable classroom pack.When AI Goes Wrong: A Case Schools Cannot Ignore ↗Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models ↗
Hard to monetize: the leading competitor offerings (Stanford, OECD/EC, Quill+aiEDU, Teach for All teacher guide) are all free, and TeachersPayTeachers already lists AI ethics lesson plans crowding the niche at low price points, suppressing willingness to pay.Building AI Knowledge: A New AI Literacy Curriculum from Quill + aiEDU ↗Artificial intelligence and ethics - TPT ↗Teacher Guide: Artificial Intelligence (AI) in the Classroom (Teach For All) ↗
AI literacy is being baked into national curricula (OECD/EC framework) so the category is durable, but the specific 235 failure samples will age as models improve, requiring continual curation.Empowering Learners for the Age of AI (OECD/EC AILit Framework) ↗
The dataset is already public on arXiv, sample sizes are tiny (235 cases), and lesson decks + discussion prompts are straightforward content production — a small team or solo educator can ship it.Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models ↗