id: PIR-2026-0004title: Vendor adds generative AI to a rule-based eating-disorder support bot; bot recommends calorie deficits and weekly weigh-ins to the population it was built to protectdate_occurred: first documented harmful outputs October 2022 (screenshots sent to NEDA by MEDA director Monika Ostroff); harmful advice still live late May 2023date_detected: 2022-10 internally (NEDA had Ostroff's screenshots); public detection 2023-05-29 (activist Sharon Maxwell's posts)date_disclosed: 2023-05-30 (NEDA announces Tessa taken down, <24h after Maxwell's screenshots)status: corroborated - details corrected from candidate intake, see Verification notesagent_description: "Tessa," a wellness chatbot offered by the National Eating Disorders Association, originally a closed, rule-based Body Positive program built by academic researchers; operated as a free service by vendor Cass (formerly X2AI), which added generative-AI answer capability in a systems upgrade. Deployed against a clinically vulnerable population just as NEDA wound down its human helpline (unionized staff laid off, effective June 1, 2023).operator_type: enterprise (nonprofit operator, commercial vendor)autonomy_level: autonomous-within-policy (unreviewed replies to users; no tool use)model_stack: unknown (Cass generative layer atop the scripted program; model never disclosed)harness: Cass chatbot platformauthority_scope: external comms only - but to users for whom bad words are clinically dangerous; advice-giving authority over a vulnerable population is the real exposurefunds_at_risk_usd: 0blast_radius: customers/third partiesroot_cause: model-update-regression - primary; a vendor-side upgrade changed behavior of a validated rule-based program. Contributing: plain-error (the generative layer's harmful advice) and operator-error (NEDA had no change control or output monitoring over its vendor)failure_locus: model-provider (vendor Cass, the bot's provider relative to NEDA; note this is a chatbot vendor, not a foundation-model lab)mechanism: Tessa was built as a closed-script prevention tool. Cass's upgrade let it generate novel answers. To eating-disorder users it then recommended, per screenshots and press accounts: 500-1,000 calorie daily deficits, losing 1-2 lbs per week, regular weigh-ins, and skinfold-caliper body-fat measurement - standard diet-culture advice that clinicians state fuels eating disorders, delivered even after users disclosed their condition. It kept doing so for at least ~7 months after NEDA received the first evidence.adversary_present: no (Maxwell and Ostroff probed it, but as good-faith testers of a bot they had reason to distrust; the harmful behavior required no manipulation)exploitation_status: in-wild-malfunction (production system, real vulnerable users exposed, no adversary; v0.2 token per the gap flagged in PIR-2026-0033)severity: degradeddirect_loss_usd: unknown; indirect_loss_usd: unknown (clinical risk to an unknown number of users; loss of both the human helpline and its replacement - the organization was left with neither)downtime: Tessa suspended indefinitely 2023-05-30data_exposure: nonedetected_by: third-party (Ostroff Oct 2022, ignored in effect; Maxwell May 2023, actioned within 24h once public)time_to_detect: ~0 (evidence in hand Oct 2022); time-to-act ~7 months, compressed to <24h by publicitytime_to_recover: n/a - never restoredremediation: bot taken down "until further notice" for investigationstructural_fix: none - the program endedcontrols_that_worked: none identified; the scripted-program boundary that was the original safety design is precisely what the upgrade dissolvedtelemetry_grade: none (user screenshots + NEDA/vendor statements; no logs public)sources:Independence: good - multiple outlets, screenshots from two unrelated testers.
- confidence: high on outputs and suspension; medium on root cause - "Cass changed Tessa without NEDA's awareness or approval" is NEDA CEO Liz Thompson's claim, and vendor statements partially conflict on whether the upgrade was within contract