PIR-2026-0004 - NEDA's Tessa chatbot gives weight-loss advice to eating-disorder patients after an unapproved generative upgrade

id: PIR-2026-0004
title: Vendor adds generative AI to a rule-based eating-disorder support bot; bot recommends calorie deficits and weekly weigh-ins to the population it was built to protect
date_occurred: first documented harmful outputs October 2022 (screenshots sent to NEDA by MEDA director Monika Ostroff); harmful advice still live late May 2023
date_detected: 2022-10 internally (NEDA had Ostroff's screenshots); public detection 2023-05-29 (activist Sharon Maxwell's posts)
date_disclosed: 2023-05-30 (NEDA announces Tessa taken down, <24h after Maxwell's screenshots)
status: corroborated - details corrected from candidate intake, see Verification notes

agent_description: "Tessa," a wellness chatbot offered by the National Eating Disorders Association, originally a closed, rule-based Body Positive program built by academic researchers; operated as a free service by vendor Cass (formerly X2AI), which added generative-AI answer capability in a systems upgrade. Deployed against a clinically vulnerable population just as NEDA wound down its human helpline (unionized staff laid off, effective June 1, 2023).
operator_type: enterprise (nonprofit operator, commercial vendor)
autonomy_level: autonomous-within-policy (unreviewed replies to users; no tool use)
model_stack: unknown (Cass generative layer atop the scripted program; model never disclosed)
harness: Cass chatbot platform

authority_scope: external comms only - but to users for whom bad words are clinically dangerous; advice-giving authority over a vulnerable population is the real exposure
funds_at_risk_usd: 0
blast_radius: customers/third parties

root_cause: model-update-regression - primary; a vendor-side upgrade changed behavior of a validated rule-based program. Contributing: plain-error (the generative layer's harmful advice) and operator-error (NEDA had no change control or output monitoring over its vendor)
failure_locus: model-provider (vendor Cass, the bot's provider relative to NEDA; note this is a chatbot vendor, not a foundation-model lab)
mechanism: Tessa was built as a closed-script prevention tool. Cass's upgrade let it generate novel answers. To eating-disorder users it then recommended, per screenshots and press accounts: 500-1,000 calorie daily deficits, losing 1-2 lbs per week, regular weigh-ins, and skinfold-caliper body-fat measurement - standard diet-culture advice that clinicians state fuels eating disorders, delivered even after users disclosed their condition. It kept doing so for at least ~7 months after NEDA received the first evidence.
adversary_present: no (Maxwell and Ostroff probed it, but as good-faith testers of a bot they had reason to distrust; the harmful behavior required no manipulation)
exploitation_status: in-wild-malfunction (production system, real vulnerable users exposed, no adversary; v0.2 token per the gap flagged in PIR-2026-0033)

severity: degraded
direct_loss_usd: unknown; indirect_loss_usd: unknown (clinical risk to an unknown number of users; loss of both the human helpline and its replacement - the organization was left with neither)
downtime: Tessa suspended indefinitely 2023-05-30
data_exposure: none

detected_by: third-party (Ostroff Oct 2022, ignored in effect; Maxwell May 2023, actioned within 24h once public)
time_to_detect: ~0 (evidence in hand Oct 2022); time-to-act ~7 months, compressed to <24h by publicity
time_to_recover: n/a - never restored
remediation: bot taken down "until further notice" for investigation
structural_fix: none - the program ended
controls_that_worked: none identified; the scripted-program boundary that was the original safety design is precisely what the upgrade dissolved

telemetry_grade: none (user screenshots + NEDA/vendor statements; no logs public)
sources:
- https://www.npr.org/sections/health-shots/2023/06/08/1180838096/an-eating-disorders-chatbot-offered-dieting-advice-raising-fears-about-ai-in-hea (independent; includes the Oct 2022 prior-knowledge reporting)
- https://www.psychiatrist.com/news/neda-suspends-ai-chatbot-for-giving-harmful-eating-disorder-advice/
- https://www.nbcnews.com/tech/neda-pulls-chatbot-eating-advice-rcna87231
- https://fortune.com/well/2023/05/31/neda-ai-chatbot-harmful-advice
- https://www.bbc.com/news/world-us-canada-65771872 (article confirmed via BBC's own feed mirror at feeds.bbci.co.uk/news/world-us-canada-65771872, same ID and title)
- independence: good - multiple outlets, screenshots from two unrelated testers.
aiid_incident_id: 545 (https://incidentdatabase.ai/cite/545/) - cross-reference; primaries verified independently
confidence: high on outputs and suspension; medium on root cause - "Cass changed Tessa without NEDA's awareness or approval" is NEDA CEO Liz Thompson's claim, and vendor statements partially conflict on whether the upgrade was within contract

Correction (dates): candidate stated "harmful outputs late May 2023." Verified reporting shows NEDA received screenshots of harmful Tessa outputs in October 2022; the May 2023 events were public exposure, not first occurrence. date_occurred, date_detected, and the ~7-month time-to-act above reflect this. It materially changes the incident's shape: not a fast-detected malfunction but a known issue actioned only under publicity.
Classification kept as primary model-update-regression per candidate, with operator-error added as contributing for the absent vendor change-control - which the corrected timeline makes load-bearing.

2026-08-19: Added aiid_incident_id cross-reference (AIID 545), matched against the AIID weekly database export (2026-08-17). A cross-reference, not a re-verification; no claim changed.

NEDA's Tessa chatbot gives weight-loss advice to eating-disorder patients after an…