Physical or psychological harm to a user
Wrong clinical output, missed crisis path, unsafe device command.
Severity: harm. Attached to 13 of 46 published product types, backed by 17 evidence records.
Product types that carry it
| Product type | Grade | Note for this type | Evidence |
|---|---|---|---|
| telemedicine app | practitioner single incident, no second source | Online visit and refill flows are used to distribute a stimulant such as Adderall over the internet, the scheme federal prosecutors alleged against one digital health company's founder and clinical president. | |
| electronic health record system | secondary | A lab or consult order with no valid routing target is accepted on screen and lands in a queue nobody works, so the patient's test never happens, or an update takes the chart down during care and the downtime view does not show current medications and orders. | |
| software as a medical device | primary | A dose typed as ".5" without the leading zero is read as a whole number, or a crash loop drains the paired pump's battery until insulin delivery stops. | |
| clinical trial management software | primary | A participant entered in an untested stratum combination is allocated from the wrong randomisation table, or the weight-based dose calculation dispenses the wrong kit. | |
| wearable companion app | primary | A device within Bluetooth range crashes the band or stops it from working, and the health alerts the user relies on stop. | |
| machine learning model service | primary | A false match from the model labels a real person as a suspect, and staff confront or remove them on the alert. | |
| LLM-based assistant | primary | A child or teenager treats the assistant as a companion and gets answers unsuitable for their age, and the operator has never tested what it says in those chats. | |
| smart home or IoT app | primary | A stranger who took over the account talks to a child through the camera's two-way audio from the app. | |
| fleet and logistics tracking | primary | An unauthenticated command reaches a tracker wired to a fuel cut-off relay and cuts the vehicle's fuel supply. | |
| social network | primary | The recommendation feed keeps serving self-harm posts the algorithm selected to a teenage account, with no separation of child and adult content. | |
| messaging app | primary | A child account added to a group chat or group call by an approved contact ends up talking to an adult the parent never approved. | |
| mobile game | practitioner single incident, no second source | Voice and text chat are on by default, so a child matched with strangers in a squad is exposed to harassment from the first session. | |
| kids app | primary | Voice or text chat switched on by default lets adult strangers contact a child player, the default the FTC tied to children being harassed. |
Tests that address it
- end-to-end functional tests: mandatory for this risk on 11 product types.
- release smoke tests: mandatory for this risk on 2 product types.
- integration tests: mandatory for this risk on 2 product types.
- regression suite: mandatory for this risk on 2 product types.
- resilience and failure testing: mandatory for this risk on 2 product types.
- device and browser compatibility: mandatory for this risk on 1 product types.
- unit tests: mandatory for this risk on 2 product types.
- penetration testing: mandatory for this risk on 3 product types.
- exploratory testing: mandatory for this risk on 6 product types.
- data integrity and migration testing: mandatory for this risk on 1 product types.
- analytics and event tracking tests: mandatory for this risk on 1 product types.
- compliance checks: mandatory for this risk on 1 product types.
Evidence
ev-0036regulatorprimary
Age appropriate design: a code of practice for online services (Children's code), Information Commissioner's Office
The UK Children's code sets 15 standards, including high-privacy defaults, for online services likely to be accessed by children.
published undated · verified 2026-10-07
ev-0054regulatorprimary
Guideline on computerised systems and electronic data in clinical trials (EMA/INS/GCP/112288/2023), European Medicines Agency, GCP Inspectors Working Group
Where the randomisation is stratified by factors inputted by the user, all the combinations of the strata should be tested to confirm that the allocation is occurring from the correct randomisation table.
EMA inspectors expect trial systems to keep a secure audit trail, restrict unblinded data to named roles, test emergency unblinding and test randomisation strata and dose calculations in response technology, which covers data corruption and participant harm in trial software.
published 2023-03-09 · verified 2026-10-07
ev-0076regulatorsecondary
The New Electronic Health Record's Unknown Queue Caused Multiple Events of Patient Harm, U.S. Department of Veterans Affairs, Office of Inspector General
Government inspection report: clinician orders the EHR could not route went to a queue no one monitored, and patients were harmed. Supports user-harm for EHR systems, where an order that never reaches the lab or specialty service produces no error on screen.
published 2022-07-14 · verified 2026-10-07
ev-0078regulatorsecondary
VA Needs to Strengthen Controls to Address Electronic Health Record System Major Performance Incidents, U.S. Department of Veterans Affairs, Office of Inspector General
The OIG found that VA and Oracle Health did not have adequate controls in place to prevent system changes from causing major incidents, to respond to those incidents when they did occur, and to mitigate their impact.
Government audit of EHR outages, degradations and incomplete functionality caused by system changes, plus downtime viewers that did not suit every facility. Supports user-harm for EHR systems, where clinicians lose the chart during care after an update.
published 2024-09-23 · verified 2026-10-07
ev-0163regulatorprimary
Tandem Diabetes Care, Inc. Recalls Version 2.7 of the Apple iOS t:connect Mobile App Used in Conjunction with t:slim X2 Insulin Pump with Control-IQ Technology Prompted by a Software Problem Leading to Pump Battery Depletion, U.S. Food and Drug Administration
This cycle intermittently repeats, which leads to excessive Bluetooth communication that may result in pump battery drain.
FDA recall of a phone app that controls an insulin pump: the app crashed and was relaunched by iOS in a loop, the Bluetooth traffic drained the pump battery and the pump could stop delivering insulin. Supports user-harm for SaMD that talks to a body-worn device over Bluetooth.
published 2024-08-28 · verified 2026-10-07
ev-0164regulatorprimary
Insulet Corporation Recalls Omnipod 5 Android App due to a Software Error, U.S. Food and Drug Administration
FDA recall of an insulin delivery app: a bolus entered without a leading zero before the decimal point was not read as a decimal, so the app could deliver far more insulin than intended. Supports user-harm for SaMD whose input fields set a dose.
published 2024-01-08 · verified 2026-10-07
ev-0175court-or-fineprimary
Founder/CEO And Clinical President Of Digital Health Company Arrested For [amount] Adderall Distribution And Health Care Fraud Scheme, U.S. Department of Health and Human Services, Office of Inspector General
Federal enforcement summary: the founder of a digital health company and its clinical president were arrested over an alleged scheme to distribute Adderall over the internet and submit false claims for it. Supports user-harm and regulatory-penalty for telemedicine apps whose visit and refill flows issue controlled-substance prescriptions.
published 2024-06-13 · verified 2026-10-07
ev-0189regulatorprimary
FDA Informs Patients, Providers and Manufacturers About Potential Cybersecurity Vulnerabilities in Certain Medical Devices with Bluetooth Low Energy, U.S. Food and Drug Administration
These cybersecurity vulnerabilities may allow an unauthorized user to wirelessly crash the device, stop it from working, or access device functions normally only available to the authorized user.
FDA states the Bluetooth Low Energy flaws also reach consumer wearables, the radio link every companion app pairs over, so unauthorised pairing and device crashes are risks of this product type.
published 2020-03-03 · verified 2026-10-07
ev-0239regulatorprimary
ICS Advisory ICSA-22-200-01: MiCODUS MV720 GPS tracker (Update A), Cybersecurity and Infrastructure Security Agency (CISA)
Successful exploitation of these vulnerabilities could allow an attacker control over any MV720 GPS tracker, granting access to location, routes, fuel cutoff commands, and the disarming of various features (e.g., alarms).
CISA reports a vehicle GPS tracker whose API accepted a hard-coded master password and unauthenticated commands and, on a web server endpoint, arbitrary device IDs without further verification, exposing location and routes and allowing fuel cut-off, the core functions of fleet telematics.
published 2022-07-19 · verified 2026-10-07
ev-0253regulatorprimary
FTC Says Ring Employees Illegally Surveilled Customers, Failed to Stop Hackers from Taking Control of Users' Cameras, U.S. Federal Trade Commission
Credential stuffing involves the use of credentials, such as usernames and passwords, obtained from a consumer’s breached account to gain access to a consumer’s other accounts.
The FTC charged a home camera company whose accounts were taken over by credential stuffing and brute force, after which attackers watched video and used two-way audio to harass people, including children, in their homes.
published 2023-05-31 · verified 2026-10-07
ev-0263court-or-fineprimary
Fortnite Video Game Maker Epic Games to Pay More Than Half a Billion Dollars over FTC Allegations of Privacy Violations and Unwanted Charges, Federal Trade Commission
Children and teens have been bullied, threatened, harassed, and exposed to dangerous and psychologically traumatizing issues such as suicide while on Fortnite.
Supports user-harm for a kids app with chat or multiplayer: the FTC tied harm to children to voice and text chat being on by default, tied unwanted charges to confusing purchase buttons, including purchases triggered when waking the game from sleep, on a loading screen or while previewing an item, and found personal information collected from children under 13 without notifying parents or getting their consent.
published 2022-12-19 · verified 2026-10-07
ev-0269regulatorprimary
Artificial intelligence: stop to ChatGPT by the Italian SA, Garante per la protezione dei dati personali
A data breach affecting ChatGPT users’ conversations and information on payments by subscribers to the service had been reported on 20 March.
The Italian data protection authority limited a chat assistant over a breach of user conversations, no legal basis for training on personal data, inaccurate personal data in answers and no age check for children.
published 2023-03-31 · verified 2026-10-07
ev-0271regulatorprimary
FTC Launches Inquiry into AI Chatbots Acting as Companions, U.S. Federal Trade Commission
The FTC opened an inquiry into how chatbot operators test and monitor negative effects on children and teens who treat the assistant as a friend or confidant.
published 2025-09-11 · verified 2026-10-07
ev-0285regulatorprimary
FTC Proposes Blanket Prohibition Preventing Facebook from Monetizing Youth Data, Federal Trade Commission
Supports user-harm for a messaging app with child accounts: a group chat and group call path let children reach contacts their parents had not approved.
published 2023-05-03 · verified 2026-10-07
ev-0288regulatorprimary
Rite Aid Banned from Using AI Facial Recognition After FTC Says Retailer Deployed Technology without Reasonable Safeguards, Federal Trade Commission
Test, assess, measure, document, or inquire about the accuracy of its facial recognition technology before deploying it
Regulator order: a face-matching model flagged shoppers as suspects with more false positives in some communities, nobody tracked the false-positive rate, and employees acting on false-positive alerts followed, searched or removed shoppers. Supports wrong-decisions, user-harm and regulatory-penalty for model services.
published 2023-12-19 · verified 2026-10-07
ev-0320court-or-fineprimary
Molly Russell: Prevention of future deaths report (Ref 2022-0315), HM Coroner, North London (Courts and Tribunals Judiciary)
There was no age verification when signing up to the on-line platform.
The coroner's report after a teenager's death lists no separation of child and adult content, no age verification and algorithm-driven content as matters of concern for social platforms. The report found that the platform's algorithms selected and provided images, video and text about self-harm and suicide to the teenager without her requesting them.
published 2022-10-13 · verified 2026-10-07
ev-0321regulatorprimary
Online Safety Act 2023, UK Government (legislation.gov.uk)
The Act places duties of care on user-to-user services, including illegal content and children's risk assessments, overseen by the UK communications regulator.
published undated · verified 2026-10-07