How to test software as a medical device
Direct answer
Shipped untested, software as a medical device reads a dose typed without its leading zero as a whole number or lets a crash loop drain the paired pump until insulin delivery stops, and ships updates without the verification records its clearance requires, in the dosing and release flows. 10 tests are mandatory here, each tied to a risk in the map below. After the release smoke check, start the risk tests with integration tests: each command the app sends must reach the test device once, and the app must show the confirmed value rather than the requested one.
10 mandatory testsfirst: integration tests
What counts as software as a medical device
Software as a medical device here means phone, desktop or embedded software whose output is used for diagnosis or treatment decisions, often paired with a sensor or pump. It counts when the software is regulated as a medical device and reads from or controls hardware. A companion app that shows wellness readings with no clearance is out of scope, and so is a record system that stores clinicians' decisions without producing them.
Adjacent product types not covered here
- wearable companion app: A wearable companion app shows wellness readings, and its regulatory risk is a new alert drifting into device claims rather than an update shipped outside a clearance.
- electronic health record system: A record system routes clinicians' orders and results, while device software produces the dose, reading or alarm the decision rests on.
- telemedicine app: A telemedicine app connects a patient to a clinician who decides, with no paired device whose commands or alarms can fail silently.
Risk map
| Risk | Grade | Note | Evidence | |
|---|---|---|---|---|
| Physical or psychological harm to a user user-harm | harm | primary | A dose typed as ".5" without the leading zero is read as a whole number, or a crash loop drains the paired pump's battery until insulin delivery stops. | |
| Fine or enforcement by a regulator regulatory-penalty | legal | primary | A software update ships to patients without the verification records and cybersecurity documentation the device clearance and life cycle standard require, and the update does not meet the clearance. | |
| Personal or sensitive data exposed data-breach | data | secondary by design analysis | An unauthenticated Bluetooth or cloud command channel lets a nearby or remote attacker read therapy history or send a setting to the paired device. | |
| User actions lost on bad networks or devices lost-actions | trust | primary | A low-glucose or sensor-failed alarm raised by the device never sounds because the phone's focus mode or a new OS version silences the app. |
Mandatory tests (10)
Step 1: release smoke tests
Done when: On the release build, the app pairs with a test device, shows a live reading and sounds a forced test alarm on each supported phone.
Tool class: mobile automation. Who usually runs it: QA engineer.
Step 2: integration tests
Done when: Each command the app sends reaches the test device once, the device confirms it, and the app shows the confirmed value rather than the requested one.
Tool class: API client. Who usually runs it: developer.
Step 3: regression suite
Why:user-harmregulatory-penalty
Done when: The regression set traced to each software requirement passes on the release candidate and the results are filed with the release record.
Tool class: test management. Who usually runs it: QA engineer.
Step 4: device and browser compatibility
Done when: On each supported phone model and OS version, including the latest OS beta, critical alarms sound with focus mode, low power mode and notification summaries turned on.
Tool class: device cloud. Who usually runs it: QA engineer.
Step 5: resilience and failure testing
Done when: When the sensor or transmitter is made to fail on the test rig, the app raises the failure alarm, and during a forced app crash loop the paired device's battery level drops no faster than with the app closed.
Tool class: chaos tool. Who usually runs it: developer.
Step 6: unit tests
Why:user-harm
Done when: Dose and threshold inputs written as ".5", "0.5", "5" and with a comma decimal each produce the intended value or a rejection, never a different dose.
Tool class: unit framework. Who usually runs it: developer.
Step 7: end-to-end functional tests
Why:user-harm
Done when: A test user goes from pairing through a full therapy day on a test rig, and every reading, alarm and delivered dose in the app matches the device log.
Tool class: mobile automation. Who usually runs it: QA engineer.
Step 8: compliance checks
Done when: A reviewer confirms that every requirement in the release has a linked verification result, the risk file lists each change, and the cybersecurity documentation matches the shipped build.
Tool class: test management. Who usually runs it: product manager.
Step 9: security scanning
Why:data-breach
Done when: The scan report and software bill of materials for the app, firmware interface and cloud API list no open high-severity finding on the release date.
Tool class: dependency scanner. Who usually runs it: security specialist.
Step 10: penetration testing
Why:data-breach
Done when: A tester within radio range or on the network cannot read therapy history or send a setting to the test device without pairing credentials.
Tool class: proxy. Who usually runs it: security specialist.
Recommended and optional tests
exploratory testingrecommended
Becomes mandatory when a field complaint describes a dose or alarm path the requirements did not cover.
usability testingrecommended
Becomes mandatory when the regulator's submission requires human factors validation for this device class.
offline and poor-network testingrecommended
Becomes mandatory when readings or alarms depend on a cloud service rather than the phone alone.
data integrity and migration testingrecommended
Becomes mandatory when clinicians make treatment decisions from the uploaded history.
accessibility testingoptional
May help users with low vision read values and alarms, which also supports human factors work.
performance and load testingoptional
May help measure phone battery and radio use over a full day of background readings.
Not needed and why
Not needed while the device is cleared in one market with one language and unit system. Becomes mandatory when a market with different glucose units or decimal format is added.
analytics and event tracking tests
Not needed while the app sends no product analytics. Becomes recommended when an analytics SDK is added and must be checked for health data leaving the device.
Order of work
First release
- release smoke tests: Pair the release build with a test device on each supported phone, show a live reading and sound a forced test alarm.
- integration tests: Send each command from the app to the test device and confirm it arrives once, the device confirms it, and the app shows the confirmed value.
- regression suite: Run the regression set traced to each software requirement on the release candidate and file the results with the release record.
- device and browser compatibility: Sound critical alarms on each supported phone model and OS version, including the latest OS beta, with focus mode, low power mode and notification summaries turned on.
- resilience and failure testing: Make the sensor or transmitter fail on the test rig, confirm the failure alarm sounds, and force an app crash loop to confirm the paired device's battery level drops no faster than with the app closed.
- unit tests: Feed dose and threshold inputs with and without a leading zero and with a comma decimal, and confirm each gives the intended value or a rejection.
- end-to-end functional tests: Run a full therapy day on the test rig from pairing onward and match every reading, alarm and delivered dose in the app to the device log.
- compliance checks: Check that every requirement in the release has a linked verification result, the risk file lists each change, and the cybersecurity documentation matches the shipped build.
- security scanning: Scan the app, firmware interface and cloud API, produce the software bill of materials, and clear every open high-severity finding before the release date.
- penetration testing: Try to read therapy history or send a setting to the test device from within radio range and from the network without pairing credentials.
Ongoing cadence
- release smoke tests: On every release build.
- device and browser compatibility: When a new phone OS version or OS beta is published.
- regression suite: On every release candidate under the device clearance.
- compliance checks: Before each software update ships to patients.
- unit tests: After each change to dose, threshold or setting input handling.
- integration tests: After each firmware or Bluetooth protocol change.
- security scanning: When a component in the software bill of materials publishes a vulnerability.
Tools
Listed alphabetically inside each test type, filtered to the tests above and to the platforms of software as a medical device (ios, android, desktop, embedded). Not recommendations.
release smoke tests
- Firebase Test LabRuns app tests on hosted devices. iosandroidfreemium
- TestRailTest case and run management. webiosandroidapicommercial
- XrayTest management inside Jira. webiosandroidapicommercial
integration tests
- Charles ProxyHTTP proxy for inspecting and throttling traffic. iosandroidwebcommercial
- JUnitTest framework for the JVM. apiandroidopen-source
- XCTestApple's framework for unit and UI tests. iosopen-source
regression suite
device and browser compatibility
- AppiumCross-platform mobile UI automation over WebDriver. iosandroidopen-source
- ApplitoolsVisual comparison of screens across builds. webiosandroidcommercial
- BrowserStackCloud of real devices and browsers. webiosandroidcommercial
- Firebase Test LabRuns app tests on hosted devices. iosandroidfreemium
- LambdaTestCloud browser and device grid. webiosandroidcommercial
- Sauce LabsCloud test infrastructure for web and mobile. webiosandroidcommercial
unit tests
end-to-end functional tests
- AppiumCross-platform mobile UI automation over WebDriver. iosandroidopen-source
- ApplitoolsVisual comparison of screens across builds. webiosandroidcommercial
- BrowserStackCloud of real devices and browsers. webiosandroidcommercial
- DetoxGray-box end-to-end testing for React Native apps. iosandroidopen-source
- EspressoAndroid UI testing framework from Google. androidopen-source
- LambdaTestCloud browser and device grid. webiosandroidcommercial
- MaestroMobile UI flows written in YAML. iosandroidfreemium
- Sauce LabsCloud test infrastructure for web and mobile. webiosandroidcommercial
- XCTestApple's framework for unit and UI tests. iosopen-source
security scanning
offline and poor-network testing
- Charles ProxyHTTP proxy for inspecting and throttling traffic. iosandroidwebcommercial
- mitmproxyInteractive HTTPS proxy. iosandroidwebapiopen-source
- Network Link ConditionerApple tool for simulating poor network conditions. iosopen-source
What to send to the person who will test
Send the checklist and the brief below. The printable checklist for software as a medical device has the same 10 items with checkboxes.
Download the checklistMarkdownCSVBrief (Markdown)
Acceptance criteria wording
A release of the device software is accepted when every command reaches the test device once and the app shows the confirmed value, and critical alarms sound on each supported phone with focus mode and low power mode on. Every requirement has a linked verification result filed with the release record, and the cybersecurity documentation matches the shipped build.
Brief
# Test brief: software as a medical device Product type: software as a medical device. Software whose output is used for diagnosis or treatment decisions. Risk map and evidence: https://howtotestaproduct.com/how-to-test/samd/ Printable checklist: https://howtotestaproduct.com/checklists/samd/ Run these 10 mandatory tests in this order. A test is done only when its done condition is true on the build you will release. 1. release smoke tests (risks: lost-actions, user-harm). Tool class: mobile automation. Usually run by: QA engineer. Done when: On the release build, the app pairs with a test device, shows a live reading and sounds a forced test alarm on each supported phone. 2. integration tests (risks: user-harm, lost-actions). Tool class: API client. Usually run by: developer. Done when: Each command the app sends reaches the test device once, the device confirms it, and the app shows the confirmed value rather than the requested one. 3. regression suite (risks: user-harm, regulatory-penalty). Tool class: test management. Usually run by: QA engineer. Done when: The regression set traced to each software requirement passes on the release candidate and the results are filed with the release record. 4. device and browser compatibility (risks: lost-actions, user-harm). Tool class: device cloud. Usually run by: QA engineer. Done when: On each supported phone model and OS version, including the latest OS beta, critical alarms sound with focus mode, low power mode and notification summaries turned on. 5. resilience and failure testing (risks: user-harm, lost-actions). Tool class: chaos tool. Usually run by: developer. Done when: When the sensor or transmitter is made to fail on the test rig, the app raises the failure alarm, and during a forced app crash loop the paired device's battery level drops no faster than with the app closed. 6. unit tests (risks: user-harm). Tool class: unit framework. Usually run by: developer. Done when: Dose and threshold inputs written as ".5", "0.5", "5" and with a comma decimal each produce the intended value or a rejection, never a different dose. 7. end-to-end functional tests (risks: user-harm). Tool class: mobile automation. Usually run by: QA engineer. Done when: A test user goes from pairing through a full therapy day on a test rig, and every reading, alarm and delivered dose in the app matches the device log. 8. compliance checks (risks: regulatory-penalty). Tool class: test management. Usually run by: product manager. Done when: A reviewer confirms that every requirement in the release has a linked verification result, the risk file lists each change, and the cybersecurity documentation matches the shipped build. 9. security scanning (risks: data-breach). Tool class: dependency scanner. Usually run by: security specialist. Done when: The scan report and software bill of materials for the app, firmware interface and cloud API list no open high-severity finding on the release date. 10. penetration testing (risks: data-breach). Tool class: proxy. Usually run by: security specialist. Done when: A tester within radio range or on the network cannot read therapy history or send a setting to the test device without pairing credentials. Acceptance criteria wording: A release of the device software is accepted when every command reaches the test device once and the app shows the confirmed value, and critical alarms sound on each supported phone with focus mode and low power mode on. Every requirement has a linked verification result filed with the release record, and the cybersecurity documentation matches the shipped build. Report back for each test: pass or fail against its done condition, the build or version tested, and the proof (screenshot, log line or report file). Licence: CC BY 4.0.
For founders: what to forward and what to ask back. For product managers: turning this page into tickets.
Questions about testing software as a medical device
What testing does software as a medical device need?
10 mandatory tests cover the 4 risks in the map: wrong doses and drained pumps, silenced alarms, open Bluetooth or cloud command channels, and updates shipped without verification records. They run as release smoke, integration, regression, compatibility, resilience, unit, end-to-end, compliance, security scanning and penetration tests. Usability testing becomes mandatory when the regulator's submission requires human factors validation for the device class.
What goes on a testing checklist for software as a medical device before launch?
The checklist is the 10 mandatory tests with their done conditions, from a paired release smoke run to the penetration test against the test device. It includes alarms with focus mode and low power mode on, dose inputs with and without a leading zero, a failed sensor on the test rig, and a reviewer's check that every requirement has a verification result.
Which tests in a vendor test plan for software as a medical device are not needed?
Localization and analytics tests are marked not needed while the device is cleared in one market with one language and unit system, and the app sends no product analytics. Localization becomes mandatory when a market with different glucose units or decimal format is added, and analytics testing becomes recommended when an analytics SDK is added.
How is the audit trail tested in software as a medical device?
The regression suite files results traced to each software requirement with the release record. The compliance check confirms every requirement has a linked verification result and the risk file lists each change. The end-to-end test matches every reading, alarm and delivered dose in the app to the device log.
What should I send to the person who will test software as a medical device?
Send the 10 mandatory tests with their done conditions, a test rig with the paired device and sensor, and the supported phone list including the latest OS beta. Include the software requirements list the regression set traces to and the risk file the compliance check reads.
Evidence
ev-0163regulatorprimary
Tandem Diabetes Care, Inc. Recalls Version 2.7 of the Apple iOS t:connect Mobile App Used in Conjunction with t:slim X2 Insulin Pump with Control-IQ Technology Prompted by a Software Problem Leading to Pump Battery Depletion, U.S. Food and Drug Administration
This cycle intermittently repeats, which leads to excessive Bluetooth communication that may result in pump battery drain.
FDA recall of a phone app that controls an insulin pump: the app crashed and was relaunched by iOS in a loop, the Bluetooth traffic drained the pump battery and the pump could stop delivering insulin. Supports user-harm for SaMD that talks to a body-worn device over Bluetooth.
published 2024-08-28 · verified 2026-10-07
ev-0164regulatorprimary
Insulet Corporation Recalls Omnipod 5 Android App due to a Software Error, U.S. Food and Drug Administration
FDA recall of an insulin delivery app: a bolus entered without a leading zero before the decimal point was not read as a decimal, so the app could deliver far more insulin than intended. Supports user-harm for SaMD whose input fields set a dose.
published 2024-01-08 · verified 2026-10-07
ev-0031regulatorprimary
Software as a Medical Device (SaMD), U.S. Food and Drug Administration
The FDA regulates software intended for medical purposes that is not part of a hardware device as a medical device.
published undated · verified 2026-10-07
ev-0167regulatorprimary
Cybersecurity in Medical Devices: Quality Management System Considerations and Content of Premarket Submissions, U.S. Food and Drug Administration
FDA guidance on cybersecurity design, labeling and premarket documentation for devices with cybersecurity risk, including section 524B cyber devices. Supports data-breach (by design) and regulatory-penalty for SaMD that connects to phones, clouds or networks.
published undated · verified 2026-10-07
ev-0168standardprimary
IEC 62304:2006 - Medical device software - Software life cycle processes, International Organization for Standardization / International Electrotechnical Commission
Life cycle standard for medical device software, covering development, verification and maintenance processes. Supports regulatory-penalty for SaMD that must show documented verification to a regulator.
published undated · verified 2026-10-07
ev-0013standardprimary
OWASP Mobile Application Security Verification Standard (MASVS), OWASP Foundation
MASVS defines security requirements for mobile apps, including secure storage, network communication and authentication.
published undated · verified 2026-10-07
ev-0165regulatorprimary
Continuous Glucose Monitor Apps Correction: Dexcom, Inc. Issues Correction for G7 Apps and ONE+ Apps due to a Software Design Error That Does Not Alert Users of Unexpected Sensor Failure, U.S. Food and Drug Administration
FDA correction notice: when the sensor transmitter failed, the glucose monitor app ended the session and stopped reporting values without the Sensor Failed alert, so users missed readings and alarms. Supports user-harm and lost-actions for SaMD that relays sensor data and alarms through a phone.
published undated · verified 2026-10-07
ev-0166regulatorprimary
FDA Alerts Patients of Potential to Miss Critical Safety Alerts Due to Phone Settings When Using Smartphone-Compatible Diabetes Devices, U.S. Food and Drug Administration
FDA safety communication: phone settings and operating system updates not supported by the device app can stop critical alerts from reaching the user. Supports lost-actions and user-harm for SaMD that depends on a consumer phone to deliver alarms.
published 2025-02-05 · verified 2026-10-07
Disclaimer
This page lists tests and risks for a product category, not for your product. Risks are attached only with the evidence shown in the Evidence section. Tools are listed alphabetically and are not recommendations. Nothing here is legal, regulatory or security advice; where a regulator or a store sets a requirement, their page is linked and takes precedence.