How to test a data pipeline

Direct answer

Shipped untested, a data pipeline feeds regulatory reports with unreconciled source data until the supervisor fines the firm for weak data quality controls, and stops loading in every job when the managed stream it reads from stalls. 9 tests are mandatory here, each tied to a risk in the map below. After the release smoke check, start the risk tests with data integrity and migration testing: reconciling row counts and control totals from source to target on each run stops dropped rows from reaching reports and dashboards.

9 mandatory testsfirst: data integrity and migration testing

What counts as a data pipeline

A data pipeline here is a set of ETL or ELT jobs that move and transform data on a schedule through an API, from source systems through staging into target tables that reports and dashboards read. It counts when extracts contain personal data, when output feeds regulated reporting, or when the pipeline depends on a managed stream or warehouse. A dashboard or BI tool that only reads finished tables is out of scope, and so is a one-off manual export.

Adjacent product types not covered here

  • machine learning model service: A machine learning model service returns a score that acts on a person, so false matches and missing reasons lead instead of dropped rows and reconciliation.
  • SDK or client library: An SDK ships code into other developers' apps and crashes their launch, while a data pipeline fails silently and the damage shows in downstream totals.
  • accounting and invoicing SaaS: An accounting SaaS imports bank feeds into a small business's books, where a re-imported day overstates a tax return, while a pipeline feeds many reports from many sources at once.

Risk map

5 risks attached to data pipeline with evidence. Sorted by severity.
RiskGradeNoteEvidence
Fine or enforcement by a regulator
regulatory-penalty
legalprimaryRegulatory reports built from pipeline tables carry unreconciled source data, and the supervisor fines the firm for weak data quality controls.
Personal or sensitive data exposed
data-breach
datasecondary
by design analysis
Raw extracts with personal fields land in a staging bucket or temp table that the masking step never reaches and analysts can query.
Records silently wrong
data-corruption
dataprimaryA load step silently drops rows past a file size limit, or the nightly dump job fails and its alert email bounces, and downstream tables look complete.
Third-party dependency fails
dependency-failure
outagepractitioner
single incident, no second source
The managed stream the pipeline reads from stalls after a provider capacity change, and every job consuming it stops loading until the provider recovers.
Users or operators act on wrong numbers
wrong-decisions
trustprimaryA dashboard fed by the pipeline publishes a daily count missing every row the load step dropped, or reports rest on data whose known quality problems were never remediated.

Mandatory tests (9)

  1. Step 1: release smoke tests

    Why:data-corruptionwrong-decisions

    Done when: After each deploy, a run on a fixed sample input produces the expected row count and totals in every target table.

    Tool class: data validation. Who usually runs it: developer.

  2. Step 2: data integrity and migration testing

    Why:data-corruptionwrong-decisionsregulatory-penalty

    Done when: For each run, row counts and control totals reconcile between source extract, staging and target, and a mismatch stops publication.

    Tool class: data validation. Who usually runs it: QA engineer.

  3. Step 3: compliance checks

    Why:regulatory-penaltydata-breach

    Done when: Every regulated report traces each figure back to its source tables, and personal fields are masked or dropped before the shared layer.

    Tool class: test management. Who usually runs it: product manager.

  4. Step 4: integration tests

    Why:data-corruptiondependency-failure

    Done when: An end-to-end run against staging copies of each source and target loads a file above the largest size seen in production with no rows lost, and when the source stream stops sending mid-run, the run is reported as stalled instead of complete.

    Tool class: data validation. Who usually runs it: developer.

  5. Step 5: regression suite

    Why:data-corruptionwrong-decisions

    Done when: Reconciliation and edge-file cases rerun on every change, and any change in sample output totals blocks the merge.

    Tool class: test management. Who usually runs it: QA engineer.

  6. Step 6: resilience and failure testing

    Why:dependency-failuredata-corruption

    Done when: With the source stream or warehouse paused mid-run, the job retries, resumes from its last checkpoint and loads each record once.

    Tool class: chaos tool. Who usually runs it: developer.

  7. Step 7: unit tests

    Why:data-corruption

    Done when: Each transform returns the documented output for empty files, oversized files, null keys, duplicate keys and timezone boundaries.

    Tool class: unit framework. Who usually runs it: developer.

  8. Step 8: security scanning

    Why:data-breach

    Done when: The scan of buckets, staging schemas and service accounts shows no public access and no personal fields outside approved tables.

    Tool class: DAST scanner. Who usually runs it: security specialist.

  9. Step 9: analytics and event tracking tests

    Why:wrong-decisions

    Done when: Each job emits start, end, rows read and rows written, and a run with no end event or a write count of zero raises an alert.

    Tool class: data validation. Who usually runs it: developer.

Recommended and optional tests

  • contract testsrecommended

    Becomes mandatory when a source team can change its schema without notice.

  • performance and load testingrecommended

    Becomes mandatory when runs overlap or outage-under-load evidence is attached.

  • penetration testingrecommended

    Becomes mandatory when analysts outside the data team get warehouse access.

  • exploratory testingoptional

    Optional: finds scheduling edge cases scripted cases miss.

Not needed and why

  • end-to-end functional tests

    Not needed: there is no user interface; the integration run covers the whole flow. Becomes recommended when the pipeline ships a self-service upload screen.

  • usability testing

    Not needed: no attached risk comes from users misreading screens. Becomes recommended when business users configure jobs themselves.

  • accessibility testing

    Not needed: there is no user interface. Becomes recommended when a pipeline admin screen ships to public sector staff.

  • device and browser compatibility

    Not needed: the pipeline runs on one managed platform. Becomes recommended when it must run on a second cloud or warehouse.

Order of work

First release

  1. release smoke tests: After the deploy, run the pipeline on the fixed sample input and compare the row count and totals in every target table with the expected values.
  2. data integrity and migration testing: Reconcile row counts and control totals between source extract, staging and target for the run, and stop publication on any mismatch.
  3. compliance checks: Trace each figure in every regulated report back to its source tables, and confirm personal fields are masked or dropped before the shared layer.
  4. integration tests: Run end to end against staging copies of each source and target with a file above the largest size seen in production and confirm no rows are lost, then stop the source stream mid-run and confirm the run is reported as stalled instead of complete.
  5. regression suite: Rerun the reconciliation and edge-file cases on the change and block the merge on any change in sample output totals.
  6. resilience and failure testing: Pause the source stream or warehouse mid-run and confirm the job retries, resumes from its last checkpoint and loads each record once.
  7. unit tests: Feed each transform empty files, oversized files, null keys, duplicate keys and timezone boundaries and compare with the documented output.
  8. security scanning: Scan buckets, staging schemas and service accounts for public access and for personal fields outside approved tables.
  9. analytics and event tracking tests: Check each job emits start, end, rows read and rows written, and that a run with no end event or a zero write count raises an alert.

Ongoing cadence

Tools

Listed alphabetically inside each test type, filtered to the tests above and to the platforms of data pipeline (api). Not recommendations.

28 tools shown

release smoke tests

  • TestRailTest case and run management. webiosandroidapicommercial
  • XrayTest management inside Jira. webiosandroidapicommercial

data integrity and migration testing

  • dbt testsTests on warehouse models inside dbt. apiopen-source
  • Great ExpectationsData validation with declarative expectations. apiopen-source

integration tests

  • JestJavaScript test runner with mocking and snapshots. webapiopen-source
  • JUnitTest framework for the JVM. apiandroidopen-source
  • PostmanAPI client with collections and scripted assertions. apifreemium
  • pytestTest framework for Python with fixtures and plugins. apiopen-source
  • VitestTest runner for Vite-based JavaScript and TypeScript projects. webapiopen-source

regression suite

  • DeepEvalUnit-style evaluation for LLM applications. apiopen-source
  • PromptfooEvaluation and regression tests for LLM prompts and outputs. apiwebopen-source
  • TestRailTest case and run management. webiosandroidapicommercial
  • XrayTest management inside Jira. webiosandroidapicommercial

resilience and failure testing

  • Chaos MeshFault injection for Kubernetes. apiopen-source
  • GremlinManaged fault injection. apicommercial
  • ToxiproxyTCP proxy for simulating network failures. apiopen-source

unit tests

  • JestJavaScript test runner with mocking and snapshots. webapiopen-source
  • JUnitTest framework for the JVM. apiandroidopen-source
  • pytestTest framework for Python with fixtures and plugins. apiopen-source
  • VitestTest runner for Vite-based JavaScript and TypeScript projects. webapiopen-source

security scanning

  • Burp SuiteProxy and scanner for web application security testing. webapifreemium
  • GarakProbing LLM applications for prompt injection and leakage. apiopen-source
  • mitmproxyInteractive HTTPS proxy. iosandroidwebapiopen-source
  • OWASP ZAPDynamic application security scanner. webapiopen-source
  • SchemathesisProperty-based testing from OpenAPI and GraphQL schemas. apiopen-source
  • SemgrepStatic analysis with custom rules. apiwebiosandroidfreemium
  • SnykDependency and code vulnerability scanning. apiwebiosandroidfreemium
  • TrivyScanner for containers, dependencies and configuration. apiopen-source

contract tests

  • PactConsumer-driven contract testing with a broker. apiopen-source
  • PostmanAPI client with collections and scripted assertions. apifreemium
  • SchemathesisProperty-based testing from OpenAPI and GraphQL schemas. apiopen-source

performance and load testing

  • ArtilleryLoad testing with YAML scenarios. apiwebfreemium
  • GatlingLoad testing with Scala, Java or Kotlin scenarios. apiwebfreemium
  • JMeterLoad testing tool from the Apache Foundation. apiwebopen-source
  • k6Load testing scripted in JavaScript. apiwebopen-source
  • LocustLoad testing scripted in Python. apiwebopen-source

penetration testing

  • Burp SuiteProxy and scanner for web application security testing. webapifreemium

What to send to the person who will test

Send the checklist and the brief below. The printable checklist for data pipeline has the same 9 items with checkboxes.

Download the checklistMarkdownCSVBrief (Markdown)

Acceptance criteria wording

A release of the data pipeline is accepted when a run on the fixed sample input produces the expected row count and totals in every target table, and row counts and control totals reconcile between source extract, staging and target with publication stopped on a mismatch. Every figure in a regulated report traces to its source tables, personal fields are masked before the shared layer, and a job with no end event or a zero write count raises an alert.

Brief

# Test brief: data pipeline

Product type: data pipeline. ETL or ELT jobs that move and transform data on a schedule.
Risk map and evidence: https://howtotestaproduct.com/how-to-test/data-pipeline/
Printable checklist: https://howtotestaproduct.com/checklists/data-pipeline/

Run these 9 mandatory tests in this order. A test is done only when its done condition is true on the build you will release.

1. release smoke tests (risks: data-corruption, wrong-decisions). Tool class: data validation. Usually run by: developer.
   Done when: After each deploy, a run on a fixed sample input produces the expected row count and totals in every target table.
2. data integrity and migration testing (risks: data-corruption, wrong-decisions, regulatory-penalty). Tool class: data validation. Usually run by: QA engineer.
   Done when: For each run, row counts and control totals reconcile between source extract, staging and target, and a mismatch stops publication.
3. compliance checks (risks: regulatory-penalty, data-breach). Tool class: test management. Usually run by: product manager.
   Done when: Every regulated report traces each figure back to its source tables, and personal fields are masked or dropped before the shared layer.
4. integration tests (risks: data-corruption, dependency-failure). Tool class: data validation. Usually run by: developer.
   Done when: An end-to-end run against staging copies of each source and target loads a file above the largest size seen in production with no rows lost, and when the source stream stops sending mid-run, the run is reported as stalled instead of complete.
5. regression suite (risks: data-corruption, wrong-decisions). Tool class: test management. Usually run by: QA engineer.
   Done when: Reconciliation and edge-file cases rerun on every change, and any change in sample output totals blocks the merge.
6. resilience and failure testing (risks: dependency-failure, data-corruption). Tool class: chaos tool. Usually run by: developer.
   Done when: With the source stream or warehouse paused mid-run, the job retries, resumes from its last checkpoint and loads each record once.
7. unit tests (risks: data-corruption). Tool class: unit framework. Usually run by: developer.
   Done when: Each transform returns the documented output for empty files, oversized files, null keys, duplicate keys and timezone boundaries.
8. security scanning (risks: data-breach). Tool class: DAST scanner. Usually run by: security specialist.
   Done when: The scan of buckets, staging schemas and service accounts shows no public access and no personal fields outside approved tables.
9. analytics and event tracking tests (risks: wrong-decisions). Tool class: data validation. Usually run by: developer.
   Done when: Each job emits start, end, rows read and rows written, and a run with no end event or a write count of zero raises an alert.

Acceptance criteria wording:
A release of the data pipeline is accepted when a run on the fixed sample input produces the expected row count and totals in every target table, and row counts and control totals reconcile between source extract, staging and target with publication stopped on a mismatch. Every figure in a regulated report traces to its source tables, personal fields are masked before the shared layer, and a job with no end event or a zero write count raises an alert.

Report back for each test: pass or fail against its done condition, the build or version tested, and the proof (screenshot, log line or report file).

Licence: CC BY 4.0.

For founders: what to forward and what to ask back. For product managers: turning this page into tickets.

Questions about testing a data pipeline

What testing does a data pipeline need?

9 tests are mandatory, starting with a fixed-sample smoke run after each deploy and source-to-target reconciliation on every run. Each is tied to one of the 5 risks in the map: rows silently dropped past a size limit, dashboards publishing low counts, fines for unreconciled regulatory reports, a stalled managed stream, and personal fields left unmasked in staging. Contract, performance and penetration tests are recommended, each with the event that makes it mandatory.

What does GDPR mean for testing a data pipeline?

The map cites the General Data Protection Regulation for the by design analysis data breach risk, where raw extracts with personal fields land in a staging bucket or temp table the masking step never reaches. Compliance checks confirm personal fields are masked or dropped before the shared layer, and security scanning looks for public access and personal fields outside approved tables.

What goes on an API testing checklist for a data pipeline?

Integration runs load staging copies of each source and target with a file above the largest size seen in production. Resilience runs pause the source stream or warehouse mid-run and check the job resumes from its last checkpoint and loads each record once. Contract checks on each source and target schema become mandatory when a source team can change its schema without notice.

Which tests in a vendor test plan for a data pipeline are not needed?

End-to-end UI and accessibility testing are not needed because there is no user interface and the integration run covers the whole flow, until a self-service upload screen or an admin screen for public sector staff ships. Compatibility testing is not needed while the pipeline runs on one managed platform, until it must run on a second cloud or warehouse.

When is test automation worth it for a data pipeline?

Automation pays from the first scheduled run, because reconciliation of row counts and control totals must happen on every run and the job alerts must fire without anyone watching. The traceability review of regulated reports is signed off by a product manager.

Evidence

  1. ev-0215regulatorprimary

    Federal Reserve Board fines Citigroup for violating the Board’s 2020 enforcement action, Board of Governors of the Federal Reserve System

    Citigroup has made insufficient progress remediating its problems with data quality management and failed to implement compensating controls to manage its ongoing risk.

    Regulator fine for unremediated data quality management problems. Supports regulatory-penalty for data pipelines that feed regulated reporting.

    published 2024-07-10 · verified 2026-10-07

  2. ev-0025regulatorprimary

    Regulation (EU) 2016/679 (General Data Protection Regulation), EUR-Lex, European Union

    The GDPR sets lawful basis, consent, data subject rights, security of processing (Article 32) and breach notification duties for anyone processing personal data of people in the EU.

    published undated · verified 2026-10-07

  3. ev-0193post-mortemprimary

    PHE statement on delayed reporting of COVID-19 cases, Public Health England (GOV.UK)

    Every one of these cases received their COVID-19 test result as normal and all those who tested positive were advised to self-isolate.

    Official government statement: result files exceeded a size limit in the data load process, so positive cases were left out of the daily reported figures. Supports data-corruption and wrong-decisions for analytics dashboards and BI tools fed by file loads, since both report from the loaded data.

    published 2020-10-04 · verified 2026-10-07

  4. ev-0205post-mortemprimary

    Postmortem of database outage of January 31, GitLab

    This means we were never aware of the backups failing, until it was too late.

    Official post-mortem: a scheduled dump command exited with a version error, its failure emails were rejected, and a manual delete command ran against the wrong host. Supports data-corruption for command-line tools and for scheduled data jobs whose failures go unseen.

    published 2017-02-10 · verified 2026-10-07

  5. ev-0214post-mortemprimary

    Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region, Amazon Web Services

    Amazon Cognito uses Kinesis Data Streams to collect and analyze API access patterns.

    Official post-mortem: a capacity addition pushed the streaming service front-end past an operating system thread limit, and services that consume the stream were impaired. Supports dependency-failure for data pipelines built on a managed stream.

    published 2020-11-25 · verified 2026-10-07

Disclaimer

This page lists tests and risks for a product category, not for your product. Risks are attached only with the evidence shown in the Evidence section. Tools are listed alphabetically and are not recommendations. Nothing here is legal, regulatory or security advice; where a regulator or a store sets a requirement, their page is linked and takes precedence.