Testing data pipelines at three layers

Context

Data pipelines are difficult to test for a specific reason: the input changes constantly. Unlike a web application where requests follow predictable formats, a pipeline processing daily data must handle schema drift, missing fields, type changes, and values that fall outside expected ranges.

Most teams catch these issues at the dashboard level — the chart looks wrong, so someone investigates. By that point, bad data has already propagated through the system.

The three-layer approach

I test data pipelines at three distinct layers, each catching a different class of failure.

Source layer — validate at the boundary

The most cost-effective test is rejecting malformed data before it enters the pipeline. Every source has a schema contract, and any record that violates it is quarantined:

def test_source_schema(df):
    expected_columns = {'date', 'revenue', 'customer_id', 'region'}
    assert expected_columns.issubset(set(df.columns))
    assert df['revenue'].dtype in ('float64', 'int64')

In production, these checks run as part of the ingestion step. Failed records are written to a quarantine table with the rejection reason, enabling later analysis of source quality trends.

Transform layer — unit test each function

Every transformation function should be independently testable with known inputs and expected outputs:

def test_normalize_region():
    assert normalize_region('NY') == 'New York'
    assert normalize_region('ny') == 'New York'
    assert normalize_region('') is None

The presence of a test for edge cases (empty string, mixed case, abbreviations) forces the implementation to handle them. Functions without such handling are a common source of silent data corruption.

Output layer — assert business rules

The final dataset must satisfy business-level constraints regardless of what the source data looked like:

• Row counts within expected ranges (not zero, not 10x the weekly average)
• No NULLs in key columns
• Referential integrity maintained across joined tables
• Metric values within physically possible ranges

These output-layer tests are the safety net that catches integration issues — cases where individual transformations work correctly but produce incorrect results when combined.

What this achieves

In the DashLogistics project, this three-layer approach caught a schema change from the AAA fuel API within minutes of it being deployed. The source-layer test failed, the pipeline quarantined the affected records, and the operations team was notified before any dashboard displayed incorrect data.

The same approach enabled confident refactoring of the transformation pipeline from Pandas to Polars — the transform-layer tests validated that the output was identical after the migration.

Priority order

If resources are limited, invest in source-layer validation first. Most pipeline failures originate from unexpected input. Transform tests are the second priority. Output-layer tests are valuable but tend to be more brittle — they fail when legitimate business changes affect expected ranges, requiring regular maintenance.