Data problems are quiet. A job finishes, the dashboard loads, and nobody notices that a few thousand records were dropped or truncated along the way.
Early in my career I built an automated test suite that compared data across Oracle and PostgreSQL hosts to catch exactly these problems. The lessons still apply to modern pipelines.
What to test
- Row counts at every stage
- Field-level comparisons between source and target
- Differences in SQL dialects and data types
- Checksums on files in transit
- Reconciliation reports that people actually read