One of the most valuable lessons I've learned while building enterprise data platforms wasn't about pipelines, ETL, or cloud architecture. It was about alert fatigue.
We had invested significant effort into building a comprehensive monitoring and alerting framework. We monitored pipeline failures, data freshness, schema changes, data quality validations, and infrastructure health. Technically, it worked. But we overlooked one important question: who actually needs to know?
Some alerts were being sent directly to business stakeholders when they only required engineering action. Other alerts highlighted temporary data quality mismatches caused by upstream dependencies outside our control. Every notification created uncertainty. Questions started appearing:
- Can we trust today's dashboard?
- Is the data broken again?
- Should we delay the report?
Ironically, our monitoring system — designed to increase confidence — was slowly reducing it.
The shift that changed everything
Instead of asking "what should we monitor?" we started asking "who should be notified, when, and what action can they actually take?" That simple shift changed everything.
Today, I think of alerting in three layers:
Actionable & immediate
Technical issues requiring immediate investigation by the team that can actually fix them.
Awareness & coordination
Issues affecting dependent teams that require awareness or coordination, not a fire drill.
Material impact only
Only incidents that materially impact business decisions, reporting, or customer experience.
Observability is measured in trust, not notification count
Because data observability isn't measured by how many notifications you send. It's measured by how much trust you preserve.
Have you experienced alert fatigue in your data platform? How do you decide who should receive which alerts?