Why Fixed Alert Thresholds Don’t Work for Error Monitoring
The simplest way to create an alert seems straightforward: “notify me when the number of errors exceeds 100 per hour.” The problem starts when the application changes. Traffic grows, new features are introduced, the number of users changes, and suddenly a threshold set several months ago no longer means much.
Error monitoring should primarily answer one question: is something unusual happening in this particular application? The number of events alone does not always provide a good answer.
The Same Threshold Doesn’t Fit Every Application
Imagine an online store processing thousands of orders every day. Even when everything is working correctly, individual errors may occur due to interrupted sessions, failed payment attempts, timeouts from third-party services, or automated bot traffic.
If such a store generates dozens or even hundreds of errors within an hour, it does not necessarily indicate an outage. Setting an alert threshold at 100 errors could therefore result in constant notifications about situations that are actually part of normal traffic.
The opposite scenario is even more problematic. An administration panel used by only a few people may go most of the day without generating a single error. A fixed threshold of 100 events might therefore never be reached, even if the application starts returning a 500 error on every request.
Alerts Based on Deviations from Normal Are More Effective
When designing monitoring alerts, it is useful to separate two factors: the minimum number of events and the deviation from the typical error rate.
The minimum threshold helps prevent false alarms. If a quiet application normally generates one or two errors per day, a single additional event should not wake up the entire team.
The second factor helps detect sudden changes. If an application typically generates 20 errors during a comparable period and suddenly produces several times that number, this is much more valuable information than the raw number without any context.
The Baseline Must Be Independent of the Current Incident
This is an important detail when designing such a mechanism. The baseline should not be calculated solely from the last few minutes or hours.
If an incident develops gradually and the number of errors increases over several hours, current data will begin to inflate the baseline. The system may then start treating the increasing error rate as the new normal at exactly the moment when you need an alert the most.
That is why it is better to establish the baseline using longer-term historical data and aggregated statistics. This prevents a single incident from immediately changing the system’s understanding of normal application behavior.
Why Alert Suppression Is Just as Important
Even a good anomaly detection mechanism can become frustrating if every subsequent monitoring window generates another notification.
An incident may last an hour, several hours, or even all night. If an alert is sent every few minutes, after dozens of messages it becomes difficult to distinguish when the problem actually started from all the repeated notifications that followed.
That is why monitoring should use repeated alert suppression. The first alert notifies you about the problem, while subsequent occurrences remain visible in the monitoring history without flooding your inbox.
A Good Alert Shouldn’t Tell You About Everything
The goal of monitoring is not to send as many notifications as possible. The goal is to deliver information when it actually matters.
A fixed threshold answers the question, “did more than X errors occur?” Anomaly-based monitoring answers a much more important question: “is the application behaving significantly differently from what is normally expected?”
That difference determines whether alerts help your team respond to real problems or, after a few weeks, become just another source of notifications that everyone learns to ignore.
