Detecting sudden spikes in error counts

The number of errors on its own doesn't always mean something has gone wrong. An application handling heavy traffic can generate more events than a small project, and a temporary rise in exceptions doesn't always mean an outage.

That's why alerts about errors should take into account not just the raw event count, but also the typical error level for that particular project.

Why doesn't a fixed threshold always work?

A threshold like "100 errors" might suit one project and be completely useless for another. For a large application, 100 errors might be a normal level, while for a small service even a dozen or so similar events could signal a serious problem.

Detecting a deviation from the application's typical behaviour is far more useful.

What does the response to a spike look like?

When the number of events clearly diverges from the observed level, the system can treat this as a signal that needs attention. The mechanism also takes into account a minimum number of events, so that a single random exception doesn't trigger an unnecessary alarm.

Why don't you get an email for every error?

A recurring problem can generate hundreds of events. Sending a separate notification for each one would quickly turn monitoring into a source of noise.

That's why notifications are throttled over time. One ongoing outage shouldn't generate identical messages every few seconds.

What should you do after receiving an alert?

First, check which error is responsible for the spike and when it first appeared. Then compare that moment with recent deployments and configuration changes.

The history of occurrence counts lets you quickly assess whether the problem is still growing or was just a brief spike.