How Much Error Data Do You Really Need for Application Monitoring?
When it comes to monitoring, it’s easy to fall into the trap of thinking that more data always means better monitoring. Every stack trace, every request, and every transaction may seem worth keeping forever.
In practice, long-term storage of all raw events comes at a cost. Data volumes grow, storage costs increase, and analyzing historical information becomes more difficult.
That’s why a well-designed monitoring system should separate the data needed to solve a specific problem from the data used to track long-term trends.
What Are Raw Events Used For?
A raw event is most valuable when you’re trying to answer the question: “what exactly happened in the application?”
A stack trace, exception message, request URL, environment details, and additional context can help identify the cause of a problem and reproduce what happened. This data is primarily useful when investigating a specific error.
However, you usually don’t need a full stack trace from two years ago to determine whether your application was more stable this month than it was six months ago.
Raw Data and Statistics Serve Different Purposes
It therefore makes sense to separate detailed data retention from historical statistics.
Raw events can be stored for a limited period. During that time, you have access to the details needed for debugging, including the error message, stack trace, and request context.
At the same time, essential information can be aggregated and stored for much longer. Instead of keeping one million individual events, you can simply retain the information that 12,438 errors occurred on a given day.
What Is Worth Keeping in Your History?
Long-term history doesn’t have to be complicated. Depending on the type of monitoring, it may be enough to retain information such as:
- the number of errors on a given day,
- the number of requests handled,
- the number of occurrences of a specific issue,
- the average or representative response time,
- uptime and performance monitoring results,
- when an issue started and when it ended.
This type of data takes up significantly less space than full events while still providing enough information to build charts and analyze trends.
Why Are Aggregates Important for Error Monitoring?
Aggregation isn’t just about saving storage space. It is also useful for anomaly detection.
If you want to determine whether today’s error count is unusual, you need historical data to compare it against. However, you don’t need to keep every detailed event from every previous day.
A daily error count can serve as a baseline for future alerts even after the detailed events from that period have already been deleted.
Data Retention and Limits
There is another important consideration: event limits should not depend on raw data retention.
If detailed events are regularly deleted, the number of events currently stored in the database no longer tells you how many events were received during a given billing period. Usage limits should therefore have an independent counter that is not affected by data retention.
This ensures that deleting old errors does not “reverse” previously recorded usage or change the usage history.
How Do You Find the Right Balance?
A good approach is to ask yourself a simple question: will this information be needed to solve a specific problem, or only to analyze a trend?
If you need it for debugging, keep the detailed event for an appropriate amount of time. If you’re interested in the application’s long-term history, aggregated statistics are usually enough.
Separating the two allows you to reduce the amount of stored data without losing the information that actually matters for maintaining your application.
Monitoring Doesn’t Need to Store Everything
The best monitoring system isn’t the one that remembers every detail forever. It’s the one that keeps the right data for the right amount of time.
Detailed data helps you fix a specific error. Aggregated data helps you understand your application’s history. Separating these two layers gives you useful monitoring, sensible data retention, and access to meaningful long-term statistics.
