Top 3 Issue Alert Tips to Stop Noisy Notifications
Sentry Alerts ping you on Slack, Microsoft Teams, or Pager Duty when something goes needs your attention. However, too many alerts can turn your notification channel into an endless noise feed.
I spoke with dozens of Sentry customers in the past 6 months, and something I heard over and over again was “Sentry can get noisy at times” and “There are days I can’t keep up with Sentry notifications because we get so many of them”. Does this sound familiar?
Alerts should be actionable, not cause you to roll your eyes and clear out the notification without reading it. This blog shares three simple Issue Alert Rule configuration tips that should help reduce noise and improve signal quality
Oh, and one last thing. Nobody knows what matters to your business better than you, so you’ll likely have to customize specific thresholds based on business logic and app traffic.
Tip 1: Prioritize New Issues
Chances are you want to know when new issues happen, not events associated with an issue that’s months old. The big question is what defines new? And when should you start to care?
When you set up a new Issue Alert, you can define new based on the first time an issue occurred and receive a notification once a set number of events and/or users are affected.
Tip 2: Annoy Notify the Right Person
No one likes being told to fix something they didn’t break. Ideally, the person who receives the notification should be the person who is assigned to the issue. You may do this today by setting a THEN condition to notify the Issue Owner. However, if the alerting issue isn’t assigned, Sentry notifies all project members. Maybe this isn’t a big deal if you’re a small team, but if dozens of people are notified about something they can’t fix…somebody isn’t going to be happy.
To stop notifying your entire team about unassigned issues, create an IF condition for when the issue is assigned to no one. For the THEN condition, route the notification to a channel dedicated to triaging.
Alternatively, you can turn off sending a notification to all project members for unassigned issues by going into Project Settings > Issue Owners and toggling the notification off.
Tip 3: “Level Up” Your Alerts
Similar to logging levels, Sentry automatically associates levels (i.e. fatal, error, warning, info, debug, and sample) to events. One of the most effective ways to only receive high-priority issue notifications is by adding the IF condition - event’s level is equal to fatal. You can define fatal and the other levels in your SDK by following instructions in the docs (for example, here’s how for JavaScript).
But maybe there are fatal errors sending alerts that you don’t care about, or filtering by event level is too restrictive. For example, you may not want to know about operation errors like ReadTimeout exceptions. Instead of customizing the event level, you can filter out specific exception types like in the screenshot below.
Wrap Up
When you first set up your project's Alert Rules, you may have opted for the “alert me on every new issue” option and are now overwhelmed by the volume of alerts Sentry sends. With some simple updates to your existing Alert Rules, you can make Alerts less noisy and more actionable.
Here’s a quick summary of what we covered above to improve the signal to noise ratio for your Alerts: 1) Define what “new” means based on event frequency or impacted users, 2) Don’t rely on notifying the issue owner for new issues, so everyone on your team doesn’t get a notification for something they didn’t break, and 3) Filter for high priority issues you care about based on event level or filter out specific issue types you don’t care about.
There’s no “one size fits all” for making your Alerts more actionable. However, after incorporating the tips above, you should be on your way to relying on Sentry Alerts to tell you when those "Oh $%&^" moments happen.
