Your email inbox is not your monitoring dashboard! In this article, we explain how to reduce the notifications sent when issues are detected.
Notifications are the results of events that are triggered based on the thresholds configured in the monitoring templates. For example, the default value for a Packet Loss Error is 5%. If you have enabled email notifications for errors, this means you would receive an email as soon as a monitoring session in that template has more than 5% packet loss. Whether a notification is sent, delayed or ignored, the issues and events are always stored and available in the App.
Where can I change the Notification Settings?
Organization Notification Settings Details
The following settings apply to all the users within an organization. These settings can only be changed by organization Administrators. Each user can set their own Minimum Notification Severity. More details at User Notification Settings.
A severity level is applied for each triggered event and the notification created is based on those severity levels. The following is the list of possible severity levels:
- Ok (level 0)
- Information (level 1)
- Warning (level 2)
- Error (level 3)
- Critical (level 4)
Minimum Notification Severity (default value: Error)
When an event is triggered, the system compares its severity level with this Minimum Notification Severity setting. If the setting is higher or equal to the event severity level, a notification is triggered. This setting has no impact on notifications with severity level Ok.
For example, if the setting is set to Error, a notification will not be sent if the event severity level is Warning. However, if it reaches Error, a notification will be sent. Then, if a Warning event is received, no notification will be sent. Finally, when an event with severity Ok is received, a notification will be sent because it's back to normal.
Notification Summary (enabled by default)
If enabled, each time a notification is about to be sent to a user, the system will aggregate all the notifications that should be sent to that user into a single email.
With this setting, if an agent with 10 sessions goes down, all the critical notifications for the monitoring sessions will be sent in a single summary email (as long as all the events are received within a 5-second window).
Note that the summary is per organization only. So if a user is in two organizations, notifications from the two organizations will never to be aggregated together in a single summary.
Notification Limit (1 notification per minute by default)
The notification limit is configured as a number of X emails in a period of Y minutes. Within that period of Y minutes, a maximum of X emails will be sent. In the meantime, if notifications are queued to be sent, they will be sent as a summary after the Y period expires.
For example, if one configures a maximum of 2 emails every 5 minutes, as soon as a first notification is ready, it will be sent. The same thing will happen for the second notification. During the 5-minute period (which started when the first notification was sent), if a third and fourth notifications are queued to be sent, they will be delayed and sent as a summary after the 5-minute period expires.
Notification Delays (enabled by default)
When enabled, the system will wait (for up to 15 minutes) before sending a notification. This setting is configurable per severity level. If the notification is no longer valid after the delay (i.e. the issue severity level is lower than the notification severity level), the notification will be ignored.
The default values are:
- Ok: 2 minutes
- Information: 15 minutes
- Warning: 5 minutes
- Error: 2 minutes
- Critical: 15 seconds
For example, if a Warning notification is created, it will be queued for 10 minutes. When the system is ready to send the notification, it will make sure the notification is still valid (i.e. the issue must be in Warning severity level) otherwise it will be ignored.
With this setting, small events that occur only for a few minutes can easily be ignored even if issue thresholds are still aggressive enough to see small issues in the App.
Notification Watermark (enabled by default)
Without this setting, every time the issue severity level is changed, a notification is created. For example, when it changes from Critical to Error, a notification is sent.
With the Notification Watermark setting enabled, the system keeps track of the highest notification sent for the issue and will only send new notifications if the severity is higher or when it is back to Ok (level 0). The Notification Watermark is reset when an issue gets back to Ok (level 0).
As with the other settings discussed above, this setting only affects notifications and all the events are recorded and available in the App.
Impact on PagerDuty Integration
All the settings above impact the PagerDuty Notifications except for the summary. Notifications to PagerDuty are never summarized and will be sent individually.