Alert Suppression Windows
Last updated
Last updated
Alert suppression windows (ASWs) can be accessed and activated via two different processes on the ThousandEyes platform. The first is scheduled ASWs, which you can apply to tests ahead of a known event, such as planned maintenance, during which you don’t need or want to receive alerts. The second is real-time ASWs, which you can apply to alert rules after the alerts have already triggered and start from the current moment.
Scheduled ASWs apply to Cloud and Enterprise Agent tests only. Real-time ASWs apply to all alert sources (e.g., Cloud, Enterprise, and Endpoint Agent tests, BGP routing tests and Internet Insights outage tests), with the exception of agent and device notification alerts.
To configure an ASW, you'll need the Edit alert suppression windows permission. To view suppression windows, you must have the View alert suppression windows permission.
The ASW is displayed as a blue bar on the timeline of any test or alert to which the ASW is assigned. Hovering over the bar displays the description.
For scheduled ASWs, data from tests assigned to the window is not evaluated against alert rule criteria. Data that meets alerting conditions will not trigger an alert until the ASW has ended. This is not the case with real-time ASWs (see Continuous and Intermittent Alert Notifications for more information). However, in both cases, notifications (such as emails, custom webhooks, or custom-built integrations) which would normally occur will not occur during the window. Due to this, the following edge cases are important to note and apply to both types of ASW:
If an alert rule's criteria require two or more rounds in a row to trigger, then any rounds that trigger during the ASW cannot count towards the rounds required to trigger the alert. For example, if an ASW is scheduled to operate between 12 and 1 a.m., and an agent triggers at 12:59 (within the 12:56-1:00 5-minute interval) with another triggering at 1:03 (within the 1:01-1:05 5-minute interval) the first agent does not count towards the two rounds of agents required to trigger the alert. Another agent would have to trigger within the scheduled interval after 1:05 for the alert to activate.
If an alert rule's criteria require two or more agents in the same round to trigger, it is possible for the ASW to start or end at a time within a round of data when some agents running the test have triggered but others have not. As a result, conditions that would normally trigger an alert do not trigger the alert at the edge of the window. For example, for an ASW whose finish time ends at 12:05 a.m., and which is assigned to tests that run at 2-minute intervals, some agents may trigger at 12:04 while others trigger at 12:06. Those agents that triggered at 12:04 will not contribute to the alert rule evaluation because they triggered within the ASW.
If an alert is active at the beginning of an ASW, the alert is not automatically cleared once the ASW begins. For scheduled ASWs, since no data is evaluated for alerts during the ASW, the alert cannot be cleared until the ASW has ended, and data gathered that would clear the alert. See Continuous and Intermittent Alert Notifications for how real-time ASWs treat cleared alerts during a suppression window.
The ThousandEyes platform runs on Coordinated Universal Time (UTC). Because UTC does not experience a period of Daylight Savings Time, the ASWs do not take into account Daylight Savings Time. If an ASW is activated prior to the beginning or end of Daylight Savings Time to run at a time after the change in Daylight Savings Time, the window will be off by the amount of the time shift. Please check your scheduled ASWs prior to the beginning or end of Daylight Savings Time and make any necessary adjustments.
Only tests or alerts to which an ASW is assigned are affected by an ASW. Other tests or alerts are not affected. Also, notifications for non-test or -alert related events such as Enterprise Agents going offline or coming online are not affected by an ASW.
The ThousandEyes platform allows you to schedule alert suppression windows against tests for periods such as planned maintenance. Windows can be scheduled for one-time events or for recurring events to handle regular occurrences such as monthly downtime for maintenance.
To configure a scheduled ASW, navigate to Alerts > Alert Suppression. Click + New Alert Suppression Window to configure a new window, or click an existing window's row to change settings:
Name: Name the alert suppression window.
Tests: Assign tests to this ASW.
Enabled: Check this box to enable the ASW or uncheck the box to disable. This is useful for recurring ASWs, for example if you want to disable the ASW for one scheduled event without removing the ASW for all the other scheduled events within the series.
Starts: Set the start date and time of the ASW, and the time zone for the time. The start time of the window must be at least two minutes ahead of the current time.
Duration: Set the length of time that the ASW will last.
Repeat: Select to repeat the ASW once per day, week, month or on a custom number of days, weeks or months. If month is chosen, text displays indicating what day of the month the repetition will take place, based on the day selected in the Starts menu. Once you select a frequency, the End Repeat field appears.
End Repeat: Select no end to the number of repetitions, a finite number of repetitions, or a date on which to stop the repetition of the window.
Create New Window/Cancel: Click the Create New Window button to save your configuration or click Cancel to exit without saving.
Search: Enter a text string to search for an ASW by name, in the table below the search field.
Table of existing ASWs: The table displays rows of previously configured ASWs. The table has the following columns:
Name: The name of the ASW.
Next Schedule: The date and time of the upcoming window (if any). Note that a window may currently be active, but the next window's date and time will be displayed.
Duration: The length of time that the ASW will last.
Tests: The number of tests to which this ASW is assigned.
Status: The status of the current window. Statuses are:
Enabled: The ASW is configured and will become active at the next configured date and time.
Active: The ASW is currently in effect.
Ended: The last occurrence of the ASW has ended.
Real-time ASWs differ from scheduled ASWs in some important ways, but behave in very similar ways once activated. See Effects on Data and Alert Notifications for more information about how ASWs work. The ways they differ include:
When they are activated: You configure real-time ASWs after an alert has triggered, but scheduled ASWs before an alert has triggered.
Where they are configured: You configure real-time ASWs via the Active Alerts screen, whereas you configure scheduled ASWs via the Alert Suppression screen.
Where they are recorded: Real-time ASWs are visible on the Alerts History screen once the ASW is activated, while you can find all of your scheduled ASWs visible on the Alert Suppression screen.
The types of configurable entities: Real-time ASWs apply to all manner of alert sources, for example Cloud, Enterprise, and Endpoint Agent test alerts and BGP routing alerts, with the exception of agent and device notification alerts; however, scheduled ASWs apply to the tests themselves, specifically Cloud and Enterprise Agent tests only.
An additional feature of real-time ASWs is that they can apply to alerts of different kinds, which increases their range of use cases, described below:
Remove noise: Alerts may start to trigger and clear frequently, generating alert noise. These are called "intermittent alerts". Intermittent alerts may be a sign that an outage or event is causing the agent or monitor to trigger and clear more often than usual. Setting a real-time ASW can remove this noise.
Known outage: If you receive an alert about an outage you are already aware of, you can suppress the alert. This is likely to be what is called a "continuous alert", or one that stays active for a longer period of time.
Prioritize alerts: If an alert triggers that is currently a low priority for the team, either of the intermittent or continuous variety, you may wish to remove it from the Active Alerts screen until it can be better prioritized.
Late-scheduled maintenance: If you’re performing maintenance outside of an already scheduled scheduled ASW, and an alert triggers due to this known-but-unscheduled maintenance, you can suppress the alert for as long as the maintenance is being carried out. This is another type of continuous alert.
With each kind of alert (continuous or intermittent), as soon as you suppress the alert, you will receive a notification that it has been "cleared" through suppression. This simply means that it has been moved from the Active Alerts screen to the Alerts History screen. However, it does not imply resolution of the alert. The notification will contain suppression metadata such as suppression by, duration of suppression, etc., to indicate that the alert was suppressed rather than resolved. From this point, the different kinds of alerts may resolve differently, as described below.
Real-time alert suppression windows use a number of alert identifiers to help determine how you will be notified when the alert clears through resolution or retriggers during or after an ASW. This is important because:
With a continuous alert: The alert may still be active once an ASW ends. In this case, the alert will move from the Alerts History screen back to the Active Alerts screen. You will recieve a notification that the alert has triggered, even though it was the same alert as before and never resolved. Importantly, the original Alert ID, Start time and Metrics at start will be the same as before it was suppressed. If the alert cleared through resolution during the suppression window, the alert will remain on the Alerts History screen, the Suppressed label will be removed, and no new notification will be sent that it has cleared.
With an intermittent alert: If an intermittent alert resolves and retriggers during an ASW, even multiple times, you will not receive any notifications about these events during the ASW. This is the point of the ASW. Once the ASW ends, if the alert is active at the time of ending, you will recieve a notification that the alert has triggered, much like with the continuous alert. However, the Alert ID, Start time and Metrics at start will refer to the most recent trigger event, not the one that was recorded at the time of creating the ASW. If the alert is cleared through resolution at the time the suppression window ends, you will find the latest trigger event in the Alerts History page, the Suppressed label removed, and no new notification sent that it cleared.
In both cases, if a fresh alert triggers after the end of the suppression window during which the previous alert cleared through resolution, you will be notified of the new alert activation as normal.
While you configure scheduled ASWs via Alerts > Alert Suppression, you configure real-time ASWs via Alerts > Alert List > Active Alerts.
On the Active Alerts screen, the final column in the table of active alerts is called Action, with a clock symbol on each row. For any alert that you would like to suppress and temporarily remove from the Active Alerts screen:
Click the clock symbol, which on hover says Suppress. Alternatively, click anywhere on the row of the alert, which opens the details panel; click the Suppress button in the upper right corner of the panel.
A pop-up opens where you can configure the alert suppression window.
In the first dropdown, choose the length of time you would like to suppress the alert. Choose from:
1 hour
8 hours
1 day
5 days
7 days
Custom
If you choose Custom, additional fields display where you can set the date and time to end the suppression.
The second dropdown allows you to set the reason for the suppression. Choose from:
Alert Too Noisy
Maintenance Event
Low Priority Alert
Known Outage
Other
Once you choose a reason, a notes section displays where you can optionally type further information about the alert suppression.
Click Confirm.
A pop-up in the lower right of the screen confirms that the alert has been suppressed.
The alert itself disappears from the Active Alerts tab. You will instead find it on the Alerts History tab, with the label Suppressed.
If you hover over the Suppressed label, a tooltip tells you how long the ASW is configured for, and who initiated or last edited the ASW.
You can edit the amount of time a real-time ASW lasts or you can remove the suppression altogether, with the following steps.
Go to Alerts > Alert List > Alerts History.
Find the suppressed alert you wish to edit. This is marked by a Suppressed label in the Alert Rule column.
On hover, the label displays the current end time and date, and who last modified the suppression.
In the Action column, click Manage Suppression. Alternatively, click anywhere on the row of the alert, which opens the details panel; click the Manage button in the upper right corner of the panel.
Choose Edit. A pop-up opens, where you can:
Change the date and time of the ASW.
Change the reason for the ASW and make any new notes.
Click Confirm to save the changes.
Go to Alerts > Alert List > Alerts History.
Find the alert you wish to remove from suppression. This is marked by a Suppressed label in the Alert Rule column.
On hover, the label displays the current end time and date, and who last modified the suppression.
In the Action column, click Manage Suppression. Alternatively, click anywhere on the row of the alert, which opens the details panel; click the Manage button in the upper right corner of the panel.
Choose Remove Suppression.
A pop-up opens that summarizes the current suppression timeframe.
Click Remove Suppression. This ends the suppression window of the alert.
