Creating and Editing Alert Rules

You can configure alert rules for different conditions including both synthetic tests as well as alerts that don’t rely on tests such as WAN Insights or event detection alerts. This article walks you through the common parts of alert rule configuration, plus those parts that are unique to each kind of test.

As for common characteristics, each alert rule has:

• A name.

• A series of tests against which it is enabled (for alerts that rely on tests).

• A scope of alert triggers (such as agents or monitors) to which the alert rule applies (with the exception of Endpoint Agent scheduled tests, WAN Insights, and event detection).

• Criteria defining the alert conditions.

• The number of alert triggers that the alert conditions must meet in order to activate an alert.

Alert rules also include a notification mechanism, such as a list of email recipients (recipients do not need to be users of ThousandEyes in order to receive email notifications), a PagerDuty service or one or more webhooks.

Adding a New Alert Rule

To create a new alert rule, click Manage > Alert Rules. The Alert Rules page opens.

From the tabs at the top of the page, select the desired alert source:

• Cloud and Enterprise Agents

• Endpoint Agents

• BGP Routing

• Devices

• Internet Insights

• WAN Insights (for information about WAN Insights alert rules, see WAN Insights and ThousandEyes Alerts).

• Cloud Insights

• Traffic Insights

• Event Detection

Then click Add New Alert Rule. The Add New Alert Rule panel opens. The image below shows the panel that opens for Cloud and Enterprise Agents.

Configuring the Alert Rule

Every new alert panel within each alert source opens with three sections. The top section is where you choose the type of alert you wish to configure and give it a name. The bottom two panels consist of the Settings tab, where you specify the alert triggers (middle section) and alert conditions (bottom section). For information about the Notifications tab, see Alert Notifications.

Naming the Alert Rule

In the top section of the panel for each new alert, you will find:

• Alert Type: Select the test layer for this alert rule.

• Compatible Test Types (for Cloud, Enterprise, and Endpoint Agents only): As you select the test layer in the Alert Type field, the dropdown field to the right displays the test types to which this alert rule can be assigned.

• Rule Name: Specify a name for the alert rule.

Selecting the Alert Triggers

The middle and bottom sections of the panel consist of the Settings tab. The middle section is where you configure your alert triggers (such as agents, monitors, or catalog providers). The fields in this section vary depending on the alert source and type, set out below.

Cloud and Enterprise Agents

• Direction (only for Network: Agent to Agent and Network: Path Trace tests): Enables you to choose whether the alert triggers in the Source-to-Target, Target-to-Source, or Both (Agent to Agent) or Either (Path Trace) direction.

• Tests: A dropdown menu listing all the tests set up in your account group. Select one or more tests to assign them to this alert rule.

• Agents: Select the agents to which you will assign this alert rule. The options are:

  • All agents: All agents will be assigned this alert rule.

  • All agents except: All agents will be assigned this alert rule except for the ones selected.

  • Specific agents: Only the selected agents will be assigned to this alert rule.

    Note: Selecting All agents except or Specific agents opens another dropdown menu where you can select the agents you do or don't want to alert on.

• Severity: Choose from Info, Minor, Major, and Critical.

Endpoint Agents

Real User Tests

• Agents: Select the agents to which you will assign this alert rule. The options are:

  • All agents: All Endpoint Agents belonging to the account group will be assigned this alert rule.

  • Specific agents: Only the selected Endpoint Agents will be assigned to this alert rule.

  • Agent labels: Only the Endpoint Agents with the specified label will be assigned to this alert rule.

    Note: Selecting Specific agents or Agent labels opens another dropdown menu where you can select the agents or labels you want to alert on.

• Visited Sites: Select the sites for which this alert will be triggered. The options are:

  • Any visited site: Any site within the monitored domain set that a user visits will be assigned to this alert rule.

  • Specific visited sites: Only the selected visited sites will be assigned to this alert rule. If you select this option, a dropdown menu appears where you can select from a number of suggested domains or type in a custom domain.

• Severity: Choose from Info, Minor, Major, and Critical.

Scheduled Tests

• Tests: A dropdown menu listing the all compatible Endpoint tests set up in your account group. Select one or more tests to assign them to this alert rule.

• Severity: Choose from Info, Minor, Major, and Critical.

BGP Routing

• Tests: A dropdown menu listing the all the tests set up in your account group. Select one or more tests to assign them to this alert rule.

• Prefix Length: A dropdown menu allowing you to specify the length of prefix for both IPv4 and IPv6. The length defaults to between 16-32 for IPv4 and 32-128 for IPv6.

• Monitors: Select the monitors to which you will assign this alert rule. The options are:

  • All monitors: All monitors will be assigned this alert rule.

  • All monitors except: All monitors will be assigned this alert rule except for the ones selected.

  • Specific monitors: Only the selected monitors will be assigned to this alert rule.

    Note: Selecting All monitors except or Specific monitors will open another dropdown menu where you can select the monitors you do or don't want to alert on.

• Severity: Choose from Info, Minor, Major, and Critical.

Devices

• Devices (for the Device alert type only): A dropdown menu listing all the monitored devices set up in your account group. Select one or more devices to assign them to this alert rule.

• Interfaces (for the Interface alert type only): A dropdown menu listing all the monitored interfaces set up in your account group. Select one or more interfaces to assign them to this alert rule.

Internet Insights

• Affected Tests: Select the affected tests to which you will assign this alert rule. The options are:

  • Any: Any affected tests will be assigned this alert rule.

  • Specific: Only the selected affected tests will be assigned to this alert rule. If you select this option, a dropdown menu will appear where you can select the affected tests you want to alert on.

• Catalog Providers: Select the catalog providers to which you will assign this alert rule. The options are:

  • Any: Any catalog providers will be assigned this alert rule.

  • Specific: Only the selected catalog providers will be assigned to this alert rule. If you select this option, a dropdown menu will appear where you can select the catalog providers you want to alert on.

• Severity: Choose from Info, Minor, Major, and Critical.

Cloud Insights

• Provider: Select the provider to which you will assign this alert rule. The current options are:

  • Amazon Web Services

• Scope Types: Select the scope type to which you will assign this alert rule. The selected scope type will be monitored from both inbound and outbound directions. The options are:

  • Account

  • Virtual Private Cloud

  • Availability Zone

  • Region

  • Subnet

• Severity: Choose from Info, Minor, Major, and Critical.

Traffic Insights

• Scope Types: Select the scope type to which you will assign this alert rule. The selected scope type will be monitored from both inbound and outbound directions. The options are:

  • Devices

  • Applications

  • Subnet Tags

  • Geolocations

• Severity: Choose from Info, Minor, Major, and Critical.

Setting the Alert Conditions

As with the alert triggers, the alert conditions vary depending on alert type, but also test type. First, we'll explain how to apply alert conditions to alert triggers (the first line under Alert Conditions in the image below); these are called global alert conditions. Then we'll explain how to set the alert conditions themselves (those items with a "-/+" next to them in the image below); these are called location alert conditions. For more information about global and location alert conditions see Global and Location Alert Conditions.

Setting Global Alert Conditions

The global alert condition is where you specify how your location alert conditions will be applied to your alert triggers, including how many location alert conditions, the number or percent of alert triggers, and how many test rounds must be met before alerting. Except for Insights products (Internet, WAN, Cloud), all alert rules have similar options for configuring global alert conditions. The following list will explain how to configure each configurable field as you read the global condition from left to right.

• All/Any: Select All when all the specified location alert conditions must be met (AND) or select Any when any one of the specified location alert conditions must be met (OR).

• any of/the same (for Cloud and Enterprise Agents and BGP routing only): Select any of if you want an alert activated when any set of alert triggers (agents or monitors) meet the alert condition(s) in consecutive rounds. Select the same if you want an alert activated only if the same set of alert triggers meet the alert conditions(s) in consecutive rounds. When you select the same, this is called selecting "sticky triggers".

  Sticky triggers: For example, an alert rule is configured for the same agent to trip a specified threshold in three consecutive rounds. If the Atlanta Cloud Agent trips the rule in round one, the Ashburn Cloud Agent trips it in round two, and the San Francisco Cloud Agent trips it in round three, an alert would not be activated. Either Atlanta, Ashburn, or San Francisco would need to trip the rule in three consecutive rounds to activate an alert.

• Threshold value (not applicable to Devices): Specify the threshold value for alert triggers that must meet the alert conditions in order to trigger this alert rule. This value will be either a number of alert triggers or a percentage of alert triggers, as specified in the next setting.

  Note: When a percentage of alert triggers is used, and the percentage results in a non-whole number threshold value of actual alert triggers, the fractional part of the value is significant. For example, when an alert rule with a threshold of 25% of all agents is applied to 13 agents, the threshold is 3.25 agents. This threshold will require 4 agents to meet the alert criteria in order to activate the alert rule.

• Threshold units: Select either the alert trigger, or the percentage thereof. The options for each alert rule are:

  • Cloud and Enterprise Agents (all test types): agent(s) or % of agents.

  • Endpoint Agent Real User Tests - Application test type: agent(s) or % of agents.

  • Endpoint Agent Real User Tests - Endpoint test type: visited site(s) or % of visited sites.

  • For Endpoint Scheduled Tests (all test types), you select a threshold value for both number and percentage of Endpoint Agents.

  • BGP Routing: monitor(s) or % of monitors.

  • For Devices, the threshold unit is part of the location alert condition, where the options are: Any interface or Any interface matching.

• Rounds (met): Select the number of test rounds that the subsequent location alert condition(s) must meet out of a total number of rounds in order to activate the alert rule. See also the Rounds (total) entry below.

• Rounds (total): Select the total number of test rounds against which the Rounds (met) selection is evaluated. For example, if Rounds (met) = 2 and Rounds (total) = 3, then for every three rounds, the alert rule will activate if the condition(s) were met twice.

• Time interval (for BGP Routing only): Select the time interval, in minutes, that the alert triggers on. For example, if 1 monitor must meet the location condition (e.g. reachability is less than 80%) to trigger an alert, and the time interval is set to 3, the alert triggers when 1 monitor's reachability is less than 80% for 3 consecutive minutes. If you change the time interval to 5, reachability must be less than 80% on any 1 monitor for 5 consecutive minutes before the alert triggers. Time intervals are available in 1-minute increments, starting with 1 minute up to a maximum of 180 minutes.

• any/at least (for Event Detection only): Defaults to "any", where an alert triggers after any length of time. If you change the drop-down menu to "at least", a text box appears allowing you to set the number of minutes required before the alert triggers. The default value is 5 minutes.

Setting Location Alert Conditions

Location alert conditions are where you set the specific metrics on which an alert becomes active. You can set any number of metrics for an alert, though bear in mind that the more metrics you set, the less likely it is an alert will activate. Location alert conditions are configured by choosing at least one metric (the test characteristic against which you're measuring change) and one operator (the type of measure). Depending on the metric, other configurable options include threshold values and units. Reading left to right, location alert conditions include the following configurable fields:

• Metric: Select a test metric for this condition. See Alert Metrics Reference for relevant metrics.

• Operators: Select an operator for this condition. There are many operators to choose from, some of which are self-explanatory. Below is a selection with more explanation. For a full list of metrics, operators and units, see Alert Metrics Reference.

  • >, <, ≥, ≤ : Numerical comparisons for greater than, less than, greater than or equal to, and less than or equal to. Available for all numerical (integer only) measures, such as packet loss percentage on network layer tests, or error count on page load tests.

  • is, is not: Non-numeric comparison for values that are not continuous ranges (e.g., HTTP response codes) or that are a fixed string value, such as the error type (e.g., "DNS", "Connect", "SSL"). Also, when suffixed with "empty", determines whether a metric has a value or has no value.

  • in, not in: Numeric or string comparison to a list of values. For example, a BGP routing rule compares a test metric's AS number (integer) to a list of one or more AS numbers to determine if the test metric is found or not found in the list. Use a wildcard * when matching against word spaces. For example, "10 * aspmx3.googlemail.com."

  • is incomplete: Determines whether a test completed the operations for a given metric. For example, this metric can be used to determine whether a path trace reached its destination, or a page load test fully loaded a page.

  • is present: Used when an error condition is present.

  • matches, does not match: Determines whether the POSIX regular expression in the alert rule is found within the string produced by the test metric (i.e., a substring will produce a match). For example, an alert rule for the Error metric of an HTTP server test with the following alert condition

    

    will alert when the test's Error Details text is "SSL certificate problem: certificate has expired":

    

    because the regular expression "certificate\s*\w*:" matches the sub-string "certificate problem:". The operators available per type of alert rule are also shown in the table below.

• Threshold: The value that the metric setting will be compared against, using the chosen operator. Note that some operators do not have a value field.

• Unit: Often, the unit is fixed once an operator is chosen, such as threshold value, %, ms, or kbps, but sometimes you can choose the unit, such as for dynamic baselines or for device interface thresholds.

• Add/Delete: Click the + or - icon to add or delete location alert criteria to this alert rule. Criteria can be nested for some types of alert rule.

Available Metrics, Operators, and Units

See Alert Metrics Reference.

Editing an Alert Rule

Editing an alert rule follows the same configuration steps set out above for adding a new alert rule. The only difference is that to edit an alert rule, you click an existing alert rule (instead of clicking the Add New Alert Rule button). A panel appears with the current alert rule configuration; you can then change any of the field settings to your desired configuration.

Duplicating and Deleting Alert Rules

In the editing pane of an alert rule, you also have the option to delete the alert rule or duplicate it. Duplicating an alert rule is an easy way to configure a new alert rule where you only want to change one or two parameters; for example, if you want to alert on the existence of an error separately from resolution time in a DNS server alert rule. You can duplicate the alert rule specifying the error condition and just change the condition to resolution time without having to configure the entire rule again from scratch.

You will find the delete and duplicate symbols (trash bin and two overlapping pages) in the bottom left of your editing pane. Tooltips appear on hover (see image below). When you click the trash bin, you are prompted to confirm you wish to delete the alert rule. When you click the overlapping pages, a fresh Add New Alert Rule pane opens with the same configuration as the current alert rule.

Reducing Noise in Alerts

If an alert is throwing notifications that exceed your operational requirements, you can adjust the alert condition thresholds.

1. Go to Manage > Alert Rules.

2. Select the name of the alert rule that you want to adjust.

3. On the Settings tab, in the Alert Conditions section, review the current thresholds.

4. Make changes to these settings to reduce the frequency of alerts, according to your requirements.

For the best way to reduce noise, try using dynamic baselines in your alert configuration instead of static thresholds. To learn more about dynamic baselines, see Dynamic Baselines.

After you adjust a noisy alert to meet your service-level expectations, the alert should begin to clear. An active alert that clears is moved to the Alerts History tab. To view cleared alerts, go to Alerts > Alerts History.

DNS Server Alert Rules

DNS server tests differ from other ThousandEyes tests in that multiple servers can be explicitly targeted in a single test. As a result, DNS server alert rules are evaluated on a per-server basis. That is, for each server in the DNS Servers field of the test configuration, the alert conditions are evaluated separately from all other servers in the DNS Servers field. For example, consider an alert rule that has the following alert conditions:

When assigned to a DNS server test with two servers configured as the targets, each server will be evaluated independently against the above alert condition. To activate the alert rule, at least four agents must receive an error against the same DNS server. The alert rule would not be triggered if, for example, three agents received an error when testing the first DNS server and a fourth agent received an error when testing the second DNS server.

BGP Alert Rules

BGP alert rules can be applied to Routing tests that explicitly monitor BGP as well as any test that has the Collect BGP data option enabled. Alert rule conditions can be applied differently depending on which type of test the rule is assigned to.

The default BGP alert rule will activate when 10% of monitors have less than 100% Reachability for at least 1 minute. You can use the time selection range to customize your alert configuration.

Prefix Length

BGP alert rules have a parameter named Prefix Length, which is used to determine the length of prefixes evaluated by the rule. The Prefix Length can be individually configured for IPv4 and IPv6 protocols.

Covered Prefixes

For example, a BGP test has only a single target prefix that will be evaluated against the alert conditions. If the Covered Prefixes box is checked, any covered prefixes found are not evaluated against the alert conditions except the explicit Covered Prefix condition. -->

In contrast, a non-BGP test type can have one or more targets. DNS server tests can explicitly test multiple DNS servers. An agent-to-server test target's domain name can resolve to multiple servers' IP addresses. When creating the BGP path visualization, the Prefix selector shows these multiple target prefixes, and evaluates each prefix against any BGP alert rules assigned to the test. Thus, prefixes that would be considered covered prefixes under a BGP test and not evaluated by the alert rule (unless by a Covered Prefix condition) are evaluated when assigned to the non-BGP test. Similarly, the Covered Prefix condition does not have any relevance when assigned to a non-BGP test.