Global and Location Alert Conditions
There are two different types of alert conditions in an alert rule: global alert conditions and location alert conditions. It is important to note that these conditions are not, in fact, triggered based on physical location, but rather on the conditions they meet.
A location alert condition assesses the presence of an anomaly, such as an error being present, throughput being under 1000 kbps, or response time breaching the medium sensitivity of a dynamic baseline (see Quantile Dynamic Baselining for more information about dynamic baselines).
A global condition assesses whether that anomaly (and/or others) constitutes an alertable issue, such as the same agent detecting the error for more than 3 test rounds, any agent detecting low throughput for 2 out of 3 test rounds, or response time breaching the low sensitivity threshold for adaptive alerting (see Adaptive Alert Detection for more information about adaptive alerting).
For the most common alert rules, quantile dynamic baselining (QDB) and adaptive alerting are the default methods for setting location and global alert conditions. However, you can also choose to make manual adjustments to both condition types. For those alert rules for which QDB or adaptive alerting are not currently available, you must manually set these alert rules (see Creating and Editing Alert Rules to understand which alert rules use QDB and adaptive alerting). This article explains how to set global and location alert conditions manually.
For information about adaptive alerting and quantile dynamic baselining, see an overview at Creating and Editing Alert Rules, or deeper explanations at Adaptive Alert Detection and Quantile Dynamic Baselining.
Setting Manual Alert Conditions
For those alert rules where adaptive alerting is the default, select Manual as your alert detection method to view your set of manual global alert condition options; these constitute the first line of dropdowns under Alert Conditions. You will not see an Adaptive/Manual toggle on alert rules that do not currently offer adaptive alerting. Location alert conditions are the metrics you select and set below this line.
Setting Global Alert Conditions Manually
The global alert condition is where you specify how your location alert conditions will be applied to your alert triggers, including how many location alert conditions, the number or percent of alert triggers, and how many test rounds must be met before alerting. All alert rules have similar, though not exact, options for configuring global alert conditions. The following list explains how to configure each configurable field as you read the global condition from left to right.
All/Any: Select All when all the specified location alert conditions must be met (AND) or select Any when any one of the specified location alert conditions must be met (OR).
any of/the same (for Network & App Synthetics and Routing rules only): Select any of if you want an alert activated when any set of alert triggers (agents or monitors) meet the alert condition(s) in consecutive rounds. Select the same if you want an alert activated only if the same set of alert triggers meet the alert conditions(s) in consecutive rounds. When you select the same, this is called selecting "sticky triggers".
Sticky triggers: For example, an alert rule is configured for the same agent to trip a specified threshold in three consecutive rounds. If the Atlanta Cloud Agent trips the rule in round one, the Ashburn Cloud Agent trips it in round two, and the San Francisco Cloud Agent trips it in round three, an alert would not be activated. Either Atlanta, Ashburn, or San Francisco would need to trip the rule in three consecutive rounds to activate an alert.
Threshold value (not applicable to Devices rules): Specify the threshold value for alert triggers that must meet the alert conditions in order to trigger this alert rule. This value will be either a number of alert triggers or a percentage of alert triggers, as specified in the next setting.
Note: When a percentage of alert triggers is used, and the percentage results in a non-whole number threshold value of actual alert triggers, the fractional part of the value is significant. For example, when an alert rule with a threshold of 25% of all agents is applied to 13 agents, the threshold is 3.25 agents. This threshold will require 4 agents to meet the alert criteria in order to activate the alert rule.
Threshold units: Select either the alert trigger, or the percentage thereof. The options for each alert rule are:
Network & App Synthetics (all test types): agent(s) or % of agents.
Endpoint Experience Real User Tests - Application test type: agent(s) or % of agents.
Endpoint Experience Real User Tests - Endpoint test type: visited site(s) or % of visited sites.
For Endpoint Experience Scheduled Tests (all test types), you select a threshold value for both number and percentage of Endpoint Agents.
Routing: monitor(s) or % of monitors.
For Devices, the threshold unit is part of the location alert condition, where the options are: Any interface or Any interface matching.
Rounds (met): Select the number of test rounds that the subsequent location alert condition(s) must meet out of a total number of rounds in order to activate the alert rule. See also the Rounds (total) entry below.
Rounds (total): Select the total number of test rounds against which the Rounds (met) selection is evaluated. For example, if Rounds (met) = 2 and Rounds (total) = 3, then for every three rounds, the alert rule will activate if the condition(s) were met twice.
Time interval (for Routing only): Select the time interval, in minutes, that the alert triggers on. For example, if 1 monitor must meet the location condition (e.g. reachability is less than 80%) to trigger an alert, and the time interval is set to 3, the alert triggers when 1 monitor's reachability is less than 80% for 3 consecutive minutes. If you change the time interval to 5, reachability must be less than 80% on any 1 monitor for 5 consecutive minutes before the alert triggers. Time intervals are available in 1-minute increments, starting with 1 minute up to a maximum of 180 minutes.
any/at least (for Event Detection only): Defaults to "any", where an alert triggers after any length of time. If you change the drop-down menu to "at least", a text box appears allowing you to set the number of minutes required before the alert triggers. The default value is 5 minutes.
Setting Location Alert Conditions Manually
Location alert conditions are where you set the specific metrics on which an alert becomes active. You can set any number of metrics for an alert, though bear in mind that the more metrics you set, the less likely it is an alert will activate. Location alert conditions are configured by choosing at least one metric (the test characteristic against which you're measuring change) and one operator (the type of measure). Depending on the metric, other configurable options include threshold values and units. Reading left to right, location alert conditions include the following configurable fields:
Metric: Select a test metric for this condition. See Alert Metrics Reference for relevant metrics.
Operators: Select an operator for this condition. There are many operators to choose from, some of which are self-explanatory. Below is a selection with more explanation. For a full list of metrics, operators and units, see Alert Metrics Reference.
>, <, ≥, ≤ : Numerical comparisons for greater than, less than, greater than or equal to, and less than or equal to. Available for all numerical (integer only) measures, such as packet loss percentage on network layer tests, or error count on page load tests.
is, is not: Non-numeric comparison for values that are not continuous ranges (e.g., HTTP response codes) or that are a fixed string value, such as the error type (e.g., "DNS", "Connect", "SSL"). Also, when suffixed with "empty", determines whether a metric has a value or has no value.
in, not in: Numeric or string comparison to a list of values. For example, a BGP routing rule compares a test metric's AS number (integer) to a list of one or more AS numbers to determine if the test metric is found or not found in the list. Use a wildcard * when matching against word spaces. For example, "10 * aspmx3.googlemail.com."
is incomplete: Determines whether a test completed the operations for a given metric. For example, this metric can be used to determine whether a path trace reached its destination, or a page load test fully loaded a page.
is present: Used when an error condition is present.
matches, does not match: Determines whether the POSIX regular expression in the alert rule is found within the string produced by the test metric (i.e., a substring will produce a match). For example, an alert rule for the Error metric of an HTTP server test with the following alert condition
Matches operator example will alert when the test's Error Details text is "SSL certificate problem: certificate has expired":
Test showing error match details because the regular expression "certificate\s*\w*:" matches the sub-string "certificate problem:". The operators available per type of alert rule are also shown in the table below.
Threshold: The value that the metric setting will be compared against, using the chosen operator. Note that some operators do not have a value field.
Unit: Often, the unit is fixed once an operator is chosen, such as threshold value, %, ms, or kbps, but sometimes you can choose the unit, such as for dynamic baselines or for device interface thresholds.
Add/Delete: Click the + or - icon to add or delete location alert criteria to this alert rule. Criteria can be nested for some types of alert rule.
Location Alert Considerations
It is important to note that location alerts trigger and clear independently from the global alert. If you see multiple location alerts triggered under a global alert, you cannot assume that all the listed location alerts met the initial alert criteria from a per-round basis. They could have been added for meeting the condition for only one round. To verify which location alerts initially triggered the global alert condition, it is best to check the test data.
It is also important to note that the only location alerts that will be displayed in the UI at the start of an active global alert will be the location alerts active at the time of trigger. This can lead to scenarios where a flapping alert trigger was involved in the evaluation criteria of a location alert being triggered, but has since cleared before the global alert becomes active. For example, imagine alert criteria that states "Any 2 agents have an error 3 out of 3 rounds." And the following occurs:
Agent A - meets condition in rounds 1, 2, 3
Agent B - meets condition in rounds 2 and 3
Agent C - meets condition in round 1
In the scenario listed above, 2 agents meet the criteria 3 out of 3 rounds: round 1 is agent A and C, rounds 2 and 3 are agents A and B. At the global alert trigger, only agents A and B will be listed in the location alerts, since agent C cleared before the global alert triggered, even though agent C contributed to the trigger of the alert. This will only happen when the alert conditions have multiple agents that need to meet an alert criteria multiple rounds in a row.
Alert Triggers
A location alert is included within a global alert when a single alert trigger meets the location alert conditions for at least one round, regardless of the thresholds set for the global alert. An alert trigger is the element that a specific test type is set to examine, and includes:
For Network & App Synthetic tests, the alert is triggered by agents.
For Endpoint Experience tests, the alert is triggered by visited sites or by Endpoint Agents.
For routing tests, the alert is triggered by BGP monitors.
For device tests, the alert is triggered by interfaces.
For Internet Insights tests, the alert is triggered by affected tests or by catalog providers.
Available Metrics, Operators, and Units
Manual Alert Condition Example
In the example below, a global alert is triggered on the HTTP connect response alert if the following conditions are met:
Any location conditions are met by 10% of agents associated with 8 tests for 2 of 2 times in a row.
Global and location conditions
The location conditions are:
Connect Time is greater than or equal to 150 ms.
Response Time is greater than or equal 100 ms.
Once the global alert condition has been triggered, any agent which meets the location alert conditions in a single round will be included as “active” in the alert as long as the global alert remains active. When an agent no longer meets the location alert conditions, it will no longer show as “active” but will remain associated with the alert.
For example, in the image below, you can see that for the HTTP connect response alert the Panama City agent triggered the global alert first at 11:25 (see Start column). While it was still active, the Copenhagen, Palermo, and Seoul agents also met the location alert conditions at 11:40, but then became inactive once their response times decreased to below the local alert conditions (as seen in the Current metric column). The alert remains active until Panama City - or the last remaining active agent - no longer meets the location alert conditions.
Reducing Noise in Manual Alerts
If a manually set alert is throwing notifications that exceed your operational requirements, you can adjust the alert condition thresholds.
Go to Manage > Alert Rules.
Select the name of the alert rule that you want to adjust.
On the Settings tab, in the Alert Conditions section, review the current thresholds.
Make changes to these settings to reduce the frequency of alerts, according to your requirements.
For the best way to reduce noise, try using dynamic baselines in your alert configuration instead of static thresholds. To learn more about dynamic baselines, see Dynamic Baselines.
After you adjust a noisy alert to meet your service-level expectations, the alert should begin to clear. An active alert that clears is moved to the Alerts History tab. To view cleared alerts, go to Alerts > Alerts History.
Manual DNS Server Alert Rules
DNS server tests when created manually differ from other ThousandEyes manual tests in that multiple servers can be explicitly targeted in a single test. As a result, manual DNS server alert rules are evaluated on a per-server basis. That is, for each server in the DNS Servers field of the test configuration, the alert conditions are evaluated separately from all other servers in the DNS Servers field. For example, consider an alert rule that has the following alert conditions:
When assigned to a DNS server test with two servers configured as the targets, each server will be evaluated independently against the above alert condition. To activate the alert rule, at least four agents must receive an error against the same DNS server. The alert rule would not be triggered if, for example, three agents received an error when testing the first DNS server and a fourth agent received an error when testing the second DNS server.
Last updated