Alerts

The ThousandEyes platform allows you to configure highly customizable alert rules and assign them to tests, in order to highlight or be notified of events of interest.

For example, a new, start-up bank is offering internet-only accounts at competitive rates. If their system goes down, even for a short period, it could leave their customers unable to make vital payments and the bank could be penalized. Alerts allow the bank’s networking teams to be notified immediately when their tests detect any disruptive issue, thus allowing them to diagnose and fix the issue before it causes harm.

Likewise, companies in many other customer service industries such as healthcare, broadband, logistics, and energy need always-on, reliable connectivity to access medical records or keep the lights on, for example. ThousandEyes alerts enable these customers to know precisely when issues surface that threaten that stability so they can take immediate action

For those who want simplicity in alert configuration and management, the ThousandEyes platform ships with default alert rules configured and enabled for each test.

For a hands-on overview where you can configure a basic alert rule, see Getting Started with Alerts. For a more detailed guide, see Creating and Editing Alert Rules.

Some alert types do not rely on ThousandEyes test data. For example, WAN Insights alerts rely on the same data that WAN Insights uses, which comes from Cisco Catalyst SD-WAN Analytics, formerly known as vAnalytics. See the documentation page WAN Insights and ThousandEyes Alerts for more information.

Assignment to Tests

Once you have created an alert rule, it can be assigned to any test that has the Enable box checked on the test configuration page (e.g., Network & App Synthetics > Test Settings > Tests for Cloud and Enterprise Agent test configuration). By default, each test has the rule "Default <test type> Rule" assigned to it. To add or remove alert rules from tests, click the pull-down menu below the Enable box, and select or deselect alert rules. Deselecting a default alert rule from a test removes the alert rule from the test, but does not stop the alert rule from being added by default to new tests. See Default Alert Rules for information about how to stop an alert rule from being added by default to tests.

Alerts dropdown on test settings screen

To create a new alert rule, click the Edit Alert Rules link to access the Add New Alert Rule screen, and create your rule. You will then return to the test configuration page, and use the pull-down menu to assign your new rule to the test.

Alert Exclusions

In some circumstances, agents are excluded from the calculations that trigger alerts. There are two reasons this may happen.

Offline Agents

Should an agent be offline, it is unable to send data to the ThousandEyes platform. The time-sensitive alerting feature cannot therefore wait to receive its data once it's back online. Only online agents that can send live data to the platform can form part of an alert trigger calculation.

Agents with Local Problems

Cloud Agents that display a "Local Problems" message on a test results page are excluded from alert calculations.

Alerting Basics

This section explains the fundamental concepts behind the ThousandEyes alerting system. Understanding these concepts will help you configure effective alerts and interpret them correctly when they trigger.

What Is an Alert? (And How Is It Different from an Anomaly or a Notification?)

Understanding the distinction between an event, an alert, and a notification is the key to using the ThousandEyes platform effectively.

• An Anomaly is an individual data point of interest from a single test round. For example, if a test runs every 5 minutes and one agent gets an HTTP 500 error, that is an event. You will see events immediately in a test's timeline view because they are raw test data.

• An Alert is a stateful condition that becomes active on the platform only when a pattern of events meets the specific conditions you define in an Alert Rule. An alert rule gives you the power to decide what is important. You can configure a rule to be highly sensitive (triggering an alert on a single event) or more robust (triggering only after many events persist over several rounds).

• A Notification is a message (such as an email, PagerDuty incident, or Slack message) that is automatically sent when an alert's state changes (for example, when it becomes active or is cleared). You must configure notifications separately from alert rules.

In short: Events trigger Alerts, and Alerts can trigger Notifications.

The Alert Lifecycle: Triggered, Active, and Cleared

Every alert goes through a lifecycle. Understanding this helps you know what you're seeing in the platform.

1. Triggered: This is the moment an alert is first created. It happens on the exact test round where the conditions of your alert rule are met for the first time. For example, if your rule is "alert when 3 agents see an error for 2 consecutive rounds," the alert is "triggered" at the end of that second round. Note that an alert being triggered doesn't necessarily mean a problem is "serious"—it means the conditions you defined have been met.

2. Active: This is the state of an alert after it has been triggered. An alert remains "active" for as long as the trigger conditions continue to be met. The Alerts > Alert List page in the platform shows only active alerts.

3. Cleared: This is the moment an alert is resolved. An alert "clears" on the first test round where the trigger conditions are no longer met. Once cleared, the alert is moved from the "Active" list to the "Alert History".

Common Questions and Troubleshooting

This section answers common questions when setting up alerts.

Why don't I see anything in the Alert List, but I see events in my test view?

The reason is that the events you see have not yet met the conditions defined in your test's alert rule.

• Test Views show you raw data, including every single event from every agent in real-time.

• The Alert List is a curated dashboard that shows only active alerts—conditions where the pattern of events has met the specific thresholds defined in an alert rule.

If you see events but no active alert, it means your alert rule is waiting for the pattern of events to meet its trigger conditions, such as more agents failing or the failure persisting for more rounds.

How Can I Test an Alert Rule?

To verify that your entire alerting workflow (including notifications) is working correctly, you can create a temporary alert rule designed to trigger immediately.

The most reliable way to do this is to use a performance-based condition that will almost always be true. Create a custom alert rule with the following settings:

• Condition: Latency > 1 ms (for network tests) or Response Time > 1 ms (for web tests)

• Alert when: 1 of All Agents assigned to test

• For at least: 1 rounds

When you assign this rule to a test, it will trigger an alert on the next completed test round. This allows you to confirm that an active alert appears in the Alert List and that your configured notifications are sent as expected.

Important: This type of rule is intended for temporary testing only. Because the condition will almost always be met, the alert will remain active and could generate continuous notifications. Remember to disable or remove this rule after you have finished testing.

How long does it take for an alert to clear?

An alert clears automatically on the first test round where the conditions of the alert rule are no longer met. For example, if a rule requires 3 agents to fail, the alert will clear as soon as a test round completes with 2 or fewer agents failing.

Can I manually clear an alert?

Alerts are data-driven and clear automatically based on test results. This ensures that the alert status always reflects the actual, measured state of your monitored service. To clear an alert manually, you can remove the alert rule from the test or you can adjust the alert conditions. You can also set a real-time suppression window.

To learn more about clearing alerts, see Alert Clearing.

If I set up a notification, how many will I get for a single outage?

By default, you will receive two notifications for each alert instance:

1. One notification when the alert is first triggered.

2. One notification when the alert is cleared.

You can configure notifications to be sent repeatedly for alerts that remain active for an extended period. To learn more about configuring alert notifications, see Alert Notifications.

How do my alerts relate to the alerts in Internet Insights?

They are completely separate.

• Your Alerts (in Alert List): These are generated by your tests based on your alert rules. They are specific to the services you choose to monitor.

• Internet Insights Alerts: These are generated automatically by Cisco's global monitoring system when it detects large-scale public internet or service provider outages. They provide broad context about the state of the internet.