PagerDuty for Alert Notifications

In order to integrate with PagerDuty, you need the following components in place.

ThousandEyes test(s)
ThousandEyes alert rule(s)
A PagerDuty service (which is based on a PagerDuty escalation policy)

Think of a PagerDuty service as a destination for your alerts. Just as with an email notification, the alerts need to be sent somewhere: the reason you'd integrate into a system like PagerDuty allows you to manage notification destinations, repeat notifications, and escalation rules for each recipient of the alert.

Creating a PagerDuty Service

If you don't have an Escalation Policy defined for your alerts coming from ThousandEyes, log in to PagerDuty and create an Escalation Policy. This defines how frequently users are notified, and who receives the notifications.

Create an Escalation Policy

On the top nav, click Escalation Policies, then on the upper right side of the page, click New Escalation Policy. In an escalation policy, you can define who gets notified, how frequently, and using which methods. Depending on the configuration of your team, you can configure an On-Call Schedule - which allows users to rotate through an on-call schedule, allowing handoffs of active incidents, or a user-based Escalation Policy. If you want to configure your Escalation Policy to use a schedule, create this schedule (Top Nav > On-Call Schedules) before attempting to create the Escalation policy.

Give your policy a name
Add users or on-call schedules to the policy
Determine how long before the alert gets escalated
Repeat steps 2-3 for escalations
Set repeat notifications until acknowledged
Save your Escalation policy

Once you have an escalation policy defined, you can configure a service which will accept notifications from ThousandEyes and begin the escalation policy.

Creating a Service Using ThousandEyes

There are two methods of PagerDuty service creation, but we'll only show you the ThousandEyes side here.

In ThousandEyes, go to the Integrations screen.
Click + New Integration.
Select PagerDuty. A link will appear:

This link takes you to an authorization page on the PagerDuty site, which requests your username and password, then logs you in.

Once you've authenticated and authorized the integration, you can select multiple existing PagerDuty services. Note You can only create integrations for services that have both alerts and incidents enabled in PagerDuty.

In the example below, we've created a new service based on a ThousandEyes test service policy and Dev App.

Finally, click Connect. You can ignore the warning about being "almost done". You're done, and alerts triggering on the basis of this alert rule will be routed to users in your escalation policy.

If you select multiple services, multiple integrations will be created on the ThousandEyes end. We will provide feedback on each integration as to whether they were successfully created. If you already have an integration with the selected service, we will prevent you from creating a duplicate.

How the Integration Works

Once you've configured a PagerDuty service, and an alert is generated, the alert will trigger a new Incident in PagerDuty, which will follow the Escalation policy upon which the service is based. This escalation policy may:

Notify by email
Notify through mobile push notification (iOS/Android)
Notify by SMS
Notify by calling you and using Text to Speech

In most cases, the alert must be acknowledged, at which point the escalation policy stops. Users can add notes following an acknowledgement, and can clear the mark the alert as Resolved as required. If the alert rule conditions are no longer met by your test (or the alert rule is updated such that the conditions no longer meet the criteria of the alert), the alert rule is marked as cleared within PagerDuty via an incident update.

Mapping of ThousandEyes Alert Severity to PagerDuty Incident Severity

Once you've configured a PagerDuty service and an alert is generated, the alert will trigger a new incident in PagerDuty with one of the following severity levels: