# Splunk ITSI Integration

This guide explains how to integrate Cisco ThousandEyes with Splunk IT Service Intelligence (ITSI). The integration enables you to ingest test data, send alert notifications, and visualize Splunk episodes directly in ThousandEyes.

## Prerequisites

Ensure you have the following components installed:

* [Splunk ITSI](https://www.splunk.com/en_us/products/it-service-intelligence.html) version 4.20.x or later. For installation instructions, see [Splunk ITSI: Install and Upgrade Manual](https://github.com/thousandeyes/docs/blob/prod/product-documentation/integration-guides/custom-built-integrations/splunk-app/\(https:/docs.splunk.com/Documentation/ITSI/4.20.1/Install/Install\)/README.md).
* [Cisco ThousandEyes App for Splunk](https://splunkbase.splunk.com/app/7719) version 0.1.0 or later.
* [Splunk App for Content Packs](https://splunkbase.splunk.com/app/5391) version 2.3.0 or later. For installation instructions, see [Splunk ITSI: Install the Splunk App for Content Packs](https://help.splunk.com/en/splunk-it-service-intelligence/content-packs/2.3/install-the-splunk-app-for-content-packs).
* [Content Pack for ITSI Monitoring and Alerting](https://docs.splunk.com/Documentation/CPITSIMonitorAlert/2.3.0/CP/About) version 2.3.0 or later. For more information, see the [section below](#configure-the-content-pack-for-itsi-monitoring-and-alerting).

## Install the Content Pack for Cisco ThousandEyes

1. Follow the instructions in [Install the Content Pack for Cisco ThousandEyes](https://help.splunk.com/en/splunk-it-service-intelligence/content-packs-for-itsi-and-ite/cisco-thousandeyes/1.0/installation/install-the-content-pack-for-cisco-thousandeyes#ariaid-title3) until you reach the step that says **“Follow the on-screen instructions to install the content pack.”**
2. Then return to this guide to complete the remaining steps.
3. Click **Proceed**.
4. Click **Import as enabled**. Optionally disable backfill if you do not want to import historical data.
5. Click **Install selected**.

   ![Cisco ThousandEyes Content Pack Installation](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-7197124d90cfa3f49d495a000678f60bfa84cfb1%2Fitsi-content-pack-cisco-thousandeyes.png?alt=media)
6. In the confirmation dialog, click **Install**.
7. Wait for the installation to complete. The dialog will show the installed KPI base searches, entity types, service templates, and services.
8. Return to the **Data Integrations** page. The Cisco ThousandEyes content pack should display a green checkmark with the status “All Saved Searches activated.”
9. In ITSI, go to **Configuration > Service Monitoring > Service and KPI Management** to confirm that the ThousandEyes services are enabled.

## Update the Index Used by the Content Pack

If your test stream does not use the `thousandeyes` index, update the content pack's search macro to match your selected index.

1. In Splunk Enterprise, go to **Settings > Advanced search > Search macros**.
2. In the **App** drop-down menu, select **Cisco ThousandEyes (DA-ITSI-CP-thousandeyes)**.
3. In the **Filter** field, type `itsi_cp_thousandeyes_index` to locate the macro.
4. Click **itsi\_cp\_thousandeyes\_index** to open it.
5. In the **Definition** field, update the value of `index="thousandeyes"` to match the index used by the Cisco ThousandEyes App for Splunk.
6. Click **Save** to apply your changes.

## Configure the Content Pack for ITSI Monitoring and Alerting

1. In Splunk ITSI, go to **Configuration > Data Integrations**.
2. Select the **Content library** tab.
3. Select the **ITSI Monitoring and Alerting** content pack.
4. Review what's included in the content pack, then click **Proceed**.
5. Leave the default **Add all 45 objects**.
6. Click **Import as enabled**.
7. Optionally backfill your ITSI environment with the previous seven days of KPI data.
8. Click **Install selected**.

   ![ITSI Monitoring and Alerting Content Pack Installation](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-300d77679f0bdd2edc9bb68f5bfa5e99dbddb21c%2Fitsi-monitoring-and-alerting-content-pack.png?alt=media)
9. Click **Install**.

For more information, see [Install and configure the Content Pack for ITSI Monitoring and Alerting](https://docs.splunk.com/Documentation/CPITSIMonitorAlert/2.3.0/CP/Install).

## Stream ThousandEyes Test Data to Splunk ITSI

1. Ingest ThousandEyes network data to Splunk using the [Cisco ThousandEyes App for Splunk](https://splunkbase.splunk.com/app/7719).
2. [Add your ThousandEyes user](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/splunk-app/configuration#add-a-thousandeyes-user).
3. [Create a test stream (metrics input)](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/splunk-app/inputs).

**Note:** The default expected index is `thousandeyes`. If your data stream uses a different index, you must update the macro as described in the [Update the index](#update-the-index-used-by-the-content-pack) section.

![ThousandEyes Test Stream Metric Input Creation](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-9291e999a07ae9fd336736925ad3da24486203fd%2Fthousandeyes-test-stream-metric-input.png?alt=media)

## Send ThousandEyes Alert Notifications to Splunk ITSI

### Step 1: Configure a Custom Webhook in ThousandEyes

Follow the steps in [Splunk Alert Notification](https://docs.thousandeyes.com/product-documentation/integration-guides/custom-webhook-examples/splunk-alert-notifs) to create a custom webhook for sending ThousandEyes alerts to Splunk ITSI.

![ThousandEyes Custom Webhook Integration](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-9761f7f34b6d5d9f3ff2b64182fa82afe4b5b40b%2Fthousandeyes-custom-webhook.png?alt=media)

### Step 2: Enable Cisco ThousandEyes Inbound Notifications in Splunk ITSI

1. In Splunk ITSI, go to **Configuration > Data Integrations**.
2. Under **Alerts**, click **Cisco ThousandEyes**.
3. In the connections table, click the **⋮** (more actions) menu for `thousandeyes_default`, then click **Activate**.
4. The connection status should update to **Active**.

## Send Splunk ITSI Episodes to ThousandEyes

### Configure Alert Rule for Aggregation Policies

1. In Splunk ITSI, go to **Configuration > Event Management > Notable Event Aggregation Policies**.
2. For each enabled aggregation policy:
   1. Open the **Action Rules** tab.
   2. Click **+ Add Rule**.
   3. Under **If**, select:
      * **The number of events in this episode is**
      * **Greater than or equal to**
      * **1**
   4. Under **Then**, select:

      * **Send to ThousandEyes**
      * **Repeat every event while episode is active**

      ![Alert Rule for Aggregation Policies](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-c5866a499c643a9b207a55e1c2953afb25f3916d%2Fsplunk-alert-rule-for-aggregation-policies.png?alt=media)
   5. Click **Configure**.
      * Enter the public host URL of your Splunk instance.
      * This URL is prepended to the ITSI episode URI and used to generate a direct link from ThousandEyes to the episode details page.
   6. Find the alert rule for closing episodes: *If the episode is broken, then change status to Closed for the episode ...*. Edit the rule to add an extra action:

      * Select the rule.
      * Click **+Add**.
      * Under **And**, select **Send to ThousandEyes**.
      * Click **Configure** and enter the same public host URL as configured in the previous step.

      ![Alert Rule for closing episode](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-3fdc0657218ce5678002f89ef9c5ae394487f043%2Fsplunk-alert-rule-for-closing-episode.png?alt=media)
3. Click **Done**, then **Save**.

### Configure Search Macros for the Content Pack for ITSI Monitoring and Alerting

If you're using the [Content Pack for ITSI Monitoring and Alerting](https://docs.splunk.com/Documentation/CPITSIMonitorAlert/2.3.0/CP/About), update the search macros to enrich alerts with the ThousandEyes test ID, as described in the [Splunk documentation](https://docs.splunk.com/Documentation/CPITSIMonitorAlert/2.3.0/CP/Install#Enable_enrichment_for_notable_events).

1. From the Splunk Enterprise main menu, go to **Settings > Advanced search > Search macros**.
2. In the **App** filter drop-down, select **IT Service Intelligence (itsi)**.
3. In the **Filter** field, search for the macro `enrich_entity_notables_with_entity_alias_and_info(1)`.
4. Click the macro name to view the full details.
5. In the **Definition** field, uncomment the existing block by doing the following:
   * Remove the first line beginning with `eval spl_comment ...`.
   * On the new first line, remove the leading ` | noop ``` `.
   * On the last line, remove the trailing ` ``` `.
6. After making these changes, the **Definition** should contain:

   ```
   lookup itsi_entities _key as $entity_key$ OUTPUT _itsi_informational_lookups _itsi_identifier_lookups
   | eval _itsi_entity_enrichment_fields=mv_to_json_array(mvappend(mvmap(_itsi_informational_lookups, "entity.info."._itsi_informational_lookups), mvmap(_itsi_identifier_lookups, "entity.alias."._itsi_identifier_lookups)))
   | rex mode=sed field=_itsi_entity_enrichment_fields "s/=/\":\"/g  s/\[/{/g s/\]/}/g"
   | spath input=_itsi_entity_enrichment_fields
   | foreach "entity.alias.*" "entity.info.*" [| eval keep.<<FIELD>> = if(in("<<MATCHSTR>>", `entity_enrichment_fields_list`), '<<FIELD>>', null())]
   | fields - entity.info.* entity.alias.*
   | rename keep.entity.* as entity.*

   | eval spl_comment="After enrichment is done, pull certain CPMA fields automatically from the entity enrichment logic"
   | eval alert_group=coalesce(alert_group, 'entity.info.alert_group')
   ```

   Alternatively, you can replace the entire definition with the snippet above.
7. Click **Save** to apply the changes and return to the **Search macros** page.
8. In the **Filter** field, search for the macro `entity_enrichment_fields_list`.
9. Click the macro name to open it.
10. In the **Definition** field, insert `"thousandeyes_test_id",` between `"alert_group",` and `""`.

    After the update, the **Definition** should contain:

    ```
    "alert_group", "thousandeyes_test_id", ""
    ```
11. Click **Save** to apply the change.

## Visualize ITSI Episodes in ThousandEyes

1. Open a ThousandEyes test impacted by an ITSI episode.
2. In the test timeline, Splunk ITSI episodes appear as yellow swimlane annotations below the primary metric.

   ![Splunk ITSI Episode Visualization](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-d44fff539fca3baa4e95f79f01cdbdf6a2db0c0f%2Fsplunk-itsi-episode-visualization.png?alt=media)
3. Hover over an annotation to see a summary of the number of episodes at that point in time and the total number of notable events across all episodes.
4. Click the **Splunk ITSI** tab to view episode details.

   ![Splunk ITSI Tab](https://1112912342-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M4QARF6s57qxMrOHDTZ%2Fuploads%2Fgit-blob-ce909eb2c9c52d2fc678b84d05058eb99a24e917%2Fsplunk-itsi-tab.png?alt=media)
5. Use the ITSI URL to navigate back to the episode in Splunk.
6. To go back to the episode in Splunk ITSI, click the **ITSI URL**.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.thousandeyes.com/product-documentation/integration-guides/custom-built-integrations/splunk-app/itsi.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.