Splunk ITSI Integration

This guide explains how to integrate Cisco ThousandEyes with Splunk IT Service Intelligence (ITSI). The integration enables you to ingest test data, send alert notifications, and visualize Splunk episodes directly in ThousandEyes.

Prerequisites

Ensure you have the following components installed:

• Splunk ITSI version 4.20.x or later. For installation instructions, see Splunk ITSI: Install and Upgrade Manual.

• Cisco ThousandEyes App for Splunk version 0.1.0 or later.

• Splunk App for Content Packs version 2.3.0 or later. For installation instructions, see Splunk ITSI: Install the Splunk App for Content Packs.

• Content Pack for ITSI Monitoring and Alerting version 2.3.0 or later. For more information, see the section below.

Install the Content Pack for Cisco ThousandEyes

1. Follow the instructions in Install the Content Pack for Cisco ThousandEyes until you reach the step that says “Follow the on-screen instructions to install the content pack.”

2. Then return to this guide to complete the remaining steps.

3. Click Proceed.

4. Click Import as enabled. Optionally disable backfill if you do not want to import historical data.

5. Click Install selected.

   

   Cisco ThousandEyes Content Pack Installation
6. In the confirmation dialog, click Install.

7. Wait for the installation to complete. The dialog will show the installed KPI base searches, entity types, service templates, and services.

8. Return to the Data Integrations page. The Cisco ThousandEyes content pack should display a green checkmark with the status “All Saved Searches activated.”

9. In ITSI, go to Configuration > Service Monitoring > Service and KPI Management to confirm that the ThousandEyes services are enabled.

Update the Index Used by the Content Pack

If your test stream does not use the thousandeyes index, update the content pack's search macro to match your selected index.

1. In Splunk Enterprise, go to Settings > Advanced search > Search macros.

2. In the App drop-down menu, select Cisco ThousandEyes (DA-ITSI-CP-thousandeyes).

3. In the Filter field, type itsi_cp_thousandeyes_index to locate the macro.

4. Click itsi_cp_thousandeyes_index to open it.

5. In the Definition field, update the value of index="thousandeyes" to match the index used by the Cisco ThousandEyes App for Splunk.

6. Click Save to apply your changes.

Configure the Content Pack for ITSI Monitoring and Alerting

1. In Splunk ITSI, go to Configuration > Data Integrations.

2. Select the Content library tab.

3. Select the ITSI Monitoring and Alerting content pack.

4. Review what's included in the content pack, then click Proceed.

5. Leave the default Add all 45 objects.

6. Click Import as enabled.

7. Optionally backfill your ITSI environment with the previous seven days of KPI data.

8. Click Install selected.

   

   ITSI Monitoring and Alerting Content Pack Installation
9. Click Install.

For more information, see Install and configure the Content Pack for ITSI Monitoring and Alerting.

Stream ThousandEyes Test Data to Splunk ITSI

1. Ingest ThousandEyes network data to Splunk using the Cisco ThousandEyes App for Splunk.

2. Add your ThousandEyes user.

3. Create a test stream (metrics input).

Note: The default expected index is thousandeyes. If your data stream uses a different index, you must update the macro as described in the Update the index section.

ThousandEyes Test Stream Metric Input Creation

Send ThousandEyes Alert Notifications to Splunk ITSI

Step 1: Configure a Custom Webhook in ThousandEyes

Follow the steps in Splunk Alert Notification to create a custom webhook for sending ThousandEyes alerts to Splunk ITSI.

ThousandEyes Custom Webhook Integration

Step 2: Enable Cisco ThousandEyes Inbound Notifications in Splunk ITSI

1. In Splunk ITSI, go to Configuration > Data Integrations.

2. Under Alerts, click Cisco ThousandEyes.

3. In the connections table, click the ⋮ (more actions) menu for thousandeyes_default, then click Activate.

4. The connection status should update to Active.

Send Splunk ITSI Episodes to ThousandEyes

Configure Alert Rule for Aggregation Policies

1. In Splunk ITSI, go to Configuration > Event Management > Notable Event Aggregation Policies.

2. For each enabled aggregation policy:

   1. Open the Action Rules tab.

   2. Click + Add Rule.

   3. Under If, select:

      • The number of events in this episode is

      • Greater than or equal to

      • 1

   4. Under Then, select:

      • Send to ThousandEyes

      • Repeat every event while episode is active

      

      Alert Rule for Aggregation Policies
   5. Click Configure.

      • Enter the public host URL of your Splunk instance.

      • This URL is prepended to the ITSI episode URI and used to generate a direct link from ThousandEyes to the episode details page.

   6. Find the alert rule for closing episodes: If the episode is broken, then change status to Closed for the episode .... Edit the rule to add an extra action:

      • Select the rule.

      • Click +Add.

      • Under And, select Send to ThousandEyes.

      • Click Configure and enter the same public host URL as configured in the previous step.

      

      Alert Rule for closing episode
3. Click Done, then Save.

Configure Search Macros for the Content Pack for ITSI Monitoring and Alerting

If you're using the Content Pack for ITSI Monitoring and Alerting, update the search macros to enrich alerts with the ThousandEyes test ID, as described in the Splunk documentation.

1. From the Splunk Enterprise main menu, go to Settings > Advanced search > Search macros.

2. In the App filter drop-down, select IT Service Intelligence (itsi).

3. In the Filter field, search for the macro enrich_entity_notables_with_entity_alias_and_info(1).

4. Click the macro name to view the full details.

5. In the Definition field, uncomment the existing block by doing the following:

   • Remove the first line beginning with eval spl_comment ....

   • On the new first line, remove the leading | noop ```.

   • On the last line, remove the trailing ```.

6. After making these changes, the Definition should contain:

   lookup itsi_entities _key as $entity_key$ OUTPUT _itsi_informational_lookups _itsi_identifier_lookups
   | eval _itsi_entity_enrichment_fields=mv_to_json_array(mvappend(mvmap(_itsi_informational_lookups, "entity.info."._itsi_informational_lookups), mvmap(_itsi_identifier_lookups, "entity.alias."._itsi_identifier_lookups)))
   | rex mode=sed field=_itsi_entity_enrichment_fields "s/=/\":\"/g  s/\[/{/g s/\]/}/g"
   | spath input=_itsi_entity_enrichment_fields
   | foreach "entity.alias.*" "entity.info.*" [| eval keep.<<FIELD>> = if(in("<<MATCHSTR>>", `entity_enrichment_fields_list`), '<<FIELD>>', null())]
   | fields - entity.info.* entity.alias.*
   | rename keep.entity.* as entity.*
   
   | eval spl_comment="After enrichment is done, pull certain CPMA fields automatically from the entity enrichment logic"
   | eval alert_group=coalesce(alert_group, 'entity.info.alert_group')

   Alternatively, you can replace the entire definition with the snippet above.

7. Click Save to apply the changes and return to the Search macros page.

8. In the Filter field, search for the macro entity_enrichment_fields_list.

9. Click the macro name to open it.

10. In the Definition field, insert "thousandeyes_test_id", between "alert_group", and "".

    After the update, the Definition should contain:

    "alert_group", "thousandeyes_test_id", ""

11. Click Save to apply the change.

Visualize ITSI Episodes in ThousandEyes

1. Open a ThousandEyes test impacted by an ITSI episode.

2. In the test timeline, Splunk ITSI episodes appear as yellow swimlane annotations below the primary metric.

   

   Splunk ITSI Episode Visualization
3. Hover over an annotation to see a summary of the number of episodes at that point in time and the total number of notable events across all episodes.

4. Click the Splunk ITSI tab to view episode details.

   

   Splunk ITSI Tab
5. Use the ITSI URL to navigate back to the episode in Splunk.

6. To go back to the episode in Splunk ITSI, click the ITSI URL.