Troubleshooting

Follow these steps to troubleshoot issues with the Cisco ThousandEyes App for Splunk.

Check Logs

• View log files in the following location: $SPLUNK_HOME/var/log/splunk/*thousandeyes*.log

• Run the following search queries in Splunk to check logs:

  • General logs:

    index="_internal" sourcetype="ciscoThousandEyes:log" source="*thousandeyes*.log"

  • Error logs:

    index="_internal" sourcetype="ciscoThousandEyes:log" source="*thousandeyes*.log" ERROR

• To view detailed logs in Splunk Web UI:

  1. Go to Cisco ThousandEyes App for Splunk > Configuration > Logging.

  2. Set the log level to debug.

  3. Disable and enable the input to recollect data.

  4. Check the logs for detailed information.

Resolve Data Collection Issues

• If data collection is not working:

  • Ensure that the internet connection is active where the input is configured, or verify the proxy configuration (if applicable).

  • Confirm that the kvstore is enabled. You can check its status by running the following command from $SPLUNK_HOME/bin:

    splunk show kvstore-status

    • The output should show no errors, and the status should be Ready.

    • Alternatively, check the Messages section in the Splunk menu bar. If the KV store is not working properly, you may see KV store-related errors there.

  • For Tests Stream Input:

    • Verify that the HTTP Event Collector (HEC) global settings are enabled.

    • Ensure that the HEC token is enabled.

• If data collection is working but dashboards are not populated:

  • Verify that the index macros are configured with the correct index values.

  • Update the macros if necessary to match the input configurations.

Fix Error: "The Server Name, Host Name, and Host is not reachable from Cisco ThousandEyes"

1. Set the Server Name or Host Name correctly:

   1. Navigate to Settings > Server Settings > General Settings.

   2. Under Index Settings, set the Splunk server name or Default host name to the correct value required in the HEC collector URL.

2. Restart Splunk after making these changes.

Workaround for HEC Token Error

If you encounter the following error:

Unexpected error "<class 'Exception'>" from python handler: "Error while fetching HECs: 'token' Please check the logs.". See splunkd.log/python.log for more details.

This error may indicate a misconfigured or missing Splunk HEC token.

To resolve the issue:

• Remove the misconfigured HEC token, or

• Fix the HEC token configuration.

Note: This issue will be resolved in version 0.0.22.

Unsupported Splunk Cloud Platform Classic Experience Version

The Cisco ThousandEyes App for Splunk does not support the Splunk Cloud Platform Classic Experience Version. It is designed to work with the Splunk Cloud Platform Victoria Experience Version.

If you try to create an input, you may encounter the following error:

Unexpected error "<class 'Exception'>" from python handler: "Error while fetching HECs: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/servicesNS/nobody/-/data/inputs/http?output_mode=json&count=0 Please check the logs.". See splunkd.log/python.log for more details

You can still use the dashboards to visualize data by creating the Splunk HEC integration through the ThousandEyes platform or API. Once configured, the data is displayed in the dashboards for the Cisco ThousandEyes App for Splunk. Use the following values when setting up the integration:

• source: cisco:thousandeyes:stream

• sourceType: cisco:thousandeyes:test

Expired Token Error or 401 Authorization Scope Error

You may encounter one of the following issues due to an outdated authorization scope:

• Expired token error: The user is blocked from using the Cisco ThousandEyes App for Splunk.

• 401 error (authorization scope): In version 0.0.25, the user cannot use the Tags feature in Test Stream Inputs.

These errors occur when previously authorized users have an outdated authorization scope that doesn't include permission to use the Tags feature in Test Stream Inputs.

Unexpected error "<class 'Exception'>" from python handler: "Error occurred while fetching tags: Response status code: 401. Please check the logs.". See splunkd.log/python.log for more details.

To resolve these issues, re-authorize the user in the app. This refreshes the token and updates the authorization scope with the required permissions. It also restores the user’s access to the Cisco ThousandEyes App for Splunk.

Refresh the Token and Update the Authorization Scope

1. Update the app to version 0.0.25 or later.

2. Go to the Configuration tab.

3. Click Edit for the affected user.

4. Click Re-Authorize and wait for the authorization flow pop-up in the ThousandEyes app.

5. Complete the authorization flow and wait for it to finish before closing the window.

6. Click Update to refresh the token and update the authorization scope.

You only need to complete these steps once per expired token. Starting with version **0.0.24**, the app automatically refreshes tokens weekly.