Troubleshooting

Follow these steps to troubleshoot issues with the Cisco ThousandEyes App for Splunk.

Check Logs

• View log files in the following location: $SPLUNK_HOME/var/log/splunk/*thousandeyes*.log

• Run the following search queries in Splunk to check logs:

  • General logs:

    index="_internal" sourcetype="ciscoThousandEyes:log" source="*thousandeyes*.log"

  • Error logs:

    index="_internal" sourcetype="ciscoThousandEyes:log" source="*thousandeyes*.log" ERROR

• To view detailed logs in Splunk Web UI:

  1. Go to Cisco ThousandEyes App for Splunk > Configuration > Logging.

  2. Set the log level to debug.

  3. Disable and enable the input to recollect data.

  4. Check the logs for detailed information.

Resolve Data Collection Issues

• If data collection is not working:

  • Ensure that the internet connection is active where the input is configured, or verify the proxy configuration (if applicable).

  • Confirm that the kvstore is enabled. You can check its status by running the following command from $SPLUNK_HOME/bin:

    splunk show kvstore-status

    • The output should show no errors, and the status should be Ready.

    • Alternatively, check the Messages section in the Splunk menu bar. If the KV store is not working properly, you may see KV store-related errors there.

  • For Tests Stream Input:

    • Verify that the HTTP Event Collector (HEC) global settings are enabled.

    • Ensure that the HEC token is enabled.

• If data collection is working but dashboards are not populated:

  • Verify that the index macros are configured with the correct index values.

  • Update the macros if necessary to match the input configurations.

Fix Error: "The Server Name, Host Name, and Host is not reachable from Cisco ThousandEyes"

1. Set the Server Name or Host Name correctly:

   1. Navigate to Settings > Server Settings > General Settings.

   2. Under Index Settings, set the Splunk server name or Default host name to the correct value required in the HEC collector URL.

2. Restart Splunk after making these changes.

Fix Issues with Custom Proxy Certificates

If the configured proxy uses a custom certificate, add the certificate to $SPLUNK_HOME/etc/apps/ta_cisco_thousandeyes/lib/certifi/cacert.pem.

Run Splunk Enterprise Locally Without Public Exposure

If you are running Splunk Enterprise locally and cannot expose the HTTP Event Collector (HEC) endpoint publicly, you must proxy data from ThousandEyes to your local Splunk Enterprise instance. You can achieve this by deploying an otel-collector as an intermediary.

Since the Cisco ThousandEyes App for Splunk doesn't allow specifying a custom endpoint when creating an integration, you must configure this in the ThousandEyes platform or via the ThousandEyes API. For more information, see Configuring ThousandEyes for Splunk Cloud or Enterprise.

When configuring the integration, ensure the following values are set to correctly visualize ThousandEyes data in the Cisco ThousandEyes App for Splunk dashboards:

• source: cisco:thousandeyes:stream"

• sourceType: cisco:thousandeyes:test

By following these steps, ThousandEyes data will stream into your local Splunk Enterprise instance without exposing the HEC endpoint publicly.

Workaround for HEC Token Error

If you encounter the following error:

Unexpected error "<class 'Exception'>" from python handler: "Error while fetching HECs: 'token' Please check the logs.". See splunkd.log/python.log for more details.

This error may indicate a misconfigured or missing Splunk HEC token.

To resolve the issue:

• Remove the misconfigured HEC token, or

• Fix the HEC token configuration.

Note: This issue will be resolved in version 0.0.22.

Unsupported Splunk Cloud Platform Classic Experience Version

The Cisco ThousandEyes App for Splunk does not support the Splunk Cloud Platform Classic Experience Version. It is designed to work with the Splunk Cloud Platform Victoria Experience Version.

If you try to create an input, you may encounter the following error:

Unexpected error "<class 'Exception'>" from python handler: "Error while fetching HECs: [HTTP 403] Client is not authorized to perform requested action; https://127.0.0.1:8089/servicesNS/nobody/-/data/inputs/http?output_mode=json&count=0 Please check the logs.". See splunkd.log/python.log for more details

You can still use the dashboards to visualize data by creating the Splunk HEC integration through the ThousandEyes platform or API. Once configured, the data is displayed in the dashboards for the Cisco ThousandEyes App for Splunk. Use the following values when setting up the integration:

• source: cisco:thousandeyes:stream

• sourceType: cisco:thousandeyes:test