Troubleshooting

Follow these steps to troubleshoot issues with the Cisco ThousandEyes App for Splunk.

Check Logs

• View log files in the following location: $SPLUNK_HOME/var/log/splunk/*thousandeyes*.log

• Run the following search queries in Splunk to check logs:

  • General logs:

    index="_internal" sourcetype="ciscoThousandEyes:log" source="*thousandeyes*.log"

  • Error logs:

    index="_internal" sourcetype="ciscoThousandEyes:log" source="*thousandeyes*.log" ERROR

• To view detailed logs in Splunk Web UI:

  1. Go to Cisco ThousandEyes App for Splunk > Configuration > Logging.

  2. Set the log level to debug.

  3. Disable and enable the input to recollect data.

  4. Check the logs for detailed information.

Resolve Data Collection Issues

• If data collection is not working:

  • Ensure that the internet connection is active where the input is configured, or verify the proxy configuration (if applicable).

  • Confirm that the kvstore is enabled. You can check its status by running the following command from $SPLUNK_HOME/bin:

    splunk show kvstore-status

    • The output should show no errors, and the status should be Ready.

    • Alternatively, check the Messages section in the Splunk menu bar. If the KV store is not working properly, you may see KV store-related errors there.

  • For Tests Stream Input:

    • Verify that the HTTP Event Collector (HEC) global settings are enabled.

    • Ensure that the HEC token is enabled.

• If data collection is working but dashboards are not populated:

  • Verify that the index macros are configured with the correct index values.

  • Update the macros if necessary to match the input configurations.

Fix Error: "The Server Name, Host Name, and Host is not reachable from Cisco ThousandEyes"

1. Set the Server Name or Host Name correctly:

   1. Navigate to Settings > Server Settings > General Settings.

   2. Under Index Settings, set the Splunk server name or Default host name to the correct value required in the HEC collector URL.

2. Restart Splunk after making these changes.

Fix Issues with Custom Proxy Certificates

If the configured proxy uses a custom certificate, add the certificate to $SPLUNK_HOME/etc/apps/ta_cisco_thousandeyes/lib/certifi/cacert.pem.

Run Splunk Enterprise Locally Without Public Exposure

If you are running Splunk Enterprise locally and cannot expose the HTTP Event Collector (HEC) endpoint publicly, you must proxy data from ThousandEyes to your local Splunk Enterprise instance. You can achieve this by deploying an otel-collector as an intermediary.

Since the Cisco ThousandEyes App for Splunk doesn't allow specifying a custom endpoint when creating an integration, you must configure this in the ThousandEyes platform or via the ThousandEyes API. For more information, see Configuring ThousandEyes for Splunk Cloud or Enterprise.

When configuring the integration, ensure the following values are set to correctly visualize ThousandEyes data in the Cisco ThousandEyes App for Splunk dashboards:

• source: cisco:thousandeyes:stream"

• sourceType: cisco:thousandeyes:te

By following these steps, ThousandEyes data will stream into your local Splunk Enterprise instance without exposing the HEC endpoint publicly.

Workaround for HEC Token Error

If you encounter the following error:

Unexpected error "<class 'Exception'>" from python handler: "Error while fetching HECs: 'token' Please check the logs.". See splunkd.log/python.log for more details.

This error may indicate a misconfigured or missing Splunk HEC token.

To resolve the issue:

• Remove the misconfigured HEC token, or

• Fix the HEC token configuration.

Note: This issue will be resolved in version 0.0.22.