> For the complete documentation index, see [llms.txt](https://docs.thousandeyes.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thousandeyes.com/product-documentation/internet-and-wan-monitoring/troubleshooting/palo-alto-firewalls-break-agent-to-agent-clock-synchronization.md).

# Palo Alto Firewalls Break Agent-to-Agent Clock Synchronization

When a ThousandEyes agent-to-agent test fails to complete or does not display one-way network metrics, such as latency and jitter, even though the underlying network path is available, the cause might be a stateful firewall, such as a Palo Alto Networks firewall, dropping or modifying the necessary test packets.

Agent-to-agent tests rely on a [clock synchronization preamble of five packets in each direction](https://docs.thousandeyes.com/product-documentation/tests/network-tests/network-tests-explained#agent-to-agent-measurements) before sending the actual probe packets. This synchronization is required to report accurate one-way latency and jitter. However, deep packet inspection (DPI) mechanisms, such as Palo Alto's App-ID, often intercept these packets. Because the ThousandEyes clock synchronization preamble does not match a known application signature, the firewall might strip the data payload from these initial packets, causing the clock synchronization to fail.

## Identifying This Issue

In the ThousandEyes **Network** view for an affected agent-to-agent test, impacted source or target agents can show a warning similar to `source/target warning: clock synchronization failed` or `clock synchronization failed`.

![Clock synchronization failed warning in the measurements panel](/files/CLiS6kOXqRuw8GcRCuhj)

Use the following signals together:

* The test path remains reachable, but the test does not complete clock synchronization.
* One-way latency and jitter might be unavailable while loss can still appear valid.
* Affected rounds coincide with a Palo Alto firewall between the source and target agents, or between an Enterprise Agent and Cloud Agent.

The agent details can also show source and target clock synchronization warnings for the same affected round:

![Agent details showing source and target clock synchronization failed warnings](/files/zGnr3kK1qPvyUSuOqvgH)

After you identify this warning, use the section that matches your role. Support teams can collect the escalation details needed by the network or firewall owner. Network and firewall teams can validate the packet behavior and update the Palo Alto policy.

## Collecting Escalation Details as a Support Team

Use this checklist if you are gathering evidence for a network or firewall owner but do not own the Palo Alto configuration:

* Capture the warning from the affected test round.
* Record the test name, round timestamp, direction, protocol, source agent, target agent, and server port. For tests that use NAT traversal, also record the public port discovered during NAT traversal.
* Confirm whether a Palo Alto firewall is between the source and target agents, or between an Enterprise Agent and Cloud Agent.
* Note whether an Application Override policy already exists for the agent-to-agent traffic, including the custom application it uses.
* Include any available packet captures, but do not block escalation if you cannot collect captures from both sides of the firewall.

## Reviewing Packet Capture Evidence as a Network or Firewall Owner

Use this section if you can collect packet captures, review firewall policy, or validate that Palo Alto inspection is changing the packets. To confirm that a firewall is stripping or dropping the clock synchronization payload, compare packet captures from the same failed test round on both sides of the Palo Alto firewall. Use the agent IP addresses and the server port configured in the agent-to-agent test. For tests that use NAT traversal, use the public port discovered during NAT traversal.

For tests that include a Cloud Agent, you cannot collect packet captures directly from the Cloud Agent side. If you need Cloud Agent capture examples or help validating the packet-capture evidence, contact [support](https://docs.thousandeyes.com/product-documentation/getting-started/getting-support-from-thousandeyes).

In Wireshark, filter the capture to the two agents and the test port, for example:

```
ip.addr == <source-agent-ip> && ip.addr == <target-agent-ip> && (tcp.port == <server-port> || udp.port == <server-port>)
```

Compare the source-side and destination-side captures:

* **Source-side capture:** The initial clock synchronization packets leave the source agent with a data payload.
* **Destination-side capture:** The matching packets arrive with the payload missing or shortened, or they do not arrive before the `clock synchronization failed` warning appears.
* **Post-remediation capture:** After the Palo Alto Application Override policy is applied, the packet lengths and payload presence should match on both sides of the firewall, and the clock synchronization warning should clear.

In the following failed packet capture example, one side shows the expected clock synchronization payload and the other side shows the matching packet without payload:

![Failed packet capture example showing missing clock synchronization payload](/files/b6Ig2mZMpuV3q9kiZTTQ)

The following failed packet capture example also shows a payload mismatch between the two sides of the firewall. The packet with `Len: 0` has no data payload, while the corresponding packet on the other side has `Len: 30` and displays `Data (30 bytes)`:

![Failed packet capture example comparing packets with and without payload](/files/UMjq2QP81pzZ6kZq0F8K)

By contrast, a healthy or remediated packet capture shows the clock synchronization payload on both sides:

![Healthy packet capture example showing payload on both sides](/files/2NQrlYNtlqUhyF04tYhG)

## Reasons for Clock Synchronization Failure

As outlined in [Network Tests, Explained](https://docs.thousandeyes.com/product-documentation/tests/network-tests/network-tests-explained#agent-to-agent-measurements), the source and target agents exchange an initial set of five packets in each direction to synchronize their internal clocks. After they account for the clock offset and drift, the agents send the actual probe packets.

When a Palo Alto firewall is positioned between the two Enterprise Agents (or between an Enterprise Agent and a Cloud Agent), it performs deep packet inspection on the data stream to identify the application. Because the ThousandEyes agent-to-agent clock synchronization packets contain proprietary timing payloads, the firewall might fail to identify the application and either drop the packets or strip the payload entirely.

Without the payload data, the receiving ThousandEyes agent cannot determine the clock offset. Consequently, the agents cannot measure one-way latency or jitter, and the test fails to complete the synchronization phase.

## Remediation

To prevent the Palo Alto firewall from modifying or dropping the agent-to-agent clock synchronization packets, you must configure the firewall to bypass App-ID inspection for ThousandEyes test traffic.

Create an **Application Override** policy in PAN-OS that matches the ThousandEyes agent-to-agent traffic, using the specific TCP/UDP ports configured in the test settings. For tests that use NAT traversal, match the public port discovered during NAT traversal instead of only the configured server port. If you use a custom application object for the traffic, attach it to the Application Override policy so the matched sessions bypass App-ID and layer 7 inspection.

For detailed instructions on configuring your Palo Alto firewall to use Application Override and custom applications, see the following Palo Alto Networks documentation:

* [Palo Alto Knowledge Base: Application Override](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClVLCA0)
* [PAN-OS Administrator's Guide: Create a Custom Application](https://docs.paloaltonetworks.com/ngfw/administration/app-id/use-application-objects-in-policy/create-a-custom-application)

For more information about NAT traversal port behavior, see [NAT Traversal for Agent-to-Agent Tests](https://docs.thousandeyes.com/product-documentation/global-vantage-points/enterprise-agents/configuring/nat-traversal-for-agent-to-agent-tests).

When creating the policy, make sure the policy matches the IP addresses of the source and target ThousandEyes agents and the destination port configured in your agent-to-agent test settings, or the discovered public port for tests that use NAT traversal. By allowing this traffic to bypass deep packet inspection, the clock synchronization preamble of five packets in each direction can complete successfully, and the agent-to-agent test can calculate accurate one-way metrics.

## Verifying the Fix

After the Palo Alto policy is applied, verify the following:

* The affected agent-to-agent test no longer reports `clock synchronization failed` warnings for the source or target agent.
* One-way latency and jitter are available again for affected source-to-target or target-to-source measurements.
* If you collect packet captures, the clock synchronization packets retain their data payload on both sides of the Palo Alto firewall.
* The Palo Alto policy matches the expected agent IP addresses, protocol, and server port for the agent-to-agent test. For tests that use NAT traversal, the policy also matches the discovered public port.