Monitoring Webex Meetings with Endpoint Agents
Summary
The ThousandEyes - Webex collaboration provides visibility into the network health of Webex meetings, utilizing ThousandEyes Endpoint Agents to monitor the Webex experience and provide insights into call quality.
In this article, we’ll look at how to monitor Webex meetings by leveraging Endpoint Agents, Webex Cloud Agents, and the Webex Control Hub.
Recommended Steps
Deploy ThousandEyes Endpoint Agents.
Understand Webex architecture.
Monitor Webex Web Zone with HTTP tests.
Monitor Webex Meeting Zone with automated session tests (AST).
Integrate with Webex Control Hub for advanced troubleshooting practices.
Exclude any local connection issues.
Use Webex Cloud Agents for monitoring UDP traffic.
Deploy Endpoint Agents
For leveraging the endpoint monitoring, we recommend installing Endpoint Agents for all Webex users. ThousandEyes offers two ways to deploy Endpoint Agents.
The first option, download the installer msi or zip file, and use a MDM policy or Windows Group Policy for multi-deployments. This solution is recommended for installation on IT-managed devices.
The second option, provide the download link to anyone with devices not managed by the IT team. The download link contains your Account Group token, which would automatically register the installed Agents to your Account Group.
Any new Endpoint Agents will consume an Agent license. So you’ll want to ensure you have unused licenses in your Account. The download link is valid provided the “Allow anyone with the link to download” option is enabled. To avoid over-subscription of licenses, limiting the users who have access to the link is recommended. In case you would like to limit access to this link, use “Generate new links” to refresh the link.
Once the Agent installation is complete, you will see the new Endpoint Agents under https://app.thousandeyes.com/endpoint/agent-settings/?section=agents including their associated status.
Understand Webex Server Structure
Now that you have deployed Endpoint Agents to the end user's hosts, you can configure Webex performance monitoring. To plan the best tests, let’s look into the Webex meeting structure first.
Webex is a cloud-based service hosted in various Cisco-owned data centers and cloud service providers worldwide. As components are regionally hosted, this adds extra challenges to troubleshooting specific to Webex components.
The Webex data center is logically divided into the Meeting Zone and the Web Zone. The Web Zone is responsible for what happens before and after a web meeting, along with scheduling, user management, billing, reporting, and streaming recordings. The Meeting Zone is responsible for switching the actual meeting once it is in progress between the endpoints and consists of two subsystems; the Collaboration Bridge (CB) and the Multimedia Platform (MMP).
Monitor Webex Web Zone
When users click “Join the meeting” link of Webex, their browser launches to the specific Webex meeting page that indicates “Starting the meeting”. The meeting URL usually consists of {instance}.Webex.com/{meetingIdentifier}.
At this stage, the client connects to the nearest configured “Web Zone” Data Center which is responsible for what happens before and after a web meeting. The Web Zone for an instance is always located in the primary data center, no matter where the client connects from, the client will connect to the same server. If the Web Zone is non-functional or inaccessible, the Webex meeting will not be able to launch and the below error message will pop up.
In other words, the availability of Web Zone decides whether the client can launch the meeting, schedule the meeting, or access the Webex instance. Monitoring Webex Web Zone is very straightforward as every Webex instance uses the unique CNAME: {instance}.Webex.com, i.e. cisco.webex.com. Simply create a scheduled HTTP server test targeting https://cisco.Webex.com with Prefer TCP mode (SACK is recommended, see the explanation in this guide).
Webex Web Zone monitoring provides visibility into issues associated with the session initiation, such as ‘why my Webex meeting doesn’t connect?’, but not about the bridge quality.
Monitor Webex Meeting Zone
When you schedule a Webex meeting, session initiation traffic is sent to the Web Zone, but when a call is in progress, audio, and video traffic is directed to the collaboration, control and media servers in a completely different location. Meeting Zone data center hosting the ‘Collaboration bridge’ and ‘Multimedia Platform’ servers which are located globally and usually will connect to the one that is closest to the end user.
If you want to monitor Audio/Video traffic, you need to identify the MMP server. So how can this be accomplished? Unlike the Web Zone's data centers, the Media Zone's servers are hidden from the end users. Running a packet capture during Webex meetings, the destination address for “UDP” traffic will show the MMP server that serves the audio/video traffic. As you can see, Webex Client uses the random port for the source UDP port, and 9000 or 5004 for the destination UDP port.
Based on the above packet capture, you can see 64.68.96.110 is the target MMP Webex server for this client's session. Though, troubleshooting using packet captures for every Webex session is not ideal for the lean IT team, the MMP servers are dynamic and could change depending on the end-user connection ie. VPN. Endpoint Dynamic Tests extended the ability to monitor this ‘moving target’ based on the actual audio/video traffic.
Create Dynamic Test under Endpoint Agent following this guide. To achieve the best outcomes, we recommend using the below configurations.
Select “Webex” for Target
Select “Auto-detect” protocol to allow Endpoint Agents to automatically select best protocol for monitoring. Note: As UDP is not supported, TCP or ICMP would be selected.
Select “1 minute” interval to ensure the meeting sessions are consistently monitored.
Select “All Agents”.
Select the number of Agents that cover all of the end-users using the Webex client.
Using this test configuration the Endpoint Agent will automatically detect the MMP server for the active Webex meeting and execute the test for the same targets in parallel. When the meeting concludes the tests will be shut down. If the Endpoint’s host IP address changes, the Webex client changes the connection to the new server geographically assigned, then the test will also automatically change the target to the new IP address. In the below example, the Webex Client connected to 4 Webex servers on this round, and one of the paths clearly shows the forwarding loss.
Currently, Dynamic Test only can be executed using TCP or ICMP, thus the end-to-end loss doesn't exactly correspond to the meeting’s audio/video quality but latency/jitter metrics are shown inline. The most useful information from Dynamic Test is the network path that shows the node causing the loss as well as you can tell how many Endpoints have been affected by this node/hop.
Integrate with Webex Control Hub Troubleshooting
If you are the administrator of your Webex Organization instance, you can monitor the active or past meetings performance via Control Hub Troubleshooting tool (https://admin.Webex.com/diagnostic/meeting). Endpoint Agent Dynamic Test data is integrated into Webex Control Hub to provide insights into the network path data. Refer to Webex Control Hub Integration to set it up.
Next, locate the meeting that you want to troubleshoot from either the “Live Meetings” tab or searching by Conference ID or Username under the “Meetings & Calls” tab.
From the meeting summary view, you can see each participant’s Audio / Video / Sharing’s Signal qualities in the timeline. Drilling down, if you click each participant’s tab, you can find the full metrics collected by Webex clients. If an Endpoint Agent was running on the same host, you will find the “Network Path” data, which was collected by the Endpoint Agent using AST.
The recommended steps to use this tool for troubleshooting is:
Spot dropped signals from the meeting view
Select the specific participants
Find the bad quality round, and click on Network Path
Find the network causing the forwarding loss
Please note this integration is limited to the voice over IP encoded sessions only. Traditional PSTN encoded voice won't show Network Path data.
Exclude Local Connection Issues
While Network path visualization is very helpful for diagnosing the issues on the ‘route’, quite often the local connection is the bottleneck. Wireless signal, gateway loss, VPN loss can contribute to the poor quality of a user’s meetings and it’s hard for a network test to isolate the issue. We recommend also leveraging the Agent View of the Endpoint Agent where you can correlate test data with the local host metrics in the same timeline. Please refer to Endpoint Agent Views for the navigation path and more detailed information. In the below case it’s clear that VPN loss is the culprit.
Use Webex Cloud Agents for Monitoring UDP Traffic
Since Endpoint Agents are limited to TCP or ICMP monitoring, Enterprise and Cloud Agents allow UDP traffic monitoring through Agent to Agent tests. Moreover, ThousandEyes deployed Cloud Agents in the Webex Data Centers solely for monitoring the traffic into the Webex infrastructure. Once you find the target MMP server that serves your Webex clients, you can set up a UDP test from Enterprise Agent from the same network toward the Webex Cloud Agents located in the same data center.
In AST’s Path Visualization view, you can find the destination server’s location. In the below example, the next step will be testing to ‘Dallas Webex Cloud Agent’.
Please refer to Monitoring Webex Meetings with Cloud and Enterprise Agents.
Last updated
