Speed Tests
Speed is the primary measure by which broadband connections are sold. We offer a variety of different mechanisms to measure broadband speed.
Supported Tests
TCP speed tests: our standard speed tests.
Hardware-accelerated UDP speed tests: these speed tests can take advantage of specific Broadcom and Qualcomm hardware to measure extremely high speeds.
Lightweight capacity tests (UDP): alternative speed tests that are ultra lightweight in terms of bandwidth usage and time.
TCP speed tests are available on all Device Agents including agents installed on routers and on phones running Android or iOS. All UDP speed tests are only available on Device Agents within routers.
TCP Speed Tests
TCP (Transmission Control Protocol) speed tests measure the download and upload speed of the broadband connection in bits per second. The transfer is conducted over one or more concurrent HTTP connections (using the GET method for download and the POST method for uploads).
In the download speed test, the agent fetches a portion of an infinitely sized binary (non-zero, randomly generated) payload hosted on an HTTP server on the target test server. The content is discarded as soon as it is received.
In the upload speed test, the agent generates the payload itself (using “/dev/random” as a non-blocking source of random content) to send to the server. The upload speed is measured at the server side (the receiver).
TCP Test Methodology
Both download and upload speed tests operate for a fixed duration by default. We recommend a duration of five seconds. Optionally, we can apply a transfer size limit (specified in MB). We recommend a fixed-duration test as it will cater well for all broadband access speeds. However, you may prefer a fixed-volume test to measure bandwidth usage. Speak to your account manager to make any of these adjustments to your speed tests.
We support four separate variants of the test:
Single TCP connection download speed test.
Multiple TCP connection download speed test.
Single TCP connection upload speed test.
Multiple TCP connection upload speed test.
For multiple TCP connection tests, we typically recommend using eight concurrent connections. In some cases (e.g., where the product of bandwidth and delay between agent and server is very high), it may be necessary to increase the number of concurrent connections.
The HTTP test can be configured to force the use of the ‘cubic’ TCP congestion control algorithm (which better represents real-world conditions) when testing against our HTTP server application, and this is our default.
Factors such as TCP slow-start are accounted for through the use of a “warm-up” period. This period begins as soon as the test starts and seeks to establish that the throughput has reached a stable rate before starting the real test (which will continue over the same TCP connection(s)). The warm-up period ends after a minimum period (default 1.5 seconds) when either a stable rate is achieved, or when a preconfigured time limit (default 5 seconds) is hit. It is important to note that we exclude the data transferred in the warm-up period from the main test results, but we still record it separately as a supplementary metric that we can make available to you on request.
TCP Test Metrics
The TCP speed test client captures the following key metrics:
The speed, in bytes per second.
The amount of data transferred.
The duration of the test.
Details about the warm-up period.
The hostname and IP address of the test server.
The number of TCP retransmissions that occurred during the test.
The agent may also record these values at multiple intervals during the test. This is commonly used to help characterize the difference between 'burst' and 'sustained' throughput (where transfer speeds may be inflated at the start of a TCP connection). Speed is measured at the application layer, so on a 100 Mbit/s physical link the maximum measurable speed is approximately 95 Mbit/s, due to packet overheads.
Finally, our TCP speed test makes use of hardware-specific functionality on some hardware devices. This is typically used on devices where the CPU is insufficiently powerful to saturate the WAN link in software mode alone. The nature of these implementations varies; sometimes it uses a dedicated hardware co-processor, and sometimes it uses a special kernel driver to avoid unnecessary copying data. An example of this is where we have integrated with Broadcom's "BSPEED" kernel driver in their DOCSIS chipsets and SPDT (single pole double throw) accelerators, as well as Airoha's TC3162 accelerator.
For TCP speed test configuration options, see TCP/UDP Speed Test Configuration Options, below.
TCP Test Example
We noticed Fibre Max underperforming in our study Measuring Broadband New Zealand. After an investigation the cause was located and a fix rolled out. In the chart below, our TCP speed test shows a significant improvement in average download speed after 4 September for RSP X. Average speeds go from under 750Mbps to over 900Mbps which is in-line with what we would expect to see on Fibre Max lines. RSP Y made their changes on 19 October; a similar step change in performance can be seen following these changes.
This chart shows the average download speed from two anonymized ISPs, X and Y, which made changes to their network following our Fibre Max investigation in New Zealand.
UDP Speed Tests
The UDP (User Datagram Protocol) speed tests measure downstream and upstream throughput at the application layer using UDP.
This tests support both software- and hardware-accelerated modes of operation. In the software-accelerated mode, the traffic is generated and measured using the main CPU in the device, in userspace. In hardware-accelerated mode, the test makes use of platform-specific hardware acceleration features to generate or receive UDP packets at a far higher speed than the main CPU can support.
Hardware acceleration is currently supported on certain generations of Broadcom and Qualcomm SoCs (systems-on-chip). The measurement traffic is not directly handled by the main CPU, and instead uses a separate network co-processor. For Broadcom, measurements of up to 10 Gbps are attainable using the licensed 'speed service' API functionality from Broadcom. On Qualcomm, measurements of up to 10 Gbps are also attainable using their Nanalog Streaming Service (NSS) UDP APIs. We have performed multiple integrations with each of these implementations.
We recommend avoiding using hardware acceleration for speed tests, unless unavoidable, because these features may be subject to additional licensing fees (as with Broadcom) or may make troubleshooting more difficult.
UDP Speed Test Methodology
Regardless of whether hardware acceleration is used or not, the measurement operates in the same way. For a download test, the agent issues a download test request to our test server over a TCP-based control channel, requesting that the server begin transmitting UDP test traffic back to the agent at a fixed rate. The agent then begins to receive the measurement packets, and assesses the throughput received. If the throughput is at least 80% of the requested rate, then the agent requests that the server increase its sending rate, and the process restarts. Eventually, the agent will stop requesting increases in transmission rates by the server, and the agent will measure the received rate at the current transmission rate. This process is controlled by a 'speed ladder', which we can configure for you.
For an upload test, the UDP speed test client issues a notification to the test server over TCP, informing it that transmission of UDP test traffic is about to commence. The agent then begins transmitting measurement packets. Packets are transmitted at full-line rate immediately and for the entire duration of the test. The packet size can be configured, and is 1400 bytes by default.
In both the upstream and downstream directions, the throughput is measured over a five-second period by default (excluding any ladder ramp-up time). The speed is always measured at the receiver side.
Finally, there is a ‘dual-channel’ extension to the hardware-accelerated mode of transmission. The dual-channel approach uses a software-based UDP measurement stream in parallel with the hardware-accelerated UDP stream. This mode is to cater specifically for certain legacy hardware that has the main CPU connected via a 1 Gbps internal interface, and the network co-processor connected via another 1 Gbps internal interface. Used individually, measurements could never exceed 1 Gbps, even on a >1 Gbps WAN link. However, when used in parallel, measurements over 1 Gbps can be achieved.
UDP Speed Test Metrics
The following key metrics are captured by the UDP test for both downstream and upstream:
Measured throughput.
Transmission rate.
Transmission bytes.
Transmission duration.
Server hostname and IP.
Packet size.
TCP/UDP Speed Test Configuration Options
Customers can configure their own TCP and UDP speed tests via Connected Devices > Settings > Test Settings. Configuration options are not currently available for Lightweight Capacity Speed Tests (UDP).
Basic Settings
The following basic configuration options apply to both TCP and UDP speed tests, and to both upload and download directions.
As well as the settings detailed in Configuring Common Test Settings, speed tests include the following configuration options:
Protocol: Defaults to TCP but you can choose UDP as well.
Target: Choose from Single Target or Target Set.
You can also choose to run tests via a regular schedule, or only on demand. Defaults to "Scheduled".
There is no default Interval setting; you must configure it.
Advanced Settings
TCP Upload and Download
IP Version
Defaults to "Prefer IPv6 (auto-detect)”. Choose also “IPv4 only” or “IPv6 only”.
Network
Transfer timeout: The network transfer timeout for the time the socket is open and throughput is being generated. Defaults to 10,000 ms, with a slider from 0-10,000 ms.
Connection count (threads): Defaults to 8. Choose also from 1-256.
Some devices may only support one connection.
Port: Defaults to 6500.
Measurement
Measurement duration: Defaults to 5,000 ms. Choose also from 3,000-60,000 ms.
Enforce max allowed bytes transferred during measurement period: Defaults to disabled. If enabled, the following field appears:
Max bytes transferred during measurement period: Defaults to empty. Choose from a range of 500-5,000,000 KB.
Warmup
Duration of each warmup sample: Ensure throughput has reached a stable rate prior to taking the full measurement with a warmup period, accounting for TCP slow-start. Defaults to 500 ms. Choose also from 250-5,000 ms.
Min warmup samples: Defaults to 3. Choose also from 1-30.
Total minimum warmup time: Sample warmup duration * minimum warmup samples.
Max warmup samples: Defaults to 10; Choose also from 1-30.
Total maximum warm up time: Sample warmup duration * maximum warmup samples.
Stop warmup if transfer size exceeds limit: Defaults to Disabled.
Transfer size limit (KB): Defaults to 100 KB. Choose from 100-1,000,000 KB.
Other
Request latency under load traffic during test: If an agent is configured to run the continuous UDP latency and loss test, this setting allows that test to continue to run in the background while a speed test is also executing. Latency results obtained during speed tests are then reported separately to the standard UDP latency and loss measurements. Defaults to Enabled.
Prefer cubic for TCP congestion control algorithm: Defaults to Enabled. If disabled, will revert to device and server default, e.g., TCP Reno.
CPE hardware acceleration: Defaults to Automatic, which means the test uses hardware acceleration for achieving maximum throughput if available. If disabled, then agents will only run in software mode.
Agent Testing Thresholds
Cross-traffic, downlink/uplink: Defaults to 25,000 bytes per second down/up.
CPU usage: Defaults to 30%. Range is from 1-100%.
UDP Upload and Download
Network
TCP port: Used for setting up the test connection. Defaults to empty.
UDP port: Used for the actual measurements where traffic is passed. Defaults to empty.
Measurement
Measurement duration: Defaults to 5,000 ms. Choose also from 250-20,000 ms.
Payload size (bytes): Defaults to 1,400 bytes. Choose also from 1,000-1,472 bytes.
Agent Testing Thresholds
Cross-traffic, downlink/uplink: Defaults to 25,000 bytes per second down/up.
CPU usage: Defaults to 30%. Range is from 1-100%.
UDP Download Only
Warmup
Pre-test speed ladder schedule: Defaults to empty. To create your own schedule, the field accepts:
A comma-separated list of speeds increasing in value. Add 2-16 speed values. Units are Mbps.
Optionally, set the duration for each test. Defaults to 1 s. We support seconds (s), milliseconds (ms) and microseconds (µs) as units. If you do not specify a unit, the default unit is seconds. Range is from 500 ms–5 s.
For example, you might create a ladder with the following speeds and 500 ms duration: 800:500ms, 900:500ms, 920:500ms, 960:500ms, 1000:500ms, 1200:500ms.
Note - The agent device integration has a max speed limit set at build-time, so will not test beyond this limit in the ramp up.
Speed ladder deviation threshold (%): Defaults to 20%. Choose also from 10-30%. Speed deviation threshold is used in two situations:
As a part of the regular ramp-up, when moving up the ladder schedule. If the throughput is within the speed ladder deviation threshold, e.g., 80% of the requested rate and 20% deviation, then the Device Agent requests that the server increase its sending rate to the next ladder speed. Once the warmup reaches either the top speed set in the ladder or gets to a speed on the ladder that it can't perform within the deviation threshold, e.g., it can only get 60% of the target speed, then the speed ladder warmup stops and the real tests begins.
To validate the cached speed ladder value: a quick test checks if it matches the current reality. A full ramp-up is forced if it deviates above the threshold.
Speed ladder cache: Only runs speed detection ladder once per week, using cache of peak speed reached for subsequent tests. The cache expires after seven days and re-tests the speed ladder, or if speed results deviate. This reduces data consumption and test execution time but can result in less accurate results. Defaults to disabled.
Last updated