Netflix Streaming Tests
The Netflix streaming test is an application-specific test, supporting the streaming of binary data from Netflix's servers using the same CDN selection logic as their real client uses. The test has been developed in direct cooperation with Netflix.
Netflix Test Methodology
The test begins by calling a Netflix-hosted web API. This API examines the client's source IP address and uses the existing proprietary internal Netflix logic to determine which Netflix server this user's IP address would normally be served content from. This logic will consider the ISP (internet service provider) and geographic location of the requesting IP address. Where the ISP participates in Netflix's Open Connect Appliances (OCA) program, it is likely that one of the OCA servers will be used. The API will return to the client an HTTP 302 redirect to a 25 MB binary file hosted on the applicable content server.
The test will then establish an HTTP connection to the returned server and attempt to fetch the 25 MB binary file. This runs for a fixed 20 seconds of real-time. HTTP pipelining is used to request multiple copies of the 25 MB binary, ensuring that if the payload is exhausted before the 20 seconds are complete, we can continue receiving more data. The agent downloads data at full rate throughout; no agent-side throttling takes place.
It's important to note that this 25 MB binary content does not contain video or audio; it is just random binary data. However, with knowledge of the bitrates that Netflix streams content at, we can treat the binary as if it were video/audio content operating at a fixed rate. This allows us to determine the amount of data consumed for each frame of video (at a set bitrate) and the duration that it represents. Using this, we then can infer when a stall occurred (by examining when our simulated video stream has fallen behind real-time). The test currently simulates videos at bitrates (in Kbps) of:
235
375
560
750
1050
1750
2350
3000
4500
6000
15600
Key Metric Measured
This approach allows us to derive the 'bitrate reliably streamed', using the same methodology as the YouTube test. A small difference here is that we do not need to restart the download at a lower bitrate if a stall is encountered; because the incoming stream of binary data is decoded at a simulated bitrate, we can simply recompute the playback characteristics of the same network stream at a different bitrate entirely on the client side. This simply means that the test uses a predictable amount of bandwidth, even in cases where stalls occur.
The key outputs from this metric are:
The bitrate reliably streamed.
The startup delay (the time taken to download two seconds of video).
The TCP connection time.
The number of stalls and their duration.
The downstream throughput achieved.
Netflix Test Example
We explored why peak-time Netflix download speeds can struggle to stream UHD movies even on super-fast connections, and super-efficient video encoding is cleverly masking the reality gap.
The chart below shows the average regular download speeds of an ISP and the average Netflix download speeds on the same ISP. The average download speeds are pretty consistent, but the Netflix speeds are a relative rollercoaster, holding up well off-peak, but plummeting during peak hours to less than half the average speed of the connection.
Last updated