Flow Consolidation
Flow Consolidation in Traffic Insights
A common occurrence in network flow records is when the same traffic conversation is reported by multiple traffic monitors as the conversation passes through them. While a natural phenomenon in network configurations, multiple records for the same conversation can yield misleading measurements and visualizations, and make some types of alerts unreliable, undermining data accuracy and trustworthiness.
To address this critical issue, Traffic Insights provides context-driven automated flow record consolidation. When toggled on (the default setting), you can observe traffic from an aggregated network-wide view where each conversation is counted just once, giving you a more accurate measurement of your network-level traffic volume. No data is discarded: the automated consolidation algorithm detects records of the same conversation and stitches them together into a single table entry.
Zoom in to a specific traffic monitoring device and you still see how many raw records are moving through it, unique or not, so you can understand disaggregated device-level traffic volume, too. Viewing device-level data (by applying filters or clicking through the table links - see How We Consolidate Flows ensures that a traffic monitor's total throughput across all its interfaces is available, accurately represented, and critical issues like packet loss are not masked.
All traffic records, including multiples of the same conversation, count towards your FPS rate.
Consolidation Example
Suppose you use a video conferencing app (for example, Webex or Zoom) that creates a flow record 8 Mbps in size. You are likely to see roughly 8 Mbps throughput value in the table entry corresponding to that app when flow consolidation is enabled. Suppose that same flow record passed through three other monitors on its journey, creating a total of four records for that one conversation. If flow consolidation is disabled, you would see the conversation for that app reporting ~8 Mbps multiplied by a factor of four to account for the number of devices that that record is passing through, or 32 Mbps.
Given that the algorithm improves measurements and visualization without discarding any data, you can enable and disable flow consolidation anytime from Traffic Insights > Settings > General Settings.
How We Consolidate Flows
Traffic Insights employs an automated algorithm to identify and mark multiple flow records identifying the same conversation, then stitches those “associated” records into one unique conversation “report” (table row) for at-a-glance digestibility. This process happens in real time as ThousandEyes ingests network flow records.
Here's how it works:
Data Collection: ThousandEyes collects all incoming network flow data, including all records associated with each individual conversation.
Conversation Identification: Within the ThousandEyes platform, each incoming record’s 4-Tuple elements (client IP, server IP, destination port, protocol) are stored in memory for a specific aggregation window, typically 5 minutes.
Metrics Consolidation: If the same 4-Tuple record is reported by more than one traffic monitor within a given aggregation window, ThousandEyes selects a single monitor as the "leader" that will provide the metrics for that particular conversation in the chart and table entry. All other records reporting the same flow from different traffic monitors are marked as “associates” of the leader. The leader is the monitor with the highest throughput, though actual conversation throughput going through all monitors varies and can be viewed at the disaggregated device-level by filtering or clicking through the table.
Rule Application: Traffic Insights takes into account your filter and grouping settings before applying consolidation rules. All data is consolidated unless either your filter or grouping settings are set to the device-level, as all device-level views are non-consolidated. Rule application happens every time you change your filter or grouping settings.
Stitching: In the Conversation view, when not filtered to the device-level, the table displays one unique conversation per row. The table “stitches” together information about multiple records so you can view at-a-glance the number of traffic monitors, interfaces, or applications that also report on the same conversation. Click on the device scaler number in the corresponding table cell, application scalar, or any other scaler to view the associated records. Notice that the filtering and grouping changes accordingly as you click through.
Data Storage: All raw records, from both "leader" and "associated" traffic monitors, are stored in the database for 30 days. This ensures that you retain maximum granularity and can query the raw data from any monitor if needed. However, when flow consolidation is enabled, the relevant consolidated values are used for calculations and alerts.
The system automatically matches flows for you based on key attributes like IP addresses, ports, protocol, and time. This process also accounts for multicast, broadcast, and anycast addresses.
How To Get Flow Consolidation
All customers have flow consolidation turned on by default. You can toggle it on or off at Traffic Insights > Settings > General Settings.
Alert rule thresholds do not automatically adjust when flow consolidation is toggled on or off. For example, you have an alert rule set to trigger if total throughput for an application falls below 1 Mbps, and flow consolidation is off. When you turn it on, throughput figures decline significantly as conversations are consolidated, potentially creating a slew of noisy alerts because throughput is now always below 1 Mbps for that application.
If you have Traffic Insights alert rules already in place that are not scoped to the device-level and you change your flow consolidation settings, you may need to adjust your alert thresholds to align with the new metric calculations; go to Manage > Alert Rules.
When enabled, the process is fully automated to provide the most accurate data without requiring complex manual rules from you. If you disable flow consolidation, queries revert to non-consolidated calculations, and aggregated metrics are not consolidated.
Where To View Consolidated Records
Consolidated records are viewable in several Traffic Insights-related areas of the platform:
Traffic Insights > Views: Aggregated charts and tables show consolidated flows per application and conversation. The Conversation view table also shows for each unique conversation the number of associated records by device, interface, and application. See Viewing Traffic Insights Data for more information.
Traffic Insights > Settings: Both the Traffic Monitors and Forwarders tabs contain Associated FPS columns that show you how many flows per second from conversations passing through the traffic monitor are also reported by other traffic monitors. Importantly, the FPS Monitoring tab shows the count of all unconsolidated flow records, which is what counts towards your FPS rate allowance (FPS limit). The General Settings tab is where you toggle consolidation on or off.
Alerts: Alert rule thresholds do not automatically update according to flow consolidation settings (on/off). If you have Traffic Insights alert rules in place and change your flow consolidation settings, see How To Get Flow Consolidation about making potential alert rule adjustments.
Consolidation Limitations
While the automated flow consolidation algorithm significantly improves network-level data navigation, accuracy, and visualization, there are cases where multiple records cannot be identified, or consolidation implemented.
Incomplete Data: When data for a single conversation is split between multiple traffic monitors, no one monitor sees the "whole" conversation, making it difficult to identify which records are multiples of that conversation. In these cases, consolidation may not be possible.
Interface-Level Records: Consolidation for interfaces across the same traffic monitor is not currently supported. This means that if a single traffic monitor reports the same traffic on multiple interfaces, those interface-level records are not consolidated by the automated process.
Flow Record Type: Automated consolidation is designed to work with 4-Tuple records compliant with standard NetFlow/IPFIX versions. See Flow Record Requirements for more information.
Dashboards: All Traffic Insights dashboard widgets display non-consolidated values.
Traffic Insights aims to provide the most accurate data possible, but understanding these limitations helps you interpret the data effectively.
Last updated