Stability Scoring

Overview

Stability Scoring measures how consistent a provider's own performance is over time. Even if a provider has excellent average latency, high variability (instability) can be problematic for real-time applications like VoIP or video conferencing.

Provider Intelligence uses a median value the the basis for calculating stability. This method is robust to outliers and produces a score between 0 (varying every hour) and 100 (the same value every hour).

Why Stability Matters

Scenario: Two providers both have an average latency of 20 ms over a month. However:

• Provider A: Latency ranges from 18 ms to 22 ms (low variability).

• Provider B: Latency ranges from 5 ms to 40 ms (high variability, with frequent spikes).

Averaging the two suggests they're equal, but Provider A is clearly more reliable for latency-sensitive applications.

Stability Score Calculation and Example

1. Data Collection: The process begins by collecting a time series of measurements for a specific provider metric over a defined period, typically the past 24 hours.

   Example (5 hours shown):

   Hour	Latency (ms)
   1	18
   2	20
   3	22
   4	19
   5	40 (spike)

2. Median-Based Reference: Instead of using an average, the method uses the median value of the collected data points as a central reference. The median is less sensitive to outliers and provides a robust baseline representing the typical behavior of the metric during the period.

   In our example, the median over 24 hours is 20 ms.

3. Deviation Analysis: Each datapoint in the time series is compared to this median reference to determine how much it deviates. This step measures the variability or spread of the metric values around the median, capturing fluctuations that indicate instability.

   In our example:

   Hour	Latency (ms)	Deviation (ms)
   1	18	2
   2	20	0
   3	22	2
   4	19	1
   5	40	20

4. Aggregation of Deviations: The deviations are aggregated using a statistical approach that emphasizes typical variation while minimizing the impact of extreme outliers. In other words, as with the performance score, stability scoring translates the deviation range into a 0-1 range, capping the most egregious deviation at 1 (where it might otherwise be higher, thus minimizing the effect of outliers).

   In our example:

   Hour	Deviation (ms)	Relative Deviation
   1	2	0.10
   2	0	0.00
   3	2	0.10
   4	1	0.05
   5	20	1.00 (capped)

5. Inversion and Scoring: The aggregated variability value is then inverted to produce a stability measure from 0-1 (a 0-100 score), where a measure closer to 1 indicates high stability with minimal fluctuations, while a measure closer to 0 indicates significant instability.

   Suppose for our example, the stability score across 24 hours is 75.

Interpretation: A stability score of 75 means the metric was fairly consistent, but the spike in Hour 5, for example, significantly reduced the score.