This article is intended solely as background information about how the values added to Statistical Process Control (SPC) trend charts by the benchmarking tab are calculated, and what they mean. For information about how to use this feature, see the guides for trend charts or the associated question types.
Confidence Limits and Intervals
Selecting for Confidence Intervals as the benchmark calculation applies a common statistical formula to gerenate the benchmark lines.
Put simply, a Confidence Interval is a range bracket around the mean (the statistical term for the most commonly-understood meaning of the word “Average”), within which we expect most of the data to be located, assuming that it fits a normal distribution. It is usually expressed in terms of a percentage – a “95%” Confidence interval is a range of values in which we would expect around 95% of all the data to sit. Data points that sit outside this range are therefore of interest, represtinging the highest or lowest 2.5% of data.
The values of the upper and lower end of this range are known as Confidence Limits. They are calculated by working out the mean and the standard deviation of the data set, then the standard deviation is multiplied by a number known as the Z-value, which is the number of standard deviations away from the mean the defined proportion (in percent) of responses will be found. This value doesn’t change in a linear way compared to the value of the confidence level, so it is drawn from a table based on the requested confidence level.
The calculated value is added to the mean to give the upper confidence limit, and subtracted from the mean to give the lower limit.
To apply the above to the trend charts, the data points on the chart are the values. It is these values that are averaged out and a standard deviation calculated for them. This does mean that some changes to the chart display options may change the values for the mean and limits – what might be the average value for daily scores may not be the average one for weekly scores.
Control Limits
An alternative to the Confidence Interval calculation is the use of “Control Limits”. These are used to generate Statistical Process Control (SPC) Charts, which are commonly used in the UK National Health Service (NHS).
These charts use a different calculation to reach a similar-looking outcome as one using confidence intervals. Instead of calculating the standard deviation, the calculation looks at the data as a sequence, and works out the difference between each pair, a “moving range” of two values. These moving ranges are averaged out and multiplied by a scaling factor of 1.128. This value, known as Sigma (σ), is multiplied by three and then added or subtracted to the mean to give an upper and lower limit.
The lower-case sigma symbol (σ) is often used in statistics as shorthand for the Standard Deviation of a normally-distributed data set. This can cause issues in the context of control limits, where it is used to denote the average of the moving range divided by the scaling factor. This can lead to some confusion, so be careful about what terms you use in communications.
Which Should I Use?
Neither metric is better than the other. Some organisations prefer that a particular method be used.
Generally speaking, the Control Limits method is arguably a better way of detecting changes to a system that you would expect to generate continuous and mostly stable results, with notable points of change that result in a new point of stability, because the sequence of data is important to how its derived. A confidence interval may be a better way to identify outlying random events in data you would expect to follow a normal distribution.