
Measurement error and reliability: how to tell whether a change is real
A 5° improvement in dynamic valgus looks like good news. Before telling the athlete, it is worth knowing how much the same test varies when nothing has changed.
- ICC measures relative reliability: below 0.5 is poor, 0.5–0.75 moderate, 0.75–0.9 good and above 0.9 excellent 1.
- SEM expresses error in the units of the measure and allows the smallest change that can be considered real to be calculated 2.
- For FPPA measured on 2D video, SEM was 2.72–3.01° and the smallest detectable difference 7.54–8.93° 3.
- In running, averaging at least 7 steps gives values that are stable between days 4.
- A real change (above the MDC) is not necessarily a clinically important change 5.
- Screening requires known sensitivity and specificity; monitoring requires reliability and a standardised set-up 6.
Why measurement error matters
Every measurement contains noise: the client does not repeat the movement identically, the camera is not placed in exactly the same spot, and the algorithm or the assessor introduces variation. In clinical practice, the problem arises when a difference that sits within that noise is interpreted as progress. Reliability indices exist to separate the two, and they are worth knowing before looking at a client’s progress chart.
There are two different questions. Relative reliability asks whether the test ranks people consistently. Absolute reliability asks how much an individual’s value varies, in degrees, millimetres or seconds. The first is answered with the ICC; the second with the SEM and the minimal detectable change 2.
ICC: relative reliability
The intraclass correlation coefficient (ICC) compares the variance between people with the total variance of the measurements. There are 10 forms of ICC, and a study should report the model, type and definition used, as well as the software 1. An essential distinction is whether the ICC includes systematic error, as forms (2,1) and (2,k) do, or excludes it, as forms (3,1) and (3,k) do 2. When comparing sessions over time, what matters is whether there was systematic error between them, for example from learning or fatigue.
| ICC | Reliability |
|---|---|
| < 0.5 | Poor |
| 0.5–0.75 | Moderate |
| 0.75–0.9 | Good |
| > 0.9 | Excellent |
The authors recommend applying these categories to the 95% confidence interval of the ICC, not just to the point estimate 1.
The ICC depends on the heterogeneity of the sample: a very diverse group tends to yield higher ICCs for the same absolute error. A high ICC therefore does not, on its own, tell you how many degrees of change are needed before an individual’s progress can be trusted. For that, you need the SEM.
SEM and minimal detectable change (MDC)
The standard error of measurement (SEM) expresses error in the units of the test itself and can be calculated from the ICC; it allows confidence intervals to be built around individual values and defines the minimum difference needed to be confident that a real change has occurred 2. A common formula is SEM = SD × √(1 − ICC), where SD is the between-subject standard deviation. The 95% minimal detectable change (MDC95, also called SDD or SDC95) is obtained as MDC95 = 1.96 × √2 × SEM, i.e. approximately 2.77 × SEM.
These illustrative values are of the same order of magnitude as those published for FPPA on 2D video: between-day ICC of 0.72–0.91, SEM of 2.72–3.01° and smallest detectable difference of 7.54–8.93° 3. For some running variables the error is smaller, with smallest detectable differences of 1.3° to 4.8° depending on the variable 4. For others it is much larger: for shoulder range of motion measured with Apple Vision, the SDC95 was 25.4° for abduction and 39.5° for adduction 7.
| Measure | Population and test | Reliability | SEM | MDC / SDD |
|---|---|---|---|---|
| 2D FPPA (video, manual measurement) | 20 students; single-leg squat, drop jump, single-leg landing | Between-day ICC 0.72–0.91 | 2.72–3.01° | 7.54–8.93° 3 |
| Lateral trunk lean, femoral adduction, foot inclination | 21 recreational runners; running | ICC 0.90–0.99 | — | 1.3–2.3° 4 |
| Knee flexion and hip adduction | 21 recreational runners; running | ICC 0.82–0.89 | — | 2.3–3.8° 4 |
| Contralateral pelvic drop | 21 recreational runners; running | ICC 0.59–0.77 | — | 2.7–2.8° 4 |
| Shoulder abduction (Apple Vision, iPhone) | 20 healthy adults | — | 9.2° | 25.4° 7 |
| Automated LESS score (OpenLESS) | Athletes; jump landing, in the field | ICC(2,k) > 0.89 | 0.98 errors | 2.72 errors 8 |
SDD: smallest detectable difference; SDC95: smallest detectable change at 95%. Each value applies to the protocol and population of the respective study.
Bland–Altman: agreement between methods
When a new method is compared with a reference, correlation is not enough: two methods can correlate perfectly and still always differ by 5°. Bland–Altman analysis separates bias (the mean difference, or systematic error) from the 95% limits of agreement (the range within which the difference for an individual is expected to fall). A small bias with wide limits looks good on average but is imprecise for each person.
Two examples illustrate the difference. For FPPA, BlazePose had a bias of −0.67° against manual video measurement, but limits of agreement of −8.76° to 7.42° 9. For 3D foot scanning with a structured-light sensor, the bias was 0.5 mm, with limits of −12.4 to 13.5 mm 10. In both cases the mean is almost perfect, but the individual value can deviate considerably.
Sensitivity and specificity: screening is not monitoring
Sensitivity is the proportion of people truly at risk whom the test identifies; specificity is the proportion of people without the problem whom the test classifies correctly. When three physiotherapists visually rated dynamic valgus in the drop jump of 40 adolescent female footballers, agreement was good (kappa of 0.75–0.85), but sensitivity was 67–87% and specificity 60–72% against 3D analysis: up to a third of high-risk athletes were not identified 6.
Screening (classifying people) requires a validated cut-off and sensitivity and specificity values against a reference. Monitoring (comparing the same person over time) depends mainly on reliability and a consistent set-up. 2D video behaves differently in the two scenarios: it correlates reasonably with 3D between athletes (r² of 0.58–0.64), but less well within the same athlete (r² of 0.25–0.36) 11. Automation can help with consistency: an automated LESS scoring system agreed with expert raters (ICC(2,k) of 0.79 in healthy participants and 0.88 after ACL reconstruction) and reduced analysis time from about 35 hours to under 25 minutes 8.
How to standardise the retest
- Same set-up: identical camera distance, height and view in every session; a different view produces systematic bias 12.
- Level phone: camera tilt changes the measured range 7.
- Same conditions: the same footwear, close-fitting clothing, similar lighting and, if possible, the same time of day.
- Same instruction and several repetitions: use the average of several repetitions, or of at least 7 steps in running 4.
- Record the context: fatigue, pain and recent training load alter movement, regardless of instrument error.
- Compare against the right MDC: use the MDC for the protocol actually used and, separately, judge whether the change matters to the client 5.
Exceeding the MDC indicates that the change is probably real; whether that change matters in the client’s life is a different question, that of the minimal clinically important difference 5. For most movement tests there are no published values for this difference, and that gap should be acknowledged rather than filled with estimates.
Frequently asked questions
What is a good ICC value?
According to Koo and Li, an ICC below 0.5 indicates poor reliability, 0.5–0.75 moderate, 0.75–0.9 good and above 0.9 excellent, judged on the 95% confidence interval 1.
How do you calculate the minimal detectable change (MDC)?
Calculate the SEM from the standard deviation and the ICC 2, for example SEM = SD × √(1 − ICC), then multiply it by 1.96 × √2 (about 2.77) to obtain the MDC95.
What is the difference between SEM and MDC?
The SEM is the typical error of a single measurement; the MDC is the smallest difference between two measurements that exceeds that error with 95% confidence. For FPPA on 2D video, SEM was 2.72–3.01° and the smallest detectable difference 7.54–8.93° 3.
What is the difference between sensitivity and specificity in a screening test?
Sensitivity is the proportion of at-risk people the test detects; specificity is the proportion of people not at risk who are correctly ruled out. For visual rating of dynamic valgus, they were 67–87% and 60–72%, respectively 6.
Is a change larger than the MDC clinically important?
Not necessarily. The MDC indicates that the change is probably real; its clinical relevance is a separate question 5.
References
- Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. doi.org/10.1016/j.jcm.2016.02.012
- Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res. 2005;19(1):231–240. doi.org/10.1519/15184.1
- Munro A, Herrington L, Carolan M. Reliability of 2-dimensional video assessment of frontal-plane dynamic knee valgus during common athletic screening tasks. J Sport Rehabil. 2012;21(1):7–11. doi.org/10.1123/jsr.21.1.7
- Dingenen B, Barton C, Janssen T, Benoit A, Malliaras P. Test-retest reliability of two-dimensional video analysis during running. Phys Ther Sport. 2018;33:40–47. doi.org/10.1016/j.ptsp.2018.06.009
- Haley SM, Fragala-Pinkham MA. Interpreting change scores of tests and measures used in physical therapy. Phys Ther. 2006;86(5):735–743. doi.org/10.1093/ptj/86.5.735
- Ekegren CL, Miller WC, Celebrini RG, Eng JJ, Macintyre DL. Reliability and validity of observational risk screening in evaluating dynamic knee valgus. J Orthop Sports Phys Ther. 2009;39(9):665–674. doi.org/10.2519/jospt.2009.3004
- van den Hoorn W, Lavaill M, Cutbush K, Gupta A, Kerr G. Comparison of shoulder range of motion quantified with mobile phone video-based skeletal tracking and 3D motion capture—preliminary study. Sensors (Basel). 2024;24(2):534. doi.org/10.3390/s24020534
- Turner JA, Reiche ET, Hartshorne MT, Lee CC, Blodgett JM, Padua DA. Open source, open science: development of OpenLESS as the automated Landing Error Scoring System. J Athl Train. 2025;60(8):590–599. doi.org/10.4085/1062-6050-0666.24
- Janisch L, Maack LM, Fohrmann D, Marques CJ. Assessment of dynamic knee angle deviations in the frontal plane in physiotherapy clinical practice: intra- and inter-rater reliability of an application and agreement with two AI-models. BMC Musculoskelet Disord. 2026;27(1):407. doi.org/10.1186/s12891-026-09958-9
- Farhan M, Wang JZ, Warncke R, Cheng TL, Burns J. Comparison of accuracy and speed between plaster casting, high-cost and low-cost 3D scanners to capture foot, ankle and lower leg morphology of children requiring ankle-foot orthoses. J Foot Ankle Res. 2024;17(3):e70006. doi.org/10.1002/jfa2.70006
- McLean SG, Walker K, Ford KR, Myer GD, Hewett TE, van den Bogert AJ. Evaluation of a two dimensional analysis method as a screening and evaluation tool for anterior cruciate ligament injury. Br J Sports Med. 2005;39(6):355–362. doi.org/10.1136/bjsm.2005.018598
- Baldinger M, Reimer LM, Senner V. Influence of the camera viewing angle on OpenPose validity in motion analysis. Sensors (Basel). 2025;25(3):799. doi.org/10.3390/s25030799
Educational content for health professionals. It does not replace clinical assessment or reading the original studies.



