The reproducibility of structured functional assessments in a social security setting. A pre-specified explanatory analysis of the RELY-studies

Abstract

Introduction: Limitations in work capacity (WC) need to be quantified in a transparent and reproducible way when insurers of social security decide whether an individual is entitled to disability benefits and to what extent. Structured assessments of work-related physical, mental and social functioning might provide an empirical basis for judgments on residual work capacity (rWC) which determines entitlement to disability benefits. This study examined the functional assessments themselves, their reliability and expert agreement when applied to claimants with mental disorders, and analyzed their relationship to rWC judgments.

Material and methods: We used RELY-data on the reproducibility of rWC judgments. A pool of 40 psychiatric experts interviewed 55 claimants for disability benefits. Interviews were videotaped and watched by three observing psychiatric experts, resulting in 280 individual ratings. All independently rated claimants’ impairments in work-related mental functions and capacity limitations using the Instrument for Functional Assessment in Psychiatry (IFAP-1 mental functions, IFAP-2a/-2b functional capacities related to the last job and alternative work, scaled 0=none to 4=worst) based on the Mini-ICF-APP, and judged rWC (in Switzerland, scaled 100% to 0%) for the last job and suitable alternative work. Analysis for reliability (ICC, intraclass-correlation coefficient) included a two-way random-effects and a linear mixed-effects model. Expert agreement was estimated as standard error of measurement, SEM.

Results: The mean score for mental functions (IFAP-1global) was 1.21 (SD 0.63) and for functional capacities in alternative work (IFAP-2bglobal) 0.87 (SD 0.56). Reliability of IFAP ratings was low to fair (IFAP-1global: ICC = 0.46; IFAP-2bglobal: ICC = 0.26), similar to the low interrater reliability of rWC. Agreement showed substantial measurement error: IFAP-1global: SEM = 0.47; IFAP-2bglobal: SEM = 0.49. The rWC judgments for claimants with identical ratings in functional limitations (IFAP-2bglobal=1) ranged from 100% to 5%.

Conclusions: Evidence indicates that Functional Assessment, if carried out well, may lead to more reproducibility. This explanatory analysis revealed low to fair interrater reproducibility in mental functions (IFAP-1), in functional capacities (IFAP-2a/b) which extends to rWC. Among various other explanations, we believe this to be mostly due to insufficient training in Functional Assessment and therefore reflects real-world variability in judgment. We recommend revising training format and intensity, and monitoring adherence in practice, followed by re-evaluation of reproducibility of expert judgements. As of today, the outcome is uncertain.