Designing fairness. best practices for gender-sensitive development of cognitive ability tests in recruitment

Abstract

Psychological tests play a pivotal role in high-stakes decisions such as recruitment, yet traditional development guidelines concentrate fairness work downstream — at test administration and post hoc statistical bias correction — while offering little concrete guidance for the design stage, where constructs are defined and operationalized. Drawing on constructivist and organizational justice theories, we argue that a “bias by design” arises when unexamined default assumptions narrow construct representation, systematically disadvantaging groups before any item is administered. We develop this for cognitive ability tests, which exhibit the largest subgroup differences among common selection instruments and routinely rely on figural matrices as proxies for general ability, thereby disadvantaging women. We contrast procedural justice (standardized administration) with distributive justice (equitable score distributions), clarifying that the latter applies to a principled class of constructs — latent and content-general, such as fluid intelligence — for which content-linked group differences signal construct-irrelevant variance rather than true differences. To address this gap, we adapt Stanford’s Gendered Innovations framework and demonstrate its application through the Modularer Kurzintelligenztest (M-KIT; Dantlgraber et al., 2015). Fairness can be addressed at three hierarchically ordered levels, where higher levels constrain lower ones: the theoretical model (broadening construct representation), the task format (removing construct-irrelevant demands), and the item (differential item functioning as final refinement, not primary remedy). We distill practical recommendations for embedding distributive justice early, including questioning default assumptions and documenting fairness deliberations. Our approach reframes fairness not as a trade-off but as integral to validity, offering a roadmap for more inclusive assessments.