Reliability and validity of threat image projection data as a measure of performance in X-ray baggage screening

Abstract

Passenger baggage is screened using X-ray machines at airports worldwide to ensure transportation security. Many airports use a technology called threat image projection (TIP) to measure detection performance of airport security officers (screeners). TIP projects prerecorded X-ray images of prohibited items into X-ray images of passenger baggage being screened, and each time a TIP image is displayed (a TIP event), the TIP system records whether the screener detected the prohibited item. Because the prohibited items and the location of their placement in bags are randomly selected, the resulting TIP images vary substantially in difficulty and do not always look realistic. It is therefore not clear whether TIP data provides a good measure of screener performance, despite the technology’s long-standing and widespread use at airports. To address this research gap, we conducted a study to estimate TIP’s psychometric properties of reliability and validity by analysing a large set of TIP data from cabin baggage screening of a European airport (1,199,838 TIP events from 728 screeners over four years). We found the reliability of performance measurement to increase with the number of TIP events in accordance with the Spearman–Brown prediction. Approximately 100 TIP events were sufficient to achieve a minimum reliability value of 0.7 when TIP was challenging enough (mean hit rate below 0.9). TIP performance predicted the covert test results (wherein instructed people tried to smuggle real prohibited items through the checkpoint; 1,184 covert tests from 474 screeners), indicating that TIP is a valid measure of detection performance in X-ray baggage screening. The results imply that TIP data provides a reliable and valid performance measure if the TIP images are challenging enough and about 100 TIP events are considered per screener.