284 Schizophrenia Bulletin vol. 47 no. 2 pp. 284–297, 2021 doi:10.1093/schbul/sbaa120 Advance Access publication 11 September 2020 © The Author(s) 2020. Published by Oxford University Press on behalf of the Maryland Psychiatric Research Center. MAJOR REVIEW Implementing Precision Psychiatry: A Systematic Review of Individualized Prediction Models for Clinical Practice Gonzalo Salazar de Pablo1–3, Erich Studerus4, Julio Vaquerizo-Serrano1–3, Jessica Irving1, Ana Catalan1,6–8, Dominic Oliver1, , Helen Baldwin1, Andrea Danese3,9,10, Seena Fazel11, Ewout W. Steyerberg12,13, Daniel Stahl5, and Paolo Fusar-Poli*,1,14–16 1Early Psychosis: Interventions and Clinical-detection Lab, Department of Psychosis Studies, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, 5th Floor, PO63, 16 De Crespigny Park, SE5 8AF London, UK; 2Institute of Psychiatry and Mental Health, Department of Child and Adolescent Psychiatry, Hospital General Universitario Gregorio Marañón School of Medicine, Universidad Complutense, Instituto de Investigación Sanitaria Gregorio Marañón, CIBERSAM, Madrid, Spain; 3Department of Child and Adolescent Psychiatry, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK; 4Division of Personality and Developmental Psychology, Department of Psychology, University of Basel, Basel, Switzerland; 5Biostatistics Department, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK; 6Department of Psychiatry, Basurto University Hospital, Bilbao, Spain; 7Mental Health Group, BioCruces Health Research Institute, Bizkaia, Spain; 8Neuroscience Department, University of the Basque Country UPV/EHU, Leioa, Spain; 9Social, Genetic and Developmental Psychiatry Centre, King’s College London, London, UK; 10National and Specialist CAMHS Clinic for Trauma, Anxiety, and Depression, South London and Maudsley NHS Foundation Trust, London, UK; 11Department of Psychiatry, University of Oxford, Oxford, UK; 12Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, the Netherlands; 13Department of Public Health, Erasmus MC, Rotterdam, the Netherlands; 14OASIS Service, South London and Maudsley NHS Foundation Trust, London, UK; 15Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy; 16National Institute for Health Research, Maudsley Biomedical Research Centre, South London and Maudsley NHS Foundation Trust, London, UK *To whom correspondence should be addressed; tel: +44-0-20-7848-0900, fax:+44-0-20-7848-0976, e-mail: paolo.fusar-poli@kcl.ac.uk Background: The impact of precision psychiatry for clin- ical practice has not been systematically appraised. This study aims to provide a comprehensive review of valid- ated prediction models to estimate the individual risk of being affected with a condition (diagnostic), developing outcomes (prognostic), or responding to treatments (pre- dictive) in mental disorders. Methods: PRISMA/RIGHT/ CHARMS-compliant systematic review of the Web of Science, Cochrane Central Register of Reviews, and Ovid/ PsycINFO databases from inception until July 21, 2019 (PROSPERO CRD42019155713) to identify diagnostic/ prognostic/predictive prediction studies that reported indi- vidualized estimates in psychiatry and that were internally or externally validated or implemented. Random effect meta- regression analyses addressed the impact of several factors on the accuracy of prediction models. Findings: Literature search identified 584 prediction modeling studies, of which 89 were included. 10.4% of the total studies included pre- diction models internally validated (n = 61), 4.6% models externally validated (n = 27), and 0.2% (n = 1) models con- sidered for implementation. Across validated prediction modeling studies (n = 88), 18.2% were diagnostic, 68.2% prognostic, and 13.6% predictive. The most frequently investigated condition was psychosis (36.4%), and the most frequently employed predictors clinical (69.5%). Unimodal compared to multimodal models (β = .29, P = .03) and di- agnostic compared to prognostic (β = .84, p < .0001) and predictive (β = .87, P = .002) models were associated with increased accuracy. Interpretation: To date, several valid- ated prediction models are available to support the diag- nosis and prognosis of psychiatric conditions, in particular, psychosis, or to predict treatment response. Advancements of knowledge are limited by the lack of implementation re- search in real-world clinical practice. A new generation of implementation research is required to address this trans- lational gap. Key words: risk/prognosis/prediction/individualized/ prevention/evidence/implementation/validation Introduction Precision medicine is an emerging approach for disease prevention, diagnosis, and treatment that considers indi- vidual variability in patient and disease characteristics, genes, environment, and lifestyle of each person.1,2 The This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/ by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://orcid.org/0000-0002-8920-3407 mailto:paolo.fusar-poli@kcl.ac.uk?subject= http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/licenses/by/4.0/ 285 Implementing Precision Psychiatry concept of precision medicine is not new; clinicians have been working to personalize care tailored to people’s in- dividual health needs throughout the history of medicine (eg, matching human blood groups across donors and recipients during blood transfusion).3 Yet, modern ad- vancements of knowledge in the field of individualized prediction modeling have allowed the consolidation of an evidence-based science of precision medicine.4 Prediction modeling can be used to forecast the probability of a certain condition being present (diagnostic models), outcomes (prognostic models), or the response to inter- ventions (predictive models) at the individual subject level. From a methodological perspective, individualized prediction modeling research includes studies that inves- tigate the development, internal or external validation of prediction models, and prediction model impact studies, which investigate the real-world effect of using prediction models in clinical practice.5 External validity is the extent to which the predictions can be generalized to the data from plausibly related settings, while internal validity is the extent to which the predictions fit the derivation data after controlling for overfitting and optimism, with the latter representing the difference in a model’s perfor- mance in the derivation data and unseen individuals (for further details see4). More recently, individualized prediction models have been developed in psychiatry,4 and a new field of pre- cision psychiatry has emerged.6–8 The area where indi- vidualized prediction models have been more extensively investigated in psychiatry relates to psychotic disorders. The high personal, clinical, and societal burden associ- ated with psychosis, coupled with the limited pathophys- iological understanding, has stimulated research into diagnostic prediction models. Incorporation of a clinical staging model for psychosis,9 together with the emergence of the clinical high-risk state for psychosis (CHR-P),10,11 has prompted research into prognostic prediction models, as well as several ongoing international collaborations.12 The associated need to stratify or personalize early in- tervention or preventive treatment for psychosis13,14 has stimulated research of predictive prediction models. Furthermore, emerging research has indicated that pre- diction modeling can benefit from transdiagnostic ap- proaches that allow methodological cross-fertilization across other nonpsychotic disorders.15–17 Despite the increasing number of records published in this area over recent years, the impact of precision psychiatry for psychosis, and more broadly for clin- ical practice, is unclear. No study to our knowledge has comprehensively reviewed the advancements and chal- lenges of prediction modeling in clinical psychiatry to date. Our primary aim was to systematically appraise diagnostic, prognostic, or predictive individualized pre- diction models that can be considered for clinical use in psychiatry, with a specific focus on psychosis; the sec- ondary aim was to test potential moderating factors. The evidence reviewed was then used to formulate pragmatic recommendations to advance knowledge in this area. To address the potential impact of precision psychiatry, we focused on diagnostic, prognostic, and predictive predic- tion model studies with at least internal or external vali- dation and implementation studies. Methods This study (study protocol: PROSPERO CRD42019155713) was conducted in accordance with the RIGHT18 and PRISMA19 statements (supplementary table 1). Search Strategy and Selection Criteria A multistep independent researcher systematic literature search strategy was used to identify the relevant articles. First, the Web of Science, Cochrane Central Register of Reviews, and Ovid/ PsycINFO database were searched, from inception until July 21, 2019 in English (specific search terms are reported in supplementary methods 1). Second, the references of the articles identified in pre- vious reviews in the field and the references from the included studies were manually searched to identify ad- ditional relevant records. Abstracts identified through the previous step were then screened and, after the ex- clusion of those not relevant to the current study, their full texts were assessed against the inclusion and exclu- sion criteria. In a fourth step, a researcher with exper- tise in risk estimation models in psychiatry (E.S.) further checked the articles against the core biostatistical inclu- sion criteria (ie, presence of appropriate internal or ex- ternal validation). The inclusion criteria were (1) original studies or study protocols published in the databases searched or gray literature; (2) studies reporting the diagnostic (principally predicts the presence of a certain con- dition), prognostic (principally predicts the clinical outcomes in the absence of therapy20), predictive (prin- cipally predicts the response to a particular interven- tion20), or implementation of risk estimation models; (3) providing estimates at the individual subject level or in subgroups; (4) studies investigating individuals affected by mental disorders or mental conditions or individuals at risk of mental disorders, defined ac- cording to established psychometric criteria, and (5) diagnostic, prognostic, or predictive studies that per- formed at least a proper internal or external validation (see below). The exclusion criteria were: (1) abstracts, conference proceedings, reviews, or meta-analyses; (2) diagnostic, prognostic, or predictive models that did not provide individualized or subgroup risk estimates; (3) diagnostic, prognostic, or predictive studies that did not perform any proper internal or external valida- tion (see supplementary methods 2); or (4) predictors- finding studies that did not report prediction models. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 286 G. Salazar de Pablo et al Descriptive Measures and Data Extraction The variables extracted in the current review included items listed in the “Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies” (CHARMS21). Additional variables were included22 as detailed in the supplementary methods 3. When more than one outcome per study was found in the same category, we extracted the information for the primary outcome, as defined in each article, unless the study reported multiple primary co-outcomes. Quality Assessment Risk of bias was assessed for each of the included studies adapting “The Prediction Model Risk of Bias Assessment Tool” (PROBAST v5/05/20195,23). PROBAST includes 4 steps and assesses the risk of bias and applicability of 4 core domains (participants, predictors, outcome, and analysis) to obtain an overall judgment of the risk of bias.5 An outcome is considered to be at high risk of bias when at least one of the questions answered is not appro- priate (no or probably no). The overall risk of bias is con- sidered high when one or more domains are considered to be at high risk24 (details can be found in supplementary methods 4). Data Analysis All the included studies were systematically summarized in tables stratified by the model type (diagnostic, prog- nostic, and predictive)—those implemented were then discussed in a separate section—and reporting core de- scriptive variables (supplementary methods 5). The top 10% of the most widely employed predictors and all the studied conditions were summarized in graphs, and the specific methodological characteristics of the studies were summarized in a separate table. These descriptive analyses were complemented by the Pearson correlation between apparent vs external accuracy within the models that reported both.16, 25–52 We further conducted meta- analytical regressions to estimate the association between accuracy and (1) the type of validation (internal vs ex- ternal); (2) the type of accuracy measure (area under the curve [AUC] vs C-statistics vs accuracy, with the latter category including accuracy measures other than AUC or C-statistics as defined by each study); (3) the type of model (diagnostic vs prognostic vs predictive model); (4) the number of specific predictors; (5) the type of pre- dictors (clinical or service use or sociodemographic vs any biomarker—neuroimaging or electroencephalog- raphy or magnetoencephalography or proteomic or ge- netic or cognitive—vs a combination of modalities); (6) the modality of predictors (unimodal, using only 1 type of predictor, eg, clinical only, vs multimodal, using more than 1 type of predictor, eg, clinical and biomarker); (7) type of analysis (machine learning vs statistical modeling, as defined in supplementary methods 6). For analyses 4–7, we also included the interaction between accuracy and meta-regressors. For analyses 2–7, we used accuracy values prioritizing external validation over internal vali- dation, in line with the previous meta-analyses of predic- tion models.53 In the case of multiple studies on the same prediction model in which the previous order of priority could not be applied, the study with the largest data set was employed. We performed a meta-regression of the difference between logit transformed accuracy (because of the bounded nature of AUC53) using a random effect meta-analysis model, taking 1–7 clustering of compari- sons into account.53 The analyses were performed with Comprehensive Meta-Analysis Version 3.54 Results Database The literature search yielded 50  698 records and, after the exclusion of nonrelevant abstracts, 1033 full-text articles were screened to identify a total of 584 predic- tion studies reporting on prediction models developed. These models were then screened for eligibility against the inclusion and exclusion criteria to identify 89 studies with individualized prediction models, which were val- idated or implemented and represented the final sample (PRISMA; figure 1): 61 were internally validated (10.4% of the total models developed), 27 were externally val- idated (4.6% of the total models developed), and 1 (0.2% of the total models developed) described a pro- tocol for the implementation of a prediction model (figure 2). Thirty point three percent (27/89) of the pre- diction models included were externally validated. 8.2% studies reported on diagnostic prediction models, 68.2% on prognostic models, and 13.6% on predictive models; 55.6% of studies employed sociodemographic pre- dictors, 69.5% employed clinical predictors, 10.2% em- ployed cognitive predictors, 13.6% employed service use predictors, 25.0% employed physical health predictors, 17.0% employed neuroimaging predictors, 0.4% em- ployed magnetoencephalography or electroencephalog- raphy predictors, 0.1% employed proteomic data, and 2.3% employed genetic predictors. The most frequently reported predictors were age (n = 38, 45.8%), sex (n = 27, 32.5%), education (n = 21, 25.3%), and depressive symp- toms (n = 18, 21.7%; figure 3). The most frequently re- ported condition was psychosis (36.4%; figure  3). The total sample size was 3 889 457 individuals, ranging from 2955 to 2 960 92956 individuals. The average age ranged from 1.857 to 64.738 years. The source of data encom- passed cohorts (46 studies, 52.3%), case-control studies (13 studies, 14.8%), clinical trials (16 studies, 18.2%), and registry data (13 studies, 14.8%). The most frequent type of external validation was geo- graphical, examining the model performance in other cen- ters or regions 24/27 (88.9%). Internal validation was more D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 287 Implementing Precision Psychiatry frequently done by cross-validation in 34/61 (55.7%). The most frequent modeling method was machine learning in 35/88 (39.8%) (supplementary table  2). In half of the studies (51.1%), there was no explicit handling of missing data; imputation (27.3%) was the most common method for data missingness (supplementary table 2). AUC was the Fig. 2. Proportion of prediction models studies developed, internally validated, externally validated, and implemented in the psychiatric literature. Records identified through database search (n = 50658) Sc re en in g In cl ud ed El ig ib ili ty Id en tif ic at io n Additional records identified through other sources (n = 40) Title and abstract screened for eligibility (n = 50698) Full-text articles assessed for eligibility (n = 1033) Records excluded during screening (n = 49665) Full-text articles excluded: no desired type of study (n = 20) no topic of interest (n = 28) Lack of proper internal or external validation (n = 447) Prediction models that have been validated or implemented and included in the systematic review (n=89) (61 internally validated, 27 externally validated, 1 implementation) Prediction models that have been developed (n=584) Full-text articles excluded due to lack of prediction models (n =449) Fig. 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart outlining study selection process. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 288 G. Salazar de Pablo et al most commonly reported measure of model performance (78.4%; supplementary table 2). Only 10.2% of the studies presented their model in full: almost half of them did not present any details of their model (46.6%) or the calibration results clearly (47.7%; supplementary table 2). Diagnostic Risk Estimation Models Four studies employed neuroimaging methods58–60 and proteomic data61 to classify individuals with schizophrenia compared to healthy controls (HC)58,59,61 or to differen- tiate schizophrenia spectrum disorder and HC with or without impaired social functioning60 (supplementary table  3). One study employed clinical predictors to dis- criminate between affective and schizophrenia spectrum psychoses.62 Two studies employed neuroimaging to differentiate unipolar vs bipolar depression25 or major depression vs dysthymia in individuals with panic disorder and ag- oraphobia.63 Another study used clinical predictors to 0 5 10 15 20 25 30 35 40 45 50 No o f s tu di es Predictors Overall Diagnos�c Prognos�c Predic�ve 0 5 10 15 20 25 30 35 No o f s tu di es Condi�ons Overall Diagnos�c Prognos�c Predic�ve Fig. 3. Most frequently reported predictors (above, top 10%) and conditions (below, all) in the included studies. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 289 Implementing Precision Psychiatry distinguish melancholic vs non-melancholic features in individuals with major depression.64 A neuroimaging study discriminated smokers and non- smoking HC.65 Another study using sociodemographic, clinical, and cognitive data discriminated individuals with cocaine dependence from HC.66 Problematic in- ternet use was discriminated from HC using clin- ical and sociodemographic predictors.26 Two studies classified posttraumatic stress disorder in veterans using sociodemographic and clinical predictors67 or magnetoencephalography.68 Three studies focused on au- tism spectrum disorders to discriminate them from atten- tion deficit hyperactive disorder69,70 or from HC57 using clinical predictors69,70 or genomic biomarkers57 (supple- mentary table 3). Prognostic Models A considerable proportion of the prognostic risk esti- mation studies16, 27–32, 71–82 (31.7%) investigated the CHR- P83 (supplementary table  4). These studies focused on the prediction of psychosis onset in CHR-P individuals (n  =  13),27–29,71–80 functional outcomes and disability in CHR-P individuals (n  =  2),81,82 psychosis onset in indi- viduals undergoing a CHR-P assessment (pretest risk n  =  1),30 and the transdiagnostic onset of psychosis in secondary mental health care (n = 3).16,31,32 Six of these studies employed sociodemographic or clinical predictors only,16,28,31,32,73,74 1 employed sociodemographic and service use data,30 2 included cognitive measures beyond sociodemographic and clinical data,27,72 3 included cog- nitive measures alone,29,71,77 1 employed electroencepha- lography predictors,75 3 neuroimaging alone,76,78,80 and 2 neuroimaging in association with clinical measures81,82 or in association with sociodemographic, clinical, and cog- nitive measures (n = 1).79 Four other studies focused on established psychosis using different combinations of sociodemographic, clinical, service use, cognitive, and physical health predictors to forecast psychotic relapses,84 hospital admission,33 employment, education or training status,34 and mortality.85 Nine studies focused on depres- sion.35–40,82,86,87 A combination of sociodemographic, clin- ical, and physical health factors was used by 3 studies to predict the onset of major depression in the general population35–37 and by 5 other studies to predict persist- ence38,86,87 or recurrence39,40 of major depression. A further study predicted disability in recent-onset depression using clinical and neuroimaging data.82 One study focused on the onset of bipolar spectrum disorders in youth at family risk using sociodemographic and clinical factors,88 while another one predicted cognitive impairment in bipolar disorder using sociodemographic and cognitive factors.89 Six studies used a combination of sociodemographic, clinical, physical health, and service use to predict sui- cidality, focusing on suicide ideation in the general pop- ulation,41,90 suicide attempts after outpatient visits,56 suicide attempts in adolescents,91 suicidal behavior,92 or deaths by suicide after hospitalization in soldiers.93 Seven studies focused on posttraumatic stress disorder (PTSD).94–100 Three studies employed a combination of sociodemographic, clinical, physical health, and service use factors to predict the onset of PTSD94–96 or the re- mission of PTSD (n = 3 studies),97–99 and a further study used clinical predictors alone to forecast PTSD features in soldiers.100 Sociodemographic, clinical, and physical health data were used by 2 studies42,43 to predict the onset of generalized anxiety disorders and panic disorder in the general population and by another study to predict the recurrence of panic disorder.44 Two studies predicted alcohol use in young people using sociodemographic and clinical45,46 predictors in combination with cognitive46 predictors, while another 2 studies predicted abstinence from heavy drinking using sociodemographic and/or clinical47,101 data. A prediction model forecasted offending behavior in schizophrenia and delusional disorder using forensic information.102 Compulsory admission into psychiatric wards was pre- dicted by a combination of sociodemographic, clinical, and service use factors,103 and medication-induced al- tered mental status in hospitalized patients was pre- dicted by sociodemographic, clinical, service use, and physical health data.104 Other models predicted the onset of common mental disorder in a working popu- lation using sociodemographic, clinical, and physical health105 variables, mental health hospital readmission using sociodemographic, clinical, and service use106 data, and violent offending in severe mental disorders using sociodemographic, clinical, and service use48 data. Predictive Models Two studies employed a combination of clinical, sociodemographic, or physical health features to pre- dict remission49,50 or response to antidepressants107,108 in major depression. Three studies predicted the onset of treatment-resistant depression using clinical and sociodemographic variables,51,52,109 service use data,52,109 and physical health data.109 A  study employed clinical and sociodemographic data to predict the level of func- tioning at 4 and 52 weeks after antipsychotic treatment in patients with first-episode psychosis.110 Two studies predicted the clinical response to transcranial magnetic stimulation combining neuroimaging and electroenceph- alography factors.55,111 A further study employed clinical and physical health data to predict treatment dropout from psychotherapy in anxiety disorders112 (supplemen- tary table 5) Implementation of Prediction Models Among externally validated models, the transdiagnostic model predicting psychosis onset in secondary mental D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 290 G. Salazar de Pablo et al T ab le 1 : R ep lic at ed p re di ct io n m od el s (a ll pr og no st ic ) S tu di es D at a so ur ce , O ut co m e D ev el op m en t sa m pl e si ze (m ea n ag e, L oc at io n) V al id at io n sa m pl e si ze (m ea n ag e, L oc at io n) P re di ct or s (n ) P er fo rm an ce (m ea su re )a P re di ct or c at eg or ie s S O C C L IN C O G P H Y C an no n et  a l 2 01 674 P sy ch os is o ns et in C H R -P 59 6 (1 8. 5, N A P L S- 2, U SA ) A ge , f am ily h is to ry , u nu su al th ou gh ts a nd s us pi ci ou sn es s, lo w er v er ba l l ea rn in g an d m em or y pe rf or m an ce , s lo w er sp ee d of p ro ce ss in g, d ec lin e in so ci al f un ct io ni ng ( 6) 0. 71 ( C -i nd ex ) C ar ri ón et  a l. 20 16 28 17 6 (1 6. 6, E D IP P P, U SA )a 0. 79 ( A U C , E xt ) X X X F us ar -P ol i et  a l. 20 17 32 T ra ns di ag no st ic ps yc ho si s on se t in se co nd ar y m en ta l he al th ca re p at ie nt s 33 82 0 (3 4. 4, SL A M b , U K ) 54 71 6 (3 2. 0, S L A M c , U K ) A ge , g en de r, et hn ic it y, a ge by g en de r, an d IC D -1 0 in de x di ag no si s (5 ) 0. 8 (C -i nd ex , D ev ); 0 .7 9 (C -i nd ex , E xt ) F us ar -P ol i et  a l 2 01 934 33 82 0 (3 4. 4, SL A M b , U K ) 13 70 2 (4 0. 9, C & I N H S T ru st ) 0. 73 ( C -i nd ex , E xt ) X X K in g et  a l. 20 08 38 M D o ns et in p ri m ar y ca re 52 16 ( 48 .9 , E ur op ed ) 17 32 ( 47 .0 , C hi le ) A ge , s ex , c ou nt ry e du ca ti on al st at us , d iffi cu lt ie s in w or k, hi st or y of d ep re ss io n in fi rs t- de gr ee r el at iv es , e xp er ie nc e of di sc ri m in at io n, li fe ti m e m aj or de pr es si on e pi so de , m en ta l qu al it y of li fe , p hy si ca l q ua lit y of li fe ( 10 ) 0. 79 ( C -i nd ex , D ev ); 0 .7 1 (C -i nd ex , E xt ) N ig at u et  a l. 20 16 39 52 16 ( 48 .9 , E ur op ed ) 29 62 1 (4 3. 8 N E SA R C , U SA ) 0. 71 ( A U C , E xt ) X X X K in g et  a l. 20 11 44 G A D a nd P D o ns et in p ri m ar y ca re 49 05 ( n. a. , U K , S pa in , Sl ov en ia , P or - tu ga l) 51 40 ( n. a. N et he rl an ds E st on ia , C hi le ) A ge , s ex , c ou nt ry , d iffi cu lt ie s in pa id a nd u np ai d w or k, h is to ry of d ep re ss io n in fi rs t- de gr ee re la ti ve s, fo llo w -u p pe ri od , lif et im e m aj or d ep re ss io n ep is od e, m en ta l q ua lit y of li fe , ph ys ic al q ua lit y of li fe ( 9) 0. 75 ( C -i nd ex , D ev ); 0 .7 1- 0. 81 (C - in de x, E xt ) N ig at u et  a l. 20 19 45 49 05 ( n. a. , U K , S pa in , Sl ov en ia , P or - tu ga l) 24 62 6 (n .a . N E SA R C , U SA ) 0. 62 ( A U C , E xt ) X X X a T ra um a an d lif e ev en ts p re di ct or s no t in cl ud ed in t he e xt er na l v al id at io n; b L am be th a nd S ou th w ar k; c A ny o th er b or ou gh ; d U ni te d K in gd om , S pa in , S lo ve ni a, P or tu ga l, T he N et he rl an ds A U C : a re a un de r th e cu rv e; C & I: C am de n an d Is lin gt on ; C L IN : c lin ic al ; C O G : c og ni ti ve ; D ev : d ev el op m en t; E D IP P P : E ar ly D et ec ti on , I nt er ve nt io n, a nd P re ve nt io n of P sy ch os is P ro gr am ; E xt : e xt er na l v al id at io n; G A D : g en er al is ed a nx ie ty d is or de r; I C D : I nt er na ti on al C la ss ifi ca ti on o f D is ea se s; M D : m aj or d ep re ss io n; n .a .: no t av ai la bl e; N A P L S- 2: N or th A m er ic an P ro dr om e L on gi tu di na l S tu dy - 2; N E SA R C : U S N at io na l E pi de m io lo gi c Su rv ey o n A lc oh ol a nd R el at ed C on di ti on s; P D : p an ic d is or de r P H Y : ph ys ic al h ea lt h; S L A M : S ou th L on do n an d M au ds le y; S O C : s oc io de m og ra ph ic . D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 291 Implementing Precision Psychiatry health care,16,31,32 the model predicting psychosis onset in CHR-P,27,72 the model predicting the onset of generalized anxiety disorders and panic disorder in the general popu- lation,42,43 and the model predicting the onset of major de- pression in the general population36,37 were all replicated twice (table 1). None of the models included in the cur- rent systematic review were fully implemented in clinical practice. However, 1 study113 described the protocol for the implementation of the transdiagnostic risk calculator to detect individuals at risk of psychosis in secondary mental health care.16,31,32 The core aim of this study was to integrate the prediction model in the local electronic health register and evaluate the clinician’s adherence to the recommendations made by the risk calculator.113 Accuracy of Prediction Models and Meta-Regressions Accuracy of prediction models was highly variable, ran- ging from 0.5640 to 1.071 (0.6925–0.9669 for diagnostic models, 0.5640 to 1.071 for prognostic models, and 0.66108 to 0.92114 for predictive models) (supplementary tables 3–5). Within the nonoverlapping prediction model studies that reported apparent and external accuracy (n  =  18), the 2 measures were strongly correlated (r  =  .78, 95% CI: 0.39–0.95, P < .001; figure 4). Meta-regressions revealed that accuracy was higher in unimodal (n = 25) vs multi- modal (n = 71) prediction models (β = .29, P = .03), di- agnostic (n = 14) vs prognostic (n = 51; β = .84, P < .001) models, and diagnostic (n  =  14) vs predictive (n  =  11; β = .87, P = .002) models, but no other significant meta- regressions or interactions were detected (supplementary results and supplementary table 6). Quality of Prediction Models Applying PROBAST, 94.3% of the included studies were found to be at high risk of bias. The results from the different domains were heterogeneous: 1.1% were at high risk of bias in the participants domain, 65.9% in the predictors domain, 90.9% in the outcomes domain, and 81.8% in the analysis domain (supplementary table 7; supplementary figure 1). Discussion This is the first large-scale systematic review to summa- rize the transdiagnostic and life span-inclusive evidence regarding diagnostic, prognostic, or predictive prediction models that have been internally and externally validated and, thus, can be considered for clinical implementation in psychiatry. Currently, only 10.4% of the total models developed are internally validated, 4.6% are externally validated, and 0.2% are considered for implementation. Most of the models validated were prognostic, followed by diagnostic and more infrequently predictive models. Most research in this area focused on psychosis and was life span inclusive. Most prediction models employed clinical predictors. Many studies were at high risk of bias and accuracy was mediated by several factors. 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 N OIT A DIL AV L A NRETXE DEVELOPMENT r=0.78, 95% CI:0.39-0.95, P<0.001 Fig. 4. Correlation between apparent and external accuracy (n = 18). D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 292 G. Salazar de Pablo et al The main finding of this study is that precision psy- chiatry has developed into a consolidated area of clin- ical research, with a substantial number of individualized prediction models developed and validated on data from 3 889 457 participants aged from 1.8 to 64 years. These substantial advancements in the field of precision psy- chiatry reflect a life span-inclusive approach. Several val- idated individualized prediction models are nowadays available, transdiagnostically targeting many psychiatric conditions encompassing psychotic disorders, affective disorders, substance use disorders, anxiety disorders, neurodevelopmental disorders, and several clinically relevant outcomes as well. However, to date, psychosis research has mostly led (36.4%) precision psychiatry. Notably, the majority (68.2%) of the current psychi- atric prediction models were prognostic, with CHR-P studies representing a leading field (31.7%) in this domain (21.6% across all prognostic, diagnostic, and predictive models). This finding confirms the traction role of psy- chosis research, as well as the close link between preci- sion psychiatry and preventive approaches. Psychiatry as a discipline is essentially “Hippocratic,” whereby the prediction of outcomes becomes more relevant than the ascertainment of cross-sectional diagnostic categories.4 The validity of diagnostic categories in psychiatry has al- ways been criticized and it has recently been further ques- tioned by transdiagnostic approaches, which challenged discrete and fixed self-delimitating boundaries across International Classification of Diseases or Diagnostic and Statistical Manual of Mental Disorders entities.15,17 These considerations are particularly valid for early psy- chosis, where the prediction of outcomes can inform treatment approaches and can explain why diagnostic models were not so frequent (18.2%). Predictive models were even less frequently investigated (13.6%), presum- ably because these types of studies are inherently more complex to run owing to the intervention-related compo- nent. Despite these speculations, accuracy in diagnostic models remained superior to prognostic and predictive models, presumably because diagnostic models rely on more established gold standards to define outcomes. Despite the substantial progress in developing and validating individualized prediction models for psychi- atry, this study also highlighted some important barriers to the advancement of knowledge. The first barrier is that, across the overall pool of prediction models developed and published in the broader psychiatric literature (n = 584), we found only about 15% (n = 88) to be properly valid- ated (n = 61: 10.4% internal validation and n = 27: 4.6% independent external validation). Within those included in the review, about one-third were validated in external databases (supplementary limitations). This finding aligns with a previous review suggesting that external validation of prediction models is infrequent.115 A  growing body of evidence has confirmed a replicability crisis in several areas of scientific knowledge, such as cancer research,116 economics,117 behavioral ecology economics,117 and ge- netic behavior research.118 Since precision psychiatry is a relatively emerging paradigm compared to other precision medicine approaches, research to date may have priori- tized the development of new models over the external validation of models already established. For example, systematic reviews in chronic obstructive pulmonary dis- ease identified a similar number of prediction models with internal (n = 100) and external (n = 38) validation to the ones reported here.24 However, several of these models were externally validated between 5 and 17 times.24 The next generation of prediction modeling in psychiatry should, therefore, consider, along with the development of new prediction models, the replication of existing algo- rithms across different scenarios. This would necessitate collaborative data-sharing efforts to reach critical mass (studies’ sample size ranged from 2955 to 2 960 92956 in- dividuals) and the establishment of international clinical research infrastructures, as well as specific support from funders and stakeholders. The current study should also educate editors and reviewers who too often devalue repli- cation studies because they feel that these studies have lim- ited advancement of knowledge compared to the original publications. In reality, focusing on the reproducibility of existing prediction models and updating existing prog- nostic models, as opposed to dropping these models and developing new ones from scratch, is the recommended procedure to maximize the efficiency of research.4 This study also provides relevant methodological ev- idence. For example, to date, most models (69.5%) are based on clinical predictors and there is no evidence that more complex models encompassing biomarkers or a large number of predictors (which may be more prone to overfitting issues) or advanced analytical methods, such as machine learning, outperform other types of pre- diction models. These findings align with recent studies indicating that complex machine learning models do not outperform more parsimonious clinically based models developed through standard statistical approaches.53,119 The current study adds further methodological value by showing that, in psychiatry, for a given apparent accuracy (we found no difference across various accuracy meas- ures), the expected external accuracy can be estimated with a correlating factor of .78 (95% CI: 0.39–0.95; figure 4). Editors and reviewers can use this factor to as- sess the external accuracy of prediction models that have not been internally/externally validated. However, cur- rent guidelines recommend performing at least internal validation,4 which, if properly performed, can accurately index the true external generalizability of the model (as shown in our meta-regressions). An associated problem is that 94.3% studies included in the current review—which adopted stringent inclusion criteria focusing on validated studies—were eventually classified at high risk of bias, mostly because of the high risk of bias in the outcomes and analysis domain. These D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 http://academic.oup.com/schizophreniabulletin/article-lookup/doi/10.1093/schbul/sbaa120#supplementary-data 293 Implementing Precision Psychiatry biases may potentially be even more substantial in the wider literature, limiting the implementation of precision psychiatry. Although the PROBAST threshold for this bias may be too strict, our findings are consistent with an independent review, which applied PROBAST and found that 98.3% of the prediction models were at high risk of bias.24 Facilitating the external validation of individual- ized prediction models is also the most robust approach to address the currently largest barrier for precision psy- chiatry: real-world implementation. The current systematic review identified only one imple- mentation study, corresponding to 0.2% of the total pool of models developed and published, which did not report data but only described the research protocol of an on- going project120 (the full implementation results have been published upon completion of our literature review).121,122 At the moment, precision psychiatry is severely limited by a translational gap. The implementation pathways of precision psychiatry is a perilous journey,123 compli- cated by obstacles related to patients (eg, making their data available or accepting the outputs of the risk calcu- lator), clinicians (eg, adherence to the recommendations made by prediction models and communicating risks), providers (eg, confidentiality and accessibility of data and interpretability and utility of outputs), and funders and organizations (implementing an infrastructure ena- bling standard prediction procedures). Because of these challenges, most prediction models that are validated are then lost in the dearth of real-world implementation sci- ence, even for psychosis research. Implementation science itself, although much needed, is contested and complex, with the unpredictable use of results from routine clinical practice.124,125 For example, the Consolidated Framework for Implementation Research (CFIR)30 is rather theo- retical124 and does not offer specific pragmatic guidance to precision psychiatry. A recent systematic review con- cluded that only 6% of studies acknowledging the CFIR used the CFIR in a meaningful way.126 Thus, the paucity of implementation studies of individualized prediction models in psychiatry can be secondary to the lack of a general implementation framework and practical guid- ance. The next generation of empirical research in the field of prediction modeling in psychiatry and psychosis research should primarily aim at filling in the implemen- tation gap by developing a coherent and practical imple- mentation framework, methodological infrastructures, and international implementation infrastructures. Conclusions To date, several validated prediction models are available to support the diagnosis and prognosis of psychiatric con- ditions, in particular, psychotic disorders, or to predict the response to treatments. Advancements of knowledge are mostly limited by the limited replication and lack of implementation research in real-world clinical practice. The next generation of precision psychiatry research is required to address this translational gap. Supplementary Material Supplementary material is available at Schizophrenia Bulletin. Funding This study was supported by the King’s College London Confidence in Concept award from the Medical Research Council (MC_PC_16048) to Dr Fusar-Poli. Dr Salazar de Pablo and Dr Vaquerizo-Serrano are supported by the Alicia Koplowitz Foundation. Dr Danese was funded by the Medical Research Council (grant no. P005918) and the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust, and King’s College London. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. Conflict of interest: The authors have declared that there are no conflicts of interest in relation to the subject of this study. References 1. Terry  SF. Obama’s precision medicine initiative. Genet Test Mol Biomarkers. 2015;19(3):113–114. 2. Genetics Reference. What is precision medicine? https://ghr. nlm.nih.gov/primer/precisionmedicine/definition. Accessed March 1, 2020. 3. Farhud  DD, Zarif  Yeganeh  M. A brief history of human blood groups. Iran J Public Health. 2013;42(1):1–6. 4. Fusar-Poli  P, Hijazi  Z, Stahl  D, Steyerberg  EW. The sci- ence of prognosis in psychiatry: a review. JAMA Psychiatry. 2018;75(12):1289–1297. 5. Wolff RF, Moons KGM, Riley RD, et al.; PROBAST Group. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51–58. 6. Fernandes BS, Williams LM, Steiner J, et al. The new field of “precision psychiatry.” BMC Med. 2017;15(1):80. 7. Khanra  S, Khess  CRJ, Munda  SK. “Precision psych- iatry”: a promising direction so far. Indian J Psychiatry. 2018;60(3):373–374. 8. Williams  LM. Precision psychiatry: a neural circuit tax- onomy for depression and anxiety. Lancet Psychiatry. 2016;3(5):472–480. 9. Fusar-Poli P, McGorry PD, Kane JM. Improving outcomes of first-episode psychosis: an overview. World Psychiatry. 2017;16(3):251–265. 10. Fusar-Poli  P, Cappucciati  M, Borgwardt  S, et  al. Heterogeneity of psychosis risk within individuals at clinical high risk: a meta-analytical stratification. JAMA Psychiatry. 2016;73(2):113–120. 11. Fusar-Poli P, Salazar de Pablo G, Correll CU, et al. Prevention of psychosis: advances in detection, prognosis, and interven- tion. JAMA Psychiatry. 2020;77(7):755–765. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 https://ghr.nlm.nih.gov/primer/precisionmedicine/definition https://ghr.nlm.nih.gov/primer/precisionmedicine/definition 294 G. Salazar de Pablo et al 12. Tognin  S, van  Hell  HH, Merritt  K, et  al.; PSYSCAN Consortium. Towards precision medicine in psychosis: benefits and challenges of multimodal multicenter studies- PSYSCAN: translating neuroimaging findings from research into clinical practice. Schizophr Bull. 2020;46(2):432–441. 13. Davies C, Cipriani A, Ioannidis JPA, et al. Lack of evidence to favor specific preventive interventions in psychosis: a net- work meta-analysis. World Psychiatry. 2018;17(2):196–209. 14. Correll CU, Galling B, Pawar A, et al. Comparison of early intervention services vs treatment as usual for early-phase psychosis: a systematic review, meta-analysis, and meta- regression. JAMA Psychiatry. 2018;75(6):555–565. 15. Fusar-Poli  P. TRANSD recommendations: improving transdiagnostic research in psychiatry. World Psychiatry. 2019;18(3):361–362. 16. Fusar-Poli P, Davies C, Rutigliano G, et al. Transdiagnostic individualized clinically based risk calculator for the detec- tion of individuals at risk and the prediction of psychosis: model refinement including nonlinear effects of age. Front Psychiatry. 2019;10:313. 17. Fusar-Poli  P, Solmi  M, Brondino  N, et  al. Transdiagnostic psychiatry: a systematic review. World Psychiatry. 2019;18(2):192–207. 18. Chen Y, Yang K, Marušic A, et al.; RIGHT (Reporting Items for Practice Guidelines in Healthcare) Working Group. A re- porting tool for practice guidelines in health care: the RIGHT statement. Ann Intern Med. 2017;166(2):128–132. 19. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta- analyses: the PRISMA statement. BMJ. 2009;339:b2535. 20. Clark  GM. Prognostic factors versus predictive factors: examples from a clinical trial of erlotinib. Mol Oncol. 2008;1(4):406–412. 21. Moons KG, de Groot JA, Bouwmeester W, et al. Critical ap- praisal and data extraction for systematic reviews of predic- tion modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744. 22. Studerus  E, Ramyead  A, Riecher-Rössler  A. Prediction of transition to psychosis in patients with a clinical high risk for psychosis: a systematic review of methodology and reporting. Psychol Med. 2017;47(7):1163–1178. 23. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. 2019;170(1):W1–W33. 24. Bellou  V, Belbasis  L, Konstantinidis  AK, Tzoulaki  I, Evangelou  E. Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: system- atic review and critical appraisal. BMJ. 2019;367:l5358. 25. Redlich  R, Almeida  JJ, Grotegerd  D, et  al. Brain morpho- metric biomarkers distinguishing unipolar and bipolar de- pression. A  voxel-based morphometry-pattern classification approach. JAMA Psychiatry. 2014;71(11):1222–1230. 26. Ioannidis K, Chamberlain SR, Treder MS, et al. Problematic internet use (PIU): associations with the impulsive-compulsive spectrum. An application of machine learning in psychiatry. J Psychiatr Res. 2016;83:94–102. 27. Carrión RE, Cornblatt BA, Burton CZ, et al. Personalized prediction of psychosis: external validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project. Am J Psychiatry. 2016;173(10):989–996. 28. Perkins  DO, Jeffries  CD, Cornblatt  BA, et  al. Severity of thought disorder predicts psychosis in persons at clinical high-risk. Schizophr Res. 2015;169(1-3):169–177. 29. Corcoran  CM, Carrillo  F, Fernández-Slezak  D, et  al. Prediction of psychosis across protocols and risk cohorts using automated language analysis. World Psychiatry. 2018;17(1):67–75. 30. Fusar-Poli  P, Rutigliano  G, Stahl  D, et  al. Deconstructing pretest risk enrichment to optimize prediction of psych- osis in individuals at clinical high risk. JAMA Psychiatry. 2016;73(12):1260–1267. 31. Fusar-Poli  P, Rutigliano  G, Stahl  D, et  al. Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis. JAMA Psychiatry. 2017;74(5):493–500. 32. Fusar-Poli P, Werbeloff N, Rutigliano G, et al. Transdiagnostic risk calculator for the automatic detection of individuals at risk and the prediction of psychosis: second replication in an independent national health service trust. Schizophr Bull. 2019;45(3):562–570. 33. Addington DE, Beck C, Wang J, et al. Predictors of admission in first-episode psychosis: developing a risk adjustment model for service comparisons. Psychiatr Serv. 2010;61(5):483–488. 34. Leighton  SP, Krishnadas  R, Chung  K, et  al. Predicting one-year outcome in first episode psychosis using machine learning. PLoS One. 2019;14(3):e0212846. 35. Bellon JA, Luna JD, King M, et al. Predicting the onset of major depression in primary care: international validation of a risk prediction algorithm from Spain. Psychol Med. 2011;41(10):2075–2088. 36. King M, Walker C, Levy G, et al. Development and valid- ation of an international risk prediction algorithm for epi- sodes of major depression in general practice attendees: the PredictD study. Arch Gen Psychiatry. 2008;65(12):1368–1376. 37. Nigatu YT, Liu Y, Wang JL. External validation of the inter- national risk prediction algorithm for major depressive epi- sode in the US general population: the PredictD-US study. BMC Psychiatry. 2016;16:256. 38. Maarsingh OR, Heymans MW, Verhaak PF, Penninx BWJH, Comijs  HC. Development and external validation of a prediction rule for an unfavorable course of late-life de- pression: a multicenter cohort study. J Affect Disord. 2018;235:105–113. 39. Wang  JL, Patten  S, Sareen  J, et  al. Development and val- idation of a prediction algorithm for use by health profes- sionals in prediction of recurrence of major depression. Depress Anxiety. 2014;31(5):451–457. 40. Klein  NS, Holtman  GA, Bockting  CLH, Heymans  MW, Burger H. Development and validation of a clinical predic- tion tool to estimate the individual risk of depressive relapse or recurrence in individuals with recurrent depression. J Psychiatr Res. 2018;104:1–7. 41. Liu Y, Sareen J, Bolton JM, Wang JL. Development and val- idation of a risk prediction algorithm for the recurrence of suicidal ideation among general population with low mood. J Affect Disord. 2016;193:11–17. 42. King M, Bottomley C, Bellón-Saameño JA, et al. An inter- national risk prediction algorithm for the onset of generalized anxiety and panic syndromes in general practice attendees: predictA. Psychol Med. 2011;41(8):1625–1639. 43. Nigatu YT, Wang J. External validation of the international risk prediction algorithm for the onset of generalized anx- iety and/or panic syndromes (the predict A) in the US general population. J Anxiety Disord. 2019;64:40–44. 44. Liu Y, Sareen J, Bolton J, Wang J. Development and valid- ation of a risk-prediction algorithm for the recurrence of panic disorder. Depress Anxiety. 2015;32(5):341–348. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 295 Implementing Precision Psychiatry 45. Ngo DA, Rege SV, Ait-Daoud N, Holstege CP. Development and validation of a risk predictive model for student harmful drinking—A longitudinal data linkage study. Drug Alcohol Depend. 2019;197:102–107. 46. Afzali  MH, Sunderland  M, Stewart  S, et  al. Machine- learning prediction of adolescent alcohol use: a cross-study, cross-cultural validation. Addiction. 2019;114(4):662–671. 47. Gueorguieva R, Wu R, O’Connor PG, et al. Predictors of ab- stinence from heavy drinking during treatment in COMBINE and external validation in PREDICT. Alcohol Clin Exp Res. 2014;38(10):2647–2656. 48. Fazel S, Wolf A, Larsson H, et al. Identification of low risk of violent crime in severe mental illness with a clinical prediction tool (Oxford Mental Illness and Violence tool [OxMIV]): a deriv- ation and validation study. Lancet Psychiatry. 2017;4(6):461–468. 49. Chekroud AM, Zotti RJ, Shehzad Z, et al. Cross-trial predic- tion of treatment outcome in depression: a machine learning approach. Lancet Psychiatry. 2016;3(3):243–250. 50. Furukawa  TA, Kato  T, Shinagawa  Y, et  al. Prediction of remission in pharmacotherapy of untreated major depres- sion: development and validation of multivariable prediction models. Psychol Med. 2019;49(14):2405–2413. 51. Perlis  RH. A clinical risk stratification tool for predicting treatment resistance in major depressive disorder. Biol Psychiatry. 2013;74(1):7–14. 52. Kautzky  A, Dold  M, Bartova  L, et  al. Clinical factors predicting treatment resistant depression: affirmative results from the European multicenter study. Acta Psychiatr Scand. 2019;139(1):78–88. 53. Christodoulou E, Ma J, Collins GS, et  al. A systematic re- view shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12–22. 54. Borenstein M, Hedges L, Higgins J, Rothstein H. Comprehensive Meta-Analysis Version 3 [Computer Program]. Englewood, NJ: Biostat; 2013. 55. Zandvakili  A, Philip  NS, Jones  SR, et  al. Use of machine learning in predicting clinical response to transcranial mag- netic stimulation in comorbid posttraumatic stress disorder and major depression: a resting state electroencephalography study. J Affect Disord. 2019;252:47–54. 56. Simon GE, Johnson E, Lawrence JM, et al. Predicting sui- cide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951–960. 57. Pramparo  T, Pierce  K, Lombardo  MV, et  al. Prediction of autism by translation and immune/inflammation coexpressed genes in toddlers from pediatric community practices. JAMA Psychiatry. 2015;72(4):386–394. 58. Kalmady SV, Greiner R, Agrawal R, et al. Towards artificial intelligence in mental health by improving schizophrenia pre- diction with multiple brain parcellation ensemble-learning. npj Schizophr. 2019;5(1):2. 59. Rozycki M, Satterthwaite TD, Koutsouleris N, et al. Multisite machine learning analysis provides a robust structural im- aging signature of schizophrenia detectable across diverse patient populations and within individuals. Schizophr Bull. 2018;44(5):1035–1044. 60. Viviano  JD, Buchanan  RW, Calarco  N, et  al.; Social Processes Initiative in Neurobiology of the Schizophrenia(s) Group. Resting-state connectivity biomarkers of cognitive performance and social function in individuals with schizo- phrenia spectrum disorder and healthy control subjects. Biol Psychiatry. 2018;84(9):665–674. 61. Cooper JD, Han SYS, Tomasik J, et al. Multimodel inference for biomarker development: an application to schizophrenia. Transl Psychiatry. 2019;9(1):83. 62. Jauhar S, Krishnadas R, Nour MM, et al. Is there a symp- tomatic distinction between the affective psychoses and schizophrenia? A machine learning approach. Schizophr Res. 2018;202:241–247. 63. Lueken  U, Straube  B, Yang  Y, et  al. Separating depressive comorbidity from panic disorder: a combined functional magnetic resonance imaging and machine learning approach. J Affect Disord. 2015;184:182–192. 64. Parker  G, McCraw  S, Hadzi-Pavlovic  D. The utility of a classificatory decision tree approach to assist clinical differ- entiation of melancholic and non-melancholic depression. J Affect Disord. 2015;180:148–153. 65. Ding XY, Yang YH, Stein EA, Ross TJ. Combining multiple resting-state fMRI features during classification: optimized frameworks and their application to nicotine addiction. Front Hum Neurosci. 2017;11:362. 66. Ahn  WY, Ramesh  D, Moeller  FG, Vassileva  J. Utility of machine-learning approaches to identify behavioral markers for substance use disorders: impulsivity dimensions as pre- dictors of current cocaine dependence. Front Psychiatry. 2016;7:34. 67. Harrington  KM, Quaden  R, Stein  MB, et  al.; VA Million Veteran Program and Cooperative Studies Program. Validation of an electronic medical record-based algorithm for identifying posttraumatic stress disorder in U.S. veterans. J Trauma Stress. 2019;32(2):226–237. 68. James LM, Belitskaya-Lévy I, Lu Y, et al. Development and application of a diagnostic algorithm for posttraumatic stress disorder. Psychiatry Res. 2015;231(1):1–7. 69. Duda M, Ma R, Haber N, Wall DP. Use of machine learning for behavioral distinction of autism and ADHD. Transl Psychiatry. 2016;6(2):e732. 70. Duda M, Haber N, Daniels J, Wall DP. Crowdsourced valid- ation of a machine-learning classification system for autism and ADHD. Transl Psychiatry. 2017;7(5):e1133. 71. Bedi G, Carrillo F, Cecchi GA, et al. Automated analysis of free speech predicts psychosis onset in high-risk youths. npj Schizophr. 2015;1:15030. 72. Cannon  TD, Yu  C, Addington  J, et  al. An individualized risk calculator for research in prodromal psychosis. Am J Psychiatry. 2016;173(10):980–988. 73. Ciarleglio  AJ, Brucato  G, Masucci  MD, et  al. A predictive model for conversion to psychosis in clinical high-risk pa- tients. Psychol Med. 2019;49(7):1128–1137. 74. Malda A, Boonstra N, Barf H, et al. Individualized predic- tion of transition to psychosis in 1,676 individuals at clinical high risk: development and validation of a multivariable pre- diction model based on individual patient data meta-analysis. Front Psychiatry. 2019;10:345. 75. Ramyead  A, Studerus  E, Kometer  M, et  al. Prediction of psychosis using neural oscillations and machine learning in neuroleptic-naïve at-risk patients. World J Biol Psychiatry. 2016;17(4):285–295. 76. Koutsouleris N, Borgwardt S, Meisenzahl EM, et al. Disease prediction in the at-risk mental state for psychosis using neuroanatomical biomarkers: results from the FePsy study. Schizophr Bull. 2012;38(6):1234–1246. 77. Koutsouleris N, Davatzikos C, Bottlender R, et al. Early rec- ognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification. Schizophr Bull. 2012;38(6):1200–1215. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 296 G. Salazar de Pablo et al 78. Zarogianni  E, Storkey  AJ, Johnstone  EC, Owens  DG, Lawrie  SM. Improved individualized prediction of schizo- phrenia in subjects at familial high risk, based on neuro- anatomical data, schizotypal and neurocognitive features. Schizophr Res. 2017;181:6–12. 79. Chung Y, Addington J, Bearden CE, et al.; North American Prodrome Longitudinal Study Consortium. Adding a neuro- anatomical biomarker to an individualized risk calculator for psychosis: a proof-of-concept study. Schizophr Res. 2019;208:41–43. 80. Koutsouleris N, Meisenzahl EM, Davatzikos C, et al. Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transi- tion. Arch Gen Psychiatry. 2009;66(7):700–712. 81. de  Wit  S, Ziermans  TB, Nieuwenhuis  M, et  al. Individual prediction of long-term outcome in adolescents at ultra-high risk for psychosis: applying machine learning techniques to brain imaging data. Hum Brain Mapp. 2017;38(2):704–714. 82. Koutsouleris N, Kambeitz-Ilankovic L, Ruhrmann S, et al.; PRONIA Consortium. Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: a multimodal, multisite machine learning analysis. JAMA Psychiatry. 2018;75(11):1156–1172. 83. Fusar-Poli P. The clinical high-risk state for psychosis (CHR- P), Version II. Schizophr Bull. 2017;43(1):44–47. 84. Fond  G, Bulzacka  E, Boucekine  M, et  al.; FACE-SZ (FondaMental Academic Centers of Expertise for Schizophrenia) Group. Machine learning for predicting psychotic relapse at 2 years in schizophrenia in the na- tional FACE-SZ cohort. Prog Neuropsychopharmacol Biol Psychiatry. 2019;92:8–18. 85. Austin PC, Newman A, Kurdyak PA. Using the Johns Hopkins Aggregated Diagnosis Groups (ADGs) to predict mortality in a population-based cohort of adults with schizophrenia in Ontario, Canada. Psychiatry Res. 2012;196(1):32–37. 86. Dinga  R, Marquand  AF, Veltman  DJ, et  al. Predicting the naturalistic course of depression from a wide range of clin- ical, psychological, and biological data: a machine learning approach. Transl Psychiatry. 2018;8(1):241. 87. Rubenstein  LV, Rayburn  NR, Keeler  EB, et  al. Predicting outcomes of primary care patients with major depression: development of a depression prognosis index. Psychiatr Serv. 2007;58(8):1049–1056. 88. Hafeman DM, Merranko J, Goldstein TR, et al. Assessment of a person-level risk calculator to predict new-onset bipolar spectrum disorder in youth at familial risk. JAMA Psychiatry. 2017;74(8):841–847. 89. Bauer IE, Suchting R, Van Rheenen TE, et al. The use of component-wise gradient boosting to assess the possible role of cognitive measures as markers of vulnerability to pediatric bipolar disorder. Cogn Neuropsychiatry. 2019;24(2):93–107. 90. Ryu  S, Lee  H, Lee  DK, Park  K. Use of a machine learning algorithm to predict individuals with suicide ideation in the general population. Psychiatry Investig. 2018;15(11):1030–1036. 91. Walsh  CG, Ribeiro  JD, Franklin  JC. Predicting sui- cide attempts in adolescents with longitudinal clinical data and machine learning. J Child Psychol Psychiatry. 2018;59(12):1261–1270. 92. Tran T, Luo W, Phung D, et al. Risk stratification using data from electronic medical records better predicts suicide risks than clinician assessments. BMC Psychiatry. 2014;14:76. 93. Kessler  RC, Warner  CH, Ivany  C, et  al.; Army STARRS Collaborators. Predicting suicides after psychiatric hospital- ization in US Army soldiers: the Army Study To Assess Risk and rEsilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2015;72(1):49–57. 94. Russo  J, Katon  W, Zatzick  D. The development of a population-based automated screening procedure for PTSD in acutely injured hospitalized trauma survivors. Gen Hosp Psychiatry. 2013;35(5):485–491. 95. Papini  S, Pisner  D, Shumake  J, et  al. Ensemble machine learning prediction of posttraumatic stress disorder screening status after emergency room hospitalization. J Anxiety Disord. 2018;60:35–42. 96. Rosellini  AJ, Dussaillant  F, Zubizarreta  JR, Kessler  RC, Rose S. Predicting posttraumatic stress disorder following a natural disaster. J Psychiatr Res. 2018;96:15–22. 97. Galatzer-Levy  IR, Ma  S, Statnikov  A, Yehuda  R, Shalev  AY. Utilization of machine learning for prediction of post-traumatic stress: a re-examination of cortisol in the prediction and pathways to non-remitting PTSD. Transl Psychiatry. 2017;7(3):e0. 98. Karstoft KI, Galatzer-Levy IR, Statnikov A, et al. Bridging a translational gap: using machine learning to improve the prediction of PTSD. BMC Psychiatry. 2015;15:30. 99. Galatzer-Levy  IR, Karstoft  KI, Statnikov  A, Shalev  AY. Quantitative forecasting of PTSD from early trauma re- sponses: a Machine Learning application. J Psychiatr Res. 2014;59:68–76. 100. Karstoft KI, Statnikov A, Andersen SB, Madsen T, Galatzer- Levy  IR. Early identification of posttraumatic stress fol- lowing military deployment: application of machine learning methods to a prospective study of Danish soldiers. J Affect Disord. 2015;184:170–175. 101. Gueorguieva R, Wu R, Fucito LM, O’Malley SS. Predictors of abstinence from heavy drinking during follow-up in COMBINE. J Stud Alcohol Drugs. 2015;76(6):935–941. 102. Hickey N, Yang M, Coid J. The development of the Medium Security Recidivism Assessment Guide (MSRAG): an ac- tuarial risk prediction instrument. J Forensic Psychiatry Psychol. 2009;20(2):202–224. 103. Hotzy F, Theodoridou A, Hoff P, et al. Machine learning: an approach in identifying risk factors for coercion com- pared to binary logistic regression. Front Psychiatry. 2018;9:258. 104. Muñoz MA, Jeon N, Staley B, et al. Predicting medication- associated altered mental status in hospitalized patients: de- velopment and validation of a risk model. Am J Health Syst Pharm. 2019;76(13):953–963. 105. Fernandez A, Salvador-Carulla L, Choi I, et al. Development and validation of a prediction algorithm for the onset of common mental disorders in a working population. Aust N Z J Psychiatry. 2018;52(1):47–58. 106. Barker  LC, Gruneir  A, Fung  K, et  al. Predicting psychi- atric readmission: sex-specific models to predict 30-day re- admission following acute psychiatric hospitalization. Soc Psychiatry Psychiatr Epidemiol. 2018;53(2):139–149. 107. Serretti  A, Olgiati  P, Liebman  MN, et  al. Clinical predic- tion of antidepressant response in mood disorders: linear multivariate vs. neural network models. Psychiatry Res. 2007;152(2–3):223–231. 108. Maciukiewicz M, Marshe VS, Hauschild AC, et al. GWAS- based machine learning approach to predict duloxetine response in major depressive disorder. J Psychiatr Res. 2018;99:62–68. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025 297 Implementing Precision Psychiatry 109. Kautzky  A, Dold  M, Bartova  L, et  al. Refining predic- tion in treatment-resistant depression: results of machine learning analyses in the TRD III Sample. J Clin Psychiatry. 2018;79(1):16m11385. 110. Koutsouleris  N, Kahn  RS, Chekroud  AM, et  al. Multisite prediction of 4-week and 52-week treatment outcomes in patients with first-episode psychosis: a machine learning ap- proach. Lancet Psychiatry. 2016;3(10):935–946. 111. Erguzel  TT, Ozekes  S, Gultekin  S, et  al. Neural net- work based response prediction of rtms in major depres- sive disorder using QEEG cordance. Psychiatry Investig. 2015;12(1):61–65. 112. Niles  AN, Wolitzky-Taylor  KB, Arch  JJ, Craske  MG. Applying a novel statistical method to advance the personal- ized treatment of anxiety disorders: a composite moderator of comparative drop-out from CBT and ACT. Behav Res Ther. 2017;91:13–23. 113. Fusar-Poli P, Oliver D, Spada G, et al. Real world implemen- tation of a transdiagnostic risk calculator for the automatic detection of individuals at risk of psychosis in clinical rou- tine: study protocol. Front Psychiatry. 2019;10:109. 114. Koutsouleris  N, Wobrock  T, Guse  B, et  al. Predicting re- sponse to repetitive transcranial magnetic stimulation in pa- tients with schizophrenia using structural magnetic resonance imaging: a multisite machine learning analysis. Schizophr Bull. 2018;44(5):1021–1034. 115. Siontis  GC, Tzoulaki  I, Castaldi  PJ, Ioannidis  JP. External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol. 2015;68(1):25–34. 116. Nosek  BA, Errington  TM. Making sense of replications. Elife. 2017;6:e23383. 117. Ioannidis J, Stanley T, Doucouliagos H. The power of bias in economics research. Econ J. 2017;127(605):F236-F265. 118. Jennions MD, Møller AP. A survey of the statistical power of research in behavioral ecology and animal behavior. Behav Ecol. 2003;14(3):438–445. 119. Fusar-Poli  P, Stringer  D, M  S  Durieux  A, et  al. Clinical- learning versus machine-learning for transdiagnostic pre- diction of psychosis onset in individuals at-risk. Transl Psychiatry. 2019;9(1):259. 120. Fusar-Poli P, Oliver D, Spada G, et al. Real world implemen- tation of a transdiagnostic risk calculator for the automatic detection of individuals at risk of psychosis in clinical rou- tine: study protocol. Front Psychiatry. 2019;10:109. 121. Wang  T, Oliver  D, Msosa  Y, et  al. Implementation of a real-time psychosis risk detection and alerting system based on electronic health records using Cogstack. J Vis Exp. 2020(159). 122. Oliver D, Spada G, Colling C, et al. Real-world implementa- tion of precision psychiatry: transdiagnostic risk calculator for the automatic detection of individuals at-risk of psych- osis. Schizophr Res. 2020;S0920-9964(20)30259-0. 123. Chekroud AM, Koutsouleris N. The perilous path from pub- lication to practice. Mol Psychiatry. 2018;23(1):24–25. 124. Rapport  F, Clay-Williams  R, Churruca  K, et  al. The struggle of translating science into action: Foundational concepts of implementation science. J Eval Clin Pract. 2018;24(1):117–126. 125. Damschroder LJ. Clarity out of chaos: Use of theory in im- plementation research. Psychiatry Res. 2020;283:112461. 126. Kirk MA, Kelley C, Yankey N, et al. A systematic review of the use of the consolidated framework for implementation re- search. Implement Sci. 2016;11:72. D ow nloaded from https://academ ic.oup.com /schizophreniabulletin/article/47/2/284/5903901 by Fachhochschule N ordw estschw eiz (FH N W ) user on 24 January 2025