How WEIRD is CHI? Sebastian Linxen Center for General Psychology and Methodology, University of Basel, Basel, Switzerland sebastian.linxen@unibas.ch Vincent Cassau Hamm-Lippstadt University of Applied Sciences, Lippstadt, Germany vincent.cassau@stud.hshl.de Christian Sturm Hamm-Lippstadt University of Applied Sciences, Lippstadt, Germany christian.sturm@hshl.de Klaus Opwis Center for General Psychology and Methodology, University of Basel, Basel, Switzerland klaus.opwis@unibas.ch Florian Brühlmann Center for General Psychology and Methodology, University of Basel, Basel, Switzerland forian.bruehlmann@unibas.ch Katharina Reinecke Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, Washington, United States reinecke@cs.washington.edu ABSTRACT Computer technology is often designed in technology hubs in West- ern countries, invariably making it “WEIRD”, because it is based on the intuition, knowledge, and values of people who are Western, Educated, Industrialized, Rich, and Democratic. Developing tech- nology that is universally useful and engaging requires knowledge about members of WEIRD and non-WEIRD societies alike. In other words, it requires us, the CHI community, to generate this knowl- edge by studying representative participant samples. To fnd out to what extent CHI participant samples are from Western societies, we analyzed papers published in the CHI proceedings between 2016-2020. Our fndings show that 73% of CHI study fndings are based on Western participant samples, representing less than 12% of the world’s population. Furthermore, we show that most par- ticipant samples at CHI tend to come from industrialized, rich, and democratic countries with generally highly educated popu- lations. Encouragingly, recent years have seen a slight increase in non-Western samples and those that include several countries. We discuss suggestions for further broadening the international representation of CHI participant samples. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI. KEYWORDS WEIRD, sample bias, generalizability, HCI research, geographic diversity ACM Reference Format: Sebastian Linxen, Christian Sturm, Florian Brühlmann, Vincent Cassau, Klaus Opwis, and Katharina Reinecke. 2021. How WEIRD is CHI?. In CHI Conference on Human Factors in Computing Systems (CHI ’21), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA, 14 pages. https://doi.org/ 10.1145/3411764.3445488 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). CHI ’21, May 8–13, 2021, Yokohama, Japan © 2021 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-8096-6/21/05. https://doi.org/10.1145/3411764.3445488 1 INTRODUCTION CHI is widely regarded as the premier venue for Human-Computer Interaction, often infuencing technology innovations that were inspired by its publications on the design and use of computer tech- nology. Such technology innovations are being used by increasingly large numbers of people from diverse countries around the world. Commonly, the research fndings produced by the CHI community that are driving such innovations may be assumed to be universally applicable to the entire human population. However, CHI is not as international as the users of technology are. In fact, the SIGCHI Executive Committee has made it one of its key missions to ‘foster‘ HCI growth around the world” [59], recog- nizing that its members, including those who contribute research to CHI, are primarily from North America and Europe. Growing HCI around the world will be especially needed given that we are only beginning to understand how people difer in their use and acceptance of technology. Prior work in HCI has started to show that people’s use and perception of technology varies across countries and (national) cultures (e.g., [64, 70]). To name only a few examples, a user’s country of origin can afect their interaction with MOOCs [33], people from richer countries tend to be more likely to schedule meetings online, but tend to be less likely to fnd mutually agreeable times, than people from less afuent countries [72], a person’s country of origin can infuence the adoption of smartwatches [28], and a person’s culture can afect their trust in specifc website designs [20]. This past work suggests that many of the fndings about the design of technology that we have accumulated over many years of studying largely Western samples may not generalize to other countries and cultures. A factor contributing to this problem is that researchers in HCI are predominantly located in Western countries [5, 57], which sug- gests that the majority of samples likely consist of Westerners. That the lack of geographic diversity in both authors and participants of published articles is a problem has been widely recognized in the behavioral sciences. For example, already in 1984 Triandis and Bris- lin [87] pointed out the relevance of cross-cultural studies and that not only highly industrialized nations should be studied, but also societies with diferent technological developments and diferent forms of political organization. In 1999, Sue [82] raised the need to cross-validate principles and measures with diferent populations https://doi.org/10.1145/3411764.3445488 https://doi.org/10.1145/3411764.3445488 https://doi.org/10.1145/3411764.3445488 mailto:reinecke@cs.washington.edu mailto:florian.bruehlmann@unibas.ch mailto:klaus.opwis@unibas.ch mailto:christian.sturm@hshl.de mailto:vincent.cassau@stud.hshl.de mailto:sebastian.linxen@unibas.ch http://crossmark.crossref.org/dialog/?doi=10.1145%2F3411764.3445488&domain=pdf&date_stamp=2021-05-07 CHI ’21, May 8–13, 2021, Yokohama, Japan Linxen, et al. and ethnicities. In 2008, Arnett [3] published an empirical analy- sis of the prevalence of US American authors and participants in APA Journals, concluding that contributors, samples, and editorial leadership of the journals are predominantly US American, neglect- ing 95% of the world’s population. Henrich and colleagues [36] showed in 2010 that US American participants are frequent outliers when compared to the rest of humanity because they skew white and afuent. US American participants (commonly undergraduates recruited through universities’ psychology subject pools [3]) are common outliers on many psychological measures [36]. Accord- ing to Henrich et al., this makes these participants “the WEIRDest people in the world” [36], an acronym for Western, Educated, Indus- trialized, Rich, and Democratic. Research fndings based on studies with these participants may not be generalizable, despite a common assumption that published fndings apply to all human beings. Henrich et al.’s article on WEIRD subjects demonstrates that the oversampling of American undergraduates in the behavioral sci- ences impacts studies’ external validity (i.e., whether fndings can be generalized to another context) and has triggered widespread calls for studying more diverse samples and replicating prior studies in other contexts (e.g., [62, 74]), and in particular in non-Western countries [44, 94]. Similar discussions have been started in the CHI community, such as in workshops and symposia discussing the generalizability of fndings [52, 73, 81, 98]. In this paper, our goal is to further these discussions by answering the following main re- search questions: (1) To what extent are participant samples in CHI papers from Western, Educated, Industrialized, Rich, or Democratic societies?; and (2) Which countries are over- and understudied? As such, we are primarily interested in characterizing the inter- national breadth of HCI samples. HCI researchers have certainly studied non-traditional samples in Western countries, such as peo- ple of low income [9, 27, 95], or people with diferent ethnici- ties [29, 56, 84]. These studies are invaluable for understanding the diversity of people within Western countries where large parts of the population do not correspond to the typical undergraduate student that Henrich et al. referred to as “WEIRD participants” [36]. Our focus instead lies on identifying in which countries HCI partic- ipant samples are being recruited in and whether these countries, overall, tend to be more Western, Educated, Industrialized, Rich, or Democratic. To analyze the international breadth of CHI, we conducted a systematic content analysis of all papers included in the CHI pro- ceedings between 2016 and 2020. Following previous call-to-action papers published at CHI (e.g., on intersectionality [78]), we chose the WEIRD acronym developed by Henrich et al. [36] in 2010 as a framework for assessing the percentage of participant samples from Western, Educated, Industrialized, Rich, or Democratic soci- eties. The choice enables us to compare results with the feld of psychology, where a similar framework has previously been ap- plied [3]. However, it is important to note that such frameworks tend to oversimplify. In the case of Henrich et al.’s WEIRD acronym, it should be especially emphasized that countries do not consist of homogeneous populations. People within the same country can be highly diverse; not everyone in Western countries, for example, enjoys a high education and income level. Other identities, such as class, sexuality, or race, also vary across a country’s population. The WEIRD framework ignores these nuances and instead focuses mainly on the diferentiation between Westerners and the rest of the world. In our work, we go beyond this binary classifcation of the world by analyzing both the combination of all WEIRD vari- ables and each WEIRD variable individually. By looking at each WEIRD variable separately, we reveal which societies that CHI study participants come from tend to be more Western, educated, industrialized, rich, OR democratic, rather than making broader claims. Our contributions are as follows: (1) We provide the frst empirical analysis of the degree of geo- graphic breadth of CHI participant samples, showing that at least 73% of CHI study fndings in the past fve proceed- ings are based on Western participant samples. While the past two years have seen slight gains in the number of non- Western participant samples, CHI is studying and designing technology for 11.8% of the world’s population. More than half of the world’s countries (102) have not seen their people being studied over the past fve years. (2) Our analysis also revealed that most participant samples at CHI tend to come from industrialized, rich and democratic countries with generally highly educated populations. While only a third of all papers described the education of their participants, those that did suggest that around 70% of CHI study participants are college-educated. (3) We provide empirical insights into current practices of de- scribing the identity of CHI study participants and the com- position of samples. Our results show that detailed informa- tion about participants’ country is mentioned only in 39% of CHI papers, and rarely if samples can be assumed to be in the US. (4) Based on our results, we provide actionable suggestions for broadening the diversity of participant samples, including ideas for facilitating recruitment of non-Western samples, and tracking the international representation of participants in the future. (5) We also make available our data set compiled from our sys- tematic content analysis of the CHI proceedings between 2016-2020, which can be used for the replication of our re- sults, answering additional research questions, and for de- veloping strategies to increase geographic diversity. 2 RELATED WORK Sample size, diversity and generalizability. ensure that the conclusions drawn from participants in their experi- ments generalize to those who did not participate. Key methodolog- ical factors that infuence this generalizability are the sample size (coupled with participants’ diversity), and the representativeness of participants (e.g., as infuenced by sample bias). Small sample sizes have been increasingly dismissed as insuf- fciently representative of a general population. A typical sample size of 40 subjects (as found by Marszalek at al. in 2011 [58] for conventional laboratory studies) means that these studies are often underpowered and fail to replicate [4]. This is because low sample sizes provide only an extremely rough estimate of the population, one that is far too noisy to reliably detect typically-sized efects. But even larger sample sizes can fail to ensure the representativeness of participants. Arnett [3], for example, showed that, independent of sample size, most fndings in the feld of psychology are based on Researchers strive to How WEIRD is CHI? CHI ’21, May 8–13, 2021, Yokohama, Japan American undergraduate students, which tend to be more afuent, and are more likely to be white, than the general U.S. population (and much more so than the average person in the world). In addi- tion Henrich et al. found that results drawn from North-American student samples often do not generalize across cultures and de- mographics [36]. The so-called WEIRD samples [36]—participants who come from Western, educated, industrialized, rich, and demo- cratic societies—have frequently been found to be outliers when compared to those from other countries. While the feld of psychology has been at the forefront of the discussion around such biased samples, HCI researchers have raised similar concerns. For example, Bartneck and Hu [5] stated in 2009 that “only 7.8% of countries are responsible for 80% of papers in the CHI proceedings”. In addition, they found that “nearly 80 percent of all credits go to traditionally English-speaking countries (USA, UK, Canada, Ireland, Australia, New Zealand)” [5]. Similarly, Mannocci et. al. [57] identifed an unequal global distribution of publications within both CHI and the International Journal of Human-Computer Studies. They concluded that “there are a number of countries that show a very high level of interest in what is happening in IJHCS and HCI but are unable to have a signifcant publishing presence or citation impact in these outlets.” Todi [86] confrmed these dis- tributions by publishing the general statistics of CHI conferences between 2014 and 2019. Sakamoto [75] presented similar results specifcally for Asian researchers at CHI conferences. While this prior work focused on the global distribution of au- thors, we contribute an analysis of the global distribution and rep- resentativeness of participant samples. In addition, we extend this prior work by analyzing dimensions of the WEIRD acronym that have previously not been analyzed, namely whether participants come from countries that are more educated, industrialized, rich, or democratic compared to the average world population. CHI eforts to foster replicability and generalizability. Over the last decade, CHI has seen a signifcant number of activities that underline the recognition that considering diverse user characteris- tics, ensuring generalizability of HCI research results, and including previously not-included groups of researchers and participants is important. In 2011, Wilson and colleagues started the initiative RepliCHI with a panel followed by several workshops with the focus on the “solid foundations” of HCI research by replicating studies in various forms [96–99]. In 2015, Sturm et al. [81] orga- nized a WEIRD-workshop at CHI to identify HCI studies that might be “unlikely to apply to users in other countries and cultures.” Ku- mar and colleagues [48–52] have been coordinating “HCI Across Borders” workshops and symposia at CHI since 2016 with the aim of including under-served communities and diverse populations into the CHI community. During their workshop “CHInclusion” at CHI 2019, Strohmayer et al. [80] focused on “social and community issues, as well as various grassroots communities”. The importance of broadening the diversity at CHI has also been recognized by the SIGCHI Executive Committee, which has defned fve strategic initiatives based on the community’s concerns [59]. Three out of the fve initiatives focus on the diverse demographics and characteristics that need to be taken into account to represent our community on a global level. Our aim with this paper is to pro- vide the numbers that enable the CHI community to make progress on this front. Diversity information in papers. To enable the gathering of such data, and to better understand the potential limitations on external validity of specifc samples, it is of course required that authors provide such details about their participants. However, this is often not the case. Researchers have been found understate their sub- jects’ identities to simplify the communication of fndings and to strengthen the notion that their fnding may be generalizable [21]. For example, Himmelsbach et al. [37] reported that in 2016, an av- erage of 2.78 out of 16 diferent diversity dimensions (age, ethnicity & culture, gender & sex, mental abilities, physical abilities, race, sexual orientation, appearance & body, class, education, geographic, location, language & accent, migration, biographies, parental status, relationship status and religion) were mentioned in CHI papers. While this number increased between 2006, 2011, and 2016, it shows that much contextual information about the study participants is missing from papers. Schlesinger et al. [78] identifed 140 (out of 13,999) CHI publications (papers, notes, alt.chi) between 1982 and 2016 that contain at least some level of study participants’ identity description. The selection was based on 50 keywords assigned to the categories of "gender, race and class". They suggested to more con- sistently report contexts, demographics, and limitations based on identity for both authors and study participants. Our work extends theirs by quantifying how often CHI papers report on participant numbers, their country of origin, income levels, and education. 3 METHOD To evaluate the geographic diversity of CHI participant samples, we conducted a systematic quantitative content analysis of the proceedings of the CHI Conferences on Human Factors in Comput- ing Systems during the years 2016 to 2020. To enable comparison to related work in other felds (e.g., to [3]), we used the WEIRD acronym developed by Henrich et al. [36] in 2010 as a framework to determine if and to what extent participant samples in recent pro- ceedings of the CHI were Western, Educated, Industrialized, Rich, or Democratic. For a nuanced perspective, our analysis primarily focused on each factor individually (treating it like an OR logical operator), though we additionally analyzed how many participant samples came from countries that are considered WEIRD if using AND as a logical operator. 3.1 Dataset We selected the fve most recent proceedings of the CHI Conference on Human Factors in Computing Systems from 2016 to 2020 [11– 14, 16], containing a total of 3,269 articles. Analyzing fve years of CHI proceedings ensures that our analysis covers representative trends in the CHI community, including potential variations due to conference location. We focused on the CHI proceedings because the venue is the most prestigious in the feld of HCI [15], widely quoted [1, 79], has the highest impact factor among HCI venues (H5-index of 95 as of September 2020 [79]), and is considered to be highly infuential for new technology developments in scientifc and practical communities. Other venues and journals, such as Asian CHI [83] or ToCHI [85] likely show other patterns; our results therefore need to be seen in the context of CHI only. CHI ’21, May 8–13, 2021, Yokohama, Japan Linxen, et al. Table 1: The coding scheme used to extract variables of inter- est from all CHI papers between 2016-2020. The full dataset can be accessed in the supplementary materials. Focus Variable General information Title, year of publication Method Study method Author information Name, place of afliation, afliation Participant information Place of residence, education, income 3.2 Analysis Our quantitative content analysis was conducted by instrumenting the WEIRD acronym as follows: Western: We estimated the infuence of Western countries by classifying participant countries, derived from the methods section of each paper, into Western and non-Western using the classifcation of Huntington [42]. All countries of the European Union [89] were classifed as Western countries. A list with the categorization of the countries can be found in the supplementary materials. Educated: To address the educational level of the participant samples, we used two diferent approaches. (a) To determine diferences in the average educational level of participant samples, we used the mean years of schooling per person per country [63, 92] for our calculations. (b) In addition, we also collected specifc information on the educational level of participants wherever available and if the information provided in a paper was transferable to the International Standard Classifcation of Education (ISCED) [88]. This was the case for 667 papers (n=149,068). Industrialized: Since industrialization is typically estimated at a country-level (rather than at an individual level), we used GDP per capita [31] (gross domestic product per capita) as an indicator of industrialization for each country. The GDP is regarded as the most infuential characteristic for as- sessing the development and progress of a national economy [55]. To adjust for diferences in purchasing power between countries we applied purchasing power parities (PPP) in Int$. Rich: To determine participant wealth we used a two step ap- proach. (a) We used participant countries’ GNI per capita [32] (gross national income per capita, PPP, Int$) to approxi- mate participant wealth. This value refects all of the income within an economy, accounts for monetary fows in and out of a country, and approximates people’s standard of living. (b) We additionally collected participant income information from the methods sections wherever available. Democratic: We used the political rights rating [39] to deter- mine countries’ degrees of democracy. The term “political rights” covers democratic categories like electoral process, political pluralism and participation, as well as functioning of government [40]. Development of coding scheme: Because information about par- ticipants is difcult to extract automatically due to varying use of language, we decided to manually extract information from the papers. To develop a coding scheme, all authors frst decided on a set of variables of interest. One author then analyzed 100 ran- domly selected articles to establish the types of information that can be extracted about participants and participant samples, and also about the paper authors since we were interested in analyzing to what extent HCI researchers usually recruit locally. All authors reviewed and discussed the coding scheme and variables extracted from the frst 100 articles. The fnal coding scheme with all vari- ables is shown in Table 1. One author then analyzed a total sample of 3,269 articles and excluded 501 articles that did not report on a human subjects study. The fnal dataset includes 2768 articles (84.7% of the 2016-2020 CHI proceedings). In parallel, a second au- thor analysed 5% (n=139) of the articles to ensure consistency and to minimize inter-rater efects. The inter-rater reliability was κ = 0.947 – 0.986 (p < .001), 95% CI (0.87 – 0.98, 1.00). A Kappa value of .8 or higher is "almost perfect" according to Landis and Koch [53], indicating that there was little subjectivity in extracting the information. Normalizing by Country Population: The number of participants and participant samples do not indicate whether a country is over- represented or under-represented in the CHI proceedings compared to its population size. To answer our second research question, we therefore normalized the number of participants (ϕ) and participant samples (σ) by their country population using population fgures provided by the United Nations [91]. More specifcally, we calculated ψp , the participants ratio and ψs , the participant samples ratio by calculating: # of ϕ or σ (country) · population (worldwide) ψ = # of ϕ or σ (total) · population (country) Here, a value of 1 corresponds to a participant/sample number proportional to the country’s population. A ratio above 1 means that the country is over-represented, while a ratio below 1 means the country is under-represented. For example, a ratio of 0.5 means that only half as many participants/samples from this country were observed than expected relative to the country’s population. A ratio of 2 depicts the opposite: twice as many participants/samples from this country were observed than expected in relation to the country’s population. To check if and to what extent CHI participant samples tend to be from educated, industrialized, rich, or democratic countries, we correlated ψs with the indicators defned above (e.g. mean years of schooling, GDP). Since some of these variables were not normally distributed and the individual variables were non-linear, robust Kendall’s tau [46] rank correlation was used for all correlational analyses (see Table 2 for an overview). 4 RESULTS We found that 2,768 papers in the past fve years of CHI reported on a human subjects study. Of these, 2,611 papers (94.3%) reported on the number of study participants (n=1,134,282); the remaining 5.7% did not specify participant numbers. Only 1,076 papers (38.9%) explicitly mentioned participants’ country afliation or allowed to have it inferred from the author afliation ( e.g., if the recruitment How WEIRD is CHI? CHI ’21, May 8–13, 2021, Yokohama, Japan description referred to “local university” and all authors were at institutions in the same country). The following analysis is based on the 1,076 papers for which we had information about the country. Table 2: Kendall rank correlations of the participant sam- ples ratioψs with measures of Educated, Industrialized, Rich, Democratic. ncountry difers due to available data per coun- try. LL and UL indicate the lower and upper limits of a boot- strapped confdence interval (10,000 replicates). Signifcance levels: *p <.05, **p <.01, ***p <.001. Samples Variable rτ 95% CI rτ ncountr y [LL, UL] Educated .46∗∗∗ [.341, .593] 93 Industrialized .50∗∗∗ [.397, .624] 91 Rich .50∗∗∗ [.386, .623] 90 Democratic .50∗∗∗ [.381, .619] 93 Table 3: Western and non-Western participant samples. A single paper can report multiple samples. Mψ shows the av- erage ratio, Mdnψ represents the median. Samples Variable n % Mψ Mdnψ Western 1,102 73.13 5.92 5.72 Non-Western 405 26.87 1.62 0.45 Total 1507 100 4.1 Western Our fndings show that, averaged across the past fve proceedings, a large majority of CHI papers involve Western participant samples (73.13%, n=1,102 participant samples - see Table 3), recruited from a total of 31 Western countries. This is a conservative estimate, given that we only included papers that reported on the country of their participants or for which we were able to infer participants’ countries. Figure 1 shows a slight downward trend for Western samples. In line with this, the percentage of non-Western participant samples almost doubled between 2016 and 2020 (from 16.31% to 30.24%). Figure 1 also shows that the stark increase in non-Western par- ticipant samples can be attributed to the fact that the percentage of US samples has signifcantly dropped from 43.56% in 2018 to 27.96% in 2019 (where CHI was held in the UK) and to 24.84% in 2020 (where CHI was to be held in Hawaii, before the COVID-19 pandemic forced it to go virtual). This suggests that CHI authors increasingly recruit study participants from other countries than the US and also increasingly study samples from multiple countries. Which countries are over- and understudied? Table 4 provides an overview of the top 10 countries of participants by actual numbers (left column), and by participant samples (middle and right column, sorted by ratio and by number of samples, respectively). We include both of these because they show two diferent results: The actual participant numbers, counting each participant and their country 83.69 71.31 76.73 69.89 69.76 16.31 28.69 23.27 30.11 30.24 37.77 41.35 43.56 27.96 24.84 0.00 10.00 20.00 30.00 40.00 50.00 60.00 70.00 80.00 90.00 100.00 2016 (San Jose, CA, USA) 2017 (Denver, CO, USA) 2018 (Montreal, Canada) 2019 (Glasgow, UK) 2020 (Honolulu,  Hawaiʻi, USA) %  o f s am pl es Year Samples: Western Samples: Non‐Western Samples: USA Figure 1: Proportion of CHI Western-, non-Western and US participant samples, 2016-2020 plus conference locations. afliation individually, show that Western participants are strongly over-represented when compared to the world’s population. For example, participants from the US account for 54.84% of all CHI study participants although they only account for 4.25% of the world’s population (hence, a high participant ratio of ψp =12.91). The columns in the middle and on the right in Table 4, showing the countries of overall samples, give us a better feel for how many fndings are based on Western participant samples. US samples account for 45.82% of all participant samples, samples from Great Britain for an additional 15.71%, followed by German samples with 8.74%. While most participant samples are recruited in the US, the US is not the most over-represented country at CHI. As the middle columns in Table 4 show, 18 participant samples were from Finland, which means that fndings based on Finnish samples were strongly over-represented with respect to the world’s population (ψs =16.80). Naturally, countries with small population sizes are found in the top 10 of this table, such as Finland, Luxembourg, Switzerland, Denmark, and even St. Lucia or Bhutan. The USA would appear only in place 12 in this ranking. As a next step, we set out to analyze how the countries of partic- ipant samples are geographically distributed and which countries and regions may be understudied. Figure 2 shows the worldwide distribution of participant samples relative to the country popula- tion (countries by ratio). While participants in CHI studies between 2016 and 2020 came from 93 countries, large numbers of countries are completely missing from this map, especially in Africa, but also in Central and South America, Europe, the middle East, and Central and South Asia. More precisely, 102 of 195 countries (52%) did not have any participant samples at CHI (using the list of countries from [60, 61]). What is the reason for an increase in geographic diversity? Our results showed that CHI participant samples are predominantly recruited from Western countries, but that recent years have seen CHI ’21, May 8–13, 2021, Yokohama, Japan Linxen, et al. Table 4: Top 10 countries of CHI participants between 2016 to 2020. “Participants’ countries” shows the total number of partici- pants by country, counting each participant individually. “Countries by ratio” ranks countries by ratio of participant samples, showing their infuence on CHI relative to the country’s population size. “Countries by nsamples ” ranks countries by the number of samples that report on participants from that country. Countries marked with an asterisk are considered to be non-Western countries that are at or below the median for at least one of the EIRD criteria. Participants’ countries Countries by samples ratio Countries by nsampl es Rank Country npar t icipant s % ratio ψp Country nsampl es % ratio ψs Country ns ampl es % ratio ψs 1 USA 136,834 54.84 12.91 St. Lucia∗ 1 0.09 28.17 USA 493 45.82 7.70 2 Ireland 1,423 0.57 9.00 Finland 18 1.67 16.80 Great Britain 169 15.71 12.88 3 Switzerland 2,152 0.86 7.77 Luxembourg 2 0.19 16.53 Germany 94 8.74 5.80 4 Finland 1,336 0.54 7.53 Denmark 16 1.49 14.29 Canada 82 7.62 11.24 5 Canada 6,367 2.55 5.27 Bhutan∗ 2 0.19 13.41 China∗ 64 5.95 0.23 6 New Zealand 768 0.31 4.98 Switzerland 22 2.04 13.15 India∗ 57 5.30 0.21 7 Bhutan∗ 92 0.04 3.72 Great Britain 169 15.71 12.88 Australia 53 4.93 10.75 8 Great Britain 7,829 3.14 3.60 Canada 82 7.62 11.24 South Korea 44 4.09 4.44 9 Australia 2,544 1.02 3.12 Sweden 21 1.95 10.76 France 29 2.70 2.30 10 Denmark 577 0.23 3.11 Australia 53 4.93 10.75 Japan 29 2.70 1.19 ψ = 0 0 < ψ ≤ 0.05 0.05 < ψ ≤ 0.1 0.1 < ψ ≤ 0.5 0.5 < ψ ≤ 1 1 < ψ ≤ 5 5 < ψ ≤ 10 10 < ψ ≤ 30 Figure 2: Worldwide distribution of CHI participant samples ratio (ψs ) between 2016-2020, showing which countries are over- represented (ψ > 1) or under-represented (ψ < 1), relative to the world’s population. Countries in white (N=102) did not have study participants in the past fve CHI proceedings. a slight increase in non-Western samples. We followed up on this result by investigating whether online studies and studies of be- havioral logs from online services available in various countries could explain the increase in diverse samples. Per proceeding year, we looked at the methods used in the two papers with the most participants, the two papers with the most diverse samples, and the two papers with the most diverse author afliations (30 papers in total). The results of this additional analysis showed that the number of papers that studied participant samples from more than one country has increased in the past years—from an average of 9 papers between 2016-2018 to 29 papers in 2019 and 30 papers in 2020. Most commonly among the top 20 papers with most participants were analyses of behavioral log data, surveys, or (very few) experiments conducted on social networking sites (8/20 papers, e.g., [10, 54]), on other online services, such as on online education platforms (e.g., [24]), or on online game sites (e.g., [54]). Some of the increase in participant diversity can also be attributed to studies that have been conducted on online platforms such as Mechanical Turk (e.g., [6]) or LabintheWild (e.g., [41, 65]). How WEIRD is CHI? CHI ’21, May 8–13, 2021, Yokohama, Japan 4.2 Educated We found a positive correlation between the participant samples ratio ψs and the countries’ average duration of schooling (rτ =.46, p<.001) as shown in Figure 3: Most participant samples at CHI come from countries with generally highly educated populations. In comparison to the world population, which has 8.4 years of schooling on average [92], the countries most represented at CHI are heavily skewed towards more years of schooling on average. AFG ARG AUS AUT BEL BGD BGR BIH BOL BRA BTN CAN CHE CHL CHN CIV CMR COD COL CUB CYP CZE DEU DNK ECU EGY ESP ETH FIN FRA GBR GEO GHA GRC HND HRV HUN IDN IRL IRN IRQ ISR ITA KEN KHM KWT LBN LCA LKA LSO MAR MEX MNG NAM NGA NLD NPL PAK PER PHL POL PRT ROU RUS SAU SGP TUN TUR TWN UAE UKR URY USA VEN ZAF 1 5 10 15 20 25 3 4 5 6 7 8 9 10 11 12 13 14 Educated: Mean years of schooling P ar tic ip an t s am pl es r at io p er c ou nt ry Ψ Figure 3: Education: Relationship between the participant samples ratio ψs and the mean years of schooling per coun- try. The dotted line indicates the positioning in which a country would be represented proportionally to the coun- try’s population. The blue solid line indicates the locally- weighted regression (loess) line. Note the logarithmic scale of the y-axis. To evaluate the representativeness of education level based on participant samples, we further turned to more detailed descriptions in the papers. Of the 2768 papers with a study, 952 (34.3%) addressed the education of their participants in some way. A slightly lower number of 667 paper (24.1%) provided sufciently detailed infor- mation to convert the data into the ISCED Education levels [88] for comparison. The majority of the participants in these papers (69.93%, n=104,237) were currently enrolled at a university or had completed a university education. 24.45% had higher secondary education, 1.35% had lower secondary education, 0.16% had only primary education and 0.05% had no formal education at all. 1.87% had vocational education and training (dual education). 2.21% of the participants did not provide any information when education level was recorded. The results demonstrate that around 70% of CHI study partici- pants (in the 24% of papers that described the education level in a way sufcient for comparison) are college-educated and that over- all, participant samples are signifcantly more educated than the average world population. 4.3 Industrialized and Rich Only 4.55% (n=126) of the studies in our dataset mentioned the in- come of participants, and the vast majority of these did not mention any numbers and instead characterized their participants as “low income”. As such, we are unable to directly establish whether the wealth of participants is representative of a general population. We instead followed Arnett’s approach [3] in using GDP per capita and GNI per capita as proxies for industrialization status and wealth, as described in the methods section. While this approach cannot be used to make inferences about the industrialization status and wealth of specifc participant samples, it nevertheless allows us to gauge whether CHI samples may be skewed towards more industrialized and rich countries. Note that while Western countries are frequently industrialized, non-Western countries such as Japan or Korea are in the top 25 of the list of countries’ GDP and GNI per capita. Our results show that CHI participant samples are predomi- nantly from industrialized countries with a high GDP per capita (rτ = .50, p < .001). Seven (7.5%) of participants’ countries are among the top 10 largest economies according to their GDP [31]. Like- wise, participant samples come from signifcantly richer countries (M=27,850 Int$) (as measured by the GNI per capita per country) than the average person’s wealth (M=17,591 Int$ [32]). This is also supported by the positive correlation between the GNI per capita and the participant sample ratio per country (rτ =.50, p < .001). 4.4 Democratic CHI’s participant samples are predominantly from democratic coun- tries, with a medium correlation between the countries’ political rights and the sample ratios (rτ =.50, p <.001). The correlation can be seen in Figure 4, in which the majority of countries that participant samples came from are clustered on the right side, indicating that they are countries with the greatest degree of freedom in terms of political rights. AFG ARG AUS AUT BEL BGD BGR BIH BOL BRA BTN CHE CHL CHN CIV CMR COD COL CUB CYP DEU ECU ESP ETH FIN FRA GEO GHA HND HRV IDN IND IRL IRQ ISR JAM JPN KEN KHM KOR KWT LBN LCA LKA LSO LUX LVA MDA MEX MNG MYS NAM NGA NPL PAK PER PHL POL ROU RUS SAU SGP SYR THA TUR UAE URY USA VEN VNM ZAF SLV 1 5 10 15 20 25 0 5 10 15 20 25 30 35 40 Democratic: Political rights [−4 to 0=smallest degree of freedom, 40=greatest degree of freedom] P ar tic ip an t s am pl es r at io p er c ou nt ry Ψ Figure 4: Democratic: Relationship between participant sam- ples ratio ψs per country and the average political rights rat- ings. The dotted line indicates the positioning in which a country would be represented proportionally to the coun- try’s population. The blue solid line indicates the locally- weighted regression (loess) line. Note the logarithmic scale of the y-axis. CHI ’21, May 8–13, 2021, Yokohama, Japan Linxen, et al. 4.5 To what extent are CHI participant samples from Western, Educated, Industrialized, Rich, AND Democratic countries While our prior analyses show that most CHI participant samples are WEIRD if focusing on each WEIRD factor individually (i.e., using OR as the logical operator), we were additionally interested in investigating for how many participant samples all characteris- tics of WEIRD apply (i.e., using AND as the logical operator). We found that 26 countries of 31 Western countries in our dataset are considered WEIRD (if using AND and greater than median for each of the “EIRD” characteristics as a cut of), including all but Bhutan in the list of countries that contributed most participants relative to their population (see left column in Table 4). Of all 1507 participant samples in our dataset, 1102 (73.13%) were recruited in Western countries and 1070 (71%) were were recruited in countries that considered fully WEIRD, i.e., they are Western AND above the median for educated, industrialized, rich, and democratic. The remaining 405 participant samples (26.87%) were recruited in non-Western countries. Only 85 (5.64%) of these were samples recruited in non-Western but “EIRD” countries (Ar- gentina, Chile, Israel, Japan, and Korea.) 4.6 Sample Diversity in Western Countries and “EIRD” Samples in non-Western Countries Our dataset and analysis additionally showed that of those papers that use Western samples only a small number studied people of a lower education or income level than the more common (under- graduate student) samples. These papers, such as work by Dillahunt et al. [25, 26], Dombrowski et al. [27], Redmiles et al. [69] as well as Saksono et al. [76] are noteworthy examples of investigations into the diversity of people within Western countries. Hence, while these participant samples are Western, but not strictly "EIRD", they are currently an exception rather than the rule. Of course many non-Western countries, such as South Korea, Japan, Israel, Chile, or Argentina, are often highly educated, industrialized, rich, and/or democratic. Participant samples from these “EIRD-countries” consti- tute 20.99% of non-Western participant samples (n=85) and 5.64% of all participant samples in the past fve years at CHI. These samples can greatly contribute to the international breadth of CHI research and to our understanding of users in diverse (national) cultures. 4.7 Relationship Between Participant Samples and Author Afliations To evaluate whether the geographic breadth of participant sam- ples is broader than that of the location of authors (which would shed light on the extent to which authors recruit beyond their local area), we analyzed the author afliations reported in the 1076 arti- cles that also contained information about participants’ countries. Of those, 874 (81.23%) papers studied participants from the same country as at least one of the authors’ institutions. In 202 papers (18.77%), at least part of the participant sample was from a country diferent from the country or countries the authors are afliated with through their institution. This includes 108 papers (10.0%) that studied participants samples from countries that do not match the country of authors’ institutions. Overall, these results demonstrate that a vast majority of authors (over 80%) recruit samples “in their own backyard”, or at least in the same country as they are in. 5 DISCUSSION The primary goal of our work was to quantify the geographic breadth of CHI participant samples. Are participant samples that CHI publications report on representative of the world’s popula- tion? Our analysis of CHI proceedings between 2016-2020 shows that they are not: 73% of fndings are derived from studies with Western participant samples of which 97% come from countries to which all of the fve WEIRD variables can be applied. This means that almost 3/4 of the knowledge we produce at CHI is based on 11.8% of the world’s population. Moreover, more than half of all countries (102 of 195 countries) did not have any participant sam- ples at CHI in the past fve years, suggesting that we know very little about technology users in those countries. Unsurprisingly, a plurality of CHI’s participant samples (45.82%) in the past fve years were recruited in the US. While we were unable to derive participants’ ethnicity, US samples in the social sciences—most commonly recruited at universities—are likely to be primarily European American [3] and this may be equally the case for most participants in HCI (though it has to be acknowledged that there is a growing movement within HCI to study diverse samples within the geographic US). In addition, while samples recruited at US universities may include international students, they are a minority in these studies and may not be representative of the population in their home country. On the upside, our fndings also, for the frst time, showed that CHI participant samples are becoming more geographically diverse. Between 2016 and 2020, the fraction of US samples dropped by around 13 percentage points to 24.84%. During the same time frame, the percentage of non-Western samples almost doubled from 16.31% to 30.24%. While some of this may be explained by the choice of con- ference location, several other factors seem to have had a positive infuence: First, the numbers may be starting to refect SIGCHI’s ef- forts to diversify the CHI community and authorship, which in turn leads to the recruitment of local samples in non-Western countries. Second, we also found an increase in the number of online studies and studies of behavioral logs, many of which include participants from several diferent countries. This increase can be attributed to a growing awareness that fndings based on one population may not generalize, research eforts that have produced guidelines on conducting online experiments with diverse samples, and to the general big data trend we have seen emerge in recent years. Third, the feld of ICTD (or HCI4D) has shed light on technology use in many non-Western countries. Its growth over the past years [22] undoubtedly contributed to the increase in non-Western samples. Similarly, the past years have seen a steady increase in research on non-student samples in Western countries, such as work by Dil- lahunt et al., Dombrowski et al., Harrington et al. and Vines et al., to name just a few, who studied people with low income or people with limited education (e.g., [25, 27, 34, 95]). All of this research contributes to our understanding of variations in how technology is being used and perceived and we hope that this trend that we have been seeing will continue. How WEIRD is CHI? CHI ’21, May 8–13, 2021, Yokohama, Japan We also aimed to identify which countries are over- and under- studied. We found that participant samples from many countries are strongly over-represented at CHI compared to their country’s population size, including Finland, Denmark, Switzerland, Great Britain, Canada, Sweden, and Australia. Notable outliers in the list of over-represented countries included St. Lucia (1 sample), Lux- embourg and Bhutan (2 samples respectively), all of which have small population sizes. Overall, the list of over-represented coun- tries indicates that CHI’s participant samples are often recruited in Western societies that are also more educated, industrialized, rich, and democratic than the majority of the world’s countries. Quite strikingly, about 70% of participants were currently enrolled at a university or had completed a university education. This is unrepresentative of the world’s population, which only enjoys an average of 8.4 years of schooling [92]. Similarly, CHI participants are signifcantly wealthier than the average person (GNI per capita: 27,850 Int$ versus 17,591 Int$). When we examine these results, we might be inclined to com- pare them to other research felds. Are CHI participant samples more diverse than those in other felds? In 2008, more than 11 years ago, Arnett [3] found that 96% of participants in journals of the American Psychological Association were from Western countries, including the US, Canada, European countries, Australia, and Israel. In 2017, Nielson and colleagues showed that these numbers have changed little, critiquing that psychology research is not “readily embracing change” [62]. The 96% of Western samples in the feld of psychology stands in contrast to 73% of Western samples at CHI. In addition, 68% of the participants samples in psychology were in the United States—22% more than the 45.82% of US partic- ipant samples we found in the CHI proceedings. These numbers suggest that CHI samples are more representative than those in psychology. The result is plausible, given that participant recruit- ment in psychology is still largely dependent on student participant pools and study credits, which are not as commonly used by HCI researchers. However, Arnett also found that the 96% of psycho- logical samples only represented 12% of the world’s population, which is exactly the same percentage of the world population that we found represented in the past fve CHI proceedings. Hence, while CHI researchers recruit subjects from more countries than psychology, they are repeatedly recruiting from countries that are already over-represented. Focusing on growing CHI in countries that are currently under-represented would therefore improve the representation of the world’s population, at least geographically. Overall, our fndings show that CHI researchers are still most commonly—in at least 73% of all cases— studying participants who have been shown to difer from the average person in their behav- iors, preferences, analytic reasoning, and in their degrees of fair- ness or cooperation [36], inhibiting external validity and a broader understanding of how people use technology. Indeed, HCI studies comparing countries and cultures in recent years have started paint- ing a picture of the diversity of technology users, showing difer- ences ranging from security and privacy behavior [77], perceptions of emoji [47], to visual preferences for websites [70] and social com- parison [7]. Many of these studies have concluded that one size may not ft all and that we should be increasing eforts to understand technology use in other countries and cultures [47, 70, 77]. To do so, it will be essential to continue eforts by the CHI community to increase studies of participant samples in other countries. In the next section, we discuss ideas for further diversifying who we study and how. 6 IDEAS FOR MAKING CHI LESS WEIRD Readers may now ask themselves: Should all CHI researchers study geographically diverse samples? We believe this is neither possible nor desirable. In fact, there is immense value in focused studies that investigate specifc groups of people in specifc countries and contexts, such as commonly done by ICTD researchers [22] or by HCI researchers focusing on populations with certain ethnicities, income levels, education levels, or other identities (e.g., [25, 27]). This includes studies with very common participant samples, such as American undergraduates. All of these studies have in the past contributed insights that formed the technology we use today. How- ever, we do think that the CHI community needs to amplify its eforts to study non-WEIRD participant samples and to clearly com- municate their samples’ identities and potential implications for generalizability. Improving the representation of non-WEIRD participants, how- ever, is a complex undertaking and raises concerns of power, as suggested in Irani and colleagues’ work on Postcolonial Comput- ing [43]. A well thought-through solution should not only focus on increasing the number of non-WEIRD participants, but also on increasing the diversity of researchers and of those who are commissioning and funding the research (i.e., companies and fund- ing agencies). Considering diferent stakeholders will lead to an increase in the diversity of viewpoints, research needs, and inter- pretations of research and results. Based on our results, we compiled a list of possible ideas to address the fact that CHI research skews Western. It must be men- tioned that this paper was written by authors who work and live in three countries that all meet the WEIRD criteria. While many of us have years of experience in intercultural contexts in business and science, this background infuences our view of potential solutions that may address the WEIRD problem. We hope that these ideas are seen as a starting point to more comprehensive discussions among the global CHI community. Diversifying authorship: Our results demonstrated that 81.23% of CHI papers in the past fve years reported on locally recruited samples, suggesting that a key opportunity to achieve a greater sample diversity is to grow the geographic breadth of authors across the world. Balancing the number of CHI authors and co- authors from non-WEIRD and WEIRD countries will have various positive efects, from diversifying viewpoints and counteracting confrmation bias to facilitating the recruitment of non-WEIRD participant samples and enhancing the discussion of CHI fndings and research emphases. The most direct way to achieve an increase in non-Western authorship is to increase the number of papers au- thored by non-Western authors. In addition, the CHI community could also increase eforts to foster collaborations across Western and non-Western countries. However, it is crucial that such collab- orations result in mutually benefcial collaborations. In particular, diversifying authorship should never be a means to an end; instead, the focus should be on achieving a shared research goal and promot- ing mutual support and beneft. Given the current academic system CHI ’21, May 8–13, 2021, Yokohama, Japan Linxen, et al. of recognizing research contributions, it is particularly important that none of the important work goes unnoticed (see for instance [38] for a more in-depth discussion on crediting contributions to scientifc scholarly output). Potential avenues for diversifying authorship have already been ongoing (e.g., SIGCHI’s initiatives [59]). These eforts could be continued and extended by nurturing interactions and collabora- tions among researchers across countries, such as by (virtually) co-locating the CHI conference with SIGCHI In-Cooperation con- ferences (e.g., with Asian CHI [83], IndiaHCI [35], or CHIuXiD [17] in Indonesia and South-East Asia), by developing workshops that get together researchers from various countries and regions, or by continuing eforts like the open and scalable university laboratories as was done by Vaish et al. [93]. In addition, it will be important to reduce barriers to publishing and attending CHI, such as by more frequently seeking conference locations in non-Western countries, which our data suggests may increase the number of submissions that include non-Western participant samples. In line with this, many other approaches would need to be combined: from lowering registration fees and/or enabling virtual attendance, growing the reviewer pool to include more diverse viewpoints for evaluating submission, all the way to ensuring that potential language barriers do not skew paper acceptances towards English-speaking countries. None of these approaches are straightforward to achieve given budget and other constraints, but little steps towards some of these may already go a long way. Fostering the use of online research: While local recruitment of diverse participants remains a bottleneck for studying represen- tative samples, studying online samples can sometimes help [8]. As we found in our analysis of papers that study participants from more than one country, they usually reported on studies of behav- ioral log data and large-scale surveys and experiments. Research that is amenable to the recruitment of online participants could be more often conducted online, preferably by authors from vari- ous countries and cultures to promote research diversity and ofer various perspectives. To support researchers in conducting such online studies, HCI research should add to existing eforts that have investigated how online research can preserve data quality and allow a wider variety of experiment methodologies (e.g., [8, 66–68, 71], including qual- itative studies. In addition, HCI research has already contributed novel crowdsourcing platforms [30] and volunteer-based experi- ment platforms [45, 71] that mitigate some of the concerns about Mechanical Turk [67]; these eforts should be continued and ideally be made available to all of HCI. Moving all research online is of course neither possible nor desirable; in fact, online research ex- cludes large parts of the world population who are without Internet access (an estimated 49% [90]), who do not access specifc plat- forms and services, or who are not reached by online recruitment messages. Nevertheless, we believe that online research could be increased and that this could add to our understanding of people’s technology use in other countries and cultures. While this may sound straightforward, it is not going to be a solution to solely support Westerners in doing more research with non-WEIRD participants. In fact, such a scenario could easily aggra- vate imbalances, for example if Western companies beneft from this research by increasing sales in non-Western markets. As mentioned above, any eforts to increase the use of online research have to go hand in hand with other advances to improve the diversity of CHI authorship, tech leadership, and funding sources. Specifcally, these eforts should be focused on supporting researchers in conducting online studies in non-Western countries as much as in Western countries to ensure that we can deepen our understanding of HCI using various points of view. Developing methods for studying geographically diverse samples: Another way to increase geographically diverse par- ticipant samples is by supporting Western and non-Western re- searchers in conducting studies with participants in countries other than their own by developing appropriate methods and case studies. For example, prior research has shown a fve times increase in re- sponse bias if interviewers are foreign researchers requiring a trans- lator, and ofered guidelines for reducing this bias [23]. Researchers have also developed methods for eliciting values in non-Western societies [2], and for adapting the think-aloud method to other cultures [18]. Irani et al. [43] proposed a reframing of methods to see participants as “active participants and partners” rather than a passive knowledge resource. In short, the CHI community has a breadth of knowledge about conducting research across coun- tries and cultures, large-scale research with diverse samples, and qualitative studies in local communities. However, it is rare that we share the experiences and knowledge gained when designing, recruiting for, and conducting these studies, including what may have gone wrong. Encouraging CHI papers, experience reports, and workshops on these topics would provide a go-to-guide for authors who are interested in studying small and foreign or large and diverse samples and lower the barrier to entry to researching non-local, diverse samples. Appreciating replications and extensions of fndings: CHI has seen various discussions around the replication of research (see, e.g., [73]), which can ensure that fndings are stable, despite diferences in the makeup of participant samples or over time. RepliCHI [99], for example, is a series of workshops at CHI that has called for discussions around revisiting work for purposes of vali- dation. But replications can also uncover variations in the fndings that may be due to demographic, geographic, and/or cultural dif- ferences between samples included in the original and replication study. Such replications and extensions of studies should continue and be promoted, including eforts to raise awareness among au- thors and reviewers about the value of attempting replication and extension of prior results in other countries, in a variety of contexts, and with a variety of participants. Report and track the international breadth of participant samples: One surprising fnding in our study of CHI papers was that 5.7% of the CHI papers in the past fve years did not mention any participant numbers. A little more than 56% did not mention participants’ country afliation, nor could it be inferred from the author information. While reporting on participants’ countries in detail may not be realistic for studies with geographically diverse samples, including country information should become standard for most CHI papers to facilitate replications, extensions, meta anal- yses, and to track the international breadth of participant samples in the future. Fostering the inclusion of geographic information in CHI papers most likely requires better guidelines for reporting on the number of participants and their demographic information, How WEIRD is CHI? CHI ’21, May 8–13, 2021, Yokohama, Japan such as age, gender, education level, and country of origin. These guidelines should already be incorporated in the paper templates and in structured ways during submission, providing authors with best-practice examples of how to report on participants. Similar to the recommendations Schlesinger et al. [78] made by suggesting to “consistently report context” and “consistently report demograph- ics” in research papers, we also believe that better guidelines and standards for reporting this information will be essential for facili- tating automated analyses and better tracking of the geographic breadth of participant samples in the future. Being able to automati- cally extract participant demographics from the papers and/or meta data provided by the authors would not only facilitate automatic analysis, but also meta studies and comparisons between studies. Identifcation of constraints on generalizability: The fact that most CHI papers study Western samples in itself is not neces- sarily bad; it is only questionable in cases where fndings may not generalize, but are presented that way. Similar to our suggestion above, it may be helpful if papers detailed on the sample composi- tion, how it compares to the world population, and whether this may impact generalizability of results. One way to ensure that pa- pers adequately describe samples, address potential question of generalizability, and suggest future work to replicate or extend the study with a diferent sample, is by having a geographically diverse set of reviewers. These reviewers could be encouraged to not only describe the contribution of a paper as is already standard, but to also pay attention to the representativeness of samples and gener- alizability across countries and culture. Non-Western reviewers, in particular, may be more sensitive to fndings that may not general- ize, and may be able to suggest alternative interpretations in their reviews. Ideally, CHI should encourage at least one non-Western reviewer per paper and include recommendations on what to look for in the PCS review form. While far from complete, we hope that these initial ideas can serve as a starting point for further ideation and brainstorming among the CHI community for how to increase the diversity at CHI. 7 LIMITATIONS AND FUTURE WORK Our work has focused on the WEIRD framework and the geographic distribution of study participants. However, nationality, education, level of industrialization, economic power and the political context are only a very small subset of factors that might infuence fndings in HCI. This means that our work does not generalize beyond the WEIRD framework and the results should not be used to infer the diversity of CHI participant samples in general. Our results also do not allow inferences about individuals’ demo- graphics and identity because papers in the CHI proceedings rarely provided detailed information, such as on a participant’s country of origin, their cultural norms, or personal education and income level. A limitation of our ideas for diversifying CHI participant samples is that this paper was written by authors from countries that all meet the WEIRD criteria. As mentioned in the Discussion section, this has undoubtedly infuenced our view of potential approaches that may address the WEIRD problem. An important next step will therefore be to discuss and broaden these ideas with the global CHI community. In future work, it would also be interesting to compare our work to related conference venues and journals, such as CSCW [19] or ToCHI [85] and to more systematically test which changes to these conferences lead to a geographic broadening of participant samples. We also hope that our work will spark an interest in tracking WEIRD metrics over time, as discussed above. We are also excited about future eforts that focus on within- country representation of authors and participants. We envision such future work to rethink using the political world map as a major reference to group study participants. Instead, researchers could develop ways to quantify the impact of all diverse factors on human interaction with computers, which would necessitate regrouping people based on individual characteristics which they share across national borders, such as gender, personality, education, religion, class and experience with technology. 8 CONCLUSION Our goal with this work was to quantify the geographic breadth at the premier conference for Human-Computer Interaction, CHI. We presented an empirical analysis of the international representative- ness of participant samples between 2016 and 2020, showing that at least 73% of CHI study fndings are based on Western participants, representing less than 12% of the world’s population. Our fndings revealed that participant samples are more educated, industrialized, rich, and democratic than the average population, demonstrating that CHI is largely a conference of WEIRD participant samples. Encouragingly, our analysis also found that the number of non- Western samples has increased in recent years, and that several studies conducted in Western countries included less commonly recruited participants, such as low-income or non-college educated populations. This suggests that CHI eforts on diversity may be starting to bear fruit. Based on these results, we provided actionable suggestions on diversifying CHI authorship, facilitating recruitment of non-Western samples, and tracking geographic representation of study participants in the future. We hope that our fndings lead to further discussions around diversity and inclusion in the CHI community. 9 DATASET Our dataset including all annotated articles from the 2016-2020 CHI proceedings can be found in the supplementary materials. ACKNOWLEDGMENTS Many thanks to the anonymous reviewers for their helpful sugges- tions for improving this article. This work was partially supported by NSF award 1651487. REFERENCES [1] ACM. 2020. Conferences CHI - CHI: Conference on Human Factors in Computing Systems. https://dl.acm.org/conference/chi [2] Taghreed Alshehri, Reuben Kirkham, and Patrick Olivier. 2020. Scenario Co- Creation Cards: A Culturally Sensitive Tool for Eliciting Values. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376608 https://dl.acm.org/conference/chi https://doi.org/10.1145/3313831.3376608 CHI ’21, May 8–13, 2021, Yokohama, Japan Linxen, et al. [3] Jefrey J. Arnett. 2008. The neglected 95%: Why American psychology needs to become less American. American Psychologist 63, 7 (2008), 602–614. https: //doi.org/10.1037/0003-066X.63.7.602 [4] Marjan Bakker, Annette van Dijk, and Jelte M. Wicherts. 2012. The Rules of the Game Called Psychological Science. Perspectives on Psychological Science 7, 6 (2012), 543–554. https://doi.org/10.1177/1745691612459060 [5] Christoph Bartneck and Jun Hu. 2009. Scientometric Analysis of the CHI Proceedings. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). ACM, New York, NY, USA, 699–708. https: //doi.org/10.1145/1518701.1518810 [6] Frank Bentley, Kathleen O’Neill, Katie Quehl, and Danielle Lottridge. 2020. Explor- ing the Quality, Efciency, and Representative Nature of Responses Across Mul- tiple Survey Panels. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376671 [7] Moira Burke, Justin Cheng, and Bethany de Gant. 2020. Social Comparison and Facebook: Feedback, Positivity, and Opportunities for Comparison. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376482 [8] Krista Casler, Lydia Bickel, and Elizabeth Hackett. 2013. Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior 29, 6 (2013), 2156 – 2160. https://doi.org/10.1016/j.chb.2013.05.009 [9] Beenish M. Chaudhry, Louis Faust, and Nitesh V. Chawla. 2019. From Design to Development to Evaluation of a Pregnancy App for Low-Income Women in a Community-Based Setting. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages. https://doi.org/10.1145/3338286.3340118 [10] Justin Cheng, Moira Burke, and Elena Goetz Davis. 2019. Understanding Per- ceptions of Problematic Facebook Use: When People Experience Negative Life Impact and a Lack of Control. In Proceedings of the 2019 CHI Conference on Hu- man Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/ 3290605.3300429 [11] CHI. 2016. CHI ’16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. https://dl.acm.org/citation.cfm? id=2858036 [12] CHI. 2017. CHI ’17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. https://dl.acm.org/citation.cfm? id=3025453 [13] CHI. 2018. CHI ’18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. https://dl.acm.org/citation.cfm? id=3173574 [14] CHI. 2019. CHI ’19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. https://dl.acm.org/citation.cfm? id=3290605 [15] CHI. 2019. CHI History Page. https://sigchi.org/conferences/conference-history/ chi/ [16] CHI. 2020. CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. [17] CHIuXiD. 2020. CHIuXiD 2020 | 6th International HCI and UX Conference. https://chiuxid.org/ [18] Torkil Clemmensen, Morten Hertzum, Kasper Anders Søren Hornbæk, Qingxin Shi, and Pradeep Yammiyavar. 2008. Cultural Cognition in the Thinking-Aloud Method for Usability Evaluation. In ICIS 2008 Proceedings. Association for Infor- mation Systems, New York, NY, USA, 1–15. [19] CSCW. 2021. CSCW 2021. https://cscw.acm.org/2021/ [20] Dianne Cyr, Milena Head, and Hector Larios. 2010. Colour appeal in website design within and across cultures: A multi-method evaluation. International Journal of Human-Computer Studies 68, 1 (2010), 1 – 21. https://doi.org/10.1016/ j.ijhcs.2009.08.005 [21] Jasmine M. DeJesus, Maureen A. Callanan, Graciela Solis, and Susan A. Gelman. 2019. Generic language in scientifc communication. Proceedings of the National Academy of Sciences 116, 37 (2019), 18370–18377. https://doi.org/10.1073/pnas. 1817706116 [22] Nicola Dell and Neha Kumar. 2016. The Ins and Outs of HCI for Development. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 2220–2232. https://doi.org/10.1145/2858036.2858081 [23] Nicola Dell, Vidya Vaidyanathan, Indrani Medhi, Edward Cutrell, and William Thies. 2012. "Yours is Better!": Participant Response Bias in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI ’12). Association for Computing Machinery, New York, NY, USA, 1321–1330. https://doi.org/10.1145/2207676.2208589 [24] Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. 2018. Observations on Typing from 136 Million Keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3174220 [25] Tawanna R. Dillahunt and Joey Chiao-Yin Hsiao. 2020. Positive Feedback and Self-Refection: Features to Support Self-Efcacy among Underrepresented Job Seekers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376717 [26] Tawanna R. Dillahunt and Alex Lu. 2019. DreamGigs: Designing a Tool to Empower Low-Resource Job Seekers. In Proceedings of the 2019 CHI Confer- ence on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https: //doi.org/10.1145/3290605.3300808 [27] Lynn Dombrowski, Adriana Alvarado Garcia, and Jessica Despard. 2017. Low- Wage Precarious Workers’ Sociotechnical Practices Working Towards Addressing Wage Theft. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4585–4598. https://doi.org/10.1145/3025453. 3025633 [28] Vincent Dutot, Veera Bhatiasevi, and Nadim Bellallahom. 2019. Applying the technology acceptance model in a three-countries study of smartwatch adoption. The Journal of High Technology Management Research 30, 1 (2019), 1 – 14. https: //doi.org/10.1016/j.hitech.2019.02.001 [29] Kadeem Fuller, Lynette Kvasny, Eileen M. Trauth, and KD Joshi. 2015. Un- derstanding Career Choice of African American Men Majoring in Information Technology. In Proceedings of the 2015 ACM SIGMIS Conference on Comput- ers and People Research (Newport Beach, California, USA) (SIGMIS-CPR ’15). Association for Computing Machinery, New York, NY, USA, 41–48. https: //doi.org/10.1145/2751957.2751961 [30] Snehal Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cos- sette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jef Regino, et al. 2015. Daemo: A Self-Governed Crowdsourcing Marketplace. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (Daegu, Kyungpook, Republic of Korea) (UIST ’15 Ad- junct). Association for Computing Machinery, New York, NY, USA, 101–102. https://doi.org/10.1145/2815585.2815739 [31] The World Bank Group. 2020. GDP per capita, PPP (current international $). https://data.worldbank.org/indicator/NY.GDP.PCAP.PP.CD [32] The World Bank Group. 2020. GNI per capita, PPP (current international $). https://data.worldbank.org/indicator/NY.GNP.PCAP.PP.CD?name_desc= false&year_high_desc=true [33] Philip J. Guo and Katharina Reinecke. 2014. Demographic Diferences in How Students Navigate through MOOCs. In Proceedings of the First ACM Conference on Learning @ Scale Conference (Atlanta, Georgia, USA) (L@S ’14). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/ 2556325.2566247 [34] Christina N. Harrington, Katya Borgos-Rodriguez, and Anne Marie Piper. 2019. Engaging Low-Income African American Older Adults in Health Discussions through Community-Based Design Workshops. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300823 [35] India HCI. 2020. HCI Professionals Association of India. https://www.indiahci. org/ [36] Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences 33, 2-3 (June 2010), 61–83. https://doi.org/10.1017/S0140525X0999152X [37] Julia Himmelsbach, Stephanie Schwarz, Cornelia Gerdenitsch, Beatrix Wais- Zechmann, Jan Bobeth, and Manfred Tscheligi. 2019. Do We Care About Di- versity in Human Computer Interaction: A Comprehensive Content Analysis on Diversity Dimensions in Research. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). ACM, New York, NY, USA, 490:1–490:16. https://doi.org/10.1145/3290605.3300720 [38] Alex O. Holcombe. 2019. Contributorship, Not Authorship: Use CRediT to Indicate Who Did What. Publications 7, 3 (Sept. 2019), 48. https://doi.org/10.3390/ publications7030048 [39] Freedom House. 2020. Countries and Territories. https://freedomhouse.org/ countries/freedom-world/scores [40] Freedom House. 2020. Freedom in the World Research Methodol- ogy. https://freedomhouse.org/reports/freedom-world/freedom-world-research- methodology [41] Bernd Huber, Katharina Reinecke, and Krzysztof Z. Gajos. 2017. The Efect of Performance Feedback on Social Media Sharing at Volunteer-Based Online Experiment Platforms. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 1882–1886. https://doi.org/10.1145/ 3025453.3025553 https://doi.org/10.1037/0003-066X.63.7.602 https://doi.org/10.1037/0003-066X.63.7.602 https://doi.org/10.1177/1745691612459060 https://doi.org/10.1145/1518701.1518810 https://doi.org/10.1145/1518701.1518810 https://doi.org/10.1145/3313831.3376671 https://doi.org/10.1145/3313831.3376482 https://doi.org/10.1016/j.chb.2013.05.009 https://doi.org/10.1145/3338286.3340118 https://doi.org/10.1145/3290605.3300429 https://doi.org/10.1145/3290605.3300429 https://dl.acm.org/citation.cfm?id=2858036 https://dl.acm.org/citation.cfm?id=2858036 https://dl.acm.org/citation.cfm?id=3025453 https://dl.acm.org/citation.cfm?id=3025453 https://dl.acm.org/citation.cfm?id=3173574 https://dl.acm.org/citation.cfm?id=3173574 https://dl.acm.org/citation.cfm?id=3290605 https://dl.acm.org/citation.cfm?id=3290605 https://sigchi.org/conferences/conference-history/chi/ https://sigchi.org/conferences/conference-history/chi/ https://chiuxid.org/ https://cscw.acm.org/2021/ https://doi.org/10.1016/j.ijhcs.2009.08.005 https://doi.org/10.1016/j.ijhcs.2009.08.005 https://doi.org/10.1073/pnas.1817706116 https://doi.org/10.1073/pnas.1817706116 https://doi.org/10.1145/2858036.2858081 https://doi.org/10.1145/2207676.2208589 https://doi.org/10.1145/3173574.3174220 https://doi.org/10.1145/3313831.3376717 https://doi.org/10.1145/3290605.3300808 https://doi.org/10.1145/3290605.3300808 https://doi.org/10.1145/3025453.3025633 https://doi.org/10.1145/3025453.3025633 https://doi.org/10.1016/j.hitech.2019.02.001 https://doi.org/10.1016/j.hitech.2019.02.001 https://doi.org/10.1145/2751957.2751961 https://doi.org/10.1145/2751957.2751961 https://doi.org/10.1145/2815585.2815739 https://data.worldbank.org/indicator/NY.GDP.PCAP.PP.CD https://data.worldbank.org/indicator/NY.GNP.PCAP.PP.CD?name_desc=false&year_high_desc=true https://data.worldbank.org/indicator/NY.GNP.PCAP.PP.CD?name_desc=false&year_high_desc=true https://doi.org/10.1145/2556325.2566247 https://doi.org/10.1145/2556325.2566247 https://doi.org/10.1145/3290605.3300823 https://www.indiahci.org/ https://www.indiahci.org/ https://doi.org/10.1017/S0140525X0999152X https://doi.org/10.1145/3290605.3300720 https://doi.org/10.3390/publications7030048 https://doi.org/10.3390/publications7030048 https://freedomhouse.org/countries/freedom-world/scores https://freedomhouse.org/countries/freedom-world/scores https://freedomhouse.org/reports/freedom-world/freedom-world-research-methodology https://freedomhouse.org/reports/freedom-world/freedom-world-research-methodology https://doi.org/10.1145/3025453.3025553 https://doi.org/10.1145/3025453.3025553 How WEIRD is CHI? CHI ’21, May 8–13, 2021, Yokohama, Japan [42] Samuel P. Huntington. 2011. The clash of civilizations and the remaking of world order (simon & schuster hardcover ed ed.). Simon & Schuster, New York. [43] Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E. Grinter. 2010. Postcolonial Computing: A Lens on Design and Development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1311–1320. https://doi.org/10.1145/1753326.1753522 [44] Aliakbar Jafari, Fuat Fırat, Ahmet Süerdem, Søren Askegaard, and Daniele Dalli. 2012. Non-western contexts: The invisible half. Marketing Theory 12, 1 (2012), 3–12. https://doi.org/10.1177/1470593111424183 arXiv:https://doi.org/10.1177/1470593111424183 [45] Brian Keegan, Katherine Ognyanova, Brooke Foucault Welles, Christoph Riedl, Ceyhun Karbeyaz, Waleed Meleis, David Lazer, Jason Radford, and Jeferson Hoye. 2014. Conducting Massively Open Online Social Experiments with Vol- unteer Science. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 2, 1 (Oct. 2014), 19–20. https://ojs.aaai.org/index.php/HCOMP/ article/view/13208 [46] Maurice G. Kendall and Jean Dickinson Gibbons. 1990. Rank correlation methods (5th ed.). E. Arnold ; Oxford University Press, London : New York, NY. [47] Philippe Kimura-Thollander and Neha Kumar. 2019. Examining the "Global" Language of Emojis: Designing for Cultural Representation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300725 [48] Neha Kumar, Vikram Kamath Cannanure, Dilrukshi Gamage, Annu Sible Prab- hakar, Christian Sturm, Cuauhtémoc Rivera Loaiza, Dina Sabie, Md. Moinuddin Bhuiyan, and Mario A. Moreno Rocha. 2020. HCI Across Borders and Sustain- able Development Goals. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3375067 [49] Neha Kumar, Susan Dray, Andy Dearden, Nicola Dell, Melissa Densmore, Re- becca E. Grinter, Zhengjie Liu, Mario A. Moreno Rocha, Anicia Peters, Eunice Sari, William Thies, Indrani Medhi-Thies, William D. Tucker, Elba Valderrama Ba- hamondez, and Susan Wyche. 2016. Development Consortium: HCI Across Borders. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’16). ACM, New York, NY, USA, 3620–3627. https://doi.org/10.1145/2851581.2856507 [50] Neha Kumar, Susan M. Dray, Christian Sturm, Nithya Sambasivan, Laura S. Gaytán-Lugo, Leonel V. Morales Diaz, Negin Dahya, and Nova Ahmed. 2017. HCI Across Borders. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17). ACM, New York, NY, USA, 3065–3072. https://doi.org/10.1145/3027063.3108901 [51] Neha Kumar, Kurtis Heimerl, David Nemer, Naveena Karusala, Aditya Vashistha, Susan M. Dray, Christian Sturm, Laura S. Gaytán-Lugo, Anicia Peters, Nova Ahmed, Nicola Dell, and Jay Chen. 2018. HCI Across Borders: Paving New Pathways. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA ’18). ACM, New York, NY, USA, Sym03:1–Sym03:8. https://doi.org/10.1145/3170427.3170666 [52] Neha Kumar, Christian Sturm, Syed Ishtiaque Ahmed, Naveena Karusala, Marisol Wong-Villacres, Leonel Morales, Rita Orji, Michaelanne Dye, Nova Ahmed, Laura S. Gaytán-Lugo, Aditya Vashistha, David Nemer, Kurtis Heimerl, and Susan Dray. 2019. HCI Across Borders and Intersections. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). ACM, New York, NY, USA, 1––8. https://doi.org/10.1145/3290607.3299004 [53] J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174. https: //doi.org/10.2307/2529310 [54] Alex Leavitt, Brian C. Keegan, and Joshua Clark. 2016. Ping to Win? Non- Verbal Communication and Team Performance in Competitive Online Multiplayer Games. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machin- ery, New York, NY, USA, 4337–4350. https://doi.org/10.1145/2858036.2858132 [55] Philipp Lepenies. 2016. The power of a single number: a political history of GDP. Columbia University Press, New York. [56] Wilson E. Lozano-Rolon. 2020. CoRE Lab - An Efort to Engage College His- panic Students in STEM. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Computing Machinery, New York, NY, USA, 3583. https://doi.org/10.1145/3394486.3411065 [57] Andrea Mannocci, Francesco Osborne, and Enrico Motta. 2019. The evolution of IJHCS and CHI: A quantitative analysis. International Journal of Human-Computer Studies 131 (Nov. 2019), 23–40. https://doi.org/10.1016/j.ijhcs.2019.05.009 [58] Jacob M Marszalek, Carolyn Barber, Julie Kohlhart, and B Holmes Cooper. 2011. Sample size in psychological research over the past 30 years. Perceptual and Motor Skills 112, 2 (2011), 331–348. https://doi.org/10.2466/03.11.PMS.112.2.331-348 [59] Helena Mentis, Clif Lampe, Regina Bernhaupt, Anirudha Joshi, Susan Fussell, Susan Dray, Dan Olsen, Aaron Quigley, Julie R. Williamson, Eunice Sari, Loren Terveen, Allison Druin, and Philippe Palanque. 2018. SIGCHI EC Values and Strategic Initiatives. https://sigchi.org/2018/09/sigchi-ec-values-and-strategic- initiatives/ [60] United Nations. 2015. Non-member States. https://www.un.org/en/sections/ member-states/non-member-states/index.html [61] United Nations. n.d.. Member States. https://www.un.org/en/member-states/ index.html [62] Mark Nielsen, Daniel Haun, Joscha Kärtner, and Cristine H Legare. 2017. The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology 162 (2017), 31–38. [63] Ministry of Education. 2020. International Comparison of Education Statistical Indicators 2020 Edition (Chinese version). Technical Report 108. Ministry of Education Republic of China (Taiwan), Taipei, Republic of China (Taiwan). http: //stats.moe.gov.tw/fles/ebook/International_Comparison/2020/i2020.pdf [64] Nigini Oliveira, Nazareno Andrade, and Katharina Reinecke. 2016. Participation Diferences in Q&A Sites Across Countries: Opportunities for Cultural Adapta- tion. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (Gothenburg, Sweden) (NordiCHI ’16). Association for Computing Machinery, New York, NY, USA, Article 6, 10 pages. https://doi.org/10.1145/2971485.2971520 [65] Nigini Oliveira, Eunice Jun, and Katharina Reinecke. 2017. Citizen Science Opportunities in Volunteer-Based Online Experiments. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 6800–6812. https://doi.org/10.1145/3025453.3025473 [66] Daniel M. Oppenheimer, Tom Meyvis, and Nicolas Davidenko. 2009. Instructional manipulation checks: Detecting satisfcing to increase statistical power. Journal of Experimental Social Psychology 45, 4 (2009), 867 – 872. https://doi.org/10.1016/ j.jesp.2009.03.009 [67] Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. Judgment and Decision making 5, 5 (2010), 411–419. https://doi.org/10.1037/t69659-000 [68] Eyal Peer, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017. Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70 (2017), 153 – 163. https://doi.org/ 10.1016/j.jesp.2017.01.006 [69] Elissa M. Redmiles, Sean Kross, and Michelle L. Mazurek. 2017. Where is the Digital Divide? A Survey of Security, Privacy, and Socioeconomics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 931–936. https://doi.org/10.1145/3025453.3025673 [70] Katharina Reinecke and Krzysztof Z. Gajos. 2014. Quantifying Visual Preferences around the World. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 11–20. https://doi.org/10.1145/ 2556288.2557052 [71] Katharina Reinecke and Krzysztof Z. Gajos. 2015. LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15). ACM, New York, NY, USA, 1364–1378. https://doi.org/ 10.1145/2675133.2675246 [72] Katharina Reinecke, Minh Khoa Nguyen, Abraham Bernstein, Michael Näf, and Krzysztof Z. Gajos. 2013. Doodle around the World: Online Scheduling Behavior Refects Cultural Diferences in Time Perception and Group Decision-Making. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (San Antonio, Texas, USA) (CSCW ’13). Association for Computing Machinery, New York, NY, USA, 45–54. https://doi.org/10.1145/2441776.2441784 [73] Paul Resnick. 2013. Reasons for Replication in HCI Research. https://presnick. wordpress.com/2013/05/07/reasons-for-replication-in-hci-research/ [74] Steven O. Roberts, Carmelle Bareket-Shavit, Forrest A. Dollins, Peter D. Goldie, and Elizabeth Mortenson. 2020. Racial Inequality in Psychological Research: Trends of the Past and Recommendations for the Future. Perspectives on Psychological Science 15, 6 (Nov. 2020), 1295–1309. https://doi.org/10.1177/ 1745691620927709 Publisher: SAGE Publications Inc. [75] Daisuke Sakamoto. 2015. Asian researchers at the CHI conference. interactions 22, 1 (Jan. 2015), 52–55. https://doi.org/10.1145/2692310 [76] Herman Saksono, Carmen Castaneda-Sceppa, Jessica Hofman, Magy Seif El-Nasr, Vivien Morris, and Andrea G. Parker. 2019. Social Refections on Fitness Tracking Data: A Study with Families in Low-SES Neighborhoods. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300543 [77] Yukiko Sawaya, Mahmood Sharif, Nicolas Christin, Ayumu Kubota, Akihiro Nakarai, and Akira Yamada. 2017. Self-Confdence Trumps Knowledge: A Cross- Cultural Study of Security Behavior. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2202–2214. https: //doi.org/10.1145/3025453.3025926 [78] Ari Schlesinger, W. Keith Edwards, and Rebecca E. Grinter. 2017. Intersectional HCI: Engaging Identity Through Gender, Race, and Class. In Proceedings of the https://doi.org/10.1145/1753326.1753522 https://doi.org/10.1177/1470593111424183 https://arxiv.org/abs/https://doi.org/10.1177/1470593111424183 https://ojs.aaai.org/index.php/HCOMP/article/view/13208 https://ojs.aaai.org/index.php/HCOMP/article/view/13208 https://doi.org/10.1145/3290605.3300725 https://doi.org/10.1145/3334480.3375067 https://doi.org/10.1145/2851581.2856507 https://doi.org/10.1145/3027063.3108901 https://doi.org/10.1145/3170427.3170666 https://doi.org/10.1145/3290607.3299004 https://doi.org/10.2307/2529310 https://doi.org/10.2307/2529310 https://doi.org/10.1145/2858036.2858132 https://doi.org/10.1145/3394486.3411065 https://doi.org/10.1016/j.ijhcs.2019.05.009 https://doi.org/10.2466/03.11.PMS.112.2.331-348 https://sigchi.org/2018/09/sigchi-ec-values-and-strategic-initiatives/ https://sigchi.org/2018/09/sigchi-ec-values-and-strategic-initiatives/ https://www.un.org/en/sections/member-states/non-member-states/index.html https://www.un.org/en/sections/member-states/non-member-states/index.html https://www.un.org/en/member-states/index.html https://www.un.org/en/member-states/index.html http://stats.moe.gov.tw/files/ebook/International_Comparison/2020/i2020.pdf http://stats.moe.gov.tw/files/ebook/International_Comparison/2020/i2020.pdf https://doi.org/10.1145/2971485.2971520 https://doi.org/10.1145/3025453.3025473 https://doi.org/10.1016/j.jesp.2009.03.009 https://doi.org/10.1016/j.jesp.2009.03.009 https://doi.org/10.1037/t69659-000 https://doi.org/10.1016/j.jesp.2017.01.006 https://doi.org/10.1016/j.jesp.2017.01.006 https://doi.org/10.1145/3025453.3025673 https://doi.org/10.1145/2556288.2557052 https://doi.org/10.1145/2556288.2557052 https://doi.org/10.1145/2675133.2675246 https://doi.org/10.1145/2675133.2675246 https://doi.org/10.1145/2441776.2441784 https://presnick.wordpress.com/2013/05/07/reasons-for-replication-in-hci-research/ https://presnick.wordpress.com/2013/05/07/reasons-for-replication-in-hci-research/ https://doi.org/10.1177/1745691620927709 https://doi.org/10.1177/1745691620927709 https://doi.org/10.1145/2692310 https://doi.org/10.1145/3290605.3300543 https://doi.org/10.1145/3025453.3025926 https://doi.org/10.1145/3025453.3025926 CHI ’21, May 8–13, 2021, Yokohama, Japan 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 5412–5427. https://doi.org/10.1145/3025453.3025766 [79] Google Scholar. 2020. Human Computer Interaction - Google Scholar Met- rics. https://scholar.google.es/citations?view_op=top_venues&hl=en&vq=eng_ humancomputerinteraction [80] Angelika Strohmayer, Cayley MacArthur, Velvet Spors, Michael Muller, Morgan Vigil-Hayes, and Ebtisam Alabdulqader. 2019. CHInclusion: Working Toward a More Inclusive HCI Community. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems - CHI EA ’19. ACM Press, Glasgow, Scotland Uk, 1–10. https://doi.org/10.1145/3290607.3299012 [81] Christian Sturm, Alice Oh, Sebastian Linxen, Jose Abdelnour Nocera, Susan Dray, and Katharina Reinecke. 2015. How WEIRD is HCI?: Extending HCI Principles to other Countries and Cultures. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’15. ACM Press, Seoul, Republic of Korea, 2425–2428. https://doi.org/10.1145/2702613. 2702656 [82] Stanley Sue. 1999. Science, ethnicity, and bias: Where have we gone wrong? American Psychologist 54, 12 (1999), 1070–1077. https://doi.org/10.1037/0003- 066X.54.12.1070 [83] Asian CHI Symposium. 2020. Asian CHI Symposium 2020. https://asian- chi.github.io/2020/ [84] Jennyfer Lawrence Taylor, Jessica Tsimeris, XuanYing Zhu, Duncan Stevenson, and Tom Gedeon. 2015. Observations from Teaching HCI to Chinese Students in Australia. In Proceedings of the ASEAN CHI Symposium’15 (Seoul, Republic of Korea) (ASEAN CHI Symposium’15). Association for Computing Machinery, New York, NY, USA, 31–35. https://doi.org/10.1145/2776888.2780366 [85] TOCHI. 2020. TOCHI ACM Transactions on Computer-Human Interaction. http://tochi.acm.org/ [86] Kashyap Todi. n.d.. KashyapTodi. http://www.kashyaptodi.com [87] Harry C. Triandis and Richard W. Brislin. 1984. Cross-cultural psychology. American Psychologist 39, 9 (1984), 1006–1016. https://doi.org/10.1037/0003- 066X.39.9.1006 [88] UNESCO Institute for Statistics. 2012. International standard classifcation of education: ISCED 2011. UNESCO Institute for Statistics, Montreal, Que- bec. http://uis.unesco.org/sites/default/fles/documents/international-standard- classifcation-of-education-isced-2011-en.pdf OCLC: 879578739. [89] European Union. 2020. Countries | European Union. https://europa.eu/european- union/about-eu/countries_en [90] International Telecommunication Union. 2020. Key ICT indicators for developed and developing countries, the world and special regions (totals and penetration rates). https://www.itu.int/en/ITU-D/Statistics/Documents/facts/ITU_regional_ global_Key_ICT_indicator_aggregates_Nov_2020.xlsx Linxen, et al. [91] Population Division United Nations, Department of Economic and Social Afairs. 2019. World Population Prospects 2019, custom data acquired via website. https: //population.un.org/wpp/DataQuery/ [92] United Nations Development Programme. 2019. Human development report 2019: beyond income, beyond averages, beyond today: inequalities in human development in the 21st century. United Nations Development Programme, New York, NY. http://www.hdr.undp.org/sites/default/fles/hdr2019.pdf OCLC: 1142517260. [93] Rajan Vaish, Snehalkumar (Neil) S. Gaikwad, Geza Kovacs, Andreas Veit, Ranjay Krishna, Imanol Arrieta Ibarra, Camelia Simoiu, Michael Wilber, Serge Belongie, Sharad Goel, James Davis, and Michael S. Bernstein. 2017. Crowd Research: Open and Scalable University Laboratories. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 829–843. https://doi.org/10.1145/3126594.3126648 [94] Janet Vertesi, Silvia Lindtner, and Irina Shklovski. 2011. Transnational HCI: Humans, Computers, and Interactions in Transnational Contexts. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA ’11). Association for Computing Machinery, New York, NY, USA, 61–64. https://doi.org/10.1145/1979742.1979584 [95] John Vines, Paul Dunphy, and Andrew Monk. 2014. Pay or Delay: The Role of Technology When Managing a Low Income. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 501–510. https://doi.org/10.1145/2556288.2556961 [96] Max Wilson, Wendy Mackay, Ed Chi, Michael