ABSTRACT Title of Thesis: STUDENT EXPERIENCES WITH DIVERSITY AND INCLUSION IN TECHNOLOGY DESIGN COURSES Shannon Lenise Fitzgerald, Master of Science, Human-Computer Interaction, 2020 Thesis directed by: Dr. Bill Kules, HCIM Program Director and Principal Lecturer, College of Information Studies Technology design education does not yet teach students how to effectively avoid embedding their unconscious social and cultural biases into artifacts they design and build, despite widespread critical examination of the social impact of technology. Unintended consequences that exclude or discriminate against people as they use technology reflect an inability to acknowledge diversity and inclusion topics as integral to technology design. Through a national survey, this exploratory study examined the attitudes of 115 students studying Computer Science, Information Science, User- Centered-Design and related disciplines, yielding insights into their classroom experiences; receptiveness to and concerns about engaging in discussions of equity, diversity and inclusion; and interest in addressing these issues in their own designs. These findings inform a set of proposed curricular interventions that incorporate ethics, equity, and bias into technology design courses as a supplement to traditional lectures introducing basic diversity and inclusion concepts. STUDENT EXPERIENCES WITH DIVERSITY AND INCLUSION IN TECHNOLOGY DESIGN COURSES by Shannon Lenise Fitzgerald Thesis submitted to the Faculty of the Graduate School of the University of Maryland, College Park, in partial fulfillment of the requirements for the degree of Master of Science 2020 Advisory Committee: Dr. Bill Kules, Chair Dr. Ana Ndumu Dr. Christopher Antoun Dr. Jennifer Golbeck ? Copyright by Shannon Lenise Fitzgerald 2020 1. INTRODUCTION 1 1.1 Background and Motivation 2 1.2 Research Questions 3 2. RELATED WORK 4 2.1 Equity, Diversity, and Inclusion in Technology 4 2.1.1 Biased Technology 6 2.1.2 Inclusive Practices 7 2.2 Teaching About Equity, Diversity, and Bias 8 2.2.1 Teaching Inclusive and User-Centered Design 10 2.3 Demonstrating Personal Attitudes and Receptiveness 10 2.3.1 Receptiveness to Sensitive Discussions 12 2.4 Effects of Environment on Attitudes 13 2.5 Intervention Approaches 14 2.5.1 Environmental Change as an Intervention Measure 14 2.5.2 Attitudinal Change as an Intervention Measure 15 2.5.3 Interventions Focused on Equity, Diversity, and Inclusion 16 2.6 Summary 17 3. STUDY METHOD 18 3.1 Operationalizing Diversity 18 3.2 Survey Design 18 3.2.1 Target Population 19 3.2.2 Collecting Demographics 19 3.2.3 Questions 20 3.3 Cognitive Walkthroughs 21 3.3.1 Findings and Refinement 23 3.4 Pilot Survey 23 3.4.1 Demographics 24 3.4.2 Findings 25 Drop-off Rates 25 Respondent-Reported Issues 26 Respondent Comprehension 27 3.4.3 Refinement 27 3.5 Sampling and Recruitment 29 ii 3.6 Procedures 31 3.7 Analysis Methods 32 3.7.1 Quantitative Data Analysis 33 3.7.2 Qualitative Data Analysis 34 4. FINDINGS 35 4.1 Participant Demographics 35 4.1.1 Changes to Demographics After Survey Launch 38 4.2 Personal Attitudes 39 4.3 Shaping Perspectives on Diversity-Related Issues 45 4.4 Receptiveness to Discussions on Diversity-Related Issues 48 4.4.1 Responding to Confrontation 51 4.4.2 Working in a Group 54 4.5 Experiences and Concerns Discussing Diversity-Related Issues 59 4.6 Environmental Factors 59 4.6.1 Environmental Changes in Higher Education 63 4.7 Interest in Areas of Equity, Diversity, and Inclusion in Technology in the Classroom 67 4.8 Student Experiences with Instructors 71 4.9 Automatic Associations 76 4.10 Integration of Diversity and Inclusion Topics in Technology Design Courses 78 5. DISCUSSION 79 5.1 Limitations 79 5.1.1 Survey Question Design 79 5.1.2 Non-probability Sampling 80 5.2 Influence of Personal Attitudes on Receptiveness 81 5.2.1 Variations By Race and Gender 82 5.2.2 Variations Across Majors 84 5.2.3 Summary (RQ1) 86 5.3 Contribution of Environmental Factors to Biases 87 5.3.1 Summary (RQ2) 88 5.4 Curricular Intervention Proposals 88 5.4.1 Goal-Oriented Interventions 89 Identifying and Justifying User Populations 89 Identifying Measures to Prevent Discriminatory Outcomes from Technology 90 5.4.2 Role Model Narrative Reflection 92 ii i 5.4.3 Summary (RQ3) 94 5.5 Recommendations for Instructors 95 5.6 Addressing Student Concerns Regarding Diversity-Related Discussions 97 5.7 Future Research 100 6. CONCLUDING REMARKS 103 7. APPENDICES 104 8.1 Intervention: Identifying and Justifying User Populations 104 8.2 Intervention: Identifying Measures to Prevent Discriminatory Outcomes from Technology 105 8.3 Intervention: Role Model Narrative Reflection 106 8.4 Survey 107 8.5 Recruitment Message 113 8.6 Informed Consent 114 8. REFERENCES 117 iv 1. INTRODUCTION For years, connections have been established between the biases of technologists and the work they produce (Noble, 2018; O?Neil, 2016). These biases, often introduced without knowledge or intention on the part of the creator, may appear in the form of an algorithm used for high-stakes financial decision-making that unintentionally reflects historical discrimination on the basis of gender, ethnicity, or health status (Hajian, Bonchi, & Castillo, 2016), or as an app that unintentionally miscategorizes photos of dark-skinned people as photos of animals (Hankerson et al., 2016). Unintended consequences that exclude or discriminate against people as they use technology could reflect a fundamental disconnect between technology and its social impact that occurs when diversity and inclusion are not treated as integral factors in technology design. Though the phenomenon of biased technology is widely researched, there is less data focused on preventative measures that can be integrated into technology design education. Curricular interventions that incorporate conversations about ethics, equity, and bias into technology design courses could be a way to address the root of these issues before they reach user-facing technology. This exploratory research attempts to provide data that supports the development of such interventions. 1 1.1 Background and Motivation Our approach to this topic began with a very small-scale exploratory study during which 8 students at one institution were interviewed one-on-one about their experiences in Computer Science, Information Science, and Human-Computer Interaction classrooms. The original goal was to discover if these students had learned about the effects of creator biases on technology in a way that informed their own design and development practices as they prepared to start their careers. Findings from this initial research suggested that pairing education on recognizing and confronting implicit bias with technical and design training in post-secondary courses may help students understand and care about the social impact of the technology they create (Fitzgerald & Kules, submitted). Without having an explicit connection drawn between diversity and inclusion topics and technology, students may not recognize the relationship. To further explore these initial findings, a nationwide survey was conducted of college students studying Information Science, Computer Science, Human-Computer Interaction, User Experience Design, Web Development, and related disciplines regarding their experiences with diversity and inclusion in technology design courses. The data collected from the survey is examined in this paper for its usefulness in the development of curricular interventions. 2 1.2 Research Questions This research attempts to answer the following questions: 1. To what extent do student?s personal attitudes influence their receptiveness to discussing diversity and inclusion topics in classrooms engaged in technology design projects? 2. What environmental factors contribute to students' implicit biases? 3. What changes to the classroom environment could have a persistent impact on students? ability to recognize and confront their own biases while designing technology? 3 2. RELATED WORK This research is situated among decades of research on the intersection of technology and concerns related to equity, diversity, and inclusion. With a documented relationship between technology and social well-being (Gaggioli et al., 2017; Steers, 2016), there is a greater focus on the need for inclusivity in designing technology. The literature examined below addresses the intersection of equity, diversity, bias and technology and what it means to teach about that intersection. Proposed methods for measuring and changing attitudes and implicit biases through interventions are also explored. 2.1 Equity, Diversity, and Inclusion in Technology Feenberg (1991) argues that the fact of cultural values embodied in technology design calls for broader democratic participation in technological choices by ordinary people who use technology. When technology has a negative effect on an ordinary person, it usually occurs after the technology has been released into the public domain. To address potential negative effects before they reach the public, the experiences of people in the user population need to be reconciled with the expertise of technologists in the form of collaboration in the design process (Grimes & Freeburg, 2013; Feenberg, 1991). Feng & Feenberg (2008) propose the use of a framework for design that incorporates both technical and social considerations to challenge the cultural assumptions that are ingrained in technology. Their research emphasizes that designers engaged in purely technical tasks are still subject to cultural and value-laden rules that lead to socially 4 biased technology. For example, optimizing technology for social concerns such as cost and compatibility takes for granted that this process presupposes ?facts? about the social world that are actually assumptions based on past judgements. According to Feng & Feenberg, a more equitable approach to technology design requires an examination of the historical and cultural conditions that traditionally define the design process to allow for design possibilities that serve real users rather than cater to assumptions about users. Human-centered design is one such equitable approach that uses immersive or observational participant involvement to produce solutions to problems (LUMA Institute, 2012). Jones (2016) advocates the use of a human-centered approach to bring equitable design and critical interrogation of the design process to the forefront of technology design conversations. Her research suggests that social justice aims can be integrated into design by reexamining traditional technology design through a human-centered lens. Similarly, the ?positive technology? (or ?positive computing?) area of human-computer interaction study recognizes ethical issues in the design of interactive systems and attempts to counteract harm to users that stem from a weak ethical base by integrating the well-being of users into the design of technology (Gaggioli et al., 2017). Strategies employed under a positive technology approach encourage well-being-focused components in applications where well-being is not the overall goal. Efforts such as these combined with a focus on improving diversity among technologists at every stage of the design process are commonly believed to be necessary steps toward confronting inequitable and biased technologies (Mone, 2016). 5 2.1.1 Biased Technology Kirkpatrick (2013) describes the concept of formal bias in technological systems: Formal bias occurs when a seemingly neutral system of rules is placed in a social context where it contributes to the systematic reproduction of unequal or unfair outcomes. Technology can be designed in a neutral manner with no intention of unfairness and still contribute to the reinforcement of social injustices (Kirkpatrick, 2020). Though bias is not inherent in technology, technology reflects the cultural context in which it is developed and used (Feenberg, 1991); for example, technology that serves capitalist interests is biased in favor of the bourgeoisie (Kirkpatrick, 2020). The appearance of inherent neutrality in technology that serves certain social interests over others is what leads to bias and unfair social outcomes (Kirkpatrick, 2020; 2013). In other words, when technology that is designed to be neutral with regard to one population is then used by a second population, people in the second population will experience unintended biases in favor of the first population. Algorithms have historically been presented as neutral artifacts, but modern research acknowledges that algorithms are influenced by the social, cultural, and political contexts in which they are developed (Seaver, 2019). When used as the basis for complex algorithmic systems, algorithms have the power to reinforce oppressive social relationships and enact systematic racial profiling (Noble, 2018). For example, 6 algorithms may determine higher interest rates on loans for Black and Latino populations based on data that reflects a racist history of associating Black and Latino populations with high-risk borrowing behavior (Noble, 2018; O?Neil, 2016). The consequences of seemingly neutral systems exhibiting biases against dark-skinned users can be found in the news, such as Google Photos auto-sorting pictures of a Black user and their friend into an album labeled ?Gorillas? or two co-workers making a viral video demonstrating that the sensors used in an electronic soap dispenser registered the presence of White skin but not Black skin (Hankerson et al., 2016; Mone, 2016). Less overt biases in technology, such as web interfaces that skew toward a particular audience and exclude another, can have more nuanced effects. One study found that a masculine- skewing design for an introductory Computer Science course webpage resulted in young women who saw the page having higher concerns about other students? perception of their gender in the course compared to women who saw a gender-neutral design for the same page (Metaxa-Kakavouli et al., 2018); their sense of ambient belonging and interest in studying Computer Science was also negatively affected. 2.1.2 Inclusive Practices Changing attitudes in favor of a more inclusive society have signaled shifts towards thoughtfulness, user-awareness and inclusivity in mainstream design practices (Clarkson et al., 2013). This new landscape calls for inclusive design, a strategy of design that considers accommodations for a broad spectrum of users such as both blind and sighted users (Oleson et al., 2018). In traditional design strategies, inclusive design may involve 7 sharing prototypes with underrepresented or otherwise marginalized populations before releasing to the public. The concept of ?user-sensitive inclusive design? is an extension of inclusive design that suggests designers develop genuine empathy for their users as a way to challenge themselves to design products that are not merely accessible to marginalized groups of people but also fundamentally useful (Newell et al., 2011). Products resulting from a user-sensitive inclusive design process have the potential to include marginalized groups in their core audience compared to products where modifications for accessibility and inclusivity are appended to the end of the design process. In practice, inclusive design methods can be applied to real world cases like the masculine-skewing web interface described above (Metaxa-Kakavouli et al., 2018) by factoring users? sense of belonging into the desired outcomes of the design. 2.2 Teaching About Equity, Diversity, and Bias In a survey of psychology educators, Prieto (2018) reported several barriers that instructors believe interfere with the incorporation of diversity content into non-diversity- focused courses. Commonly reported barriers included insufficient time to incorporate diversity content when it was not thought to be relevant to the course; student apprehension to learning about diversity in a non-diversity-focused course; and a lack of diversity-oriented textbooks. Findings from this research showed that instructors of color dedicated significantly more class time to diversity content than White instructors; instructors focused more on incorporating diversity content into their courses as their 8 personal acceptance of culturally diverse students increased; and concerns about teaching culturally diverse students were related to fear of unintentional faux pax in the classroom. In the realm of Computer Science (CS) education, Vakil (2018) recommends moving toward a justice-centered approach to equity in CS classrooms by examining the role of ethics in CS curriculums, the role of identity in CS learning environments, and the political motivations behind CS education reform. Diversity and inclusion narratives promote CS education as a moral imperative to expand racial and gender equity in society but can overlook the implications of pushing underrepresented minorities toward multinational corporations that are intertwined with ethical and political injustices; Vakil calls for a radical rethinking of the purposes of equity as it relates to CS education. In a study on teaching diversity in graduate-level courses, findings indicated that student experiences could be enhanced by such methods as flexible teaching styles to reach students with different worldviews; presentations of varied perspectives on diversity including the perspectives of students in the course; and facilitation of safe learning environments by instructors (Morgan Consoli & Marin, 2016). Beyond learning about diversity content, the appeal of a safe space to learn about sensitive topics can be linked to success and retention in college students based on a sense of belonging, or the feeling of mattering and being connected to others (O?Meara et al., 2017). 9 2.2.1 Teaching Inclusive and User-Centered Design In technology design courses, instructors must balance design?deciding what to make? with engineering?deciding how to make it (Oleson et al., 2018). Fostering a positive user experience is an extension of both these skill sets; without a focus on inclusive design in technology design education, software that fails measures of accessibility, usability, and functionality for diverse populations continues to be released (Oleson et al., 2018). In a study on how to teach inclusive design, Oleson (et al) found that instructors need to thoroughly understand students? existing perspectives on inclusion to effectively answer questions and facilitate discussion. A study on teaching user-centered design (UCD) suggests that to foster deeper learning the UCD principles of empathy, ideation, prototyping, testing, and refinement should be core practices to which students and instructors are held accountable rather than just keywords and assignments (Shivers- McNair et al., 2018). 2.3 Demonstrating Personal Attitudes and Receptiveness An attitude is a relatively enduring evaluation of an entity called an attitude object where an attitude object could be an issue, behavior, event, person, or social group (Hart et al., 2009; Albarracin et al., 2005). Expressions of attitudes, e.g., ?I have strong viewpoints on diversity-related issues?, are evaluations of the relationship between the self and the attitude object. Because they are evaluations, personal attitudes as well as the strength with which they are held and their likeliness to change can be measured by social 10 psychological techniques such as self-reporting (?Attitudes, Behavior, and Persuasion,? 2015). Receptiveness is a willingness to thoughtfully and fairly consider the opposing views of others with the intent of genuinely evaluating information that may contradict strongly held beliefs (Minson et al., 2019). When choices are available between information that supports pre-existing attitudes and information that challenges pre-existing attitudes, preferences in favor of personal attitudes affect receptiveness (Hart et al., 2009). With social, political, and other identity-related issues, people in a disagreement tend to dismiss the arguments of their opponents as undesirable evidence without careful thought, automatically choosing to believe that positions out of line with their core beliefs must be rooted in ignorance or malevolence (Minson et al., 2019). This is demonstrated by a form of selective exposure called confirmation bias in which personal attitudes are defended by seeking information that supports them and avoiding information that challenges them (Hart et al., 2009). For those who are primarily motivated by defending their personal attitudes, such as to lessen the perception that they are poorly informed (Steele, 1988), bias toward information that supports pre-existing attitudes is lessened when the attitudes are not strongly held; when the newly presented information is low quality; or when the person evaluating the information is more open-minded (Hart et al., 2009). Bias toward information that challenges pre-existing attitudes emerges when newly presented 11 information is related to accomplishing a goal, such as writing an essay to justify attitudes and beliefs (Hart et al., 2009). 2.3.1 Receptiveness to Sensitive Discussions The concept of ?felt understanding? is a belief that people outside one?s own group understand and accept the beliefs, experiences, and identity of the people in their group (Livingstone et al., 2020). In discussions of sensitive topics like diversity, fostering felt understanding among students and instructors can help students be more receptive to the discussion. Morgan Consoli & Marin (2016) found several factors that contribute to positive experiences that foster felt understanding for students learning about diversity in graduate classrooms including evidence of instructor comfort discussing diversity and engagement with multicultural research; intentional facilitation of trust, comfort, and a safe space for sharing or asking questions; freedom to address political correctness and boundary consciousness; and freedom to express differing opinions from instructors and peers. In practice, students and instructors can demonstrate receptiveness in a discussion by accepting exposure to opposing views, hearing them fully, and trying to understand and process them rather than shut them out (Minson et al., 2019). Those who are sharing can practice ?conversational receptiveness?, a use of language to communicate willingness to thoughtfully engage with opposing views (Yeomans et al., 2020), to encourage receptiveness to their perspective. Starting a discussion with conversational receptiveness can prevent conflict escalation as the discussion moves forward. One example of 12 conversational receptiveness is the use of elaboration questions, e.g., asking a counterpart for additional information on their perspective, when challenged with an opposing viewpoint during a discussion; asking elaboration questions can lead counterparts to view the interaction more favorably and open them up to future interactions with the person they initially disagreed with (Chen et al., 2010). 2.4 Effects of Environment on Attitudes Rudman & Ashmore (2007) attributed the underlying causes of implicit bias to such environmental factors as cultural biases, early developmental experiences, and emotional experiences. Findings from their research on the predictive capabilities of implicit attitudes and stereotypes show that emotion-based factors are linked to changes in implicit orientation in environments with new emotional stimuli; for example, they suggest that an affinity for a Black professor or a reduced perception of Black people as threatening can be used to accurately predict reductions in implicit prejudices. This is supported by Dasgupta?s (2013) findings that changes in local environment as well as emotions elicited by changes in local environments lead to changes in implicit attitudes. Rudman & Ashmore also found a link between developmental events, particularly in the home environment, and automatic associations such as people who were raised by their mothers or prefer their mother to their father automatically preferring women to men. This reinforces Dasgupta?s interpretation of implicit attitudes as ?situational adaptations? that reflect the environments and communities where individuals are immersed; they are not acquired or discarded consciously. 13 2.5 Intervention Approaches A productive intervention should provide useful information and varied potential outcomes that allow participants in the intervention to tap into internal influences and feel committed to the choices they make rather than feeling they are bending to external pressures (Argyris, 1970). Cognitive interventions have a pointed focus on changing beliefs and attitudes; retrospection can be used to examine past experiences by which the beliefs and attitudes were formed and change how an individual perceives their past experiences (Murphy, 1985). Bennett (1987) describes three types of intervention approaches: cultural, ideological and educational; legal, policy and political; and community-based economic development. The style of interventions of interest in this research are cultural, ideological and educational. 2.5.1 Environmental Change as an Intervention Measure Vuletich & Payne (2019) explored ways to meaningfully change implicit biases by looking at an existing study (Lai, et al., 2016) that compared the effectiveness of nine interventions designed to decrease racial biases. The Lai et al. study showed a change to participant bias immediately following an intervention, but the change did not persist after a few days. Vuletich & Payne?s reanalysis of the study added the context that implicit biases reflect biases present in the environment (as described in 2.5. Effects of Environment on Attitudes). Because the social environments of participants were stable, biases would naturally return to their previous state once participants returned to their environment. This demonstrates that environmental stability is a constraint to changing 14 otherwise malleable individual attitudes such that interventions that change the environment could effectively reduce biases. 2.5.2 Attitudinal Change as an Intervention Measure Forscher et al., (2019) found that implicit measures are changed the most by interventions that associate sets of concepts, invoke goals or motivations, or tax mental resources. Implicit measures are changed the least by procedures that induce threat or undesirable emotions. Acknowledging resistance, such as to an opposing viewpoint, is an effective way to help reduce it. When resistance is acknowledged its power is diffused and it has less influence; this opens up the possibility for persuasion (Knowles & Linn, 2004). Persuasive messages can be adjusted to fit the expected characteristics of the audience. For example, strong persuasive arguments can be effectively presented to audiences who are likely to process the message with thoughtfulness and care; thoughtful message processing is likely to lead to strong and long-lasting attitude change. When the audience does not care about the message then its content is less important; tangential characteristics such as aesthetics, music, or likeability of the communicator which are processed spontaneously rather than thoughtfully can be the persuasive elements in these cases (?Attitudes, Behavior, and Persuasion,? 2015). The use of narrative persuasion, thought by some to be one of the only strategies for influencing strongly held attitudes that elicit resistance (Slater & Rouner, 2002), can appeal to both thoughtful and spontaneous processing. Persuasion with the use of narratives allows resistant individuals to identity with characters in the narrative 15 (Knowles & Linn, 2004). In the realm of science communication, persuasive narrative storytelling can be used to increase comprehension, interest, and engagement when communicating science to non-experts (Dahlstrom, 2014). Non-judgemental exchange of narratives in conversation has been associated with long-lasting reductions in exclusionary attitudes and prejudices toward people outside of one?s own group (Kalla & Broockman, 2020). Those who do not care about the message may be influenced by the likeability of the character if not the content of the narrative (?Attitudes, Behavior, and Persuasion,? 2015). 2.5.3 Interventions Focused on Equity, Diversity, and Inclusion A variety of intervention examples that address equity in technology classrooms have emerged in recent years. Skirpan et al. (2018) offer classroom activities specific to incorporating ethics into Computer Science classrooms such as participatory exercises such as being asked to physically stand at a point along a spectrum to show their alignment between two extreme viewpoints on a topic; small group workshops that require deep understanding of technical and social concepts; individual assignments such as reading reflections and applications of learned topics; lecturers focused on legal, business, psychological, and historical dimensions of technology and its social impact; and a course-long group project that integrated evaluations of the social impact of the technology. Guti?rrez & Jurow (2016) promote equity-focused interventions in the form of social design experiments that attempt to connect design processes with a sense of purpose and 16 fundamental social transformation. Van Camp et al. (2019) tested an intervention that introduced female STEM role models to female STEM students and found that students who engaged in reflections on their similarity to the role models showed a greater change in their implicit stereotypes related to women in STEM compared to students who did not engage in the same reflection. These interventions have the common goal of personalizing challenging new information to increase its impact. 2.6 Summary The literature reviewed here provides a foundation that this research builds upon by demonstrating why the objectives of this study are necessary to explore. Well-established connections between technological artifacts and social biases have led to increased research into equitable and inclusive practices in technology, yet examples of biased technology are still commonly seen. An exploration of methods for measuring personal attitudes and receptiveness allowed for the development of a survey questionnaire that could effectively measure student attitudes regarding diversity and inclusion topics and their corresponding receptiveness to discussing these topics in technology design courses. A probe of equity-focused interventions that demonstrated their ability to change attitudes and biases inspired the outcomes of this research by showing that such interventions can be effective, particularly in a classroom environment. While research exists that addresses both student experiences discussing diversity and the importance of diversity-content in technology education separately, there is an under-researched area at the intersection of these two topics; this research aims to contribute to that conversation. 17 3. STUDY METHOD We gathered data from an anonymous online survey distributed by email to technology design students throughout the United States. The use of a survey rather than interviews made it possible to reach a large and diverse sample group. It also removed some of the potential barriers that may have kept participants from sharing sensitive information when speaking directly to an interviewer. The larger dataset made possible by the use of an online survey opens up a wide application of uses for the resulting data, such as offering insights that may be useful in developing interventions that cater to different student mentalities and circumstances is of particular interest in this research. 3.1 Operationalizing Diversity The methodology of this research is informed by goals of increased inclusion in technology. This involves a focus on diversity of race and ethnicity, gender identity, sexual orientation, age, socioeconomic background, physical ability, and religion. It also includes a diversity of factors that may be difficult to measure through traditional means but shape experiences and worldview, such as mental health and intelligence, rather than just differences that can be observed. 3.2 Survey Design The survey was designed based on Punch?s (2003) recommendations for small-scale, cross-sectional, self-administered surveys where individual people are the unit of analysis. Cross-sectional surveys such as this measure data from one point in time rather 18 than at multiple points over an extended period as is done in longitudinal surveys (Punch, 2003). Having individuals as the unit of analysis?in this case students?puts the focus on the ways people vary as measured by defined variables (Punch, 2003) as opposed to the way an entire class of students or entire school might vary. 3.2.1 Target Population The population of interest for this research includes current students 18 years or older studying technology design, a term which here includes Information Science, Computer Science, Human-Computer Interaction, User Experiences Design, Web Development, and related disciplines. Of this group the scope was limited to undergraduate students or non-degree-seeking students with no degree, an associate degree, or lower; graduate students were excluded from the target population. This limited scope was chosen to simplify common themes that may emerge in the data; students without degrees or pursuing their first degree may have experiences that are more similar to each other than to graduate students. 3.2.2 Collecting Demographics Demographic information was collected so that data could be analyzed across multiple axes. Race, ethnicity, gender (including a gender non-conforming option) and sexual orientation were all collected. These factors represent areas of social bias that are of interest in this research. Other than ethnicity, each demographic featured a ?custom? option where participants could write-in their own label if it was not otherwise represented. 19 Geographic region was also collected with demographic information. Regions were separated into 9 categories (e.g., New England, Mid-Atlantic, Pacific, et cetera) to preserve a level of anonymity for respondents while allowing the resulting data to be contextualized within social and cultural norms that may differ by region. 3.2.3 Questions Survey questions were designed using the BOSS criteria which advises that questions be brief, objective, simple, and specific (Iarossi, 2006). The criteria were defined as: ? Brevity: Keep questions short to decrease the potential for misreading and ask one question at a time rather than asking compound questions. ? Objectivity: Use neutral words rather than emotionally charged words and refrain from suggesting an answer in the question. ? Simplicity: Use language that is familiar to all respondents so that questions are both understood and understood in the same way. ? Specificity: Be precise so that respondents are not left with follow-up questions. The majority of closed-ended questions made use of 5-point Likert scales. A Likert scale is a rating technique used for the assessment of attitudes (Croasmun & Ostrom, 2011). The survey primarily used scales measuring agreement from ?Strongly Disagree? to ?Strongly Agree? with a midpoint of ?Undecided?, and frequency from ?Never? to ?Always? with a midpoint of ?Sometimes?. When agreement and frequency were too general to suit the question, specific multiple-choice answers were provided. 20 Each research question was analyzed to determine what concepts would be explored in the survey. These concepts were then used as starting points for gathering insights which would address each question. For example, one question asks: To what extent do student?s personal attitudes influence their receptiveness to discussing diversity and inclusion topics in classrooms engaged in technology design projects? The key concepts in this question were identified as ?personal attitudes? and ?receptiveness?. Using this method, concepts from each research question were used as categorical groupings for the survey. Once all research questions were analyzed the survey had been broken into five categories: 1. Personal Attitudes 2. Receptiveness 3. Environment 4. Technology Design Courses 5. Impact 3.3 Cognitive Walkthroughs Three (3) cognitive walkthroughs were conducted to test the survey. A cognitive walkthrough is a method for analyzing the mental processes required to complete a task (in this case to complete a survey) to identify problems that may exist in the design and suggest why they may be occuring (Lewis & Wharton, 1997). The cognitive walkthrough method was adapted from Tourangeau?s Cognitive sciences and survey methods (1984) 21 which identified the cognitive processes as (1) question comprehension, (2) information retrieval, (3) judgment and estimation, and (4) response. The participants, chosen as a convenience sample due to their accessibility to the researchers, were a black female, white female, and white male. All were graduates of technology design programs, specifically Computer Science, Human-Computer Interaction, and Product Design. During a cognitive walkthrough the participant was given a chart (see Table 3.1) outlining the four stages of cognition that would be used to analyze every question in the survey. When presented with a question, the participant was asked to determine if that question passed or failed each stage. For example, if they were able to understand the words in a question and interpret its meaning, then that question would pass the Comprehension stage. When evaluating the Response stage, participants were asked to determine if a question mapped easily to the Likert scale being used. Table 3.1 Stages of a cognitive walkthrough Cognitive Stage Respondent Task Stage 1 Comprehension Interpret the question Stage 2 Retrieval Search memory for relevant information Stage 3 Judgment Evaluate and/or estimate answer Stage 4 Response Provide information in the format requested 22 3.3.1 Findings and Refinement Some questions that used a frequency scale, e.g., ?Never? to ?Always?, failed the Judgement stage because it was difficult for participants to accurately estimate their answer in terms of frequency. In these cases, reframing the question to use an agreement scale, e.g., ?Strongly Disagree? to ?Strongly Agree?, made questions easier to answer because participants were no longer required to recollect frequency and could report their current level of agreement instead. Based on this trend, agreement scales were used by default unless there was a significant advantage to measuring frequency. 3.4 Pilot Survey A pilot run of the survey was conducted before the full survey. Pilot studies have been shown to be necessary and useful in testing the feasibility of aspects of a research project including the recruitment of subjects, research tools, and data analysis methods (Hassan et al., 2006). Pilots used in survey research are useful to find flaws in a survey so that they may be addressed before a full study as well as in finding potential improvements to survey implementation (Hassan et al., 2006). Students recruited for the pilot group were a convenience sample of current and incoming graduate students studying Human-Computer Interaction at one university. The sample differed from the intended population defined in 3.1.1 Target Population in that graduate students were sampled rather than undergraduate and non-degree-seeking students, but some of the primary goals were to test the distribution methods of the recruitment email 23 and analysis of the resulting survey data which were not affected by the degree level of the participants. The use of graduate students may have impacted the feedback pilot participants gave in response to the content of the survey, but this difference is a minor limitation considering the small scale of the pilot. As an addendum to the main survey questions, three pilot-specific follow-up questions were added to assess the survey itself: 1. Please comment on any questions in this survey that you found confusing or had difficulty answering. 2. Did you have any issues using Qualtrics (the survey distribution software) while taking this survey? 3. Do you have any other critiques or suggestions about the survey? 3.4.1 Demographics Table 3.2 Pilot survey participant demographics (N=18) Responses Gender Race Complete 11 (61%) Female 9 (50%) Black 4 (22%) Incomplete 7 (39%) Male 2 (11%) White 4 (22%) Total 18 Not given 7 (39%) Asian 3 (17%) Gender Non- 0 (0%) Not given 7 (39%) Conforming American Indian or 0 (0%) Custom 0 (0%) Alaska Native Native Hawaiian or 0 (0%) Pacific Islander Custom 0 (0%) 24 The survey link was sent to approximately 40 students; of that group, 18 students began the pilot survey and 11 students completed the survey. Respondents were primarily female with 9 females and 2 males. All respondents were either Black, White, or Asian. No American Indian, Alaskan Native, Native Hawaiian, Pacific Islander, or other races participated in the pilot. 3.4.2 Findings The analysis techniques could be applied as expected with the pilot data. All questions on the survey were deemed useful in providing information that addressed the research questions, with one exception (see 3.3.3. Refinement). The order in which sections of the survey were presented and the way progress was tracked lead to some issues which are explored below. Drop-off Rates Of the 18 participants who began the pilot survey, 7 dropped out before reaching the end. An analysis of the points at which participants dropped off revealed that 4 of the 7 participants who did not complete the survey left when they reached the first page of questions immediately following the consent information. This group is likely to have begun the survey then decided upon seeing the questions that they were not actually interested or did not have time to complete it. Another 2 participants dropped off on the second page of questions after having completed the first page. This group likely lost interest or realized the time commitment necessary to get through several more pages of questions based on their time spent on the first page. 25 The 1 remaining participant who did not complete the survey dropped off on the last page of questions in the main survey. The questions they did not reach were the optional pilot- specific post-survey questions and the demographic information. Although this participant had nearly completed the survey, their progress at this point was marked as 52% on the progress bar in the survey software; progress in the software is measured by remaining pages rather than density of questions. This may have given the false impression that the remaining time commitment was the same as the time that had already been invested, which was not actually the case. A limitation of the survey structure is that respondents who drop off of the survey before completing it never reach the demographic information; as such, their answers can never be cross tabulated against their race, gender, or sexual orientation to help determine what personal attitudes and environmental factors are common among which groups. Respondent-Reported Issues At the end of the pilot survey respondents were asked to provide feedback about questions they had difficulty answering, technical difficulties they encountered using the survey software, and general critiques or suggestions about the survey. None of the participants reported technical issues while using the survey software. Issues that were reported included confusion about the wording of some questions, difficulty acknowledging one?s own bias to answer the questions honestly, and feelings of repetitiveness between sections of the survey. 26 One participant noted that the series of questions in the Impact section asks, ?I automatically associate some stereotypes with certain social groups (e.g., races, genders, or sexual orientations)?, then the following questions in that section assume an affirmative answer. For example, ?The automatic associations I have toward certain social groups aren?t harming anyone?. If the respondent indicated that they disagree, that is, that they don't automatically associate stereotypes with any social groups, then the remaining series of questions are irrelevant. Respondent Comprehension Respondents that reported difficulty answering questions seemed to have more trouble judging their responses than understanding the questions. A known complication leading into the survey was that some self-awareness is required to effectively evaluate the appropriate responses. Efforts were made to mitigate this by asking questions as simply as was reasonable to get to the root of the research questions, but some self-reflection is still required and intended as a function of the survey. 3.4.3 Refinement Based on analysis and user feedback some minor changes were made to the survey. 1. The Impact section of the survey following the question ?I automatically associate some stereotypes with certain social groups (e.g., races, genders, or sexual orientations)? was altered so that the rest of the questions in the series do not assume an affirmative answer. The ?The automatic associations I have toward 27 certain social groups aren?t harming anyone? was updated to ?Automatic associations toward certain social groups aren?t harming anyone? and so on. 2. The question ?I treat people from certain social groups (e.g., races, sexual orientations, etc.) differently than I perceive them? was removed from the survey. This question performed poorly during cognitive testing and was noted as confusing during the pilot. It was included in the survey to explicitly measure the respondent?s awareness of their own bias; however, the same measurement can be taken from a question asked more simply elsewhere, ?I automatically associate some stereotypes with certain social groups (e.g., races, genders, or sexual orientations)?. 3. In the Likert scale used for matrix questions measuring agreement, the ?Neutral? option was changed to ?Undecided?. Some changes were considered but not implemented. In response to the participant who dropped off on the last page of the main survey, removing the survey?s embedded progress bar was considered. The progress bar gave the false impression that only half of the survey had been completed when it was actually almost over because Qualtrics measures progress by remaining pages rather than density of questions. Removing the progress bar was ultimately disregarded as having no progress bar at all may make participants more likely to abandon the survey than the false impression of being earlier in the survey than they actually were. 28 Collecting contact information at the beginning of the survey instead of optionally at the end was considered as a way to re-engage participants who abandoned the survey, particularly those who left immediately following the consent page. This group may be receptive to a reminder message in the days following their abandonment, but the idea was disregarded as it compromised the anonymity of the survey. Moving the demographic information to the beginning of the survey was considered so that in the event of participants dropping out of the survey without completing it their partial answers could still be cross tabulated with demographics. The change was not implemented because placing demographic questions that may be perceived as boring at the beginning of a survey could contribute to participants dropping off before they reach more meaningful questions (Stoutenbourgh, 2008). 3.5 Sampling and Recruitment A convenience sample of college students studying technology design (see 3.1.1 Target Population) was recruited from a cross section of schools and programs (see Table 3.3) to participate in a nationwide survey. Faculty and staff at the selected schools were sent recruitment emails and asked to redistribute them to students enrolled in applicable programs and courses; students who received a recruitment email could then voluntarily take the survey. Participants who chose to give their email address were automatically entered into a raffle to receive a $50 gift card. 29 Table 3.3 School and Program Categories Type Mission Delivery Community/Junior/Two-Year Women?s Online Four-Year Religiously affiliated Massive Online Open Courses (MOOC) Liberal Arts Minority-Serving Institution (MSI, e.g., HBCU) Non-degree Program University Public Private Technical/Professional/Trade School Rather than targeting only technology design students at 4-year colleges and universities, intentional efforts were made to oversample less common cases such as Women?s colleges, religiously-affiliated schools, and programs offered 100% online despite students from those schools making up a minority of the target population. This type of non-random selection is useful when it is believed that a particular group may provide information that could not be obtained otherwise (Taherdoost, 2016). In an attempt to balance the sample with respect to the types of institutions and programs that were represented, respondent characteristics were monitored during data collection and recruitment efforts were redirected to the types of schools and programs that were under- represented in the data. For example, if there were significantly less Computer Science students responding to the survey than Information Science students, additional Computer Science students were targeted. Because convenience sampling is a non-probability approach, every person in the target population did not have an equal chance of being included in the sample (Taherdoost, 30 2016). As such, the sample is not representative of the total population. Limitations of the convenience sampling method are addressed in 5.1.2 Non-Probability Sampling. 3.6 Procedures Before the cognitive walkthroughs were conducted, the survey and all related materials including the recruitment message and consent form were reviewed and approved by the Internal Review Board (IRB) of the University of Maryland, College Park. Following the cognitive walkthroughs and pilot survey, any revised materials were re-submitted to the IRB. All amendments to the materials were approved before they were used in the next stage. The survey was designed to take approximately 15 minutes to complete and consist of 4 parts: (1) Eligibility Questionnaire, (2) Informed Consent, (3) Main Survey, and (4) Optional Contact Information. Before viewing the informed consent and accessing the main survey, participants answered questions to confirm their eligibility as members of the population defined in 3.1.1 Target Population. An eligibility questionnaire was deemed more useful than having students self-select themselves as members of the target population; the questionnaire accommodated the possibility that students may not carefully read the inclusion criteria or may be dishonest about their status as a member of the target population. If students were deemed eligible and were able to produce a valid survey 31 code from a recruitment email shared through their school, they were allowed to proceed to the consent form. The use of a survey code was helpful to lessen the chance of uninvited parties attempting to take the survey. Eligible participants were asked to provide informed consent before starting the main survey. The survey collected demographic information such as race, gender, and sexual orientation for later use in cross-tabulation analysis. Cross-tabulation methods are used to reveal the relationship between multiple variables in categorical data; since this research deals with diversity and inclusion topics, the ability to identify trends in responses based on social groups provided valuable insight. After demographics the survey collected information about participant?s personal experiences with diversity and inclusion topics and their receptiveness to learning about those topics in technology design classrooms. 3.7 Analysis Methods Due to the use of non-probability sampling methods (see 3.4 Sampling and Recruitment), this research does not attempt to make statistical inferences about the target population. The research is exploratory in nature and is not formally testing a hypothesis. Concerns related to the use of non-probability sampling are addressed in 5.1.2 Non-Probability Sampling. On the subject of informal survey data analysis, Punch?s (2003) research supports a logical, rather than entirely methodological, approach to analysis procedures in the 32 absence of statistical expertise. A thorough understanding of the research questions allows logical conclusions to be drawn about how survey data should be interpreted to answer those questions (Punch, 2003, p. 44). 3.7.1 Quantitative Data Analysis The quantitative data analysis methods used in this research are intended to identify recurring themes within the sample group. As an exploratory data analysis, focus is on the discovery, exploration, and empirical detection of phenomena in the data (Jebb et al., 2017). The findings, while not necessarily representative of the entire target population, are of interest in understanding the experiences that exist within the sample group. The data from this survey was analyzed following a logical three-step framework (Punch, 2003, p. 45): 1. Summarizing and reducing the data 2. Showing the distribution of variables across the sample 3. Analyzing relationships between the variables For closed-ended survey questions, data was reduced by removing variables for which there was no data. The use of an online survey tool eliminated the need for some steps traditionally used for summarizing survey data, specifically editing the data, data entry, and cleaning the data (Iarossi, 2006, p. 188) as the survey tool automated the ID?s and labels that would otherwise need to be applied manually. The total count of answers for each option were tallied to show the distribution of variables across the sample. 33 Relationships between variables were then examined by filtering results against other variables to compare how students? who answered one question in a particular way answered a second question. For example, to address our research question of how students? personal attitudes could influence their receptiveness to discussing diversity and inclusion topics, questions that measure students? personal attitudes are compared against questions that measure their receptiveness. 3.7.2 Qualitative Data Analysis A thematic analysis approach was used to analyze open-ended survey questions. Thematic codes were developed inductively in a data-driven fashion by interpreting the raw data from open-ended survey questions and giving each idea a code, or label, to describe its content; this categorization of answers allowed patterns to be identified and has a higher likelihood of obtaining validity against criteria compared to applying a pre- existing code because different people working with the same raw data are likely to encode the data in a similar way (Boyatzis, 1998). Both prominent and intricate themes can be identified with data-driven coding which can put emphasis on the perspectives of previously silenced voices (Boyatzis, 1998). This was a meaningful choice for this research as recognizing marginalized perspectives is of particular interest. 34 4. FINDINGS The findings below are presented based on the key variables identified in the research questions, including personal attitudes, receptiveness, technology design courses, environmental factors, and environmental changes. These variables are cross tabulated with intersectional demographic information such as race and gender to find trends among subgroups that differ from the overall trends. 4.1 Participant Demographics Table 4.1 Survey participant demographics (N=115) Responses Gender Race Major Completed 115 Female 60 (52%) Asian 55 (47%) Information 45 (39%) Ineligible 37 Male 50 (44%) White 46 (40%) Science Abandoned 12 Gender Non- 4 (3%) Black 9 (8%) Computer 32 (28%) Conforming Science Mestizo 3 (3%) Not given 1 (1%) User-Centered 26 (23%) MENA (Middle 2 (2%) Design and Custom 0 (0%) Eastern/North related African) disciplines American Indian or 1 (1%) Other 12 (10%) Alaska Native technology design Not given 1 (1%) Native Hawaiian or 0 (0%) Pacific Islander Custom 0 (0%) Participation Numbers Out of 164 participants who began the survey, 115 reached a completed submission. From the group that did not complete the survey, 37 people were found to be ineligible by the pre-questionnaire that determined whether they were part of the population defined 35 in 3.1.1 Target Population; the most common reason for ineligibility was holding a bachelor?s degree. 12 people abandoned the survey midway and did not return. The demographics discussed below are based on the 115 completed submissions. Gender 52% of participants were female; 44% were male; and 4% were gender non-conforming. Race and Ethnicity Sixty percent (60%) of participants were people of color (POC); the races represented in this group include 47% of total participants who were Asian; 8% who were Black; 3% who were Mestizo (mixed European/Indigenous heritage with Hispanic or Latino ethnicity); 2% who were Middle Eastern/North African (MENA); and 1% who were American Indian or Alaska Native. Forty percent (40%) of total participants were White. Four percent (4%) of participants were of Hispanic, Latino, or Spanish origin. Age Seventy-three percent (73%) of participants were between the ages of 18?21; 23% were between the ages of 22?29; and 4% were 30 or older. Sexual Orientation Seventy percent (70%) of participants were heterosexual. Nineteen percent (19%) were categorized by the researchers as queer; this group is comprised of 10% of total respondents who identified as pansexual or multisexual, 5% who identified as 36 homosexual, and 4% who identified as asexual. Ten percent (10%) of participants chose the ?Prefer not to say? option for their sexual orientation. Geographic Region Forty-one percent (41%) of participants were from the South Atlantic region of the United States; 21% were from the Pacific region; 15% were from the Mid-Atlantic region; 9% were from the West North Central region; 5% were from the Mountain region; 4% were from the East North Central region; 3% were from the New England region, 2% were from the West South Central region. No participants were from the East South Central region. Education Forty percent (40%) of participants were Information Science majors; 29% were Computer Science majors; 23% had a User-Centered Design related major such as User Experience (UX/UI) or Human-Computer Interaction; 4% were Software Engineering majors; 3% were Web Development majors; and 3% had another unspecified Technology Design major. Degrees For 79% of participants, a High school degree or equivalent was their highest level of education completed; the remaining 24% held an associate degree. Ninety-eight percent (98%) of participants were currently pursuing a bachelor?s degree. 37 Schools and Programs Though a variety of school and program types were targeted including 4-year schools; 2- year junior colleges, community colleges, and technical schools; women?s colleges; minority-serving institutions; religiously affiliated institutions; non-degree programs; online programs; and Massively-Open Online Courses (MOOCs), >99% of submitted responses came from 4-year universities, the vast majority of which had none of the other targeted attributes. 87% of the represented universities were public and 9% were private. 7% of responses came from formally designated minority-serving institutions. <1% of responses came from non-degree and online programs. 4.1.1 Changes to Demographics After Survey Launch When the survey launched there were six options listed for race in the Demographics section based on the racial categories provided by the United States Census Bureau as of the 2020 Census (About Race, n.d.): (1) American Indian or Alaska Native, (2) Asian, (3) Black, (4) Native Hawaiian or Pacific Islander, (5) White, and (6) Custom, which included the option to write-in a race. While the survey was underway, 2 participants used the Custom race option when they felt that their race was not represented in the provided list. One participant wrote in ?Middle Eastern/North African?. Another used the Custom race option as a place to explain that they identified as Hispanic and were unsure how to classify their race from the available options; this participant also asked if Hispanic participants were expected to choose White as their race. 38 In response to these write-ins, two additional race options were immediately added for selection by future participants: (1) MENA (Middle Eastern/North African), and (2) Mestizo (mixed European/Indigenous heritage with Hispanic or Latino ethnicity). Both MENA and a combined race and Hispanic origin option have been considered for use in the U.S. Census as they offer a level of detail about racial and cultural experience that is not otherwise captured, such as when Arabs are instructed to choose White as their race in the U.S. Census (Research to Improve Data on Race and Ethnicity, n.d.; Race Reporting among Hispanics, n.d.). Having an inclusive list of racial identities is in line with the focus of diversity and inclusion in this research. At the time these additional options were added, 60?80 survey responses had already been collected. The surveys that indicated MENA or Hispanic as a custom race were edited to select one of the newly added race options, thus they were no longer filterable as surveys where participants had chosen a custom race. 4.2 Personal Attitudes The survey attempted to establish a foundation of students? personal attitudes to address the question of how student?s personal attitudes influence their receptiveness to discussing diversity and inclusion topics in technology design courses. Out of all respondents, 73% indicated that they have strong viewpoints on diversity-related issues such as gender or racial bias and consider themselves well-informed on diversity-related issues. Sixty-seven percent (67%) of total respondents indicated that they are personally 39 impacted by diversity-related issues with 29% reporting strong agreement; however strong agreement increased to 39% among POC respondents compared to 18% among White respondents and 47% among female and gender non-conforming respondents compared to 10% among male respondents. Fig. 1. Measures of personal attitudes on diversity-related issues 40 Fig. 2. Race and gender comparison of students who are personally impact by diversity-related issues Eighty percent (80%) of all respondents seek out information on diversity such as articles, workshops, or conferences to educate themselves at least sometimes. Among POC respondents, 87% seek out information on diversity at least sometimes compared to 73% of White respondents. Among female and gender non-conforming respondents, 94% seek out information on diversity at least sometimes compared to 66% of male respondents. Eighty-six percent (86%) of all respondents discover new information about diversity-related issues in off-campus settings at least sometimes; compared by gender, 22% of males reported rarely discovering new information about diversity-related issues in off-campus settings compared to 6% of female and gender non-conforming respondents. Seventy-two percent (72%) of all respondents discover new information 41 about diversity-related issues in academic settings at least sometimes. When examined across majors, 42% of Computer Science majors reported that they rarely discover new information about diversity-related issues in academic settings compared to 14% of Information Science majors and 19% of User-Centered Design related majors. Fig. 3. Measures of personal attitude related to discovering information about diversity-related issues 42 Fig. 4. Students who seek out information on diversity to educate themselves compared by race and gender Fig. 5. Students who discover new information about diversity-related issues in off-campus settings compared by gender 43 Fig. 6. Students who discover new information about diversity-related issues in academic settings compared by major When asked if they had positive or negative experiences discussing diversity-related issues with a group, 43% of all participants answered that they had mostly positive experiences, while 42% answered that they had a similar amount of positive and negative experiences; 7% had mostly negative experiences and 8% were unsure or had not discussed diversity-related issues in a group. This distribution was consistent among students who were personally impacted by diversity-related issues. 44 Fig. 7. Experiences of students discussing diversity-related issues with a group 4.3 Shaping Perspectives on Diversity-Related Issues The survey addressed students? attitudes in regard to discovering new diversity-related content to establish their prior exposure and potential interest in this type of content in an academic setting. Out of all respondents, 74% reported that their perspectives on diversity-related issues are shaped by facts and statistics; this percentage increased to 82% among male respondents and decreased to 70% for female and gender non- conforming respondents, however 20?22% of both groups indicated strong agreement. Seventy-two (72%) of all respondents reported that their perspectives on diversity-related issues are shaped by experiences and emotions with 26% of all respondents indicating strong agreement; strong agreement increased to 42% among female and gender non- conforming respondents and decreased to 8% among males. The subgroup with the 45 highest percentage of respondents who disagreed that their perspectives on diversity- related issues are shaped by experiences and emotions was white males at 30% compared to women of color and gender non-conforming people of color at 7%. Eighty-two percent (82%) of all respondents reported that their perspectives on diversity- related issues can change if they discover new information that counters what they currently believe. There was no significant variation to this reporting among subgroups. Fig. 8. Measures of how diversity-related perspectives are shaped 46 Fig. 9. Students whose perspectives on diversity-related issues are shaped by facts and statistics compared by gender Fig. 10. Students whose perspectives on diversity-related issues are shaped by experiences and emotions compared by gender 47 Fig. 11. Students whose perspectives on diversity-related issues are shaped by experiences and emotions compared by a subset of racial and gender intersections 4.4 Receptiveness to Discussions on Diversity-Related Issues Students were asked to reflect on experiences they had discussing diversity-related content in a group with the goal of measuring the ways they demonstrated their receptiveness. Of all respondents, 66% were open to sharing their experiences regarding diversity-related issues in a group of classmates with 16% overall indicating strong agreement; strong agreement increased to 22% among female and gender non- conforming respondents compared to 8% among males. 48 Seventy-nine percent (79%) of all respondents were interested in hearing classmates share their experiences regarding diversity-related issues with 43% indicating a strong interest; strong interest increased to 56% among female and gender non-conforming respondents and decreased to 24% among males. Seventy-five percent (75%) of all respondents were more comfortable discussing diversity-related issues with friends than with classmates they do not know with 47% indicating strong interest; strong interest increased to 56% among female and gender non-conforming respondents and decreased to 36% among males. Of the respondents who were personally impacted by diversity- related issues, 66% were also open to sharing their experiences with classmates. Fig. 12. Measures of receptiveness to discussions of diversity-related issues 49 Fig. 13. Student openness to sharing their experiences regarding diversity-related issues in a group of classmates compared by gender Fig. 14. Student interest in hearing their classmates share their experiences regarding diversity-related issues compared by gender 50 Fig. 15. Student who are more comfortable discussing diversity-related issues with friends than with classmates they do not know compared by gender 4.4.1 Responding to Confrontation Questions about student experiences dealing with opposing viewpoints were included as a measure of receptiveness to difficult or sensitive discussions. When confronted directly with opposing viewpoints on diversity-related issues, such as in a personal conversation, 58% of respondents were more likely to engage (i.e., debate, argue, listen) than disengage (i.e., refrain, ignore); this increased to 64% among White respondents and decreased to 54% among POC respondents. While 16% of all respondents indicated strong agreement that they were more likely to engage with direct confrontation by opposing viewpoints 51 than disengage, this increased to 22% among female and gender non-conforming respondents and decreased to 8% among males. Twenty-three percent (23%) of all respondents were just as likely to engage or disengage with direct confrontation by opposing viewpoints or were unsure which response was more likely; and 19% were more likely to disengage. When confronted indirectly with opposing viewpoints on diversity-related issues, such as in the media or current events, 41% of all respondents were more likely to engage (i.e., listen, read, watch) than disengage (i.e., refrain, ignore); 30% were likely to respond either way or were unsure which response was more likely; and 30% were more likely to disengage. For White respondents, the likelihood of engaging with indirect confrontations by opposing viewpoints dropped to 29% compared to 46% of POC. 52 Fig. 16. Measures of receptiveness when responding to confrontation by opposing viewpoints on diversity-related issues Fig. 17. Student responses to direct confrontation by opposing viewpoints on diversity-related issues compared by race and gender 53 Fig. 18. Student responses to indirect confrontation by opposing viewpoints on diversity-related issues compared by race 4.4.2 Working in a Group One of the potential challenges of integrating diversity-content into a classroom setting could be an unwillingness among students to work with their peers when diversity-related issues are being discussed. Sixty-two percent (62%) of all respondents reported that demographic factors such as peers? race and gender could lessen the amount they contribute toward diversity-related discussions when working in a group at least sometimes with the remaining 38% reporting that peers? demographic factors rarely or never lessen their contributions toward diversity-related discussions. Fifty-six percent 54 (56%) of males reported that they were rarely or never influenced to contribute less toward diversity-related discussions by peers? demographic factors compared to 28% of female and gender non-conforming respondents; this margin widened even further between racial and gender intersections with 66% of White males reporting that they were rarely or never influenced to contribute less toward diversity-related discussions by peers? demographic factors compared to 25% of women of color and gender non- conforming people of color. Among different majors, 88% of respondents studying User-Centered Design or related fields reported that demographic factors about peers? could lessen the amount they contribute toward diversity-related discussions at least sometimes compared to 69% of respondents studying Information Science and 47% of respondents studying Computer Science; conversely, 39% of Computer Science majors indicated that that peers? demographic factors never lessen their contributions toward diversity-related discussions compared to 4% of User-Centered Design related majors. Forty-eight percent (48%) of all respondents reported that they worry about feeling discriminated against during discussions of diversity-related issues with a group at least sometimes; this increased to 59% among POC respondents compared to 35% of White respondents; it increased further to 71% of respondents who were women of color or gender non-conforming people of color compared to 38% of White female respondents and 31% of White male respondents. Conversely, 42% of White participants reported that 55 they never worry about feeling discriminated against during discussions of diversity- related issues with a group compared to 14% of people of color. Fig. 19. Measures of receptiveness to discussing diversity-related issues in a group 56 Fig. 20. Frequency with which demographic factors such as peers? race and gender could lessen the amount students contribute toward diversity-related discussions when working in a group compared by gender and a subset of racial and gender intersections Fig. 21. Frequency with which demographic factors such as peers? race and gender could lessen the amount students contribute toward diversity-related discussions when working in a group compared by major 57 Fig. 22. Frequency with which students worry about feeling discriminated against during discussions of diversity-related issues when working in a group compared by race and a subset of racial and gender intersections Of the respondents who have strong viewpoints on diversity-related issues, 54% worry about feeling discriminated against during discussions of diversity-related issues with a group. Of the respondents who are personally impacted by diversity-related issues, 58% worry about feeling discriminated against during discussions of diversity-related issues with a group. Of those who indicated strong agreement that diversity-related issues personally impact them, 44% often or always worry about feeling discriminated against during discussions of diversity-related issues with a group compared to 13% of those who indicated moderate agreement. 58 4.5 Experiences and Concerns Discussing Diversity-Related Issues To capture the nuances of discussing diversity-related issues for each individual, students were asked to share any personal experiences or concerns pertaining to discussing diversity-related issues in a group academic setting that may not have been reflected in other questions. This open-ended question was analyzed by assigning descriptive categories to each response that were tallied to provide quantitative data (see 3.6.2 Qualitative Data Analysis). Out of 38 responses, 11% of comments mentioned positive experiences and 45% of comments mentioned concerns or negative experiences; 50% of comments mentioned neutral information, such as examples of courses in which diversity had been discussed. Out of 25 female and gender non-conforming respondents, 56% expressed concerns related to gender such as feelings of not being taken seriously by male classmates because of their gender. Out of 21 POC respondents, 48% expressed concerns related to race such as being more comfortable discussing race related issues with other people of their own race. Though only 4 Black female participants answered this question, they all had negative experiences related to the way they were perceived as Black females in STEM, particularly by white male classmates. Similarly, only 2 gender non-conforming participants responded to this question, but both had negative experiences related to the way their gender identity was perceived by classmates. 4.6 Environmental Factors Before pursuing higher education, students? personal attitudes and biases were shaped by environmental factors such as their communities, homes, and childhood experiences. 59 Students were asked to report on their environments as a basis from which to approach potential environmental changes. Forty-seven percent (47%) of respondents reported growing up in communities where prejudices, such as different treatment based on race, class, or gender, were common; there was a bimodal distribution among respondents such that 33% indicated moderate agreement and 31% indicated moderate disagreement. There was not much variance in this reporting among races, however 59% of female and gender non-conforming respondents reported growing up in a community where prejudices were common compared to 28% of males. Twenty-nine percent (29%) of all respondents grew up in households where ignorance or prejudices, such as preferences for a certain race or gender, were common. Forty-nine percent (49%) had friends of mostly the same gender or background as them since they were teenagers; responses to this question were also bimodally distributed with 35% expressing moderate agreement and 28% expressing moderate disagreement. Of those who grew up in communities where prejudice was common, 71% were open to sharing their experiences regarding diversity-related issues in a group of classmates compared to 58% who grew up in communities where prejudice was not common. For 54% of respondents, associations they have with their childhood caregivers, such as parents, foster parents, and nannies, influence the way they currently perceive people. For 58% of respondents, representation of characters who were different from them in the media they consumed while growing up, such as shows, books, or music, influenced their 60 perception of different social groups; this increased to 70% among female and gender non-conforming respondents compared to 42% among males, and 64% among POC respondents compared to 49% among White respondents. Seventeen percent (17%) of POC respondents indicated strong agreement that representation in media influenced their perception of different social groups while growing up compared to 4% of White respondents. Thirty-one percent (31%) of all respondents agreed that an experience they have with one individual could positively or negatively influence their perception of other people of the same gender or background and 53% disagreed. Fig. 23. Environmental factors related to prejudice and friendship in childhood 61 Fig. 24. Environmental factors related to representation and automatic associations Fig. 25. Students who grew up in communities where prejudices, such as different treatment based on race, class, or gender, were common compared by gender 62 Fig. 26. Students for whom representation of characters who were different from them in the media they consumed while growing up, such as shows, books, or music, influenced their perception of different social groups compared by race and gender 4.6.1 Environmental Changes in Higher Education Students were asked to share details of their environments once they began pursuing higher education as a way to determine if their environments had remained stable or changed. Forty-eight percent (48%) of respondents sought out friends and/or faculty of the same gender or background as them when they began pursuing higher education; this increased to 55% among POC and decreased to 36% among White respondents. Sixteen percent (16%) of female and gender non-conforming respondents indicated strong agreement about seeking out friends and/or faculty of the same gender or background as them compared to 2% of males. 63 Thirty-three percent (33%) of respondents experienced or witnessed as much or more prejudice when they began pursuing higher education than before; 43% were unsure of how much prejudice they experienced or witnessed; and 25% experienced or witnessed fewer prejudices. Seventy-seven percent (77%) of respondents developed a better understanding of their own prejudices when they began pursuing higher education; this increased to 86% among females and gender non-conforming respondents and decreased to 64% among males. Of all respondents who gained a better understanding of their prejudices, 89% were interested in hearing classmates share their experiences regarding diversity-related issues. Fig. 27. Measures of environmental changes when students began pursuing higher education 64 Fig. 28. Students who sought out friends and/or faculty of the same gender or background as them when they began pursuing higher education compared by race and gender Fig. 29. Students who had a better understanding of their own prejudices when they began pursuing higher education compared by gender 65 Fifty-eight percent (58%) of respondents? environments became more diverse when they started pursuing higher education while 49% of respondents? circle of friends became more diverse. Twenty-five percent (25%) of respondents? environments stayed the same; 10% of respondents? environments became less diverse; and 8% of respondents? environments swapped from predominantly one demographic to another. Thirty-four percent (34%) of respondents? circle of friends stayed the same; 11% of respondents? circle of friends became less diverse; and 3% of respondents? circle of friends swapped from predominantly one demographic to another. Fig. 30. Diversity of students? environments when they began pursuing higher education 66 Fig. 31. Diversity of students? circle of friends when they began pursuing higher education 4.7 Interest in Areas of Equity, Diversity, and Inclusion in Technology in the Classroom As a measure to gauge general interest, students were asked which areas of equity, diversity, and inclusion they would personally be interested in learning about in a technology design course. Seventy-four percent (74%) of respondents were interested in learning about bias in machine learning and AI, such as chatbots mimicking users? racist or sexist speech patterns; 70% were interested in learning about ethical concerns of using AI for high-stakes decision-making, such as racial profiling; 63% were interested in learning about explicit and implicit bias, such as biases held by technologists; 61% were 67 interested in learning about racial discrimination in technology, such as the inaccuracy of facial recognition technology among some racial groups; 58% were interested in learning about gender discrimination in technology, such as targeted ads based on gender stereotypes and inclusive design practices for technologists, such as accessibility and usability; 57% were interested in learning about the effects of biased technology on high- stakes decision-making, such as discriminatory lending in loan and insurance systems; 52% were interested in learning about ethical concerns of releasing biased technology to the public, such as the effects of medical systems not listing symptoms common in women; and 50% were interested in learning about recruitment and hiring of diverse technologists, such as workforce representation and inclusion. Fifty-three percent (53%) of respondents reported having encountered one or more biases such the ones listed above in technology they had used. 68 Fig. 32. Students? interest in learning about areas of equity, diversity, and inclusion in a technology design course When variations in areas of interest were examined among different majors, between 72? 77% of Information Science majors were interested in learning about biases held by technologists and gender and racial discrimination in technology compared to 37?44% of 69 Computer Science majors who were interested in these less technically-focused areas. 81% of Computer Science majors were interested in the algorithm-focused topic of bias in machine learning and AI. Eighty-one percent (81%) of respondents with User- Centered Design related majors were interested in inclusive design practices and ethical concerns of releasing biased technology to the public compared to 41% of Computer Science majors. Among social subgroups, 72% of female and gender non-conforming respondents were interested in learning about gender discrimination in technology compared to 48% of males. Fig. 33. Students? interest in learning about areas of equity, diversity, and inclusion in a technology design course compared by major 70 4.8 Student Experiences with Instructors Though this research is focused on student experiences, the role of instructors is a major component that may influence student receptiveness to diversity-related content in technology design courses. Students were asked to evaluate their past experiences with instructors of technology design courses as an indication of whether they had demonstrated an ability to teach diversity-related content from the perspective of their students. Fifty-four percent (54%) of all respondents reported that issues related to biased technology had been addressed by instructors in technology design courses they had taken; this increased to 85% among User-Centered Design related majors and 64% among Information Science majors; it decreased to 25% among Computer Science majors. Sixty-nine percent (69%) of all respondents had been encouraged by technology design instructors to make sure users were included by their designs; this increased to 92% among User-Centered Design related majors and 71% among Information Science majors; it decreased to 47% among Computer Science majors. 71 Fig. 34. Students? experiences with instructors regarding course content Fig. 35. Students reporting of whether issues related to biased technology had been addressed by instructors in technology design courses compared by major 72 Fig. 36. Students reporting of whether they had been encouraged by technology design instructors to make sure users were included by their designs compared by major Sixty percent (60%) of respondents reported that the majority of technology design instructors they had taken courses with had shown that they can discuss diversity-related issues competently. Thirty-three percent (33%) of respondents felt that they had common background and experiences with the majority of technology design instructors they had taken courses with; this increased to 45% among males and decreased to 25% among female and gender non-conforming respondents. Sixty-five percent (65%) of respondents felt like they would be able to bring up diversity-related concerns in a technology design course without fear of negative consequences; this increased to 74% among White males and decreased to 59% among women of color and gender non-conforming people of 73 color. Of the respondents who felt like they would be able to bring up diversity-related concerns in a technology design course without fear of negative consequences, 76% were open to sharing their experiences regarding diversity-related issues in a group of classmates. Fig. 37. Students? experiences with instructors regarding teaching style and identity 74 Fig. 38. Students who felt that they had common background and experiences with the majority of technology design instructors they had taken courses with compared by gender Fig. 39. Students who felt like they would be able to bring up diversity-related concerns in a technology design course without fear of negative consequences compared by a subset of racial and gender intersections 75 4.9 Automatic Associations Feelings regarding automatic associations, or implicit biases, were measured to gauge whether students were aware of and interested in issues such as bias and motivated to address them. Forty-five percent (45%) of respondents automatically associate some stereotypes with certain social groups, such as races, genders, or sexual orientations. Eighty percent (80%) of respondents felt that automatic associations toward certain social groups are harmful; this increased to 91% among female and gender non-conforming respondents and decreased to 66% among males. Eighty-six percent (86%) of respondents felt it is important not to act on automatic associations and 88% felt it is important to take actions to counteract automatic associations and biases; of this group, 81% also sought out information on diversity to educate themselves. When asked how much responsibility they felt technology creators bear in the social impact of their technology, 60% of respondents felt technology creators bear some responsibility; 36% of respondents felt they bear full responsibility; 3% of respondents felt they bear no responsibility; and 2% of respondents were unsure. 76 Fig. 40. Measures of students? automatic associations Fig. 41. Students feelings on the harmfulness of automatic associations compared by gender 77 Fig. 42. Students feelings on how much responsibility they felt technology creators bear in the social impact of their technology 4.10 Integration of Diversity and Inclusion Topics in Technology Design Courses When asked to share general comments regarding diversity and inclusion in technology design courses, 16 participants responded. Of this group, 4 offered potential changes that could make the integration of diversity content into tech design courses smoother, such as training for instructors; 7 expressed that the content is important and needs to be addressed; 3 expressed disinterest in the content; and 2 expressed that they were satisfied with the exposure they already had to diversity content in technology design courses. 78 5. DISCUSSION In the discussion below, parallels are drawn between findings from the survey and existing research on personal attitudes, receptiveness, environment, and intervention techniques that enact long-lasting change. Insights from the findings are used to inform curricular interventions that could be used to integrate diversity and inclusion content into technology design courses. Recommendations for managing discussions of diversity- related issues with diverse students in a tech-focused environment are also discussed. 5.1 Limitations Two main limitations may have impacted the quality and usefulness of the findings presented in this research; namely the use of a voluntary non-probability sample rather than a randomized probability sample and limitations innate in survey questionnaires. Both limitations are discussed in detail below. 5.1.1 Survey Question Design From the outset of the survey design, several barriers and potential issues suggested by Iarossi (2006) to be inherent in survey question design were considered. The most basic barriers include difficulty recalling information, embarrassment at personal questions, and unwillingness to admit ignorance leading to a false answer, such as when a respondent does not want to admit they do not understand the question. Sensitive questions can be another barrier as they may prompt non-response such that the people with the most sensitive information to report may be the least likely to report it, thus 79 biasing results. As an example of this, though all demographic questions on the survey were optional, every participant who completed the survey disclosed their race, ethnicity and gender but 10% of participants chose the ?Prefer not to say? option for sexual orientation; this may be because sexual orientation is considered more sensitive than race or gender. Respondents are also prone to ?social desirability bias,'' a tendency to distort answers in ways that make them look better, especially when describing immoral behavior; this may account for such findings as 88% of respondents reporting that it is important to take actions to counteract automatic associations and biases; the results could be skewed by the belief by respondents that this is the moral answer. 5.1.2 Non-probability Sampling Participants for this research were recruited using a purposive style of convenience sampling where targeted participants were judged by the researchers to be a representative sample based on factors consistent among members of the target population rather than being a statistically representative sample (Wolf et al., 2016). Convenience samples are non-random and do not give every person in the target population an equal chance of being included in the sample (Taherdoost, 2016). The absence of randomization means non-probability samples are not representative of the total population and thus cannot be used to make inferences about any population beyond the sample. Non-probability sampling can be unreliable when the resulting data is analyzed using inferential procedures intended for probability samples or when researchers do not report 80 their potential selection biases (Heeringa et al., 2010). The selection methods used in this research (described in 3.4 Sampling and Recruitment) are biased in favor of people who have strong viewpoints about diversity-related issues; are enrolled in a school or program that was hand-selected by the researchers; are motivated by the possibility of receiving compensation; are interested in taking part in survey research; have an instructor who was interested in sharing the research; and can maintain focus on a 15-minute survey without losing interest. Because the survey was voluntary, students who are apathetic or averse to the topic may have been less likely to respond. Central and southern United States are also underrepresented in the sample with the most common geographic regions being those closest to the researchers. Together these factors resulted in a sample that mostly reflects the interests, concerns, and needs of students who support the integration of diversity content into technology design courses. 5.2 Influence of Personal Attitudes on Receptiveness Overall, the majority of students in the sample had personal attitudes that align with the goals of the research, such as a belief that automatically associating stereotypes with certain social groups is harmful. Research discussed in 2.3 Demonstrating Personal Attitudes and Receptiveness suggests that people are more receptive to information that supports their pre-existing attitudes (Hart et al., 2009); as such it follows that students whose attitudes already lean in favor of diversity-related topics would be receptive to discussions of diversity and inclusion topics in technology design courses. The majority of students reported that their perspectives on diversity-related issues can change if they 81 discover new information that counters what they currently believe, indicating an open mindedness and willingness to learn that was consistent across subgroups. With that in mind, the discussion below focuses on variations to receptiveness that occur across intersections of social identities such as race and gender. 5.2.1 Variations by Race and Gender Among the students represented in the sample, females, gender non-conforming students, and people of color (POC) reported being personally impacted by diversity-related issues more so than males and White students. The majority of students in groups whose attitudes were shaped by their personal impact by diversity-related issues were receptive to learning about diversity in technology design courses but had reservations about the impact of the discussion on their personal comfort. For example, POC respondents were less likely to engage with direct confrontations about diversity-related issues compared to White students, perhaps because they had more concerns about how they may be perceived during such discussions. This is supported by comments left in the survey by women, gender non-conforming students, and students of color about their experiences during diversity-related discussions in the classroom. One Black female student described: When discussing search engines and algorithms and how they return biased results for Black women, it felt like my experience was easily invalidated by those around me just because ?that?s how it works?. As a Black woman it?s hard 82 enough to navigate the world and in my majority white male classes I already feel like an outlier. The discussion referenced in this quote is about how historically, entering ?black girls? into Google search would return pages of porn sites as top results, whereas entering ?white girls? into Google search would return neutral images as top results, such as stock photos of White women?s faces (Noble, 2018). This student?s concerns exemplify what the impact on an individual person can be when other students are not sensitive to who else is present in the classroom when voicing their opinions on diversity-related issues in the classroom. Similar sentiments were expressed by several women, with such phrases as ?feeling ignored? and ?not taken seriously? recurring in comments from female participants. One female student described feeling the need to speak up more in male-dominant groups to prove herself to them. For women of color who expressed their concerns, feelings of both gender and racial discrimination were identified as barriers to diversity-related discussions in technology design courses when the demographics of the course skewed towards White males. The majority of male respondents and the majority of White respondents reported that they are rarely or never influenced to contribute less to diversity-related discussions because of demographic factors such as peers? race and gender compared to females, gender non-conforming students and people of color who were much more easily influenced by these factors; this could contribute to females and people of color having feelings of gender or racial discrimination in discussions, 83 particularly if males and White students are not sensitive to the way their comments are perceived by students with different social identities than them. 5.2.2 Variations Across Majors Several perspectives and experiences differed significantly across majors with a notable difference between Computer Science and User-Centered Design related disciplines. In general, Computer Science students had less focus on diversity content in courses they had taken with almost half of all Computer Science students reporting that they had rarely encountered new information about diversity-related issues in an academic setting; this is likely because their course goals are more tightly focused on technical topics compared to Information Science and User Centered Design related disciplines where diversity content will naturally be embedded in some courses. Computer Science majors who described their experiences discussing diversity and inclusion content in their courses mentioned a hyper-focus on accessibility issues that dealt with whether or not users, such as those with impaired sight, could access information; however, very little or no time was spent discussing impacts on other groups. Though only a few respondents left comments in the survey expressing outright disinterest in the integration of diversity and inclusion content in technology design courses, most of them were Computer Science majors. One Computer Science student simply said that such an integration would be a waste of time; another said it would be detrimental to integrate diversity and inclusion content because it would take away time that could be better spent on technical topics. Research suggests that Computer Science 84 students have responded well to the integration of ethics content into Computer Science courses when it is presented in small doses, allows them to engage with real-world problems and current events, and allows them to apply what they learn to complex ethical situations (Skirpan et al., 2018); this suggests that concerns about non-technical content being a waste of time could be appeased if the content is presented in an engaging and impactful way and feels relevant to the aims of the course. Students in the sample who studied User-Centered Design and related disciplines had a strong focus on diversity-related content compared to other groups and may be more sensitive to social context during discussions of diversity-related issues in the classroom, as is suggested by the vast majority of students studying User-Centered Design or related disciplines being influenced by peers? demographic factors when contributing to diversity-related discussions; in contrast, over a third of Computer Science majors, the majority of whom were White males, indicated that peers? demographic factors never lessen their contributions toward diversity-related discussions. One female Computer Science major described her experience: As a woman of color in a STEM major, I often feel discouraged to participate in groups in which there are no other women or non-men in the group. This is due to the fact that men feel more comfortable working with each other. Most times, I feel like an outsider to them. I work in student services and often hear stories from non-men-identifying students who feel discouraged to stay in their STEM majors because they feel left out by the men in their classes. 85 This sentiment of feeling ignored by male students in group classroom discussions, discussed in 5.2.1 Variations by Race and Gender, is echoed among other female respondents, particularly female Computer Science students. One respondent, a White female, commented that she started her college career as a Computer Science major but switched to Information Science because she was brushed aside and told that she did not belong in Computer Science courses when she tried to ask for help or needed clarification on problems; when she asked her male classmates if they had experienced this kind of treatment, they did not seem to. 5.2.3 Summary (RQ1) Personal attitudes that align with an interest in or commitment to diversity-related issues seem to indicate receptiveness to discussing diversity and inclusion topics in technology design courses among students in the survey sample; however, the intersection of different social identities, such as a female student who is also a person of color or a queer student who is also a gender minority, can affect receptiveness when students feel that their social identity makes them vulnerable in diversity-related discussions, particularly when they are one of a few students with a similar identity in the discussion. This emphasizes the need for equitable solutions which accommodate the different ways the material is received by different audiences to ensure that everyone can engage comfortably with the material. 86 5.3 Contribution of Environmental Factors to Biases About half of all survey participants reported growing up in communities where prejudices were common, although it was reported much more by females and gender non-conforming respondents than males. Female and gender non-conforming respondents also reported having a better understanding of their own prejudices when they began pursuing higher education more so than males; these findings may suggest that females and gender non-conforming students are more aware of prejudices around them and thus more likely to report them, perhaps because they are more likely to experience gender bias. Less than a quarter of respondents reported growing up in households where ignorance or prejudices, such as preferences for a certain race or gender, were common; this could correspond with how few students in the sample demonstrated overt opposition to the goals of the research. An environment free from explicit bias may have contributed to the development of personal attitudes that align with diversity and inclusion efforts. For female and gender non-conforming students in particular, representation of characters who were different from them in media shaped the way they perceived different social groups, which may correspond with the greater sensitivity to peers? race, gender and other demographic factors in diversity-related discussions reported by female and gender non-conforming students in the sample. 87 5.3.1 Summary (RQ2) A lack of overtly negative environmental factors reported among the sample, such as a lack of early developmental experiences in households where prejudices were common, seemed to afford students a willingness to confront their biases, with the vast majority of respondents feeling that it is important not to act on implicit biases and just as important to take actions to counteract them. 5.4 Curricular Intervention Proposals On topics related to responding to confrontation, all students were more likely to ignore indirect messages compared to direct messages, particularly White students who may not be as strongly motivated by indirect appeals to racial diversity, for example. Some of the few students who expressed resistance to the integration of diversity and inclusion content into technology design courses were concerned about the content taking away time that could be better spent learning technical skills, with one participant who studied Software Engineering commenting that in her experience, STEM students do not care about humanities and thus discussions of social impact would be out of scope for their courses. As discussed in 5.2.2 Variations Across Majors, engaging and impactful interventions that feel relevant to the aims of the course could lessen this kind of resistance. With this in mind, the intervention proposals below attempt to keep the focus on technology. Each intervention example is intended to be adaptable across different technology design majors from Computer Science to User-Centered Design related disciplines; they can be 88 used to supplement traditional lectures which introduce basic concepts related to diversity and inclusion. Since effectively managing discussions of diversity-related issues with a group of students is a topic that warrants its own dedicated research, the proposals offered here work as assignments that can be completed by individual students and are not discussion-based; however, student concerns related to managing discussions of diversity-related issues and some findings that address these concerns are mentioned. 5.4.1 Goal-Oriented Interventions As discussed in 2.3 Demonstrating Personal Attitudes and Receptiveness, when new information that challenges one?s personal attitudes is presented, individuals are likely to be more receptive if the information is related to accomplishing a goal or if they are asked to justify their beliefs (Hart et al., 2009). The invocation of goals and tasks that tax mental resources are what change implicit measures the most (Forscher et al., 2019; Hart et al., 2009), thus, requiring students to actively reflect on social impact as a core part of their project requirements may be more effective than a lecture on social impact that is never put into practice. Some examples of goal-oriented measures applied to diversity and inclusion interventions are given below. Identifying and Justifying User Populations A potential project-based intervention measure could be the addition of new requirements to a class project that tasks students with identifying user populations that should be targeted?hypothetically?to test their finished project. They may be given formal or 89 informal examples of potential user populations, such as different age groups, races, cultures, careers, and identities, but should be encouraged to think creatively to identify specific populations that are appropriate for their project. Following the identification of relevant users, students could be asked to justify why the groups they chose are appropriate, why contrapositive groups were excluded, and how they believe their project may be received by someone in an excluded group. Ideally when using this intervention, excluding someone from a design would be a conscious and justified choice made by the student rather than an unintended oversight. As an example, a student may be designing an app that recommends makeup palettes based on a user-uploaded photo of a person?s face or upon selection of one of several provided photos of different faces. If the student were to choose young women as a test population, they would need to justify why young women are an appropriate audience for their app; why contrapositive groups, such as older women, males, or gender non- conforming people, were excluded; and how they believe a member of a contrapositive group may experience their app. Identifying Measures to Prevent Discriminatory Outcomes from Technology A potential individual assignment-based intervention measure could involve students identifying measures that could have been taken by technology creators to prevent discriminatory or otherwise negative outcomes from technology based on real-world examples. This type of assignment could be used as a follow up to the introduction of a topic related to equity, diversity, and inclusion in technology, such as a lecture, video, or 90 article on the effects of biased technology on users. After the topic is introduced, students could be given a scenario in which a user is negatively affected by biased or inequitable technology and asked to identify measures the creators of the technology could have taken to prevent the scenario. Students could then be asked to review the description of a second piece of technology, this time with no information about outcomes, and identify what potential negative social impact it could have and what measures could be taken by the creators to avoid them. Since narrative communication can allow individuals to generalize from one case to infer what circumstances would allow for other such cases to be possible (Dahlstrom, 2014), inferring from a problem in one user scenario could help students recognize problems that commonly affect users. This concept could also be gamified into a playable educational experience as described by Shilton et al. (2020) where the decisions students make are contextualized within an online roleplaying simulation. As an example, a student may be given a scenario that has been in the news, such as the 2015 viral video in which one Black and one White user show that an electronic soap dispenser registers the presence of White skin but not Black skin (Hankerson et al., 2016; Mone, 2016); they may note such preventative measures the designers could have taken to avoid this scenario as testing the sensor with various skin tones before release. The student may then be given the description of an automatic sensor-operated door that opens into the lobby of an apartment building and asked to identify the potential negative social impacts the technology could have. 91 5.4.2 Role Model Narrative Reflection As discussed in 2.4 Effects of Environment on Attitudes, implicit orientations may be changed in environments with new emotional stimuli or by emotion-based factors; for example, a student?s implicit orientation may be influenced by a positive relationship with a professor from a different social group than them (Van Camp et al., 2019; Rudman & Ashmore, 2007). This type of relationship-based attitude change could be a factor for students in technology design courses beyond just a classroom intervention; the employment of diverse faculty may be a way to inspire a positive change in student attitudes. Among the survey sample, the majority of students did not feel that they had common background or experiences with their technology design instructors; male students were more likely to identify with their professors than female and gender non-conforming students, but the majority of males still did not identify with their professors. One White female survey participant, mentioned in 5.2.2 Variations Across Majors, described switching her major from Computer Science to Information Science when she was told that she did not belong in Computer Science courses when she tried to ask for help or needed clarification on problems. Another Black female student commented that she began as an Engineering major and switched to Information Science due to a lack of support from professors who she believed were not sensitive to the needs of students in regard to issues of diversity and inclusion in the classroom. For students like this, having a supportive instructor who could act as a role model may inspire positive attitude change, such as nurturing the idea that women are welcome in technology disciplines 92 rather than reinforcing the idea that women do not belong in STEM. In general, a variety of ages, teaching styles, backgrounds, and perspectives among faculty may help more students relate to their instructors as role models. With regard to nurturing feelings of inclusion in technology design by introducing role models, the intervention tested by Van Camp et al. (2019) described in 2.5.3 Interventions Focused on Equity, Diversity, and Inclusion provides a useful model. In the intervention tested in that study, female STEM students were given examples of female STEM role models; students who engaged in reflections on their similarity to the role models showed a greater change in their implicit stereotypes related to women in STEM compared to students who did not engage in the same reflection. This aligns with research presented in 2.5.2 Attitudinal Change as an Intervention Measure which describes that narrative persuasion methods where individuals have the opportunity to relate to characters in a narrative can help unreceptive individuals identify with new viewpoints; this approach has been associated with long-lasting reductions in prejudice (Kalla & Broockman, 2020; Knowles & Linn, 2004). One way to adapt this intervention to be useful for all students in a course could be by introducing students studying technology design to narrative profiles about diverse role models, such people of different races, genders, and backgrounds, who have made contributions to relevant technology design fields. A narrative profile could describe the person?s background, the path they took to enter their field, and how they fit into the field; students could then be asked to reflect on their own similarities and differences to multiple role model profiles. As an example, a student studying programming may be 93 assigned a grouping of three profiles of real-world programmers, such as a video game developer, a web application developer, and a software engineer who come from different backgrounds or have different social identities. The student may then be asked to reflect on their similarities and differences to the role models in terms of background, experiences, and goals. This could be a productive exercise even for students who are resistant to the aims of the task since acknowledging resistance diffuses its power and thus can be an effective way to reduce it (Knowles & Linn, 2004); students may be able to use the opportunity to confront their resistance if they are asked to justify why they may or may not identify with the role models. 5.4.3 Summary (RQ3) The interventions discussed above attempt to target environmental factors that contribute to implicit biases, such as emotional experiences and cultural biases (Rudman & Ashmore, 2007). Methods that embed interventions into assignments are supported by survey comments from students who felt that tangibly applying diversity content to their work rather than passively sitting through lectures is the only way to ensure that they engage with the content. The interventions purposefully focus on individual assignments to alleviate concerns about diversity-related discussions in the classroom (see 5.5 Recommendations for Instructors and 5.6 Addressing Student Concerns Regarding Diversity-Related Discussions for more on this topic). If the interventions are effective, they could de-normalize cultural bias in classroom environments and provide positive 94 emotional experiences for students that lead to persistent attitude changes (Vuletich & Payne, 2019; Forscher et al., 2019; ?Attitudes, Behavior, and Persuasion,? 2015). 5.5 Recommendations for Instructors A recurring theme among students who expressed concerns regarding the integration of diversity and inclusion topics into technology design courses was the role of instructors in communicating the information. Some students suggested training for faculty who may otherwise be unqualified to speak on these issues. One student used their comments on the survey to pose the questions: If [professors] come in contact with [a student] who opposes their teachings, how will they solve the scenario right then and there in class? Will they confront the student? How will they continue with class afterwards? Morgan Consoli & Marin (2016) address concerns like these by presenting Essential Instructor Characteristics and Methods that were determined by students in their study to be indispensable for classrooms focused on discussions of diversity; these characteristics were described in 2.2 Teaching About Equity, Diversity, and Bias and 2.3.1 Receptiveness to Sensitive Discussions. Some of the instructor characteristics include (Morgan Consoli & Marin, 2016): ? Evidence of comfort discussing diversity and engagement with multicultural research; ? Intentional facilitation of trust, comfort, and a safe space for sharing or asking questions; 95 ? Conveying freedom for students to address political correctness and boundary consciousness; ? Conveying freedom for students to express differing opinions from instructors and peers; ? Flexible teaching styles to reach students with different worldviews; ? Presentations of varied perspectives on diversity including the perspectives of students in the course; and ? Facilitation of safe learning environments. Instructors who adopt some of these methods, either through mandated training or personal motivation, may be better equipped to lessen confrontation and increase productive dialogue among students discussing diversity-related issues. For instructors of technology design courses in particular, part of the necessary work will be helping students understand that discussions of the social impact of technology in these courses are not a deviation from the more established aims of the course, but rather important and relevant content that bears discussing. Shilton (2010) recommends promoting social values within the technology design process by moving advocacy for social values into the design process itself; she suggests integrating considerations of social values as a positive part of the process rather than a negative part by which flaws are discovered in a finished technology. If the idea of these components working together are effectively communicated, students may then be willing to invest in learning how to participate in these types of discussions more sensitively. 96 5.6 Addressing Student Concerns Regarding Diversity-Related Discussions Seventy-five percent (75%) of students who responded to the survey were more comfortable discussing diversity-related issues with their friends than with classmates they do not know with 47% indicating a strong preference for discussing diversity-related issues with their friends. These preferences were supported by comments from respondents where some further explained that they were more comfortable discussing diversity with people who had a similar social identity to them, such as the same race or gender. One White male student expressed concern that a poorly managed discussion on diversity-related issues may make things worse for those who are most impacted by them, particularly if students who are less impacted do not know how to handle the discussion. This concern was shared by a Black female student who lamented: I fear for students like me who are Black or a woman of color or gender minority. Our feelings can be easily invalidated and made a mockery of by those who do not understand our experiences. Research shows that students' attitudes are positively affected by having teachers of the same race or gender as them with such reported benefits as feeling cared for and motivated in class and having a higher quality of communication between teacher and student (Egalite & Kisida, 2018); for students like the one quoted above and other female respondents described in 5.2 Influence of Personal Attitudes on Receptiveness, a lack of racial and gender diversity among instructors and students alike can have a significant 97 impact on their experiences in the classroom since diverse learning environments can help alleviate racial tension (Karkouti, 2016). Concerns about insensitivity among students were mentioned by several respondents, with some mentioning their reservations to participate in discussions out of concern for making others uncomfortable. One such student explained: My main concern when discussing these issues is accidentally offending someone. Even though these are discussions where I should feel free to speak my mind and learn things, sometimes I feel like I'm not qualified to share my opinion on issues that don't relate to my identity. For example, I'm Asian American, so when the discussion is about Asian American diversity topics then I'm much more willing to speak out than if it were about, for example, African Americans. In those discussions, I feel like I have to think very hard before I speak so that I don't accidentally say something insensitive. I also feel like I should let people with those identities speak out about their experiences since it relates more to them; I don't want to feel like I'm taking their voice away. Reluctance to speak on issues that are out of scope of one?s own identity highlights the potential for diversity-related discussions to put pressure on students to disclose aspects of their identity as a way to qualify their contributions to the discussion. One student explained: 98 I'm a white male who is on the Autism Spectrum. I've faced a lot of discrimination over my life because of that. But I also know that there are a number of people who will look at me and say because I'm a white male I don't have anything to add when it comes to diversity. Thus, I don't get involved because I don't want to accidentally tee off an oppression olympics. Oppression Olympics, which refers to a competition among oppressed social groups to determine which intersection of identities is the most oppressed (Rogue et al., 2012), may need to be carefully managed as students should not feel that they need to prove how oppressed they are to be able to engage in the discussion. Students who have a hidden disability, for example, should not feel pressure to disclose their disability in order to speak up on behalf of their own interests. Conversely, students who are frequently the only person with their identity participating in a discussion expressed concerns about being tokenized or treated as an ambassador of their identity. One student described: When topics surrounding the LGBTQ community surface and I'm the only gay person in a group, I often feel as if my opinion or experience then becomes the de facto answer for all other LGBTQ people in the minds of my peers. Similarly, if there's a question regarding language around a topic or correctness, there's an expectation that I'm an expert and should always be able to deem something as appropriate or inappropriate. It is important, then, not to put the weight of the discussion on students who represent marginalized groups. 99 Aside from managing interactions with others, some students were unsure how to handle navigating deeply personal topics in an impersonal environment like a classroom. One student described that the stakes can feel higher in a classroom environment which makes the possibility of a confrontation scarier; another explained the difficulty they had striking a balance between emotional discourse and standoffishness in group discussions on diversity-related topics. This may be especially difficult in technology design courses where students may not expect or be prepared to have sensitive discussions. In line with the recommendations discussed above for instructors to facilitate a safe learning environment for their students, instructors could start the semester by providing students with an agreement to address boundaries for participating respectfully in diversity-related discussion including information about techniques like using conversational receptiveness to communicate willingness to thoughtfully engage with differing views and diffuse tension during a discussion (Yeomans et al., 2020). 5.7 Future Research This research primarily focuses on concerns raised by students who are open to the integration of diversity and inclusion topics in technology design courses but may be wary of the implementation methods for such an undertaking. Not much information was discovered about students who are strongly opposed or even apathetic to the integration of diversity and inclusion topics in technology design courses. For future research it may 10 0 be worthwhile to use recruitment methods that result in a more representative sample of the target population and potentially a wider variation in responses than what is presented in this research, such as gathering randomized data from a probability sample rather than a voluntary sample or incentivizing as many students as possible from one or more technology design courses to participate in the study to achieve a cross section of perspectives. These methods could yield data that suggests the need for alternative intervention proposals to those seen here, or that could be used to infer generalizations about the entire target population. Outside of the survey method, additional research could pilot test curriculum elements focused on diversity and inclusion content, such as the proposed set of interventions above, in specific technology design courses, then evaluate them both informally through instructor assessment and more formally by surveying students or evaluating specific work products. Beyond exploring just the perspectives of students, it is just as important for future research to address the interests and needs of faculty who teach technology design courses. This could involve measures of receptiveness among instructors to determine if they are willing or comfortable integrating diversity and inclusion content into their courses; evaluating barriers to integration; and identifying necessary resources and support that could aid in such an integration, such as training for faculty on how to teach about diversity content in a technologically focused environment. For the foundation of what is taught in technology design courses to meaningfully change, universities need to 10 1 make an effort to ensure that instructors see value in the messages they are asked to share and get the support they need to make them better qualified to share those messages. A more pointed focus on managing discussions on diversity-related issues in technology design courses would also be a useful area of research, as this was one of the major concerns raised by students. Though some methods for facilitating such discussions respectfully were mentioned here, it would normally be out of scope for students in a technology-focused course to spend class time learning tactics for participating in sensitive discussions. More attention needs to be given to this issue in an attempt to find reasonable solutions that make students feel that these discussions are part of the primary aims of the course worth taking time away from other activities. 10 2 6. CONCLUDING REMARKS Overall, the sample examined in this research was open to the idea of integrating diversity and inclusion topics into technology design courses and did report environmental factors in their lives before pursuing higher education that would make them averse to such an integration; still, attitudes varied along social axes in terms of how receptive students were to discussions of diversity content with their classmates. For survey respondents who were female, gender non-conforming, or people of color, concerns related to their personal comfort during diversity discussions were commonly reported, indicating that a sensitivity to social identity is necessary when dealing with these topics. Despite limitations innate in survey data collection, this research makes use of trends identified among the sample population to propose interventions that function as independent assignments and do not require class-wide discussion. This purposeful omission of group discussions is meant to account for variations in student comfort to sensitive discussions pending further research on how to manage discussions of equity, diversity, and inclusion in technology design courses that would not typically address these topics. The proposed types of interventions and subsequent examples are a starting point from which future endeavors can be developed. 10 3 7. APPENDICES 8.1 Intervention: Identifying and Justifying User Populations Curricular Intervention: Identifying and Justifying User Populations Description: Add project requirements that task students with identifying user populations that should be targeted to test their finished project. (Note: This intervention could be appended to any technology design project, e.g., designing a database, building an app, creating an interface design, etc., as a hypothetical step to get students thinking about user testing even if actual tests with users are not required.) Steps 1. Identify Users: Give students examples of potential user populations (e.g., different age groups, races, cultures, careers, and identities) but encourage them to think creatively to identify specific populations appropriate for their project 2. Justify Choices: Ask students to justify why the user populations they chose are appropriate 3. Justify Exclusions: Ask students to justify why contrapositive groups were excluded (e.g., if they chose young women, justify why older women and men were excluded) 4. User Scenarios: Ask students to imagine how they believe their project may be received by someone in an excluded group Goal: Ideally when using this intervention, excluding someone from a design would be a conscious and justified choice made by the student rather than an unintended oversight. 10 4 8.2 Intervention: Identifying Measures to Prevent Discriminatory Outcomes from Technology Curricular Intervention: Identifying Measures to Prevent Discriminatory Outcomes from Technology Description: Task students with identifying measures that could have been taken by technology creators to prevent discriminatory or other negative outcomes from technology based on real-world examples Steps 1. Introduce EDI Topic: Before the intervention, introduce a topic related to equity, diversity, and inclusion in technology (e.g., a lecture, video, or article on the effects of biased technology on users) 2. Read Scenario with Negative Outcomes: Give students a real-world scenario in which a user is negatively affected by biased or inequitable technology 3. Identify Measures to Prevent Outcomes: Ask students to identify measures the creators of the technology could have taken to prevent the negative outcomes 4. Anticipate Negative Outcomes: Give students a description of a technology with no information about outcomes, then ask them to identify potential negative outcomes and measures that could be taken by creators to avoid them Goal: Get students thinking about what steps they can take to predict and avoid unintended negative impacts on users as part of the technology design process. 10 5 8.3 Intervention: Role Model Narrative Reflection Curricular Intervention: Role Model Narrative Reflection Description: Have students reflect on their similarity to diverse technology design role models. Steps 1. Introduce Profiles: Introduce narrative profiles about technologists with different social identities who are either well-known figures (e.g., popular game designers, people working at popular tech companies, etc.) or figures from the college community; a profile may describe the person?s background, their path into their field, and how they fit into the field 2. Engage in Reflection: Ask students to do some short writing in reflection on their similarities and differences to the role models in terms of background, experiences, and goals Goal: Help students feel included in their field as well as recognize the contributions of people who are different from them in their field. 10 6 8.4 Survey Personal Attitudes Strongly Disagree Disagree Undecided Agree Strongly Agree 1 I have strong viewpoints on diversity-related issues such as gender or racial bias 2 I consider myself well-informed on diversity-related issues 3 Diversity-related issues personally impact me Never Rarely Sometimes Often Always 4 I seek out information on diversity to educate myself (e.g., articles, workshops, or conferences) I discover new information about diversity-related 5 issues in academic settings (e.g., courses, on- campus organizations, or events) I discover new information about diversity-related 6 issues in off-campus settings (e.g., social media, online news and videos, or off-campus events) When it comes to discussing diversity-related issues with a group have you had positive or negative experiences? ? Mostly positive experiences 7 ? Mostly negative experiences ? Similar amount of positive and negative experiences 10 7 ? Unsure/I haven't discussed diversity-related issues with a group Receptiveness Strongly Strongly Disagree Disagree Undecided Agree Agree 8 My perspectives on diversity-related issues are shaped by facts and statistics 9 My perspectives on diversity-related issues are shaped by experiences and emotions My perspectives about diversity-related issues can 10 change if I discover new information that counters what I currently believe 11 I?m open to sharing my experiences regarding diversity-related issues in a group of classmates 12 I?m interested in hearing my classmates share their experiences regarding diversity-related issues I?m more comfortable discussing diversity-related 13 issues with friends than with classmates I don't know When confronted directly (i.e., personal conversation) with opposing viewpoints on 14 diversity-related issues, I am more likely to engage (i.e., debate, argue, listen) than disengage (i.e., refrain, ignore) When confronted indirectly (i.e., media and current events) with opposing viewpoints on 15 diversity-related issues, I am more likely to engage (i.e., listen, read, watch) than disengage (i.e., refrain, ignore) 10 8 Never Rarely Sometimes Often Always When working in a group, demographic factors 16 such as peers? race or gender could lessen the amount I contribute toward diversity-related discussions When working in a group, I worry about feeling 17 discriminated against during discussions of diversity-related issues 18 Please describe any experiences or concerns pertaining to discussing diversity-related issues in a group academic setting. Environment Strongly Disagree Disagree Undecided Agree Strongly Agree I grew up in a community where prejudices (i.e., 19 different treatment based on race, class, gender, etc.) were common I grew up in a household where ignorance and 20 prejudices (i.e., preferences for a certain race, gender, etc.) were common Associations I have with my childhood caregivers 21 (e.g., parents, foster parents, nanny, etc.) influence the way I currently perceive people Growing up, representation of characters who were 22 different than me in the media I consumed (e.g., shows, books, or music) influenced my perception of different social groups An experience I have with one individual could 23 positively or negatively influence my perception of other people of the same gender or background When I began pursuing higher education I sought 24 out friends and/or faculty of the same gender or background as me 10 9 When I began pursuing higher education I 25 experienced or witnessed fewer prejudices than before 26 Since I began pursuing higher education I have a better understanding of my own prejudices 27 Since I was a teenager my friends have mostly been the same gender or background as me When I began pursuing higher education, my environment? ? Became more diverse 28 ? Became less diverse ? Swapped predominant demographic (e.g., went from predominantly one race to another) ? Stayed the same When I began pursuing higher education, my circle of friends? ? Became more diverse 29 ? Became less diverse ? Swapped predominant demographic (e.g., went from predominantly one race to another) ? Stayed the same Technology Design Courses The list below represents areas of equity, diversity, and inclusion in technology. Which (if any) would you personally be interested in learning about in technology design courses? 30 ? Explicit bias and implicit bias (e.g., biases held by technologists) 11 0 ? Gender discrimination in technology (e.g., targeted ads based on gender stereotypes) ? Racial discrimination in technology (e.g., the inaccuracy of facial recognition technology among some racial groups) ? Bias in machine learning and AI (e.g., chatbots mimicking users? racist or sexist speech patterns) ? Effects of biased technology on high-stakes decision-making (e.g., discriminatory lending in loan and insurance systems) ? Ethical concerns of using AI for high-stakes decision-making (e.g., racial profiling) ? Ethical concerns of releasing biased technology to the public (e.g. the effects of medical systems not listing symptoms common in women) ? Recruitment and hiring of diverse technologists (e.g., workforce representation and inclusion) ? Inclusive design practices for technologists (e.g., accessibility and usability) ? Other (Please describe) Strongly Disagree Disagree Undecided Agree Strongly Agree 31 I have encountered one or more biases such the ones listed above in technology that I?ve used Issues related to biased technology have been 32 addressed by instructors in technology design courses I?ve taken I?ve been encouraged by technology design 33 instructors to make sure users are included by my designs The majority of technology design instructors I?ve 34 taken courses with have shown that they can discuss diversity-related issues competently 11 1 I?ve felt that I had common background and 35 experiences with the majority of technology design instructors I?ve taken courses with I feel like I would be able to bring up diversity- 36 related concerns in a technology design course without fear of negative consequences Impact Strongly Disagree Disagree Undecided Agree Strongly Agree I automatically associate some stereotypes with 37 certain social groups (e.g., races, genders, or sexual orientations) 38 Automatic associations toward certain social groups aren?t harming anyone 39 It?s important not to act on automatic associations 40 It's important to take actions to counteract automatic associations and biases How much responsibility do you feel technology 41 creators bear in the social impact of their technology Feedback 42 Please share your concerns regarding the integration of diversity and inclusion topics into technology design courses. 43 Please share any additional comments regarding diversity and inclusion in technology design courses. 11 2 8.5 Recruitment Message If you?re a student studying programming, user experience design, web development, or other technology-related disciplines we invite you to participate in a brief, 15 minute survey about your experiences with diversity and inclusion in technology design courses and in your personal life. We plan to use this research to support the development of classroom materials that integrate conversations about ethics, equity, and bias into technology design courses with the goal of helping students understand and care about the social impact of the technology they create. Everyone who submits a valid survey and provides an email address will be entered into a drawing to receive one of ten $50 Amazon e-gift cards. Please use this link to take the survey: [survey url] [survey code] This research is being conducted by Shannon Fitzgerald (lenise@umd.edu) at the University of Maryland, College Park. All survey responses are confidential. Thank you! Shannon Fitzgerald & Bill Kules 11 3 8.6 Informed Consent Project Title Student Experiences with Diversity and Inclusion in Tech Design Courses Purpose of the Study We are trying to understand how to help students studying technology design and development keep their unconscious social biases out of the technology they create. The results of this research will support the development of classroom modules and interventions that integrate conversations about ethics, equity, and bias into technology design courses. Procedures In this survey you will be asked about personal and academic experiences with diversity and inclusion topics, such as how you deal with discussing diversity-related issues with a group. The survey should take about 15 minutes to complete. If you answer the majority of survey questions and provide a valid email address you will be entered into a drawing to receive one of ten $50 Amazon e-gift cards. Potential Risks and Discomforts There are no major risks associated with this study. You will be asked to take a survey where you provide demographic information and answer questions about your experiences with diversity and inclusion in your personal and academic life. You may choose to share difficult experiences you have had surrounding diversity and feel stressed. You may discontinue the survey at any time. Potential Benefits There are no direct benefits for participating in this study. You may benefit from the opportunity to reflect on diversity and inclusion practices in technology courses. The 11 4 results of the study will provide insight into the ability for education to instill a sense of social responsibility in designers and developers when creating technology. This may lead to ideas for interventions that can be used to meet this goal in classrooms. Confidentiality All survey results are confidential and any analysis done by researchers will reference a participant number. No personally identifying information is captured in the analysis of survey data but an email address must be provided in order for you to be eligible to be entered in the drawing for an Amazon e-gift card. If we write a report or article about this research project your identity will be protected to the maximum extent possible. Your information may be shared with representatives of the University of Maryland, College Park or governmental authorities if you or someone else is in danger or if we are required to do so by law. Participant Rights Your participation in this research is completely voluntary. All questions on the survey are optional. You may choose not to take part at all. If you have questions or concerns please contact the investigator Shannon Fitzgerald at lenise@umd.edu If you have questions about your rights as a research participant, please contact: University of Maryland College Park Institutional Review Board Office 1204 Marie Mount Hall College Park, Maryland, 20742 E-mail: irb@umd.edu Telephone: 301-405-0678 11 5 IRBNet Package number: 1610804-1 For more information regarding participant rights, please visit: https://research.umd.edu/irb-research-participants This research has been reviewed according to the University of Maryland, College Park IRB procedures for research involving human subjects. Please download or save a copy of this consent information for your records. Statement of Consent Selecting the ?I CONSENT? option below indicates that you are at least 18 years of age, you have read this consent form or have had it read to you, your questions have been answered to your satisfaction, and you voluntarily agree to participate in this research study. If you agree to participate, please select the "I CONSENT" option. 11 6 8. REFERENCES 1. About race. (n.d.). The United States Census Bureau. Retrieved October 20, 2020, from https://www.census.gov/topics/population/race/about.html 2. Albarracin, D., Johnson, B. T., & Zanna, M. P. (Eds.). (2005). The handbook of attitudes. Lawrence Erlbaum Associates Publishers. 3. Argyris, C. (1970). Intervention theory and method: A behavioral science view. Addison-Wesley. 4. Attitudes, Behavior, and Persuasion. (2015). In Principles of social psychology. University of Minnesota Libraries Publishing. https://doi.org/10.24926/8668.2001 5. Bennett, E. M. (Ed.). (1987). Social intervention, theory and practice. Edwin Mellen Press. 6. Boyatzis, R. (1998). Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage. 7. Chen, F. S., Minson, J. A., & Tormala, Z. L. (2010). Tell me more: The effects of expressed interest on receptiveness during dialog. Journal of Experimental Social Psychology, 46(5), 850? 853. https://doi.org/10.1016/j.jesp.2010.04.012 8. Clarkson, J., Keates, S., Coleman, R., & Lebbon, C. (2013). Inclusive design: Design for the whole population. Springer London. 9. Croasmun, J. T., & Ostrom, L. (2011). Using likert-type scales in the social sciences. Journal of Adult Education, 40(1), 19?22. 10. Dahlstrom, M. F. (2014). Using narratives and storytelling to communicate science with nonexpert audiences. Proceedings of the National Academy of Sciences, 111(Supplement_4), 13614?13620. https://doi.org/10.1073/pnas.1320645111 11. Dasgupta, N. (2013). Implicit attitudes and beliefs adapt to situations: A decade of research on the malleability of implicit prejudice, stereotypes, and the self-concept. In P. Devine & A. Plant (Eds.), Advances in Experimental Social Psychology (Vol. 47, pp. 233?279). Academic Press. https://doi.org/10.1016/B978-0-12-407236-7.00005-X 12. Egalite, A. J., & Kisida, B. (2018). The effects of teacher match on students? academic perceptions and attitudes. Educational Evaluation and Policy Analysis, 40(1), 59?81. https://doi.org/10.3102/0162373717714056 13. Feenberg, A. (1991). Critical theory of technology. Oxford University Press. 14. Feng, P., & Feenberg, A. (2008). Thinking about design: Critical theory of technology and the design process. In P. Kroes, P. E. Vermaas, A. Light, & S. A. Moore (Eds.), Philosophy and Design: From Engineering to Architecture (pp. 105?118). Springer Netherlands. https://doi.org/10.1007/978-1-4020-6591-0_8 15. Fitzgerald, S., & Kules, B. (Submitted). Understanding How Students? Implicit Biases Affect Their Technology Designs. 16. Forscher, P. S., Lai, C. K., Axt, J. R., Ebersole, C. R., Herman, M., Devine, P. G., & Nosek, B. A. (2019). A meta-analysis of procedures to change implicit measures. Journal of Personality and Social Psychology, 117(3), 522?559. https://doi.org/10.1037/pspa0000160 11 7 17. Gaggioli, A., Riva, G., Peters, D., & Calvo, R. A. (2017). Positive technology, computing, and design: Shaping a future in which technology promotes psychological well-being. In Emotions and Affect in Human Factors and Human-Computer Interaction (pp. 477?502). Elsevier. https://doi.org/10.1016/B978-0-12-801851-4.00018-5 18. Grimes, S. M., & Feenberg, A. (2013). Critical theory of technology. In S. Price, C. Jewitt, & B. Brown, The SAGE Handbook of Digital Technology Research (pp. 121?129). SAGE Publications Ltd. https://doi.org/10.4135/9781446282229.n9 19. Guti?rrez, K. D., & Jurow, A. S. (2016). Social design experiments: Toward equity by design. Journal of the Learning Sciences, 25(4), 565?598. https://doi.org/10.1080/10508406.2016.1204548 20. Hajian, S., Bonchi, F., & Castillo, C. (2016). Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). ACM, New York, NY, USA, 2125-2126 21. Hankerson, D., Marshall, A. R., Booker, J., El Mimouni, H., Walker, I., & Rode, J. A. (2016). Does technology have race? Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 473?486. https://doi.org/10.1145/2851581.2892578 22. Hart, W., Albarrac?n, D., Eagly, A. H., Brechan, I., Lindberg, M. J., & Merrill, L. (2009). Feeling validated versus being correct: A meta-analysis of selective exposure to information. Psychological Bulletin, 135(4), 555?588. https://doi.org/10.1037/a0015701 23. Hassan, Z. A., Schattner, P., & Mazza, D. (2006). Doing a pilot study: Why is it essential? Malaysian Family Physician?: The Official Journal of the Academy of Family Physicians of Malaysia, 1(2?3), 70?73. 24. Heeringa, S. G., West, B. T., & Berglund, P. A. (2010). Applied survey data analysis (1st ed.). Chapman and Hall/CRC. https://doi.org/10.1201/9781420080674 25. Iarossi, G. (2006). The power of survey design: A user's guide for managing surveys, interpreting results, and influencing respondents. Washington, DC: World Bank. ? World Bank. https://openknowledge.worldbank.org/handle/10986/6975 License: CC BY 3.0 IGO. 26. Jebb, A. T., Parrigon, S., & Woo, S. E. (2017). Exploratory data analysis as a foundation of inductive research. Human Resource Management Review, 27(2), 265?276. https://doi.org/10.1016/j.hrmr.2016.08.003 27. Jones, N. N. (2016). Narrative inquiry in human-centered design: Examining silence and voice to promote social justice in design scenarios. Journal of Technical Writing and Communication. https://doi.org/10.1177/0047281616653489 28. Kalla, J. L., & Broockman, D. E. (2020). Reducing exclusionary attitudes through interpersonal conversation: Evidence from three field experiments. American Political Science Review, 114(2), 410?425. https://doi.org/10.1017/S0003055419000923 29. Karkouti, I. M. (2016). Black students? educational experiences in predominantly white universities: A review of the related literature. College Student Journal, 50(1), 59?70. 30. Kirkpatrick, G. (2020). Technical politics: Andrew Feenberg?s critical theory of technology. Manchester University Press. https://doi.org/10.7765/9781526105349 11 8 31. Kirkpatrick, G. (2013). Formal bias and normative critique of technology design. Techn?: Research in Philosophy and Technology, 17(1), 25?46. https://doi.org/10.5840/techne20131713 32. Knowles, E. S., & Linn, J. A. (2004). Resistance and persuasion (1st ed.). Psychology Press. https://doi.org/10.4324/9781410609816 33. Lai, C. K., Skinner, A. L., Cooley, E., Murrar, S., Brauer, M., Devos, T., Calanchini, J., Xiao, Y. J., Pedram, C., Marshburn, C. K., Simon, S., Blanchar, J. C., Joy-Gaba, J. A., Conway, J., Redford, L., Klein, R. A., Roussos, G., Schellhaas, F. M. H., Burns, M., ? Nosek, B. A. (2016). Reducing implicit racial preferences: II. Intervention effectiveness across time. Journal of Experimental Psychology: General, 145(8), 1001?1016. https://doi.org/10.1037/xge0000179 34. Lewis, C., & Wharton, C. (1997). Cognitive walkthroughs. In Handbook of Human-Computer Interaction (pp. 717?732). Elsevier. https://doi.org/10.1016/B978-044481862-1.50096-0 35. Livingstone, A. G., Fern?ndez Rodr?guez, L., & Rothers, A. (2020). ?They just don?t understand us?: The role of felt understanding in intergroup relations. Journal of Personality and Social Psychology, 119(3), 633?656. https://doi.org/10.1037/pspi0000221 36. LUMA Institute (Ed.). (2012). Innovating for people: Handbook of human-centered design methods (First edition). LUMA Institute. 37. Metaxa-Kakavouli, D., Wang, K., Landay, J. A., & Hancock, J. (2018). Gender-inclusive design: Sense of belonging and bias in web interfaces. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1?6. https://doi.org/10.1145/3173574.3174188 38. Minson, J. A., Chen, F. S., & Tinsley, C. H. (2019). Why won?t you listen to me? Measuring receptiveness to opposing views. Management Science, 66(7), 3069?3094. https://doi.org/10.1287/mnsc.2019.3362 39. Mone, G. (2016). Bias in technology. Communications of the ACM, 60(1), 19?20. https://doi.org/10.1145/3014388 40. Morgan Consoli, M. L., & Marin, P. (2016). Teaching diversity in the graduate classroom: The instructor, the students, the classroom, or all of the above? Journal of Diversity in Higher Education, 9(2), 143?157. https://doi.org/10.1037/a0039716 41. Murphy, G. E. (1985). A conceptual framework for the choice of interventions in cognitive therapy. Cognitive Therapy and Research, 9(2), 127?134. https://doi.org/10.1007/BF01204844 42. Newell, A. F., Gregor, P., Morgan, M., Pullin, G., & Macaulay, C. (2011). User-sensitive inclusive design. Universal Access in the Information Society, 10(3), 235?243. https://doi.org/10.1007/s10209-010-0203-y 43. Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. New York University Press. 44. O?Meara, K., Griffin, K. A., Kuvaeva, A., Nyunt, G., & Robinson, T. N. (2017). Sense of belonging and its contributing factors in graduate education. International Journal of Doctoral Studies, 12, 251?279. 45. O?Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy (First edition). Crown. 11 9 46. Oleson, A., Mendez, C., Steine-Hanson, Z., Hilderbrand, C., Perdriau, C., Burnett, M., & Ko, A. J. (2018). Pedagogical content knowledge for teaching inclusive design. Proceedings of the 2018 ACM Conference on International Computing Education Research, 69?77. https://doi.org/10.1145/3230977.3230998 47. Prieto, L. R. (2018). Incorporating diversity content into courses and concerns about teaching culturally diverse students. Teaching of Psychology, 45(2), 146?153. https://doi.org/10.1177/0098628318762875 48. Punch, K.F. (2003). Survey research: The basics. Sage Publications: London. 49. Race reporting among hispanics: 2010. (n.d.). The United States Census Bureau. Retrieved October 20, 2020, from https://www.census.gov/library/working-papers/2014/demo/POP- twps0102.html 50. Research to improve data on race and ethnicity. (n.d.). The United States Census Bureau. Retrieved October 20, 2020, from https://www.census.gov/about/our-research/race-ethnicity.html 51. Rogue, J., Daring, C. B., & Volcano, A. (2012). Queering anarchism: Addressing and undressing power and desire (S. Deric, Ed.). AK Press. 52. Rudman, L. A., & Ashmore, R. D. (2007). Discrimination and the implicit association test. Group Processes & Intergroup Relations, 10(3), 359?372. https://doi.org/10.1177/1368430207078696 53. Seaver, N. (2019). Knowing Algorithms. In DigitalSTS: A field guide for science & technology studies (pp. 412?422). Princeton University Press. 54. Shilton, K. (2010). Technology development with an agenda: Interventions to emphasize values in design. Proceedings of the Annual Meeting of the American Society for Information Science & Technology (ASIS&T), 47, 1. 55. Shilton, K., Heidenblad, D., Porter, A., Winter, S., & Kendig, M. (2020). Role-playing computer ethics: Designing and evaluating the privacy by design (Pbd) simulation. Science and Engineering Ethics. https://doi.org/10.1007/s11948-020-00250-0 56. Shivers-McNair, A., Phillips, J., Campbell, A., Mai, H., Yan, A., Macy, J., Wenlock, J., Fry, S., & Guan, Y. (2018). User-centered design in and beyond the classroom: Toward an accountable practice. Computers and Composition, 49, 36?47. https://doi.org/10.1016/j.compcom.2018.05.003 57. Skirpan, M., Beard, N., Bhaduri, S., Fiesler, C., & Yeh, T. (2018). Ethics education in context: A case study of novel ethics activities for the cs classroom. Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 940?945. https://doi.org/10.1145/3159450.3159573 58. Slater, M. D., & Rouner, D. (2002). Entertainment-education and elaboration likelihood: Understanding the processing of narrative persuasion. Communication Theory, 12(2), 173?191. https://doi.org/10.1111/j.1468-2885.2002.tb00265.x 59. Steele, C. M. (1988). The psychology of self-affirmation: Sustaining the integrity of the self. In Advances in Experimental Social Psychology (Vol. 21, pp. 261?302). Elsevier. https://doi.org/10.1016/S0065-2601(08)60229-4 60. Steers, M.-L. N. (2016). ?It?s complicated?: Facebook?s relationship with the need to belong and depression. Current Opinion in Psychology, 9, 22?26. https://doi.org/10.1016/j.copsyc.2015.10.007 12 0 61. Stoutenbourgh, J. W. (2008). Demographic measures. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (Vol. 1, pp. 185?186). Thousand Oaks, CA: Sage Publications, Inc 62. Taherdoost, H. (2016). Sampling methods in research methodology: How to choose a sampling technique for research (SSRN Scholarly Paper ID 3205035). Social Science Research Network. https://papers.ssrn.com/abstract=3205035 63. Tourangeau, R. (1984). Cognitive sciences and survey methods. Cognitive aspects of survey methodology: Building a bridge between disciplines, 15, 73-100. 64. Vakil, S. (2018). Ethics, identity, and political vision: Toward a justice-centered approach to equity in computer science education. Harvard Educational Review, 88(1), 26?52. https://doi.org/10.17763/1943-5045-88.1.26 65. Van Camp, A. R., Gilbert, P. N., & O?Brien, L. T. (2019). Testing the effects of a role model intervention on women?s STEM outcomes. Social Psychology of Education, 22(3), 649?671. https://doi.org/10.1007/s11218-019-09498-2 66. Vuletich, H. & Payne, B. (2019). Stability and Change in Implicit Bias. Psychological Science. 095679761984427. 67. Wolf, C., Joye, D., Smith, T. W., & Fu, Y. (2016). The Sage handbook of survey methodology. Sage Reference. 68. Yeomans, M., Minson, J., Collins, H., Chen, F., & Gino, F. (2020). Conversational receptiveness: Improving engagement with opposing views. Organizational Behavior and Human Decision Processes, 160, 131?148. https://doi.org/10.1016/j.obhdp.2020.03.011 12 1