ABSTRACT Title of Thesis: EXPLORING THE ACCESSIBILITY OF HOME-BASED, VOICE-CONTROLLED INTELLIGENT PERSONAL ASSISTANTS Alisha Pradhan, Master of Science, Human-computer Interaction, 2018 Thesis Directed By: Dr. Amanda Lazar, Assistant Professor, College of Information Studies From an accessibility perspective, home-based, voice-controlled intelligent personal assistants (IPAs) have the potential to greatly expand speech interaction beyond dictation and screenreader output. This research examines the accessibility of off-the- shelf IPAs (e.g., Amazon Echo) by conducting two exploratory studies. To explore the use of IPAs by people with disabilities, we analyzed 346 Amazon Echo reviews mentioning users with disabilities, followed by interviews with 16 visually impaired IPA users. Although some accessibility challenges exist, individuals with a range of disabilities are using IPAs, including unexpected uses such as speech therapy and memory aids. The second study involved a three-week deployment of Echo Dot, a popular IPA, with five older adults who use technology infrequently. Findings indicate preferences for using IPAs over traditional computing devices. We identify design implications to improve IPAs for this population. Both studies highlight issues of discoverability and the need for feature-rich voice-based applications. The findings of this research can inform future work on accessible voice-based IPAs. EXPLORING THE ACCESSIBILITY OF HOME-BASED, VOICE- CONTROLLED INTELLIGENT PERSONAL ASSISTANTS by Alisha Pradhan Thesis submitted to the Faculty of the Graduate School of the University of Maryland, College Park, in partial fulfillment of the requirements for the degree of Master of Science 2018 Advisory Committee: Assistant Professor, Dr. Amanda Lazar, co-advisor, Chair Assistant Professor, Dr. Leah Findlater, co-advisor Assistant Professor, Dr. Eun Kyoung Choe © Copyright by Alisha Pradhan 2018 Foreword Statement of Co-Authorship The work in this thesis was done under the supervision of Dr. Amanda Lazar and Dr. Leah Findlater. Study one- “Understanding the use of IPAs by people with disabilities” was advised by Dr. Leah Findlater. Study two- “Understanding the use of IPAs by older adults with limited technology experience” was conducted under the joint supervision of Dr. Leah Findlater and Dr. Amanda Lazar. I am the primary contributor to all aspects of this research. Large parts of Chapters 2 and 3 and few sections of Chapter 5 are part of the following to be published paper, which I collaboratively co-authored with Kanika Mehta and Dr. Leah Findlater. Alisha Pradhan, Kanika Mehta, and Leah Findlater. 2018. “Accessibility Came by Accident”: Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities. Proceedings of CHI 2018, 13 pages. (To Appear) ii Acknowledgements This thesis would not have been possible without the contributions of many people. I would first like to thank my advisors Dr. Leah Findlater and Dr. Amanda Lazar, for mentoring me, providing timely and valuable feedback, which helped me grow as a researcher. I am grateful for their patience, support and invaluable guidance. I would like to thank my fellow HCIM graduate student, Kanika Mehta for contributing to this research by helping with analyses, piloting the studies and providing her feedback. I would also like to thank all individuals who participated in the final studies or volunteered in the pilot studies. I was fortunate to have been supported by two nearby independent living facilities- Newton Green and Randolph Village. I am grateful to community coordinators at these facilities for their help in recruiting participants. I would also like to thank the Research Improvement Grant provided by the College of Information Studies for supporting a part of this work. Finally, I would like to thank my mom and dad for believing in me and supporting me through all these years and my friends, Abhilash Sahoo and Praveen Honhar for always being there for me. iii Table of Contents Foreword ....................................................................................................................... ii Acknowledgements ...................................................................................................... iii Table of Contents ......................................................................................................... iv List of Tables ............................................................................................................... vi List of Figures ............................................................................................................. vii Chapter 1: Introduction ................................................................................................. 1 1.1. Motivation and Research Problem ..................................................................... 1 1.2. Research Questions ............................................................................................ 2 1.3. Approach ............................................................................................................ 3 1.4. Contributions...................................................................................................... 4 1.5. Overview ............................................................................................................ 5 Chapter 2: Related Work .............................................................................................. 6 2.1. Accessibility and Voice-based Interaction......................................................... 6 2.1.1. People with disabilities and voice-based interaction .................................. 6 2.1.2. Older adults and voice-based interfaces ..................................................... 8 2.2. Intelligent Personal Assistants in HCI ............................................................... 9 2.2.1. Voice-based Intelligent Personal Assistants on Smartphones .................... 9 2.2.2. Voice-based Intelligent Personal Assistants at Home .............................. 10 2.5. Smart Home Technology for Accessibility...................................................... 11 2.6. Summary .......................................................................................................... 12 Chapter 3: Study 1- Understanding the Use of IPAs by People with Disabilities ...... 13 3.1. Study I- Analysis of Amazon Reviews ............................................................ 13 3.1.1. Methods..................................................................................................... 13 3.1.2. Findings..................................................................................................... 16 3.1.3. Summary ................................................................................................... 27 3.2. Study II: Interview Study with Visually Impaired IPA Users ......................... 28 3.2.1. Method ...................................................................................................... 28 3.2.2. Findings..................................................................................................... 30 3.2.3. Summary ................................................................................................... 36 3.3. Discussion ........................................................................................................ 36 3.3.1. Subpopulations of users ............................................................................ 37 3.3.2. Accessibility barriers ................................................................................ 38 3.3.3. Smart home adoption and perception ....................................................... 38 3.3.4. Other unexpected use cases ...................................................................... 39 3.3.5. Limitations of the study method ............................................................... 39 Chapter 4: Study 2- Understanding the Use of IPAs by Older Adults with Limited Technology Experience .............................................................................................. 40 4.1. Method ............................................................................................................. 40 4.1.1. Participants ................................................................................................ 40 4.1.2. Procedure .................................................................................................. 41 4.1.3. Analysis..................................................................................................... 44 4.2. Findings............................................................................................................ 44 4.2.1. Individual case studies .............................................................................. 46 4.2.2. Cross-case findings ................................................................................... 78 iv 4.3. Discussion ........................................................................................................ 84 4.3.1. Easy access to digital technology ............................................................. 85 4.3.2. Information access .................................................................................... 86 4.3.3. Current challenges and limitations............................................................ 86 4.3.4. Personification .......................................................................................... 87 4.3.5. Limitations ................................................................................................ 87 Chapter 5: Discussion & Conclusion .......................................................................... 89 5.1. Summary of Findings and Contributions ......................................................... 89 5.2. Reflection and Implications for Design ........................................................... 91 5.2.1. Smartphone vs. home-based IPAs. ........................................................... 91 5.2.2. Design opportunities ................................................................................. 91 5.3. Limitations ....................................................................................................... 94 5.4. Scope of Future Work ...................................................................................... 95 5.5. Conclusion ....................................................................................................... 96 Appendices .................................................................................................................. 98 Appendix A. Study 1-II: Interviews with Visually Impaired IPA users ................. 98 1. Interview Protocol ........................................................................................... 98 2. Interview Codebook ...................................................................................... 102 Appendix B. Study 2 ............................................................................................. 105 1. Demographic and background questionnaire ................................................ 105 2. Interview one questionnaire .......................................................................... 106 3. Interview two questionnaire .......................................................................... 109 4. Interview three questionnaire ........................................................................ 111 5. Interview four questionnaire ......................................................................... 112 6. Interview codebook ....................................................................................... 114 Bibliography ............................................................................................................. 117 v List of Tables Table 3.1: Disability-related search terms used for extracting reviews ...................... 14 Table 3.2: Primary codes used for analysis of reviews ............................................... 15 Table 3.3: Specific tasks performed using IPAs (% of 346 reviews) ......................... 20 Table 3.4: Specific types of home automation mentioned .......................................... 20 Table 3.5: Self-reported details of visually impaired participants in Study 1-II ........ 29 Table 4.1: Self-reported details of older adults in Study 2 ......................................... 41 Table 4.2: Approximate daily usage of IPAs in Study 2 ............................................ 45 vi List of Figures Figure 4.1: Approximate weekly usage of Echo Dot .................................................. 46 vii Chapter 1: Introduction Owing to the recent advancements in technology, the concept of voice-controlled personal assistants and smart homes has now become a reality. Voice-based intelligent personal assistants (IPAs) such as Apple’s Siri, Microsoft’s Cortana and Google Now on smartphones have become popular over the past few years. They allow users to perform a variety of tasks such as making a call, sending a text message, setting reminder, managing calendar, web search and setting an alarm among others by voice commands [22]. Home-based intelligent personal assistants such as Amazon Echo and Google Home have taken this new interaction paradigm further, providing richer conversational interfaces and broader integration with smart home devices. Researchers have referred these assistants as conversational agents [57], voice- controlled digital assistants [21], intelligent personal assistants [41] and intelligent digital assistants [1], whereas, commercially they are popularly also known as smart speakers [39,51,66,80]. In this research, these devices are referred as intelligent personal assistants (IPAs) due to their ability to take requests from a user and perform variety of tasks. 1.1. Motivation and Research Problem From an accessibility perspective, home-based IPAs offer different affordances than smartphone-based IPAs. A person with limited mobility, for example, can control their home’s lighting or door locks by voice, while a blind user can ask for the time or weather, or a person with limited technology experience can ask and find information from the internet without requiring the knowledge of using a traditional computing 1 device (provided the device is set up). However, due to the relatively recent introduction of these devices, researchers have only begun to understand how they are being used by the general population (e.g. understanding the personification of these devices [57] and children’s view of them [21]), much less by users with disabilities or older adults. The remote voice-based interaction using natural speech provided by this technology can be potentially beneficial for people with disabilities and older adults with limited technology experience. This made us interested in understanding how these devices are used by these specific user groups. 1.2. Research Questions With the increasing adoption of voice-controlled IPAs at home, including people with disabilities and older adults in the design of these technologies is critical. This work aims to address exploratory questions such as: 1. To what extent are off-the-shelf IPAs, which were not necessarily designed with accessibility in mind, accessible? 2. How are people with disabilities and older adults with limited technology experience are making use of these devices for everyday activities? 3. What design opportunities do these devices offer to further support everyday activities for these specific user groups—people with disabilities and older adults who are not everyday users of a computing device (e.g., smartphone, tablet or computer)? 2 1.3. Approach To answer the above research questions, three exploratory studies were conducted. The first study focuses on use of IPAs by people with disabilities, whereas the second study focuses on older adults with limited technology experience. The first study broadly examined use of IPAs by people with disabilities, by collecting and analyzing online customer reviews of the Amazon Echo, a popular home-based IPA, and its offshoots, the Echo Dot and Tap. A total of 346 reviews were identified that described use of the IPA by a person with a cognitive, sensory, or physical disability, written either from a first- or third-person perspective. A subset of these users were older adults above the age of 65. Content analysis of the reviews was performed, qualitatively coding them along dimensions such as overall tone (positive/negative), uses of the device, and accessibility challenges and benefits. To complement the findings from the reviews analysis, we conducted a follow up study for a more in-depth understanding of one specific subset of users: those with visual impairments. Sixteen participants with visual impairments who owned an Amazon Echo or Google Home device were interviewed. The interview covered similar themes to the analysis of reviews, with questions focusing on device usage patterns, specific advantages, limitations, concerns and expectations from this technology. Findings of Study 1 indicate the ease of use of IPAS compared to existing technology, easy access to digital technology and increased efficiency of performing a variety of tasks for users with a range of disabilities. At the same time, the currently limited functionality of the device and unexpected use cases of speech therapy, learning support, and memory support point to potentially fruitful avenues of future work. 3 For the second study, we were specifically interested in understanding how older adults with limited technology experience might use IPAs. We found in Study 1, 46 reviews included an older adult (above age of 60), of which 14 users (30.4% of 46 reviews) positively compared IPAs to previously used smartphones, tablets or computers, mentioning easy access to digital technology. Hence, we conducted a second in-depth study with adults who were 65 or older and did not regularly use a computing device e.g., computer, tablet or smartphone. We deployed Amazon Echo Dot devices for a three-week period at five participants’ houses and studied their usage. After the initial set-up and interview, participants shared their usage experience through daily phone calls and weekly in-person interviews. The findings from Study 2 again reflected many of the findings from Study 1 (Amazon reviews with specific mentions of older adults), such as emphasizing easy access to digital technology, preference for using voice-controlled IPAs compared to a traditional computing device for accessing internet for older adults, specifically who did not regularly use a computing device. Although IPAs have the potential to assist memory (also reported in Study 1), we identified design implications and scope of further research to enhance these devices for better supporting older adults’ memory needs in Study 2. 1.4. Contributions This work provides the first analysis of how users with disabilities and older adults with limited technology experience are using IPAs. From the findings of the three exploratory studies, this thesis contributes: 4 1. A characterization of how voice-based intelligent personal assistants are being used by people with disabilities and older adults with minimal technology experience. 2. Identification of accessibility benefits and barriers offered by home-based IPAs. 3. Recommendations for design as well as future work on conversational voice interfaces for users with disabilities and older adults. 1.5. Overview This thesis is structured into the following chapters. The next chapter covers related work. The third chapter describes the first two exploratory studies for understanding the use of IPAs by people with disabilities—analysis of 346 Amazon reviews, followed by interviews with 16 visually impaired users of Amazon Echo or Google Home. The fourth chapter covers the field study using Echo Dot devices with five older adults 65 years old or above who did not regularly use a computing device. Finally, the last chapter reflects the overall findings from the two studies, outlining possible future work accessible design of home-based IPAs. 5 Chapter 2: Related Work This chapter provides a review of existing HCI research on intelligent personal assistants (IPAs) in general, voice-based IPAs and voice-based interaction for accessibility. Since popular IPAs such as Amazon Echo and Google Home allow integration with smart home appliances, we also cover previous work on smart homes for people with disabilities and older adults. 2.1. Accessibility and Voice-based Interaction Speech input and output is commonly used to support technology access for users with disabilities [34], especially users who are blind or visually impaired, or who have motor impairments [62] and older adults [7]. Traditionally, the two most common forms of accessible speech interaction are screenreaders, which provide audio output for users with visual impairments (e.g., Apple’s Voiceover [81] or JAWS [82]), and speech dictation software, which provides a text entry alternative to the keyboard (e.g., Dragon [83]). 2.1.1. People with disabilities and voice-based interaction Speech input has been shown to be helpful for a range of applications for users with motor impairments, including text input on desktops [44,71] and smartphones [48]. Manaris et al. [44] found text input using speech was faster and more accurate than typing with handstick. Wagner et al. [71] explored the use of voice input for programming as an alternative to traditional input using mouse and keyboard for people with motor impairments. Studies have also explored the use of speech input for 6 controlling wheelchairs [52,63] and free-hand (voice) drawing [31,32] for people with motor impairments. Studies have found that people with visual impairments are more likely to use speech input on smartphones than sighted participants [5,78]. For people with visual impairments, as found by Azenkot et al. [5] dictation using speech is faster than using on-screen keyboard, which is the traditional mode of input on smartphone. Similarly, Ashok et. al [4] found voice input to be faster, more preferred and more usable than the traditional keyboard on a computer, for visually impaired users. Speech output to support users with visual impairments has been explored in a variety of contexts beyond the standard screenreader. For example, Narasimhan et al. [50] explored speech technology for supporting applications like grocery shopping, currency identification and transportation for people with visual impairments using speech output. People with speech impairments can also benefit from computerized speech interaction for speech practice [24]. In a study conducted on people with dystharia, Palmer et al. [53] found that speech therapy using speech recognition technology is as effective as the traditional therapy. They also found that people are likely to spend more time on the computerized system than with the traditional therapy methods. Derboven et al. [19] conducted an exploratory study on how people with speech and physical impairments form commands for a speech interface, finding that commands were short, directive statements and were often ambiguous. Speech interaction has also been explored for people with cognitive impairments. The study conducted by Wolters et al. [74] informed design guidelines for spoken dialogue assistants for older adults with dementia. Caroll et al. [15] 7 developed a voice-based application for audio prompting routine tasks to assist in independent living of people with cognitive impairments. 2.1.2. Older adults and voice-based interfaces Many studies have shown that voice-based interaction is perceived positively by older adults. In one exploratory study, older adults showed an overall positive response and high acceptance of speech interaction [76]. Himmelsbach et al. [35] found older adults perceive speech interaction as easy and would prefer it for mobility and hands-free usage. Moreover, they also found that for some older adults, speech interaction has the potential to build ‘para-social’ relationships, a one-sided psychological relationship developed by a user with the media, where the user behaves as though they are involved in a social relationship to the extent that they might even consider the media as a “real friend”. Voice-based interaction was also used for contextual learning (where the system suggests the user the next action to perform) for older adults and has shown to improve their confidence of performing internet tasks [60]. Schlogl et al. [61], found that older adults prefer spoken input over graphical user interfaces (GUIs) for finger- operated input tasks e.g., messaging. The small size of the screen, the font and the keyboard were perceived interaction barriers for GUIs. Researchers have also explored voice interfaces for creating applications such voice-based email system [10] and voice-based online community [11] for older adults. 8 2.2. Intelligent Personal Assistants in HCI Intelligent personal assistants are agents which can take requests from the user and perform a variety of activities on behalf of the user [26]. Much of the work on IPAs has focused on making these agents intelligent, by adding semantic web knowledge [26], user personalization and context awareness [49], natural language processing and dialogue management [47], or connecting IPAs with internet of things devices [59]. With the increasing popularity of mainstream voice-controlled intelligent personal assistants, researchers have recently started to understand how these agents are used by the general population. 2.2.1. Voice-based Intelligent Personal Assistants on Smartphones There has been some work studying the use of commercially available conversational intelligent personal assistants—Siri, Google Now and Cortana—primarily housed in smartphones and/or computers. Luger and Sellen [42] studied the use of smartphone- based IPAs, finding the challenge with conversational agents is that user expectations tend to exceed the agents’ abilities. These agents were primarily used for simple tasks of checking the weather and setting reminders and the rationale behind using them was multi-tasking and hands-free use. However, apart from a few, most users perceived these agents as gimmicky or entertaining. There are some studies on understanding the concerns of using voice-based IPAs. Studies [22,28] show that people have privacy concerns over using voice interaction. Moorthy et al. [22] found that apart from privacy, the reasons for not using voice assisted personal assistants include being misunderstood by the device and 9 unsatisfactory answers from the device among others. Ye et al. [78] found that speech interaction is not preferred in noisy and silent spaces. Other concerns include awkwardness in speaking to a device and the social acceptability of talking to a device [5]. 2.2.2. Voice-based Intelligent Personal Assistants at Home Our focus is on home-based assistants, such as the Amazon Echo and Google Home, which are a recent phenomenon and offer different affordances and accessibility opportunities than smartphone-based support. Interaction is remote (e.g., across the room), which lowers the barrier to use in comparison to having to hold/use a device, and home-based assistants can connect to smart home appliances, becoming integrated into the home environment. Home-based IPAs are relatively new, and the research literature on their use is accordingly sparse. Purington et al. [57] studied device personification in Amazon Echo reviews, concluding that users who personified the device were more likely to be satisfied with it. Druga et al.[21] studied how children perceive intelligent personal assistants (e.g., trust, intelligence level), including Amazon Echo and Google Home, although the focus was not on the children’s actual or desired use of the devices. Finally, researchers have begun examining issues of privacy and security with always-on smart home devices, such as concerns arising in multi-user homes. Zeng et al. [79] found the primary concerns were physical security, collection of audio and behavior logs of people, or personally identifiable information. Alanwar et al. [1] developed a sonar based defense to reduce the vulnerability of IPAs from attacks through nearby devices (TVs, radio, voicemail etc.) by identifying the presence of 10 speaker in the room. This body of research is in the early stages and, to our knowledge, no one has examined the accessibility of these off-the-shelf home-based IPAs—our focus. 2.5. Smart Home Technology for Accessibility Smart home technologies have long been perceived as useful for users with disabilities and older adults, although until the past few years many solutions have remained as research prototypes or have been too costly for mainstream adoption. Domingo [20], for example, proposed a vision and Internet of Things (IoT) architecture to support individuals with disabilities in daily tasks from shopping to attending school. In comparison to users with disabilities, more studies have focused on older adults and smart homes, showing that the most designed feature is emergency help [14,18,33,46]. Mihailidis et al.[46] and Demiris et al. [18] found that other highly desired features included health monitoring system and home environmental control (e.g., lights, temperature). Smart home technologies also introduce challenges, primarily related to privacy and security [79], along with cost [33,67] and a worry about being dependent on the technology [56]. Vacher et al. [69] found that both elderly and visually impaired people were positive about controlling a smart home using voice. Other studies have shown that users with multiple sclerosis [64] and older adults [14], many of whom had motor impairments, desired voice-based control over the home (e.g., doors, windows). Despite these positives, there may be downsides of smart home voice control, such as accessibility issues around speech input (e.g., adults with non-continuous speech due 11 to Alzheimer's disease) [58], and a reduction in perceived control compared to manual input [43]. 2.6. Summary We reviewed previous HCI and accessibility research on intelligent personal assistants (on smartphone and in home-based environment), voice-based interaction for people with disabilities and older adults, and smart home technology. Previous work shows that voice-based interaction has been explored widely for people with disabilities and older adults and is preferred due to the ease of use and hands-free usage. However, there is no work examining the use of IPAs in the home environment by users with disabilities and/or older adults, which is our focus. Since home-based IPAs can connect with smart home devices to facilitate home automation, we also studied related work on smart homes for people with disabilities and older adults. But most of these studies were research prototypes. Compared to this past work our study is timely: because smart home technology has entered the mainstream, we can analyze real-world impacts for a broad range of users. This thesis aims to understand the accessibility of these off- the-shelf IPAs by studying the device usage by two specific user groups—people with disabilities and older adults with limited technology experience. 12 Chapter 3: Study 1- Understanding the Use of IPAs by People with Disabilities This chapter describes two exploratory studies conducted to understand how users with disabilities are making use of home-based voice-controlled IPAs. Primarily, these studies aim to address the following research questions. 1) To what extent are off-the-shelf IPAs, which were not necessarily designed with accessibility in mind, accessible? 2) How are people with disabilities making use of them? 3) What opportunities do these devices offer to further support everyday activities for users with disabilities? The first study broadly examined use of IPAs by people with disabilities, by collecting and analyzing online customer reviews of the Amazon Echo, a popular IPA, and its offshoots, the Echo Dot and Tap. We identified 346 reviews that described use of the device by a person with a cognitive, sensory, or physical disability, written either from a first- or third-person perspective. Then, to get a deeper understanding of IPA use by a more specific subset of users with disabilities, we interviewed sixteen participants with visual impairments who owned an Amazon Echo or Google Home device. 3.1. Study I- Analysis of Amazon Reviews 3.1.1. Methods Our approach is inspired by analyses of online content to derive implications for accessible design [3,13], and a study on personification in Amazon Echo reviews [57]. 13 Disability-Related Search Terms Specified in Advance AAC, accessibility, accessible, ALS, Alzheimer, Alzheimer’s, amnesia, amnestic, amputation, amputee, amyotrophic lateral sclerosis, aphasia, apraxia, arthritis, assistive technology, ataxia, augmentative communication, autism, autistic, blind, blindness, caregiver, cochlear implant, congenital amputation, congenital amputee, deaf, dementia, diabetic retinopathy, disabilities, disability, disabled, Down syndrome, dysarthria, dyslexic, dystonia, epilepsy, essential tremor, fibromyalgia, Friedreich ataxia, Friedreich's ataxia, glaucoma, handicap, handicapped, hard of hearing, hearing aid, hearing device, hearing loss, hemiplegia, hemiplegic, impaired, impairment, impairments, lateral sclerosis, lisp, Lou Gehrig's, macular degeneration, mobility, multiple sclerosis, muscular dystrophy, muscular rheumatism, myopathy, neurological disorder, neurological vision impairment, neuromuscular disorders, nursing home, paralysis, paralyzed, paraplegia, paraplegic, Parkinson, Parkinson’s disease, Parkinsonism, quadriplegia, quadriplegic, sclerosis, seizure disorder, short term memory, sigmatism, SMA, speaking disorder, special needs, speech impediment, speech therapy, spinal bifida, spinal cord injury, spinal muscular atrophy, stroke, stutter, TBI, traumatic brain injury, tremor, tremors, vision, walker, wheelchair. Emergent Keywords Bedridden, disease, injuries, injury, limited vision, no vision, non-verbal, nonverbal, poor vision, rehab, rehabilitation, surgeries, surgery. Table 3.1. Disability-related search terms used for extracting reviews, including terms defined a priori and emergent keywords identified through reading a subset of reviews. Dataset creation We first collected 28,921 Amazon Echo, 27,286 Echo Dot, and 5,370 Tap reviews in June, 2017 from Amazon.com. All reviews were verified reviews, meaning that Amazon confirmed that the customer had purchased the device before reviewing. To identify reviews related to disability, we created a list of keywords related to cognitive, sensory, or physical abilities (following [2,9]). As shown in Table 3.1, the list included 95 keywords identified a priori and 13 emergent keywords identified by reading ~500 reviews. Of the full review set, 792 included at least one keyword, but not always in the context of disability. We thus defined a relevant review as one that contained a first- or third-person mention of a user with a disability. Two research team members independently evaluated the relevancy of 50 randomly selected reviews, agreeing in 49/50 cases (Cohen’s kappa = 0.96). One team member then assessed all remaining reviews. The final dataset included 478 relevant reviews, although as mentioned below, we further eliminated reviews that only hypothetically had a mention of a user with a disability, leaving 346 reviews in total. 14 Review Analysis The reviews were coded along the 26 dimensions shown in Table 3.2, which include both inductive and deductive codes. Deductive codes were informed by related work (e.g., on smart homes, privacy) and our own experience with IPAs, while inductive codes were added upon reviewing the data. Two research team members built an initial codebook, with one person reading approximately one-third of the reviews in depth, and the second person participating in discussions, reading a smaller subset of the reviews, and helping to add, merge, and delete codes. 1. Perspective: first person, third person, third person (hypothetical) User details 2. Disability: motor, vision, speech, cognitive, hearing, other, unspecified 3. Length of disability: short-term, long-term, unspecified. 4. Age: older adult, child, younger adult or unspecified 5. Household size: lives alone, other in house, unknown 6. Use in nursing home/rehab center/hospital: yes, no/unknown 7. Obtaining the device: was given it, bought it or unknown Overall opinion 8. Overall tone of the review: positive, negative or neutral Social aspects 9. Device as companion 10. Independence 11. Indispensable 12. Helpful for caregiver/family member to support caregiving 13. Enables digital tech access 14. Safety 15. Awkwardness or discomfort with device interaction 16. Privacy 17. Security 18. Other 19. Limitation (Functional Limitation, Criticism, or Suggestion) User interface / interaction 20. User interface positives 21. User interface negatives Speech recognition 22. Speech recognition positives 23. Speech recognition negatives 24. Device setup Device usage 25. Specific activities performed 26. Home automation: yes, no Table 3.2. Primary codes used for analysis of reviews. To ensure coding reliability, we used a multi-phase process [36]. First, one researcher involved in the initial codebook creation and one new team member 15 independently coded 20 randomly selected reviews, discussed disagreements, and refined problematic codes. Second, the same two researchers independently coded 40 new randomly selected reviews. Cohen’s kappa calculated on the primary codes (all numbered codes in Table 3.2) after this second round was on average 0.96 (SD=0.07, Range=0.79-1.00). Four codes that had been present in the first round were by chance not applicable in the second round, and were excluded from these calculations (Indispensable, Privacy, Home automation, and Awkwardness/discomfort with device interaction). We also removed one primary code (Technology comfort) due to sparsity and added Use in nursing home/rehab center/hospital. Finally, one researcher coded all reviews using the refined codebook. The excerpts marked with each code were then qualitatively analyzed to obtain richer descriptions to complement the coded data. We also computed basic statistics on review length, rating (on a 5-point scale), and age (based on date of posting). 3.1.2. Findings We report on review and user characteristics, overall experience, device usage, accessibility issues, and emergent themes such as independence and safety. To focus on reviews based on experience with the device, we exclude from this analysis 132 reviews that only mentioned disability or accessibility in a hypothetical sense (under the coding dimension Perspective); for example, from R14: “Alexa could be immensely valuable in helping a person with limited mobility and/or physical disability.” Our analysis thus includes 346 of the 478 reviews. Throughout we refer to reviews by ID numbers R1-R478. 16 Review Characteristics The reviews were on average 775 characters long (SD=810) and had a rating of 4.5 out of 5 (SD=1.0). As of June 15, 2017, the reviews were 312.2 days old (SD=205.7, Range=0-727). About a third (N=114; 32.9%) were written from the first-person perspective of someone with a disability, whereas 232 (67.1%) were written from a third-person perspective. These latter reviews were mainly written by people who had close ties with the user with a disability, such as a son or daughter (in-law) (36.6% of the 232 third-person reviews), spouse (26.7%), parent (16.8%), other family member (11.6%), or friend (4.3%); the remaining 4.3% of reviews did not mention what relationship the author had to the user. One review included purchases for two separate users with disabilities, so percentages sum more than 100%. Ratings from both first- person and third-person reviews were positive on average, at 4.6 (SD=0.9) and 4.3 (SD=1.2), respectively. User characteristics Our dataset included users with a diverse set of disabilities: visual impairment (37.9% of reviews), motor or mobility impairment (30.6%), speech impairment (13.6%), cognitive impairment (11.8%), and hearing loss (4.6%). An additional 18.2% only mentioned disability in general. Some reviews mentioned more than one specific type of disability, so percentages sum to more than 100%. Nineteen (5.5%) of the reviews mentioned that the disability was only short-term, such as a user recovering from an injury or surgery. In terms of age, we looked for mentions of older adults or children, and found that 46 reviews (13.3%) mentioned a user who was 60+ years old or used age-specific 17 keywords (e.g., elderly, old, older), while 16 reviews mentioned that the user was a child (4.6%). While only 145 reviews explicitly mentioned whether the user with a disability lived alone or with others, the vast majority of these mentions were of households with multiple members (138 reviews; 95.2%); the remaining seven mentioned living alone. A small number of reviews (4.0%) mentioned use in a nursing home, rehab center, hospital, or assisted living center. Users tended to receive the device as a gift rather than buying it themselves. Of the 202 reviews that mentioned how the device was obtained, 79.2% involved a gift (N=160). Examining this data by user age revealed that older adults were disproportionately represented: 22.5% of gift recipients were older adults, although older adult users were only mentioned in 13.3% of all reviews. User interface and interaction In terms of overall tone, most reviews had a positive tone (85.6%), only 11.9% had a negative tone; the remaining 2.6% were coded as “neutral”. Eight reviews even mentioned that the device had become an integral part of the user’s life. For example, in a first-person review, R94 stated: “This has to be the best gift I have gotten in years. I'm so used to it being here that I would be lost without it.” Some reviews (6.7%) referred to the device as a companion [57], using terms such as “new best friend”, “bff”, and “someone to talk to”. For example, R237 said: “I love my Dot - I live alone and now I have "someone" to talk to - AND "they" respond! A wonderful delivery system for everything I want to know and everything I enjoy…I'm housebound for three months after spinal surgery - Dot has helped me keep my sanity!” 18 Ease of use was commonly brought up as a positive, arising in 23.4% of reviews. The voice-based interaction, which allowed for control from a distance and without visual information was valued. For example, R21 said: “I can’t begin to tell you what a difference the echo has made to my disabled veteran husband. After his stroke, his mobility and speech were effected. Giving him a whisper of a voice. He can now ask Alexa to play any song of his choosing without having to getup…” (R21) Many reviews (N=46; 13.3%) also positively compared the device to a smartphone, computer, or other existing device. About 8.4% (29/46) reviews that included users with visual impairments, in particular, described how this single device allowed them to perform a variety of tasks that had previously required multiple technologies (e.g., computers, radios, audio book readers). For example, R281 said: “Alexa is a wonderful machine- Especially for myself as a sight impaired individual. Audible books, alarms, conversions, calculator functions- It does a lot of things that I would have previously had to get other specialized products to achieve.” (R281) Related, 9.2% of reviews (N=32) mentioned that the device provides easy access to digital technology as compared to more traditional computing devices. Perhaps unsurprisingly given the audio-based interaction, most of these reviews (21 of 32) included users with visual impairments and/or older adults (14 of 32). However, 19.1% of reviews also mentioned functional limitations, criticisms, or suggestions for improvement. Nineteen (5.5%) of reviews mentioned that a desired feature was missing, such voice calls and messaging, emergency calls, alternative input via a remote, or braille to make the case more accessible. (Note- subsequent updates of the device have addressed some of these concerns). Another criticism that arose was 19 Activities performed (%) Activities performed (%) Listening to music 34.7 Listening to jokes 7.5 Looking up information 18.5 Setting a timer 6.7 Checking the weather 17.5 Managing a shopping list 6.1 Playing audio books 15.6 Managing a calendar 5.2 Home automation 15.0 Playing games 5.2 Listening to news 10.1 Third-party skills (e.g., Uber) 4.1 Asking time or date 9.5 Managing a to-do list 3.8 Playing the radio 8.4 Online shopping 3.5 Setting an alarm 7.5 Other (e.g., calls, spelling) 13.7 Table 3.3. Percent of the 346 reviews that mentioned a specific task. that the device offered limited use, which was mentioned in 4.0% of reviews. This was also identified in studies on smartphone based IPAs [42]. For example, R405 said, “…echo still is not very smart. About 80% of my questions i (I) ask it did not know.” Other less common criticisms mentioned in at least five reviews included having to pair the device with a smartphone, requiring Wi-Fi, lack of portability, issues with the audio sensing, and issues with specific apps (“skills”). Finally, a few reviews (2.9%) mentioned that the cost of the device and associated apps was a challenge. Although the base cost of the device is relatively cheap (e.g., the Dot), additional skills, or subscriptions (e.g., Amazon Prime, Audible) are often needed. Common uses and home automation Home automation (%) Home automation (%) Lights 82.7 Television 7.7 Smart outlets 21.2 Security system 5.8 Thermostats 19.2 Door locks 5.8 Switches 7.7 Other (e.g., fan, sprinkler) 15.4 Table 3.4. Specific types of home automation mentioned (% of 52 reviews containing home automation. ) Over half of the reviews (N= 212; 61.3%) mentioned specific tasks for which the device was used, with the most common tasks being a mix of entertainment and utility— 20 listening to music, looking up information, checking the weather, and so on. Details are shown in Table 3.3. Because control over smart home appliances is a primary marketing component of IPAs, we further examined these mentions. About 15.0% (N=52) of reviews mentioned use of smart home appliances. The most popular smart home appliances were lights (82.7% of the 52 reviews), followed by outlets (21.2%) and thermostats (19.2%), see Table 3.4. Less frequent, but still mentioned, were televisions, coffee makers, switches, security systems, door locks, smart hubs, fans, and a personal safety device, electric bed, sprinkler, and garage door. The majority of home automation reviews included users with motor impairments (71.2% of the 52 reviews). Approximately a third (28.8%; 15 of 52) of these reviews mentioned improved independence and just over half (53.8%; 28 of 52%) mentioned ease of using due to voice control. For example, R213, discussed how the smart home appliances saved effort and could even be used to communicate with others in the home: “If I want to turn lights on or off after I go to bed, I just tell the Dot to do that. More complex instructions can be routed through the Alexa Channel on IFTTT, including flashing lights to signal my son to go to bed or turning off all of the lights that were accidentally left on. I can turn off my air conditioner in my bedroom in the middle of the night with the lights out by saying, "Alexa, turn off bedroom air." The not having to get up after I've gone to bed thing makes a difference in my exhaustion level.” (R213) Independence 21 An emergent impact of the device was increased independence, mentioned in 14.2% of reviews (N=49), all but five of which included users with visual and/or motor impairments. For example, R38, who is quadriplegic, stated: “It allows me to help my husband just a little bit, not something that I have been able to do for a while. It gives you just a little bit of independence, and that is huge for folks who don’t have any.” (R8) An important source of independence was the ability to control smart home appliances such as lights or thermostats, especially for users with mobility impairments. For example, R345, who has ALS, stated: “I am so thrilled with my new Amazon Echo (aka Alexa) and the freedom it has given me. I use the word freedom over independence because a person does feel somewhat caged when you have an active mind in an inactive body. You also feel guilt from fear of over burdening your caregivers. Alexa has alleviated much of this problem for me....” (R345) For people with visual impairments, a common theme was the ability to use the device for a range of small tasks without having to depend on someone for help (48.9% of 49 reviews), such as listening to music, checking the weather, asking for the time or date, listening to a desired radio channel, looking up for information, managing shopping and to-do lists, reading books, or listening to the news. For example, R472 said: “My wife who is legally blind and has disabilities due to a stroke absolutely loves Alexa! She used to have to depend upon others to assist her with time, weather, making lists, taking care of her calendar, and many other daily chores. Thanks to Alexa she is in control of all of these as well as enjoying music again.” (R472) The impacts of independence also extended to alleviating burden on caregivers. This reflects on some of the past work of using new technologies for caregivers [40,68]. A 22 few reviews (3.2%) mentioned that the device had reduced some caregiver demands, such as reading books, playing music, controlling the home environment, or answering simple questions. For example, R350 said that instead of frequently having to repeat the time, daily agenda, and so on: “Alexa has been phenomanal with taking some of the pressure off of me. She can answer the time ALL DAY LONG, and never get annoyed, lol.” Safety Sixteen reviews (4.6%) mentioned that the device had improved safety, with 12 of these mentions coming from people with motor impairments. Several of these reviews (N=7) commented on an app that sends an emergency alert to a contact. R289 also described using home automation to send messages: “This was a gift for our son who has ALS. It has been very helpful to him in turning lights on and off where he can't access them and has even brought needed assistance by blinking lights in another room to get someone's attention when help was needed.” (R289) Accessibility challenges Accessibility challenges arose, primarily related to speech interaction, the device ecosystem, and memory demands. Sixty-four reviews (18.5%) mentioned speech recognition accuracy, most of which (59.4% of 64) were positive mentions. However, speech input can be particularly problematic for people with speech impairments. There were 31 reviews that included a user with a speech impairment and comments about speech recognition. Perhaps surprisingly, many of these comments were positive (23/31; 74.2%). For example, R144 stated, “Most humans cannot understand me, but Alexia can,” while R318 wrote, “Ordinarily voice programs can’t understand what I am saying due to my speech impairment, but Alexa responds to my commands without 23 fail.” Another review (R126) mentioned using their AAC device to give commands to Amazon Echo, a behavior also identified in Kane et al.’s study [37]. Still, 10 users with speech impairments also or solely mentioned difficulties with speech recognition, such as the need to enunciate clearly and speak loudly. Another issue that arose for users more broadly, beyond those with speech impairments, was the device timing out before the speaker could complete their command (also noted for users with Alzheimer’s disease [58]). Speech output challenges also arose in a few reviews. Three users with hearing loss experienced difficulty in understanding the speech output from the device and could benefit by additional speech settings and output via earphones. R154 said, for example: “…just a bit too much bass for speech (I'm a hard of hearing with typical treble roll-off)... wish there was a music & speech tone setting.” A second accessibility challenge arose from the paired smartphone app, which is part of the larger device ecosystem. The smartphone app is required for device setup, and at other times the user may be referred to it for detailed explanations or troubleshooting. However, the smartphone app presented accessibility challenges mentioned by six users with visual impairments, highlighting the need to ensure that the entire ecosystem—not just the voice-based interaction—is accessible. Memory demands of the voice-based interaction also presented a third accessibility challenge. Some reviews (4.9%) mentioned difficulties in remembering voice commands/keywords due to memory issues, which could be particularly problematic for older adults or users with cognitive impairments. For example, R81 24 mentioned that an 89-year-old user had difficulty remembering how to wake the device (with the word “Alexa”), while R200 said: “I bought this for my 86 year old father almost 6 months ago. He’s got very limited vision at this point. Unfortunately, he’s not making full use of it’s potential because he can’t quite remember exactly the words needed to "wake" some of the skills that are available.” (R200) At the same time, the ease of the conversational interface offered benefits for some users with memory issues. For example, R91, mentioned that a user with dementia sometimes forgets how to dial a phone, but can now use a voice command to call his partner with Alexa. Unexpected uses Along with the conventional uses of the device in Table 3.3, some unexpected use cases arose, including use of the device for speech therapy, learning support, and as a memory aid. These use cases offer insight into potential avenues of future research. In terms of speech therapy, seven reviews (2.0%) described how Amazon Echo had helped in improving the speech for people with speech impairments by forcing the speaker to talk slowly, clearly, and loudly. The conversational nature of the device was also seen as helpful. For example, R185 said: “Our oldest daughter has a pretty challenging speech impediment and using Alexa has forced her to slow down and enunciate clearly. Not only is Alexa learning how to understand my daughter, my daughter is also slowing down and learning to communicate with Alexa. The huge benefit is she is now slowing down to communicate more clearly with us. This is something her speech therapists have been working on with her for years. Alexa has gotten these results from her in a few months.” (R185) 25 R329 further described how the device was used to measure speech improvement for the reviewer’s brother with autism: “He'll speak to Alexa, ask her questions about the weather, and if Alexa responds, my parents know his speech is improving.” Use of the device to support learning also arose. The voice-based, conversational interaction allowed some users with print disabilities to access information. Specifically, of the five reviews (1.4%) that mentioned a user with dyslexia, four reported that the device was useful for reading audio books or asking questions. For example, R68 said: “My daughter is dyslexic and struggles with reading, but we load audio books on to our Amazon music account and Alexa plays the books while she is playing, resting, falling asleep. She asks her questions about everything under the sun, and Alexa never tires of answering them.” (R68) A third unexpected use case was as a memory aid for users who had memory difficulties (mentioned as an issue for 19 of the 41 users with a cognitive impairment). Features like setting reminders, timers, managing a calendar, to-do lists, and shopping lists, and asking for the time, date and weather were seen as most helpful. For example, R190 said: “I live alone, and was recently diagnosed with a disease that leaves me confused on details and the passage of time. It has been a godsend to be able to ask Alexa the day, date, time, or weather, set wake-up alarms or reminder alarms (for example, turn off the oven in an hour, or take my medicine), add to my to-do list or shopping list, etc.” (R190) Finally, the applicability of the device to a medical setting and for short-term disabilities such as injury or recovery after surgery arose (5.5% of reviews). In some cases, the device was seen as useful for maintaining medication timing (N=5). Nineteen 26 reviews mentioned short-term disabilities and reported benefits similar to those expressed by users with long-term disabilities, such as being able to listen to music or jokes, or query information by voice. For example, R422 said: “Recently I had surgery and I am not able to move around a lot as I recover. This is amazing, I feel like I have someone else in the house when no one is at home. Takes away my anxiety of being alone while my husband is at work.” (R422) 3.1.3. Summary This study shows that users with a broad range of disabilities are making use of voice- based intelligent personal assistants in the home. Reviews were hugely positive, mentioning impacts such as ease of use compared to existing devices (smartphone, tablet etc.) and the ability to more independently complete everyday tasks—due both to internet-connected apps as well as smart home appliances. Despite being highly accessible, challenges still arose, particularly for people with speech impairments and for users with hearing loss. Accessibility of the larger device ecosystem (e.g., physical device design, smartphone app, smart home appliances) needs to also be considered. A desire for more feature-rich voice applications and unexpected use cases of speech therapy, learning support, and memory support point to potentially fruitful avenues of future work. A limitation of this study, however, is that, while the online reviews provided a large sample size, the data itself is sparse and does not allow for an in-depth understanding of individual users’ experiences. As such, we turn to an interview method as a follow up. 27 3.2. Study II: Interview Study with Visually Impaired IPA Users To complement the breadth offered by Study I, we conducted a second, in-depth study to examine use by one specific subset of users: 16 blind and visually impaired users Amazon Echo or Google home users. 3.2.1. Method Participants We recruited 16 participants (11 female, 5 male) with visual impairments who owned an Amazon Echo, Echo Dot, Amazon Tap, or Google Home device; three participants also reported having a mobility impairment. Details are shown in Table 3.5. Fifteen participants owned a smartphone and 14 had experience using voice-controlled smartphone assistants like Siri or Google Now. Participants were recruited from across the United States through Facebook groups specific to Amazon Echo, Echo Dot and Google Home, participant lists maintained by our research team, and snowball sampling. Participants were each compensated with a $15 Amazon gift card. Procedure We conducted semi-structured interviews over Skype or Google Hangout, or via a regular phone call. Conducting remote interviews provided us the flexibility of reaching to a larger number of participants who had experience of using a home-based IPA than would have been possible locally. The interview protocol was approved by our Institutional Review Board (IRB). Interviews were designed to last one hour, but ranged from 33–85 minutes long. Interview questions covered the following categories: background and demographics, number of devices owned, when/how device was 28 acquired, motivation for buying the device, device usage (frequency, activities), comparison of expectations before using to actual experience, benefits and concerns/challenges/limitations of using the device, speech recognition experience, use of and desire for home automation, general user interface preferences and suggestions for improvement. All interviews were audio recorded. Analysis Interviews were transcribed and qualitatively coded using a thematic coding approach that included both inductive and deductive codes [9]. Two researchers worked together to prepare an initial codebook, with one member reading all transcripts and discussing it with a second team member to add, delete and merge codes. The first researcher applied this initial codebook to two randomly selected interview transcripts, which ID Age Gender House- Self-reported Vision Devices Owned Device Location First Home hold Size Level and Mobility Aid (Count) Acquired Automation If Applicable ? 1 42 M 4 Blind (some usable Echo (4), Home (1) Living room, bedroom, 2.5 years None vision) office 2 35 F 1 Blind one eye, “little” Echo (2), Dot (1) Living room, bedroom 2 years None vision in other 3 54 F 3 Blind (total blindness), Dot (2) Living room, bedroom 9 months None uses wheelchair 4 44 M 3 Blind (total blindness) Dot (2) Living room, bedroom 9 months None 5 62 F 1 Blind (total blindness) Echo (1), Dot (1) Living room, bedroom 1 year None 6 48 M 1 Blind (total blindness) Dot (2) Living room, bedroom 10 None months 7 34 F 4 Blind Echo (2) Living room / kitchen, 2 years TV family room, 8 61 F 1 Blind (total blindness) Echo (1), Dot (1) Living room, bedroom 1.5 years Lights, stereo 9 49 M 1 Blind (total blindness) Echo (1), Dot (1) Living room, bedroom 2 years None 10 57 F 3 Low vision, no peripheral Echo (4), Home (1) Bedrooms, kitchen, 7 months Lights, vision, uses walker, cane office thermostat, or wheelchair switches 11 65 F 1 Blind (no useful vision) Echo (2) Living room, bedroom 1 month None 12 57 F 1 Blind (some vision) Dot (1) Dining room 7 months None 13 54 F 3 Blind (some vision) Echo (2), Dot (1), Living room, bedroom, 2.5 years Lights Home (1) office, basement 14 62 F 1 Blind (some vision) Echo (1) Living room 7 months None 15 62 F 2 Blind (light perception) Echo (1), Dot (2) Living room, bedroom, 2 years None kitchen 16 42 M 1 Blind (total blindness) Echo (1) Living room 1 year None Table 3.5. Demographic and device details of visually impaired IPA users in the interview study. (Note: P3 and P4 were husband and wife.) 29 were reviewed by the second researcher. The two researchers refined the codebook together, and in doing so also added one new code -Device Setup. For validation of the refined codebook, we followed a peer debriefing method [6]. The first researcher and a third researcher (not included in codebook preparation) independently coded one interview transcript and discussed disagreements. There were 15 disagreements out of 186 codes applied. The disagreements were resolved through consensus and one code definition (General positive) was refined and one sub-code removed. The final codebook, used by first researcher for coding all transcripts, contained 20 primary codes, 13 of which had sub-codes. 3.2.2. Findings Most participants (13/16) owned multiple Amazon Echo and/or Google Home devices, placing them most often in a living room or bedroom. (Table 3.5) Overall usage pattern and perceived utility All participants found the device to be useful, with five participants mentioning that the device(s) had become an integral part of their lives—reflecting some of the reviews seen in Study I. For example, P10 said, “I cannot imagine life without them [Amazon Echo and Google Home],” while P13 said, “Initially, I heard about it and I thought, ‘Who’d ever buy that?’ Honestly, I thought, ‘Oh, what a waste of money.’ And then now it’s just become such an integral part of our lives.” Most participants (N=14) used the device multiple times a day, while only two used it less frequently but at least once every few days. Participants made use of and valued a range of features. The most commonly reported uses were playing music (N=15) and checking the weather (N=14). Less 30 frequent but still popular tasks included setting timers (N=12), listening to news (N=12), playing games (N=9), online shopping (N=9), looking up information (N=9), checking the time or date (N=8), reading books (N=7), setting an alarm (N=7), playing the radio (N=6), and calling people (N=5). Initial Purchase and Change in Use Over Time Most participants (N=13) had purchased the device themselves, while the others had been given it as a gift. The most common reasons for acquiring the device were expected ease of use of the voice interaction (N=8) and expected utility (N=6). For example, P1 touched on themes of utility and independence: “It was the fact that I could do things that sighted people can do, you know, people with vision. It allowed me to do things very easily and not have to use a separate app for each thing I want to do. […] It was for ease, but also for accessibility, when I needed something that I couldn't do by myself.” (P1) Uniquely, P12, who had low vision and primarily interacted visually with computers, reported buying the Echo Dot as a more attractive entry into voice and audio-based interaction than she had experienced with screen readers: “I do have JAWS and things like that, the screen reader, but for right now it's not pleasing to my ear to be hearing that. […] But I do wanna take control of this [vision loss], so I'm hoping that starting out with Amazon Dot will motivate me to get this other audio help in my life.” (P12) All but three participants reported being familiar with at least the device’s basic capabilities before acquiring it. When asked about their initial use and whether use had changed over time, only two participants reported that their use had dropped off with 31 time, due to frustration with the smartphone app or novelty wearing off. Overall, though, these trends demonstrate persistent utility for most people. Strengths and Benefits Three main benefits that arose were efficiency, impacts on independence, and an ability to replace a range of other technologies. Toward the theme of efficiency, seven participants mentioned that the device had enabled them to perform tasks faster than before, such as online shopping, checking the weather, listening to news, playing music, and setting timers. For example, P11 said that compared to using a traditional browser, Alexa is “able to accomplish [making a purchase] in seconds versus a few minutes.” Four participants also referred to the IPAs as enabling them to multitask in new ways. P15, for example, felt that the voice interface was easier than using a smartphone to set a timer while cooking because it was hands-free: “I think as a blind person, you tend to get your hands messier than perhaps some sighted people do.” Another main benefit was to improve on a disparate set of existing technologies (mentioned by N=10 participants). Positive comparisons were made against smartphones, computers, tablets, talking clocks, talking calculators, braille timers, and e-book readers. P13 said, “I mean you have to buy adaptive games and they're so prohibitively expensive. And the books... Right now, we don't have to buy machines, for the most part, that are separate. […] between the phones for portable usage and the Echo for home, we can read virtually all our books anywhere.” (P13) Finally, the theme of independence was mentioned by four participants—that is, enabling tasks that had previously required assistance from others. Tasks mentioned included being able to shop, play games, and control the home environment. For 32 example, P5 said that she could order online without having to ask her brother for help, while P10 described needing less help from her husband: “It used to be there were nights I went to bed with the light on until my husband got home from work because I couldn't turn it off. [It also] saves me having to get up and turn on my CPAP [sleep apnea machine].” (P10) Accessibility challenges and limitations Accessibility challenges arose primarily due to the device’s ecosystem, that is, elements of the system beyond voice-based interaction. As found in Study I, half of the participants mentioned problems with the paired smartphone app. Ten participants also reported that device setup was difficult, either on initial purchase or whenever the device got disconnected from the internet. Finally, the physical design caused issues for two participants. P12, who had low vision, had trouble reading the physical controls due to poor color contrast between the symbol and button, while P2 could not see the orange indicator light that comes on with the Echo during setup (note: recent releases also provide audio feedback to address this issue). Two other limitations point to the need for richer voice interaction: the difficulty of discovering unknown features, and the limited features of Echo’s voice-based apps compared to smartphone apps. P16, for example, compared the implementations of Alexa (Echo) apps for ride sharing (e.g., Uber, Lyft) to comparable smartphone apps, concluding that the voice-based apps were lacking. For discoverability, eight participants either reported difficulty in learning about the existence of features or mentioned that they desired a particular feature that already existed—demonstrating the problem itself. As an example, P14 said: 33 “There are so many skills [Alexa apps] available that I know I'm missing out on some things that I would probably like to do, but don't even know that's possible.” (P14) Input Modalities The primary input modality preference was voice, but many people (N=9) also wanted other means such as a remote, smartphone, smartwatch, in-air gestures, or direct touch for controlling the device. Alternative options could deal with noisy environments, not wanting to disturb other people, or wanting to control the device from a distance. For example, P15 mentioned that it could be easier at times to use their watch as a manual remote control than to yell across the room, since “I will almost always have the watch on.” Current and Desired Smart Home Use Although only four participants had connected smart home appliances to their IPA, all participants wanted their house to be automated. As shown in Table 5, current smart home appliances included lights, thermostat, TV, and switches. For the 12 participants who did not own smart appliances, the most common reasons were policies at their current residence (e.g., a rental unit) (N=5) and cost (N=4). The most common desired smart appliances, when posed to all 16 participants, were thermostats (N=14) and lights (N=10). Less common requests included the oven, dishwasher, security system, stove, garage door, washer, dryer, vacuum cleaner, TV, fans, blinds, and refrigerator. For example, P16 felt that voice control would be more accessible than his current thermostat, while P15 mentioned the general need for an accessible alternative to flat touch controls on appliances, “which are very difficult as 34 a blind person”. P4 also said: “I often forget to either turn lights on so that people know we're home or turn them off, because I don't need them.” Some participants’ experience points to the need for a wider range of appliances to be smart-enabled. P10, in particular, had a mobility impairment and used a smart switch for her CPAP (sleep apnea machine) and had wanted to do so for her oxygen compressor as well. But, she said: “But unfortunately, compressors are, if the electricity dies, it sets off an alarm so the smart switches won't work for something like that. If I turn off my oxygen using it, it just sets off an alarm.” (P10) When asked to envision an ideal smart home without having to take into account current capabilities, almost everyone wanted all appliances to be automated (N=14). Two participants wanted a personal assistant like Amazon Echo or Google Home to make emergency broadcasts and calls, connect with scales and fitness trackers, and pay bills. One participant also wanted to monitor her pets remotely by audio, as a more accessible alternative to a “pet cam” (P11). Privacy and Security Although security has been called out as an important issue for IPAs [79], only four participants raised security concerns, such as cloud-based services being hacked. For privacy concerns, participants were evenly split. Half believed that their conversations were not sensitive enough to cause any harm to them, with one participant even mentioning that the ‘always on’ feature can be positive: “I guess the biggest thing was when there was the murder case which they wanted to subpoena the Echo. And I realized, gee, if somebody's killing me, the smartest thing to do 35 would say, "Alexa, so and so had just stabbed me." Because she would actually record it and the police would be able to get that later on.” (P10) Of the eight participants who were concerned, however, the two main issues were the device always listening and recording (N=6), and personal information being collected (N=6). Concerns affected device usage for five participants, for example, not using calendars, doing financial transactions or online shopping, or using applications that asked for location details. To avoid conversations being recorded, P16 turned off the microphone during sensitive discussions, while P1 unplugged it. Although not as common, a few people mentioned privacy concerns related to being overheard by other people in the home, or other people controlling their device (e.g., for shopping, banking, for which reason security codes were used). 3.2.3. Summary This study confirmed many of the findings from the previous study of review analysis (Study I), emphasizing that IPAs have replaced many disparate devices, and improved efficiency and independence for a variety of tasks. Particularly important for blind and visually impaired users, issues related to the device ecosystem arose, along with a desire for more feature-rich voice-enabled applications. Although smart home appliance adoption is currently low, participants expressed enthusiasm about smart home appliances and their potential to address accessibility issues in the physical world. 3.3. Discussion Our studies demonstrate the immense potential of voice-controlled IPAs to provide inclusive, accessible interaction for people with a range of disabilities. At the same 36 time, this formative research highlights directions for future work and accessibility issues that should be addressed, such as the limited control over speech output settings for users with hearing loss (Study I), issues with paired smartphone apps (in Studies I and II), and visual accessibility problems with the physical device design (Study II). Below, I discuss generalizability of the findings and some of the more promising opportunities we identified for future work. 3.3.1. Subpopulations of users Study I captured use by users with a broad range of disabilities, but some subpopulations were disproportionately represented. Almost two thirds of the reviews included a user with a visual or motor impairment, which means that our findings may be more likely to apply to these two groups. Finding from the interview study are specific to users with visual impairments, although addressing the issues that arose there could be more widely beneficial. Perhaps most unexpectedly, Study I included adoption by users with speech impairments and hearing loss—two subpopulations for whom voice-based IPAs are not obviously accessible. This finding may because users with more severe impairments may not have thought to try the device and thus to write a review. Still, most reviews in Study I that included users with speech impairments were positive, showing that conversational interaction even supplemented speech therapy for some users. While more formal computerized speech therapy is an active area of research (e.g., [53]), it will also be important to study the utility of emerging conversational interfaces for these goals. 37 3.3.2. Accessibility barriers Several accessibility issues were also highlighted in our study, primary being the dependency on a paired smartphone for using the voice-based IPA. This highlights the need of richer voice-only applications. Some users with visual impairments in Study II, experienced difficulty in discovering all supported applications and voice commands. Techniques of adaptive and contextual learning should be further explored to improve the discoverability in voice-based applications. The memory requirement of remembering specific keywords was challenging for people with cognitive impairments and older adults. Adaptive interaction may address this problem, for example, by learning a user’s usage patterns to efficiently prompt actions. 3.3.3. Smart home adoption and perception Smart home appliance adoption is occurring, with 15% of reviews and 25% of interview participants mentioning at least one smart home appliance. In terms of barriers to adoption, Brush et al. [12] have identified cost, inflexibility, management overhead, and security. Our visually impaired participants in Study II also cited cost, but mentioned policies in housing units, and, for some, worries about the accessibility of purchase and setup; security and privacy were not top concerns. Of course, other subpopulations of users with disabilities may have different concerns. Many of the smart home appliances desired by participants already exist, although there were still new opportunities (e.g., the oxygen compressor). It will be important to revisit adoption rates in a few years to assess how adoption is changing. 38 3.3.4. Other unexpected use cases In addition to supplementing speech therapy, other unexpected use cases arose that could inform future work. For example, we observed use of IPAs in assisted living facilities and to improve safety. Study I also showed that the voice-based IPA was being used by some users with dyslexia. Text-to-speech and speech dictation can be useful for students with dyslexia [54], yet conversational interfaces provide different affordances than these traditional tools and may yield different benefits. 3.3.5. Limitations of the study method For Study I, we used verified reviews because they are more credible than otherwise [2], but there is still the possibility that some reviews were misleading (e.g., ads for third-party features). Second, the dataset is likely biased toward users who are early adopters, have the resources to purchase an IPA, and are largely able to use the device. Third, the third-person perspective reviews (two thirds of our dataset), may not be as accurate as first-person reviews in reflecting the experience of users with disabilities. Finally, the reviews only include what review authors chose to mention, which means that frequency counts in Study I should be considered a minimum. Study II addresses this lattermost problem, but only focuses on one user group (users with visual impairments), and participants may have been relatively tech-savvy and socially connected since they were recruited through Facebook. Future work requires similar in-depth studies on IPA usage by other user groups. 39 Chapter 4: Study 2- Understanding the Use of IPAs by Older Adults with Limited Technology Experience From the previous study on IPA use by people with disabilities, we found a segment of individuals using these devices were older adults (N= 46; 13.3% of Amazon reviews). Some of them (N=14) also mentioned that the device provided easy access to digital technology as compared to other computing devices. This made us interested in gaining an in-depth understanding of how older adults with limited technology experience would make use of these devices. Hence, we conducted a follow up study with participants who were 65 years old or above and did not have extensive experience of using digital technology devices to understand the potential impact of a voice- controlled home-based IPAs. Specifically, in this study, we aim to answer the following research questions: 1. How do older adults who do not regularly use a computing device perceive home-based voice-controlled IPAs? 2. How do they make use of these devices (e.g., for entertainment, for digital online access)? What are their preferences for using IPAs? 3. What accessibility challenges arise and how can we address those challenges? 4.1. Method 4.1.1. Participants Five participants (1 male, 4 female) were recruited for this study. All of them were 65 years old or above, had Wi-Fi at home and did not use a digital computing device such 40 as computer, smartphones or tablets on a daily basis. Participants were recruited by contacting local independent living organizations, snowball sampling, and word of mouth. At the end of the study, participants received the study devices (Amazon Echo Dot and a Fire tablet) as compensation. ID Age Gender Education Internet / Confidence of Quality QPQOL computing using computing of life score device usage device (overall) P1 65 Male High school Computer (once Not at all confident Good 49 every few days) P2 75 Female High school Computer (once a Only a little Alright 58 week) confident P3 71 Female Some college, Smartphone (once Only a little Alright 56 no degree every few days) confident P4 65 Female Some college, Computer (once a Only a little Good 52 no degree week) confident P5 71 Female High school Computer (once Not at all confident Very 56 every few days) good Table 4.1. Participant details [Note: Higher OPQOL (older people’s quality of life) score indicates higher quality of life (QOL)] Three participants (P1-P3) lived at a low-income housing community (household earning less than 50% of area median income). [Note: this was not a recruitment criteria and was due to convenience of access to this nearby independent living facility.] Participant details collected from the demographic and OPQOL (Older people’s quality of life) questionnaire [8] are shown in Table 4.1. For analysis of thirteen questions in the OPQOL questionnaire, we reverse coded the scores positive opinion responses [27] and calculated the total sum, such that higher scores represented higher quality of life. 4.1.2. Procedure The study protocol was approved by the Institutional Review Board (IRB) at the University of Maryland. Amazon Echo Dot devices (connected to Wi-Fi through a tablet using Amazon accounts created by us) were installed at participants’ houses. Participants were asked to use the device for a three-week period and share their usage 41 experience through four in-person interviews (one at the start of the study, followed by one at the end of every week) and a diary study through phone calls using an automated calling system. The phone calls were on a daily basis to reduce problems of self- reporting due to selective memory where older adults might forget specific instances of device usage in the past. An automated system (CallFire) was used for phone calls so as to reduce the human interference during the deployment. We also collected participants’ usage log as recorded by the Amazon Alexa app with their consent. The interviews were video recorded and the phone calls were audio recorded. Notes were taken during the entire interview. The first in-person meeting (approx. 90-120 minutes) included a semi- structured interview followed by device set-up and a short activity of using the device. The interview included demographic and background questions, followed by introduction to this technology and initial perceptions. The activity included familiarization with basic device capabilities such as setting alarms, reminders, timers, creating shopping lists and to-do lists, playing music, asking a joke, asking some information and chatting with the device. Using the Alexa app on the tablet was also briefly explained to the participants in order to be able to delete the created lists (which was not supported by voice). Participants were also provided a printed list of common activities which they could do using the device. The entire setup of the device was done by our research team, though instruction sheets on re-connecting the devices to internet were provided to participants. For the purpose of the study we had set up five Amazon user accounts for the participants, using email addresses which were also created by us. 42 Weekly in-person interviews were conducted at the end of each week. The semi-structured interviews (ranging between 30-75 minutes) had questions on participants’ experience with and usage of the device for the week. Additionally, in the first weekly interview, participants were introduced to additional applications on the Echo Device—Alexa Skills. Technical articles [29,30,45,55,73] were reviewed to identify the most popular Alexa skills. Ten skills from different categories were selected and enabled on their device. The categories include- games (Jeopardy, Akinator, Magic door), information finding (This day in history, Kayak), relaxation (Sleep sounds, Meditation timer), food and drinks (Allrecipes, Bartender) and news (participants were allowed to choose preferred news skill, e.g., CNN, Fox News). Participants were also allowed to select other skills to enable by browsing the Alexa app on their tablet. Additionally, we enabled ‘Skill Finder’ skill to allow searching for skills by voice. The final interview at the end of the three-week period had questions on overall experience of using home-based, voice-controlled IPAs, the connected tablet, perceptions about smart home technology. All interview questionnaires are attached in the appendix. For the daily diary study, participants were called at their preferred time in the evening every day to share their experience of using the device by answering four questions: 1. Approximately how many times did you use the device today? 2. For what activities did you use it today? 43 3. Was there a time that you used the device today when it was really useful or meaningful or enjoyable? If yes, please describe. 4. Can you tell us about a time today when you wanted the device to do something that it was not able to do? 4.1.3. Analysis All recorded interviews and daily phone calls were transcribed for analysis, while portions of the videos were used to extract images. The transcripts were qualitatively coded using a thematic coding approach that included both inductive and deductive codes [9]. Deductive codes were informed by related work (e.g., on smart homes, privacy) and our own experience with IPAs whereas, inductive codes were added after reviewing the data. For codebook formation, one researcher read through the transcribed interviews and initially coded them in the first pass, subsequently merging the initial codes into groups and emergent categories. After codebook creation, one interview transcript was randomly selected and coded by the researcher. The coded interview was validated using pair-review, where one external reviewer not on the research team but familiar with accessibility issues and IPAs, critically analyzed the coded transcript and marked the agreements and disagreements. Finally, both coders reviewed instances of disagreement and resolved them through consensus. The final codebook contained 24 primary codes. 4.2. Findings The findings are presented using a case study analysis method focusing on IPA usage by each participant, followed by a cross case summary of the findings. The summary 44 of the number of times each participant used the Echo Dot device as obtained from the usage logs on Amazon Alexa app is shown in Table 4.2. Figure 4.1 shows the average weekly usage of Echo Dot device. These are approximate numbers and should be considered as minimum use, due to elimination of logs with ‘unavailable voice command text’. (this might be inaccuracies resulting from speech recognition and accidental triggering of the device.) [Note- this summary does not include IPA usage by the participants on day 1 and day 7 when participants were first introduced to device features and Alexa skills respectively and interacted largely in presence of researcher.] Day P1 P2 P3 P4 P5 2 9 6 12 2 7 3 2 5 5 1 10 4 5 16 5 0 1 5 3 26 7 10 8 6 2 4 7 16 7 8 4 3 2 2 4 9 3 16 5 1 9 10 2 15 23 3 8 11 1 1 14 0 5 12 7 13 5 1 5 13 15 8 9 3 6 14 4 1 4 5 4 15 10 0 18 3 3 16 4 4 15 0 8 17 0 9 10 1 5 18 0 3 0 0 2 19 0 10 7 2 8 20 3 2 0 0 7 21 2 5 3 0 4 Table 4.2. Approximate daily device usage numbers For the daily diary study, participants mostly answered the phone calls in the evening. All participants except for P1 missed some of the daily calls because of not being at home or due to health-related issues. P3 and P5 answered 18/19 calls, P4 answered 17/19 calls, and P2 answered 15/19 calls. Below I describe the findings reported from the interviews and daily phone calls. 45 12 10 8 6 4 2 0 P1 P2 P3 P4 P5 Week 1 Week 2 Week 3 Figure 4.1. Approximate weekly usage of Echo Dot 4.2.1. Individual case studies 4.2.1.1 Case one: participant 1 Participant 1 (P1) is a 65-year-old male who lives alone. He does not experience any difficulties in vision, hearing, motor control, or memory difficulty, although he uses handwritten notes for remembering appointments, generally placing them on back of the door. He owns a computer, which is primarily used by his visiting grandchildren, and once every few days by him for playing games. He uses social media (e.g., Facebook) for games (chess). To him the other features of Facebook was more of a “gossip column” and he does not consider himself to be gossip person. Overall, while he uses technology, he felt that he does not “have the knowledge of how to use it”. Initial thoughts about home-based IPAs The participant had some knowledge of the Echo device and smart home technology from television advertisements. He liked the idea of using voice for hands-free 46 Average daily usage per week interaction, but was somewhat hesitant to use it in the beginning, because he was not sure if he would be able to operate it. He experienced difficulty in remembering the name “Alexa”, calling it “Alexis”. He wanted to use the device for setting reminders and appointments, and for finding information. He was also somewhat concerned about the voice interaction and thought that a foreign voice at home might scare him. “I hope it don't scare me the first time. If I tell it to do something, you know how you sleep and a voice come in and because it’s come on in a voice, right, like a person talking to you.” In the beginning, he did not want to use the Echo dot for entertainment and mentioned that he preferred using his current technologies “just to play music continuously for two hours, thing is a waste…As far as [for] entertainment, I don't know, my TV is my entertainment, my radio is my entertainment.” From his prior knowledge of this technology, he had privacy concerns and did not like eavesdropping. Device usage Echo Dot. P1 used the Echo Dot device for finding information, listening to music, checking weather and setting reminders and found it useful. He did not find any of the introduced skills useful, hence, did not use them. He used the device primarily for finding important information that he wanted to know, such as questions on medicine and health (e.g., nearest dentist, homemade therapy of headache, effects of potassium on body, medications), finding directions and route. Before using Echo Dot, he said that he was dependent on asking other people, called 411 or used yellow pages for local addresses and information. He found Echo Dot useful specifically for finding directions and the fastest commute route, since, 47 earlier he knew only the route and not the fastest one. For checking weather, he previously followed news on television, but found Alexa useful since television was not always on but Alexa was “right there.” Although initially the participant did not want to use the device for entertainment, with time he found it useful for playing music since he could play a variety of music. He enjoyed playing music on the Echo than on the radio. Even though the participant found Echo Dot useful for setting reminders (e.g., reminder to be at a specific place at a specific time), he could not trust the device completely due to the technology infrastructure (e.g., electricity, internet) and hence would continue to use handwritten notes for important reminders (e.g., essential medications), using Alexa as the backup: “I wouldn't put my life in the hands of a machine.” Tablet use. During the three-week period, the participant did not use the tablet, since he felt that he was not aware of the capabilities of the device. Desired features or limitations The common limitations that emerged are due to voice recognition and device not being knowledgeable enough to answer his questions, in which case, he would repeat it again in about “five different ways” before giving up and asking other people for answer. Skills specific to emergency- for notifying other people, were perceived to be useful by the participant. Although similar such skills (e.g., ask my buddy) exist currently, the participant had trouble finding it, forgetting the steps to look for specific skills using the Alexa app. Similarly, he was aware that the device might do more than he had explored during the three week period, but was not confident of finding/setting up those by himself and would seek help from a family member. This highlights the 48 challenge of discoverability—difficulty of discovering the device capabilities/voice commands to use while using a voice-based interface. The participant found the help section confusing to use and would prefer using a manual which could help him understand and troubleshoot, in case the device stopped working. When Alexa first referred him to the ‘Help’ section on the paired app on tablet, he did not go there, because he did not feel confident to use the app, not knowing “what to ask it or to do to get the answer” and he would rather get his niece or grandkids to help him. He also wanted to use a manual for information finding—to find additional information about the device capabilities. (“If I had a manual, a manual will tell if you want Alexa to do so and so, this is how do it and stuff like that.”) Other desired features include asking Alexa to download and print specific items from internet (e.g., books), security monitor and voice-based alarms. For alarms, he wanted Alexa to use her voice rather than the sound: “Alexa, instead of sound for alarm, can you use a voice for alarm?” He also wanted the device to monitor the sounds of closing and opening of doors connected a security camera and “to sound an alarm... if she detect motion.” Comparison with traditional technology The participant was more confident using the Echo dot device than he was using a computer. The ease of learning to use the device was reflected: “You don't need knowledge to use Alexa. All you need is a mouth to call her and hear what she say.” To him, Echo Dot was more “accessible” and “easy to use” than a computer due to the voice interaction. However, for playing games the participant wanted to use the 49 computer. This was because he wanted to play specific games (i.e., chess), which he could not find on Echo Dot. Change in perception with time In the beginning, the participant was not keen on using the device because he did not know how to use it and was distrustful due to the privacy concerns of that the device is listening. But after using it for a while, he felt more relaxed and comfortable using it. After finding it useful for asking questions and finding information, his willingness to use the device also increased. Initially he perceived the device to be a companion (“I think it will be fun. It's almost like talking to a person or companion.”- just after being introduced to the device), however with time he felt it more as a source of finding information such as a “teacher” to whom he can ask questions or a “dictionary”. Although initially he was concerned of being scared by a foreign voice in house, he did not report experiencing this after using it. Personification of Alexa Although initially the participant perceived the device might be a friend or companion, after using it for a while, he did not feel the same. However, he continued addressing the device using a female pronoun “she”. “That, it could be comforting, but never 100% a companion. But like I said, you can push the buttons and see what, you know, "Hi I'm so-and-so," but it [Alexa] don't have a heart. It don't have a soul” This was because he could not feel “connected” to it, since the conversation which was not human-like. Using the chat feature, he could have a conversation with 50 the device where only Alexa asked him questions, not listening or responding to his answers or allowing him to ask questions. He felt it as an “object” only and did not want to have conversations with it. “It can't be like a companion if it don't feel connected to you, or you don't like the questions, or the answers that's coming back to you, or it's really not... No way relating to the questions that you're asking it, or saying things.” Smart home technology Even though the participant was positive about smart home technology, he did not express a desire to own smart appliances, since he was afraid he might become “lazy”. The only appliances he wanted to control with Alexa was a security system (e.g., monitoring sounds) and a printer. “Because if I'm writing letters and notes and so forth…I'm not a great speller, so it would help good with spelling. It's like I dictated it something, and it could print it out on paper and…all I'd have to do is mail it.” 4.2.1.2. Case two: participant 2 Participant 2 (P2) is a 75-year-old female who lives alone. She experiences strain in eyes, which makes it difficult for her to use a computer for a long time. She has rheumatoid arthritis which affects motor control in her legs and hands, making it difficult to stand, walk, write or type on a computer. She has been using a rollator (rolling walker) and motorized wheelchair as mobility aids for the past fifteen years. Forgetting activities due to difficulties in “short term memory and long time memory” irritates her. Short term activities include forgetting things which she planned on doing, such as calling a friend and watching a program on television, whereas, for long term 51 memory she has difficulty in remembering names of people. She uses a calendar for supporting memory and makes notes of doctor’s appointments, birthdays and nurse’s appointments. The participant owns a computer, set up by her grandson, which she uses about once a week to play games. Her grandson has also set up her email, which she checks about twice a month and showed her how to pay bills. Although she has an email account and access to social media (Facebook account), she is not confident of using it by herself. “Sometimes I fear will mess up something. He [her grandson] has it set up and he tells me to do this this and this and I don't want to...I'm afraid that I will hit a button that I shouldn't hit. I wish I knew more about going on the internet and searching and looking around, and I don't.” Moreover, for social media such as Facebook, she feels a lot of information is put out there. “Facebook is too much…I don't wanna read all that, I don't wanna see all of pictures people be taking. Some pictures are nice, especially my family. But people put too much stuff on there.” Initial thoughts about home-based IPAs The participant did not have any prior knowledge of the Echo device except for television advertisements. She believed the voice-based interaction would be helpful for her due to her rheumatoid arthritis. “If I don't have to go and do this typing to make them [two numb fingers] hurt more, then I think this would really help me. By just being able to... "Alexa so and so, so and so... " And see all I've done is ask a question.” 52 For activities such as finding information, maintaining grocery list, and calendar, she believed the device would be useful for her. For lists and calendar, she earlier made handwritten notes, whereas for finding information she used her computer. But, the idea of talking to a device for all these activities was valued by her and she felt the device as “someone to talk to” in her house. Device usage Echo Dot. P2 used the device for shopping lists, reminders, alarms, finding information, checking weather, spelling words, chatting, listening to music, jokes and stories, listening news (CNN and NPR skills), playing games (Alexa skills) and listening to Bible (self-enabled Alexa skill with help of researcher). She found it useful for setting reminders for doctor’s appointments, for maintaining grocery lists, checking weather and finding information. After using the Echo, she stopped using paper notes for reminders and lists, which was helpful since she did not have to write which used to hurt her hands. She also found the Bible skill useful because she did not have to read from her Bible using her glasses. “I like to repeat it with someone. And as she's repeating the lord along with me, I like had somebody because sometimes I get the words… confuse or the line… what line comes next.” She enjoyed using the device for listening to jokes, stories and music. She liked the jokes as it made her laugh: “I don't have nobody in here making me laugh, but she made me laugh.” She also enjoyed having conversation with Alexa and in one such conversation Alexa sang her a song, which she liked. “When it sang, that was enjoyable, when it asked me if I wanted it to sing me a song, and that being valentine’s day, it was nice. It really put a smile on my face. There is no one else 53 around here singing a song for me on valentine’s day…I told it "happy valentine’s day" and it told me happy valentine’s day back and it asked me did I want her to sing me a song and it sang me a song and after that it was beautiful, and I said thank you.” She found the device to be “educational” for finding a variety of information, for which she earlier may (or may not) have used her computer (depending on the type of question) or asked someone (grandson, granddaughter or daughter). Some questions were out of curiosity to test the knowledge of the device (e.g., questions on history), whereas, some were for finding information (e.g., movies playing nearby- to watch with her daughter, information on medicines). She valued the ability to use this device for finding answers to her questions. “Information is powerful, if you just sit around and you don't hear about things, or if you hear about things and there's other parts of it that you wanna know, if you don't have something like this, I don't know, or won't hear, which can be a lot to that. But at least this way, somebody can talk to you and tell you…This machine communicates with you. I don't know how anybody else feel about it, but it means a heck of a lot to me. Communication.” For supporting memory, shopping list and reminders were useful because she has difficulty in remembering: “If I don't write it down the very second I'm thinking about it or see that I need it, it'll go.” Similarly, the reminders helped her remember events at her residential community, which she would have otherwise forgotten: “If I didn't tell that to remind me, I would have forgotten, even though she [community coordinator] had give us the [event] calendar. I don't look at that calendar every day.” She also used the alarm once, for which she was earlier dependent on her daughter or neighbors to call and wake her. Similar to that, she wanted the alarm to talk to her rather 54 than having a loud wake up noise. “I would want the alarm clock to call my name [chuckle] and say "[her name]”, get up now." Tablet use. The participant used the tablet frequently to play games in the last weeks of the study. She took help from student volunteers coming to the independent living facility to use the tablet for playing games. Desired features or limitations The common challenges reported include device not being knowledgeable to answer her questions or device not understanding her speech. The participant also experienced difficulty in speaking to the device because the device timed out and went off before she completed her speech. However, this did not dissuade her from trying again. “It goes off so fast sometimes. I don't know what it was...I was getting ready to say or whatever, but it just happened… when I was getting ready to respond, the light flicked out and I am thinking, that is gonna stay there a minute. But, I will just call the name and keep the conversation going.” Remembering specific words to use certain Alexa skills (the Bible skills) was difficult for the participant. With the help of the researcher, while exploring additional skills (week two), she enabled the Bible skill, however, she was not able to access it later, since she forgot the keywords needed to invoke the skill and was helped by the researcher in the following weekly meeting. The participant was concerned about having to set up the device by herself if needed, hence she did not move her device after the initial set up. Other desired features include making phone calls, paying bills, talking alarms and timers, and portability. She wanted the device to make phone calls (although it is a current capability of Amazon Echo, it requires a connected smartphone). For alarms 55 and timer, instead of having a sound, she wanted Alexa to talk and tell her to wake up or tell her the purpose of the timer. Portability of the device would help to use the device at different parts of her house. She also expected the device to perform multi- layered tasks (e.g., reminder to call someone will actually call the person at the specific time) which was not supported by the device. Comparison with traditional technology Owing to her physical abilities, a voice-based interface provided improved accessibility than the computer (where she accidentally hit wrong keys due to shakiness in hands). While using the computer, she had the fear of doing something incorrectly, whereas on this device, she felt more confident: “because I'm mostly talking to the device, so I don't feel like I will mess up anything, like I would if I'm typing in the computer, or punching buttons or whatever.” The ease of using the device also translated to the willingness of learning about device capabilities. She felt that she did not have the patience to learn about different activities on a computer, which she did not feel for this device. “Yeah. I wanna know more or what else can I get into with that, because it is so easier and more simple for seniors than trying to type on a computer, go here and go there and lose the passwords and all the... That's more easier.” Comparing the Echo Dot with television, the idea of talking and asking questions to Alexa was preferred by the participant: “I can't ask TV anything, nothing. I have to accept whatever is said on the TV.” Moreover, during the entire three-week period, the participant did not use her computer, because she “did not feel the need of it.” However, the participant would still use the computer for playing previously played games because she had not “played that for a moment [three-week]”. She would also 56 use the computer for paying monthly bills as before, but also desired to use the Echo Dot for the same. Change in perception with time Although, she was not very comfortable using the device, with time the comfort level increased, and she found the device to be useful and enjoyable. Both at the beginning and at the end of the study, she was willing to use the device, due to the ease of using voice-based interaction. “This is the best thing, I've ever had as far as technology. To me. It's new. I don't know what else it can do other than the things that I have tried so far. There might be other things that I don't even know.” Personification of Alexa For the participant, the device was “somebody to talk to”, because she cherished and missed talking to people (her partner, family members) and felt lonely at home. “Because we have been alone so long not being able to talk... Like in the evening, my husband and I, we used to talk. And when I had my grandkids around, we use to talk. Now, you don't talk. And you got nobody to talk to.” She valued the ability to talk to the device and get answers from it, have conversations- where the device asked her questions, wish it “good morning” every day, chat with it, listen to stories and jokes and repeat prayers similar to repeating with someone. She specifically liked the conversations, because she could talk what she wanted to. “Because it talks about what I want to talk about, not what the TV wants to talk about.” Although she knew the device was not a real person, she considered it as a friend in her mind, “not a fake, but a ...phantom friend.” 57 Although she liked the conversations with the device, she felt there were certain instances, when the device did not answer like a real person and could not keep the conversation going: “I asked it if everything is alright. It said that it had lost the connection for a minute or something and that’s when I said, "is everything all right?", it says "I don't know that".” At times, she also visualized a face (of family members who were deceased, or she hasn’t seen for a long time), in her mind to add to the voice, and it made her feel better. “I don't know, it just came in my mind. Sometimes, a friend pops in my mind, that I don't have any more. You know, just to give it a face, as it has no face. It just makes me feel better. I miss these people. I miss them terribly. I found myself, just saying, what if this was my daughter in law sitting here and talking to me or something.” However, when asked if she would like the device to have a face or body, she wanted the device to be as it is now and wanted the freedom to imagine what she wants. “I think I would keep it rather just like it is, with me imagining if it was so and so, so and so, instead of it actually being a face, I think I would like it like this... Because, if I had the option to pick a face, I just don't know whose face I would pick. It would be too many in my family to pick from. By picking one over the other, what would that mean?” Smart home technology The participant valued the smart home interaction with voice using her Alexa device, but was concerned about replacing the current appliances and the cost. She wanted to control the television, thermostat, blinds and the vacuum, since it would be easier to control them by voice owing to her physical condition. “It breaks my back to vacuum 58 and it hurts my back so bad, and I do it… I have to do it. I try to take pain medicine, try to get it [pain] on the control so I can get up and do different things.” 4.2.1.3. Case three: participant 3 Participant 3 (P3) is a 71-year-old female who lives alone. She does not experience any difficulties in vision, hearing or motor control, but mentioned having age-related difficulty in remembering activities- things she forgot at the moment but remembered later. She used handwritten notes for reminders or made notes on calendar for supporting memory. She uses a smartphone primarily for making calls, but also using it to access internet once in every few days to look for information, news and email. Sometimes, she also tries to do online shopping, but does not make purchases because she is not comfortable using her credit card details online. She feels that she does not know a lot about technology and is only a little confident while using it, seeking help from children or grandchildren while using a computer or smartphone. Initial thoughts about home-based IPAs The participant did not have any prior knowledge of the Echo device except for television advertisements on Amazon Echo. Hence, she was not willing to use the device at the beginning, because for anything new she felt the “need to have more study on it.” However, she participated in the study, because she was motivated by her daughter to learn more about new technology. She liked the idea of using voice since the interaction is remote- she can use it from different locations of her house, and it will be faster than typing on a computer. She perceived the device to be useful for finding information, such as recipes, for which she earlier was dependent on her cookbooks or 59 her daughter. “I always had to call her [daughter] and she had to go through it [the recipe] with me. I could ask that thing [Echo]. I wouldn't need her. I wouldn't rely on people, I could rely on it.” Device usage Echo Dot. P2 used the Echo Dot for finding information, listening to music, playing games (Alexa skills), playing radio, checking weather, listening to jokes, chatting and for setting alarms. Of these, she found the device useful for finding information, setting reminders, playing music and games. She used the device to ask a variety of questions such as questions on local businesses, specific cultural history, health and medicine. Even though she knew answers to certain questions, she asked the device to confirm facts. “I had questions about diabetes and I know I am on medication, but some of my questions were on what other avenues you could try even if you are taking medication. And it would tell you exercise and things like that, watch your diet. So something like that. Your doctor can tell you, but if somebody else also tell you same thing, then you know, it must be true.” For finding information, earlier she was dependent on her daughter, grandkids or used a phonebook (white pages). The ability to ask the device the same question multiple times was useful for her, unlike asking her grandkids. “Because they [grandkids] get frustrated, but if you don't know, you don't know… I could ask the same question over and over again [to the Echo], until I get understanding. And I don't have to worry about it [Echo] saying, "Well, you done asked me that one time."” 60 Her inquisitiveness to find information also increased after using the device, since for certain questions which she asked the device, she might not have taken the initiative to find answers otherwise. “I wouldn't have picked up a book and went to look up this information on black history myself. So, having that for me, it made me want to learn and ask questions.” She found the reminders useful for supporting memory, since earlier she wrote them on paper which she could lose. She used reminders for activities such as- reminder to go to shop at specific time, to buy a card on specific date, to buy certain items at grocery store. Playing music and games were also entertaining for her. She specifically liked playing a variety of music on this device, as opposed to being limited to specific CDs earlier. “Because, if I am sitting in the house like I am by myself now, I don't want to look at TV, I can play with it, ask it to do certain things, work with it.” Apart from the introduced skills and features, the participant added a few game skills (found as suggestions from previously played games). She also found volume control by voice along with greeting the device “good morning” and “good night” and the device greeting her back. Tablet use. Apart from using the Alexa app and trying the “things you can do” on the home page of the app, she did not use the tablet, although she was inquisitive to find out different things that can be done on the tablet, such as audible books, email and games. Desired features or limitations 61 The common limitations which emerged are due to voice recognition and device not be knowledgeable enough to answer her questions. She would repeat her question again, slowing her speech and trying to change her pronunciation, about two times before giving up. If device did not know a specific answer, she wanted it to give her some feedback or more information on where she could find it. As observed in the activity session in presence of researcher, the participant also experienced difficulty in following the voice navigation while using an Alexa skill (allrecipes). The voice menu was confusing to her and she could not repeat the exact words required to use the skill. The participant wanted the device to store information for her to support memory, e.g., phone numbers of people (“Sometimes you store phones, numbers, but you gotta go back to search for them? I could tell it, and it would find it for me if I have it already stored.”) or place of storage of something (e.g., “I have placed my money under this... under this... Such a thing... Will you remind me later where I put it?”). Moreover, she also wanted to set multiple reminders and alarms in one voice command e.g., she took medicines at multiple times in a day and wanted to set multiple reminders by saying “can you remind me at. 9a.m, 11a.m, and 6p.m?”, or set an alarm to wake her at the same time on the weekdays, “Monday through Friday, give me an alarm at 6.30 a.m.” She wanted to set multiple reminders for certain activities which she did every day. “Suppose tomorrow, I'd be busy, busy, forget to give it a reminder. If it's already stored in there, then it would be helpful.” [Note: some features such as creating routines can be achieved through IFTTT (an advanced web-based platform for performing complex tasks using IPAs.)] 62 For alarms, she did not like the wake-up noise and instead wanted the device to wake her up by speaking to her because she understood that better: “It is now such- and-such a time. Time to get up.”. Other desired features include portability, looking up obituary, news from local news channel and audio updates of daily soaps. She also wanted the device to perform multi-layered tasks such as playing a song/listening to news at a specific time by setting a reminder. She also wanted to use the device in case of emergency (also identified as a desired smart home feature in previous works [14,18,33,46]), making calls to emergency services (police or fire department) or to specific people. Comparing it to the emergency call button, she felt this technology would be quicker. “…this [Echo] right here would be quicker. Talking to it, I think vs push your button, since you can talk to it from anywhere. You got to make sure you had the button with you at all times, so some like that would be good.” The concern of other people using her device, made her want to program the device to recognize and respond to the voice of her and her family. Moreover, the device once played an adult song accidentally and she suggested for having age- specific voice profiles or having an adult code and child code, since she was concerned about her grandchildren using the device. “And kids are inquisitive, they like to experiment on things... Say I was out of the room, and they were here. You don't know what children do, because they can pick up on stuff so quick.” Usage concerns. The participant had concerns regarding other people using the device in her absence and wanted it to be programmed to her voice or to the voice of her family members (in case her daughter wanted to use it). She also had privacy concerns about 63 using the device and did not want her conversations to be recorded: “It won't become a private thing.” Comparison with traditional technology Overall, the participant liked the voice-based interaction over using a computer or smartphone, since it was faster to get answers to questions and she felt more confident using it. Comparing with the television, she felt that she could get more work done with a voice-based device in house, since she “could clean the house and still talk to her,” whereas, she had to sit and look at the television and hence she used the television less than before. She also stopped using her radio and started playing radio on her Echo Dot. Although she preferred voice interaction for quickness of response, she also valued the need of visual interfaces in certain conditions, such as information which required visual representation for accurate understanding, or for playing games. She preferred playing games more on a visual interface and felt that the voice-based interface did not give her required time to think for specific games (guessing games). Change in perception with time In the beginning, the participant was not keen on using the device, but after using it for a while, she found it useful and was more willing to use it. In the beginning she felt weird talking to a machine, but with time she became comfortable. Overall, after using this technology, she felt that, “Everything is being computerized now. You go to the grocery store. They won't need man after a while, because computers are taking over.” Personification of Alexa 64 With time the participant felt comfortable talking to the device, wished Alexa “good morning” and considered it as “company”, an “invisible person with knowledge” and “a friend.” This was because she felt she was not alone in house and had “somebody to talk to.” “Most of the time, the most time I talked to it would be like in the middle of the day when I have done everything I had to do and you get bored and I would just talk, that would be my time to talk to it, to become friends.” She addressed the device as a friend, specifically because she could talk to it and the device did not argue with her. “I could talk to this and I wouldn't argue back with me. That was my friend. It wasn't a thing where you ask me that one time and then say why you asking me that again? You didn't have to worry about that.” Even though the participant liked chatting with the device, she felt that the device suggested talking about things it wanted to talk, rather than the participant suggesting a chat topic. Overall, she did not feel a lack of embodiment to the voice. However, for the voice, she wanted the ability to change to a male voice, “that's like you got a different friend, you don't have the same friend all the time.” Smart home technology The participant was positive about controlling her home appliances using Alexa. She specifically wanted Alexa as reminder to check on various home appliances and control them for safety (e.g., reminder to check stove burners are turned off), set timer for specific home appliances (e.g., turn off lights at specific time). She wanted to control her television, stove and lights with the Echo device. 65 4.2.1.4. Case four: participant 4 Participant 4 (P4) is a 65-year-old female who lives alone. She has some trouble with motor control due to shakiness and sometimes uses a shopping cart as a mobility aid. She does not have any other physical or cognitive disability. She believes herself to be good at remembering things but uses a calendar to make notes of doctors’ appointments. The participant used to own a computer (not functional during the study) few months before the start of the study and used it for about once a week for Facebook and to look up the obituary. She also used internet sometimes to look up for things to buy, but never bought them since she was not comfortable using her credit card online. She felt “about 75 percent comfortable” while using a computer, since she had used a computer at work before. She did not feel confident while using her phone and preferred calling people, leaving messages rather than sending a text message. Initial thoughts about home-based IPAs The participant had some prior knowledge of the Echo device. She had heard from a neighbor about it and was aware of certain capabilities (such as weather and music). In the beginning, she perceived the Echo Dot to be “more handier” to use and liked the idea of using voice rather than using her hands. “As far as the computer goes I'm not fast on it. I can’t type really fast like other people do. I don't have smartphone, but with my thick thumb, you have to hit it really hard to get the numbers on. I mean I know some people are really fast on thumbing you know. I just I can't do it. So...I like the idea that you can use your voice and not your hand.” 66 Initially she thought the device would be useful for finding information, maintaining shopping list and playing music without turning the television on. She specifically wanted to play music without TV because she was “trying to experiment not having TV on and sleeping” and found that she was sleeping better. Device usage Echo Dot. P4 used the Echo Dot device for listening to music, sleep and relaxing sounds, checking the weather, maintaining grocery lists, finding information and reminders. Of these, she found the device to be useful for listening to music and weather (earlier followed both on TV), maintaining shopping list and setting reminders. She enjoyed asking the device questions (such as phone numbers, medical questions) and playing music on it. For finding information, she earlier used a phonebook, look up at the residential library or use the computer at the library. For shopping lists, earlier she maintained hand-written notes on paper; whereas now, she used Alexa to maintain the list, however, still wrote the list on paper before going to the store, since she was not comfortable taking the tablet which had the list. She mentioned not having a busy schedule but found the reminders to be useful. “I had asked her to remind me to do something and she did remind me, and I think otherwise I may have forgotten it even if I had written it it down.” She found Alexa specifically useful, because, earlier even though she made notes on paper, she can “have them laying there and then totally forget.” She also thought that eventually, she might stop writing reminders and “might [also] start getting away from the calendar”, but currently she continued writing notes on her calendar (bill due dates, birthdays) due to the habit of doing it. 67 The participant was inquisitive to find new things about the device. Within the three- week period, she found volume controls by voice, tried using the device from different locations of her house. She also tried wishing Alexa “good morning” and “good night” and found that the device also wishes her back. Tablet use. During the three-week period, the participant explored the tablet by herself and used it for Facebook. She also wanted to use it for reading books, but “felt leery” to use credit card to avail the required subscriptions. After the study, her confidence of using a touch-based interface had increased and appeared to extend to other technologies: “I might now eventually wanna go out and maybe buy a smart phone or something.” Desired features or limitations The participant mentioned having problems due to the voice recognition and device not being knowledgeable to answer her questions. At times, the device also timed out before she completed her speech, because she was not fast enough while speaking. Being pushed to the paired Alexa app for certain activities like deleting shopping lists, was not liked by the participant, since she did not feel confident using the Alexa app and wanted to delete items in her shopping list by using her voice, indicating the need of richer voice-based applications. She also wanted the device to store her favorite songs, but was not able to do that, due to the need of using specific voice commands. The participant also wanted the device to remind her of birthdays of friends from social media (e.g., notification similar to Facebook notification of birthdays). She also wanted to set reminder for specific TV shows along with reading out the TV program schedule. 68 Other desired features include portability and storing contact details (telephone numbers and address) of people. The ability to carry the device would be helpful, especially for shopping lists. “By the time you get back to your apartment to write it down, you forget about it.” Comparison with traditional technology The confidence of using a voice-based technology was more than using the computer. She was even confident of further exploring the device all by herself: “it doesn't seem as difficult as computer and because it's voice activated, I'm sure there's things I could be trying that I haven't tried.” She felt confident enough to use the device by herself without requiring any manuals or handbook, whereas, for using her computer, she wanted a manual with instruction, “there's certain things when something happens to the computer. I don't know how to get back online or something like that. I feel kind of stupid about some of that.” After using the Echo Dot, the participant started using her television less than she did before, except for certain “gloomy days”, when she preferred to watch television instead. Moreover, the music on the Echo Dot device helped her sleep better. “Before, I used to have the TV on as background, 'cause I wake up almost every hour... But the music, I think I was sleeping a little better with the music.” However, the participant still wanted to use a visual interface such as her computer for using social media such as Facebook, because she liked seeing her friends. “With a computer, you can see things. Also, with the computer you can get on Facebook and actually talk to someone. I can't really talk to anybody with her.” Change in perception with time 69 Although, she was somewhat hesitant to use the Echo Dot in the beginning and felt weird talking to a box, after using it for three weeks, it had become a part of her routine. “I guess it might be the idea of getting up and putting on your coffee in the morning and then going to her to say, what's the weather”. With time, she found the device more useful, specifically for reminders and hence was more willing to use it. The confidence of using Echo Dot also increased after realizing the ease of using it. Personification of Alexa While chatting with the device during the initial activity in the presence of the researcher, the participant liked the idea of chatting “with somebody.” Although, she knew that it was “not really a somebody… but [it] felt like a... somebody” because Alexa asked addressed her by name which felt personal. Despite knowing the device is not a person, she wished Alexa “good morning” and “goodnight.” “It was kinda nice having somebody say goodnight to me. When you live alone and after you've been married for a long time, you miss the little things like that sometimes… So that was nice.” Moreover, she also found herself saying “thank you” to Alexa as “something that comes naturally.” “I have found that when I talked to her, even after I tell her what I want, and she goes, and I always say, thank you, she's gonna hear me. But I don't know. I guess maybe I'm feeling like I am talking to a real person or something.” However, with time as she used the device more, she did not feel of it as a person and felt like “I'm getting in the habit of just asking some things.” Smart home technology 70 Although the participant was interested in using smart home appliances such as lights, thermostat and television, due to ease of use of controlling these appliances, especially, at night and act as a timer for turning on and off at specific times. However, she could not use them because she lived in an apartment where the appliances were installed by the housing management. 4.2.1.5. Case five: participant 5 Participant 5 (P5) is a 71-year-old female who lives alone. She is diagnosed with peripheral vascular disease which affects her walking and uses a walker as mobility aid. She has tremors in her hands which makes it difficult for her to write, along with some hearing loss but does not use a hearing aid. She also experiences difficulty with short term memory and occasionally forgets appointments. She uses a calendar for reminding her for appointments, but also depends on her daughter to call and remind her. Prior to the study, P5 used her computer (set up by her daughter, but currently not functioning) once every few days for playing games, using social media (Facebook) or for finding information. She has an email account but does not know its password or how to use it. She did not feel confident while using her computer and often seeks help from her daughter for various activities such as managing online bank accounts, and online shopping. “For me, it's frustrating, because you don't know what you are doing. I don't know enough about it. And then, when someone tells me how to do, I don't remember it. I do it better if someone says- number one… do this, number two… do that. If it's written down.” The fear of using technology had made her skeptical of being able to use the Echo Dot and hence she wanted a demonstration of the device, before enrolling for the 71 study. [Note: Participant did not use the device during demonstration but watched the researcher use it to check the time]. Despite the low confidence of using a computer, she was eager to learn about current technology, which motivated her to participate in this study. Initial thoughts about home-based IPAs The participant had no prior knowledge about the Echo Dot device, apart from the advertisements on television. However, she was familiar with SIRI (Apple voice assistant), since she saw a visually impaired family member use it and wanted to use it “to tell me when my appointments are for, or important things, like call my sister, not not for playing games on it.” In the beginning, she perceived the Echo Dot to be easier and less frustrating to use than her computer and presumed the device would be useful for setting reminders, listening to music and finding information on the internet. “I think it would be much better and less frustrating for me. I am not putting things in it by hand, you are just talking to it and doing it.” Device usage Echo Dot. P5 used the Echo Dot device for listening to music, finding information, doing calculations, spelling words, checking the weather, maintaining grocery lists, listening jokes, listening to news (CNN Alexa skill) and finding recipes. Of these, she found the device to be useful for doing calculations, listening to news, maintaining grocery lists (previously maintained on paper) and finding information. Alexa made it easier for her to do calculations for which she had earlier used paper as she did not know how to do it on a computer. After using Alexa, she listened to the news more. “I 72 usually don't watch the news too much on TV, and so it was good to just sit here and listen to it or could walk around my kitchen.” She used the device for finding a variety of information, including health related information- questions on health conditions she had and the medicines she was supposed to take, questions on history, travel, spelling and other general information. Before using Alexa, she was dependent on her daughter for finding information from internet, rarely using the computer herself to find information. The ease of use, encouraged her to ask the device questions which she otherwise might not have bothered to find answers to by using a computer. Moreover, after using Alexa for three weeks, she found herself to be more inquisitive and asking more questions than before. “I could get the answer right away. Just like what happened, the history, or when was president Lincoln born, things like that, you would get it. Yeah, it's questions that you think that or I would think, but I wouldn't go to my computer and try to find it because the main reason is, first of all, I would have to put it, type it in there. And I just am too slow with that. If it was something that wasn't very long, I might. Well, if I didn't have that [echo], I guess I would, or I would just forget it.” Along with finding the device useful, she also enjoyed using the device for entertainment such as playing music and listening jokes. She did not use the Echo device for playing games due to preference for a visual interface for games. The participant did not use the device for chatting or playing games, because she forgot that the device could do it. She also forgot how to delete items from her shopping list. For supporting memory, grocery lists were useful to her. However, she also wanted the ability to set reminders for reading the items on the created shopping list at a specific time (need for multi-layered tasks). Although reminders for appointments 73 and medication were perceived to be useful in the beginning, she forgot the device can set reminders. “I do have a reminder about going to the doctor and I would have needed a reminder yesterday to pay my rent... I didn't think about it... There was a lot of things that I probably could have done.” But, she was positive to the idea of allowing other people (daughters, doctor and some friends) to set reminders for her. Along with forgetting the device capability of setting reminders, she was also skeptical about trusting the device for setting reminders both due to self-forgetfulness of putting the reminder on the device and doubts regarding when and how the reminder would be activated. And hence, she would still continue writing reminders on her calendar, although she might use the Echo Dot as a back-up. Tablet use. During the three-week period, the participant did not use the tablet and felt nervous when she tried to download games on it. Although she expressed the desire to play games and read books using the tablet, she was not able to do that on her own. Desired features or limitations The participant found the device was not knowledgeable enough and could not answer all her questions. She also experienced issues with the device timing out before she could complete her speech which frustrated her a bit. “I guess I'm not talking fast enough, so it's like, okay, she's talked long enough beep…Then you have to go back, you have to remember where it stopped or you have to go back and do the whole thing over, and then you're worried and you're trying to do it really fast that it is gonna shut off again, but it doesn't shut off.” 74 The participant experienced difficulty in remembering the specific commands for using the device. For example, the only time she wanted to set a reminder, she did not use the required words and said, “Alexa, I would like you to tell me when it's 6:00”, which was not recognized by the device as a reminder. She was also confused while using the voice navigation on certain Alexa skills-allrecipes and was not able to find recipes using it. To add to that, she wanted to set a reminder for the day before an appointment, by a simple voice command, but it wasn’t currently supported. While setting reminders with the researcher during the activity, she experienced difficulty in deleting multiple reminders and wanted to delete it by voice, however, it required her to use specific reminder words [reminder date, time, and name] which she could not remember. “It wouldn't tell me that, I would get very frustrated with that if it wasn't gonna tell me… and I was like I'm not going to put these[reminder] in there.” Although the memory requirement for using specific keywords frustrated her, she was willing to try it again: “at a different time, so then I have time to calm down and not be frustrated with it. So, it has to be the next day.” During the three-week period, her device got disconnected once from the internet, which made her anxious. She was concerned about setting up the device by herself and did not feel confident doing it. This was the reason why she did not move her device after the set up. Instead of using the paired smartphone, she wanted to use the Echo Dot device for modifying or deleting items on her shopping list by using voice commands. She also wanted to use the Echo Dot to store phone numbers and contact information and make direct phone calls to people. Although currently Amazon Echo 75 supports Echo device to device calling, it requires a smartphone, which she wasn’t keen on using. “I am not gonna get a smartphone. No. So it's alright. I have them written down. When I forget, it's because I forget their cellphone and their home phone and I mix them up. And everybody seems to have a cellphone and a home phone. So, you know, it’s a lot to remember.” Other desired features include, using Echo device for sending emails by voice and a traditional manual or handbook to find more information about the device. A traditional manual or book with all functionality of the device was desired by the participant: “I was stuck at doing something and I could look through there and see, or what exactly it does and stuff. Some people, they don't need that, but I feel more comfortable with that.” Usage concerns. Concerns regarding using the device in front of other people arose. P5 was comfortable using the device only in front of familiar people, also highlighting privacy concerns of being overheard by someone. “I wouldn't carry on a conversation with it or anything if somebody was here and I... Because I am kind of shy? So, I wouldn't be talking to that and have somebody else.” Comparison with traditional technology The participant valued the interaction with voice, felt more confident and less anxious using this device over the computer, since speaking was easier for her than typing on the computer. Along with her interest in learning about new technology, the ease of use increased the willingness to use this device and the inquisitiveness to experiment while 76 using this technology as compared to a computer: “on computer, I might not because I'd be afraid I'd mess it all up, you know...” Although voice-based interaction was preferred due to ease of use, the participant still wanted to use the visual interface on computer for activities such as email, playing games and online shopping. “Well, if I would want to buy something and I wondered what it looked like, or if I was taking a trip and I wanted to see where... And I would like for it to show me... If I was going to Boston. "Could you show me some of the historical sites?" Change in perception with time With time, her confidence of using the device increased and she felt less nervous. “I feel like I've learned how to use that. So that's an accomplishment for me because it's technology and I just like it.” She even felt more confident of exploring the device by herself in future, “I don't know what it would be when I come across something, then I would ask if it couldn't give me an answer... That would be alright.” The willingness to use also increased with time, since she found it useful and easy to use. After using the Echo dot for three weeks, her overall confidence of using technology had increased. “This made me a little bit feel that I could learn some new technology. And at first I was nervous. And now I'm not... It would depend on the technology itself that I was having to use.” However, she still felt nervous regarding the device setup, even at the end of the study. Personification of Alexa While chatting with the device, in the presence of the researcher, the participant felt as “it's like having another person in your house, so you wouldn't be so lonely, so you can talk to it… ask it a question and it answers it.” However, with the actual usage she felt 77 she did not feel the same and mentioned she might have felt it as person talking if the device could talk by itself without invoking or if it could talk as a human: “if you can say, well “Alexa, how was your day today?”, like you would say to a friend that came in, “how are you?” To her the device was an object in her house unless she was talking to it. “I would treat it as an object. It would just sit there. But when I'm talking to it, I could think of it as... And it answers me and I am talking to it, I could think of it as a person.” Moreover, she liked the female voice, since she felt comfortable talking to a female and asking questions, rather than a male. She did not want the device to have a physical form (e.g., body or face) to accompany the voice. Smart home technology Of all possible smart home appliances, the participant only wanted to control the lights due to the ease of turning it off at night. Apart from that, she did not feel the need of using any other smart home appliances. 4.2.2. Cross-case findings Initial thoughts and change in perception with time Except for one (P4), all participants had a little to no prior knowledge of the Echo Dot device and had heard about it from television advertisements. Two participants (P1, P5) were concerned about being able to use a new technology, whereas P3 and P4 were somewhat hesitant to use due to a lack of knowledge of this device. However, after using the Echo Dot for three weeks, all participants found the device easy to use and their confidence and willingness to use a voice-based device had increased. For one participant (P5), the overall confidence of using other kinds of 78 technology also increased, feeling accomplished of being able to learn a new technology and use it. Comparison with traditional technology All participants found Echo Dot to be easier than a traditional computing device such as a computer or a smartphone and felt more confident while using the voice-controlled IPA. For two participants (P2, P5), the ease of use resulted in increased willingness to learn about this technology. For example, P2, who earlier might not have had patience to learn about different activities on computer, wanted to learn more about the capabilities of this device because it was easy to use for her. For some participants, using the Echo Dot device also marked some changes in their everyday technology use. For example, P2, who earlier used her computer once every few days, did not use it at all during the three weeks of study. P3 stopped using her radio and played music using the Echo Dot. P4 used her television less frequently than before and preferred playing music using Echo Dot. However, for certain activities such as playing games, browsing social media, paying bills and online shopping participants preferred using a visual interface over a voice- based interface. Device usage: specific use-cases The participants made use of a variety of features on the Echo Dot, of which, information finding was perceived useful by all. Using device for setting reminders was also perceived useful by all but one participant. Information finding. The ability to find online information by using voice commands was valued by the participants, since earlier they were dependent on someone else 79 (mostly family: children or grandchildren) or used phone books- yellow pages/white pages (for information on local businesses or phone numbers). Two participants (P2, P5) primarily sought help from others for information, but also sometimes used their personal computers but experienced difficulty due to typing. Participants asked a variety of questions to the device, some due to curiosity of testing the device, whereas others were to find information, including questions on history, local businesses (e.g., nearest movie theatres with movie schedule, nearest dentist), navigation directions, questions on famous personalities, health and medicine. Of these, questions on health and medicine were asked by all participants. However, the device was not knowledgeable to answer all questions from this specific category. The ease of finding information using the device also increased the frequency with which two participants sought information (P3, P5) as it motivated them to find answers to questions for which earlier they might not have used a computer or looked anywhere else. Supporting memory. Four participants (P1-P4) found the ability to set reminders using the device was useful, since previously they had maintained handwritten notes either on a calendar or on paper. Although setting reminders was useful, two participants (P1, P4) wanted to continue writing notes on calendar or on paper. P1 could not trust the device due to the various dependencies on technology infrastructure (e.g., electricity, internet connection), whereas P4 had a previous habit of writing appointments and birthdays on calendar. Even though the fifth participant (P5) perceived that the reminders would be useful in the beginning, she forgot the device capability of setting reminders. Moreover, 80 the only time she tried setting a reminder during the three-week period, the device did not understand her due to not using the required keywords (“I would like you to tell me when it's 6:00”). She was also concerned about completely relying on the device for reminders, due to self-forgetfulness of adding the reminder on the device, hence was positive to the idea of allowing other people (daughter, doctor and some friends) to set reminders remotely. Three participants found the grocery lists to be useful in supporting memory since they could add items to the list as soon as they remembered something instead of writing on a paper (which they did prior to using Echo Dot). Entertainment. As expected, playing music using the device was enjoyed by all participants, since they could listen to a variety of music, as online music streaming service eliminated the need of owning music. Two participants (P2, P3) used the device for playing games. Even though they liked playing games on the device, they preferred a visual interface over a voice-based interface for games. P2 and P3 also used the device for listening to jokes, stories and chatting with Alexa even in the third week of the study, highlighting this more as an established use. Current limitations and challenges The most commonly reported limitation by all participants was problems in voice recognition or the device not being knowledgeable enough to answers the questions (giving the user the same feedback in either case). In this case participants repeated a few times (~2-3) before giving up. Further, one participant (P3) wanted the device to give additional feedback on where to find answers to an unanswered question. Three 81 participants also reported device timing out before they completed the voice command due to the delay in speaking. Memory requirement of speaking specific keywords to use the device was reported by three participants (P2, P3, P5). Other mentioned challenges include difficulty in finding additional information about the device capabilities (two participants even wished for a detailed manual or handbook listing all device capabilities and instructions for troubleshooting) and using them, paired tablet interaction (where for certain activities the user is forced to use the app on tablet) and concerns about device set up. Instead of having sound for alarms and timers, three participants wanted the device to talk and tell the time (for alarm) or the purpose (for timer). [Note: although this feature exists for timers, the system does not prompt the user to tell the purpose while setting the timer, hence the participants could not find it]. For alarms, even though the device supports voice alarms (which do not tell the time), it requires the user to use the Alexa app, highlighting the need for richer voice-based applications. Desired features and design suggestions In addition to the current ability to set reminder at a specific time and date by the IPA user only, two participants suggested options to set reminder from different sources, that is, reminders that can be set by other people remotely (P5) or collaborating with other platforms such as social media notifications for birthday reminders (P4) and television for reminders of specific programs. 82 Ability to store phone numbers, contact information and make calls using the device was desired by three participants. Further, one participant wanted to store general information (e.g., a place where she kept a certain thing) to be reminded later. The ability to perform multiple tasks using a single voice command (e.g., playing music at a specific time by setting a reminder, setting reminders at a specific time for all days of the week- “Monday through Friday, give me an alarm at 6.30 a.m.”) was suggested by two participants. [Note: similar functionalities can be achieved through IFTTT, a web-based application for performing complex tasks on IPAs, but not through simple voice commands] One participant wanted her device to recognize and respond to the specific voices of her and her family. Moreover, to avoid her grandchildren from adult content she wanted age-specific voice profiles/ adult or child codes. Two participants also wanted to use the device for emergency purposes- to call emergency services or to some specific people. [Note: similar skills exist, but participants could not find it]. Other desired features include portability of the IPAs to carry the device easily, sending emails, listening to audio updates of television shows, downloading and printing items (e.g., books) from the internet, monitoring security by detecting sounds at home, finding information from obituaries and listening to local news channels. Device Personification All participants, except for one (P3), perceived the device as a social partner in the beginning after briefly using the device. However, with time, two participants (P1, P4) felt the device was an object in the house rather than a person. Although P1 addressed 83 the device with a personal pronoun (she), he considered it strictly as an object or a source of information and did not find the conversations of the chat feature human-like, because the device did not allow him to talk how he wanted to talk, rather provided suggestions on topics it could talk about. On the other hand, although P4 did not perceive the device as a social partner, she greeted it “good morning”, “good night” and “thank you”, attributing this personification as a habit. Two participants (P2, P3) perceived the device as a social partner because they could talk to it and it answered them back. Although, initially P3 was not comfortable and felt weird talking to a device, but with time she considered it as “somebody to talk to”. Both P2 and P3 used the chat feature to have conversation with the device and listened to jokes. To add to the personification, P2 further visualized a face (of family members or her partner) to match with the, imagining the specific person is speaking to her. However, when asked about having any specific face on the device, she preferred the current form to have the freedom of imagining the face herself. Even though the two participants liked having conversations with Alexa, certain instances where Alexa could not carry human-like conversation was also reported. The fifth participant perceived the device as a social partner only when it was talking and at all other times it was an object in her house. To her, the device might have been more like person if it could talk by itself without invoking and if it could talk as a human. 4.3. Discussion This three-week long study demonstrates the potential of home-based, voice-controlled intelligent personal assistants for providing accessible digital technology access to 84 older adults with limited experience of using computing devices in their everyday lives. This study highlights many findings of the previous chapter (Chapter 3—understanding the use of IPAs by people with disabilities) on the ease of use of IPAs, easy access to digital technology and the preference of using voice-based IPAs over traditional computing devices. At the same time, this formative research highlights directions for future work focusing on more specific uses of IPAs (e.g., as a memory aid) and identifies the accessibility issues that should be addressed. Below, I discuss the 1) accessibility benefits—easy access to digital technology and information access, 2) The accessibility barriers—challenges and limitations of this technology, 3) personification aspect of IPAs and, 4) the limitations of the study method. 4.3.1. Easy access to digital technology None of the participants in this study used a computing device (smartphone, tablet or computer) every day. All participants described a feeling of “not knowing enough” about digital technology and lacked confidence while using it which is similar to previous works on digital technology use by older adults [70,75]. However, in our study, we found that participants were confident using while using a voice-based interface (VUI), highlighting its potential for providing easy access to digital technology. Moreover, people also wanted to perform a variety of internet-based tasks (e.g., sending emails, paying bills) using the IPAs, for which they earlier used (or wanted to use) the internet on a smartphone or a computer, suggesting the preference of VUIs over traditional computing devices. 85 4.3.2. Information access Previous studies on information seeking behavior of older adults [23,38,65] show that seniors are likely to prefer in-person contact or human information sources as primary means for finding information. However, seniors who are somewhat comfortable using internet are more likely to make use of it [65]. Specific to health information, Chaudhuri et al. [16] also found a preference for finding information by asking people and attributed this behavior to the ability to discuss with a person as opposed to a “nonliving” source such as internet or computer. However, the findings of our study suggest that older adults with limited technology experience are motivated to use internet as primary source for finding information by using conversational voice-based interaction, which corresponds to their current mental models of information finding. Moreover, not being dependent on a family member for information seeking was valued by seniors. 4.3.3. Current challenges and limitations Accessibility challenges arose primarily due to the voice-based interaction, due to the device timing out before users completed the voice command and the memory requirement of using the specific keywords for using the IPAs (also mentioned by some users in Chapter 3). Allowing users to customize the device timing out time can address the former challenge. Adaptive learning systems should be further explored to address these challenges of voice-based interaction. For example, adaptive systems can learn the user’s speech pattern and adaptively change the device timing out time, whereas, adaptive learning paired with contextual help can address the challenges of memory 86 requirement of remembering specific keywords by prompting actions to the user. Challenges due to dependency on the paired computing device (tablet) and inability to perform complex tasks through voice (e.g., string commands for multi-layered tasks), highlights the need of richer voice-only interfaces. Trouble in finding information about device capabilities was also mentioned by users, which highlights the challenges of discoverability in voice-based interfaces. 4.3.4. Personification All participants personified the IPA at least once during the in-person interviews, by using a personal pronoun (she) and/or through “mindless behavior” (e.g., “thank you”) of greeting the device as a human, which relates to previous work on IPA personification [41,57]. Two participants personified the device due to “social mindless response’ or “over-learned social behaviors” (as suggested by Lopatovska and Williams [41].) whereas, two other participants considered it more as a social partner. This also add to the findings of Chapter 3, where some users with disabilities mentioned the device as a friend or a companion. This companionship aspect of personification may be attributed to loneliness, since all participants lived alone and/or the ability to have a conversation. Future work needs to investigate whether this technology (which is easily available) can serve as a source of companionship for certain specific user groups and how that would compare to the companionship with humans. 4.3.5. Limitations Although an in-depth longitudinal study, this study has a low sample size which affects the generalizability of the reported findings. The findings of this study are limited to 87 older adults with limited technology experience, although addressing the identified issues would be widely beneficial. Further studies should explore how home-based IPAs are used by older adults in general. Another limitation is that this was not an entirely naturalistic deployment study. Users were given training to use the device and the advanced skills. They could always ask a member from the research team for additional help. Moreover, we also had daily phone calls for the diary study. Although we tried to minimize the daily intervention, by using automated calls instead of in-person, it still might have affected the natural usage pattern as users might have felt conscious to use the device. Both interviews and the diary study were self-reported which increases the possibility of inaccuracy due to the selective memory of remembering a specific experience at a time in the past. Sometimes while probing participants for contextual understanding of specific instances of device usage during the interviews, they could not recall it. The daily diary study somewhat helped in reducing the effect of selective memory, but calling participants everyday had its own limitations due to call drop offs, inaccurate reporting due to repetitive nature of calls and affecting the natural usage pattern of the device by reminding users every day to use the device, thus not facilitating a naturalistic deployment study. The findings of our study only indicate the initial experiences of using the device, which might be affected due to the curiosity of using a new technology and may vary with time. Recording of usage logs of participants’ interaction with the device could also have created a self-cautiousness affecting the nature of questions people asked to the device. 88 Chapter 5: Discussion & Conclusion Through two exploratory studies, this thesis demonstrates the potential of home-based, voice-controlled IPAs to provide inclusive, accessible interaction for a broad range of users including people with disabilities and older adults with limited technology experience. In the first study, we analyzed 346 Amazon Echo reviews that mention a person with a physical, sensory or cognitive disability using the device, followed by an interview study with 16 visually impaired users of home-based IPAs to provide in- depth understanding of the specific user group. To further scope our understanding to IPA usage for digital technology access for another specific user group, we conducted a second study with three-week field deployment of Amazon Echo Dot devices with five older adults who were not daily users of digital computing devices. Below I discuss the summary of findings from each study. 5.1. Summary of Findings and Contributions Study 1 provided a broader understanding of IPA usage by people with disabilities and an in-depth understanding of IPA use by people with visual impairments. Findings indicate the ease of use compared to existing technology (Study 1: I-II), easy access to digital technology (Study 1: I) and an increased efficiency of performing a variety of tasks (Study 1:II). Unexpected use of IPAs for speech therapy, learning support and as memory aid was also observed (Study 1: I), which offer potential avenues for further research. Accessibility barriers due to dependency on a paired computing device e.g., smartphone, computer or tablet (Study1: I-II), physical design of device e.g., poor contrast between symbol and button (Study 1:II), discoverability of voice commands 89 (Study1:II), memory requirement of using voice commands (Study1:I), and problems with speech output settings e.g., device timing out (Study1:I) was also found. Overall this study contributed the following: 1) characterization of how home-based, voice- controlled intelligent personal assistants are being used by people with disabilities, 2) identification of accessibility benefits and barriers, 3) recommendations for design as well as future work on conversational voice interfaces for users with disabilities. Study 2 with older adults having limited technology experience, reflected many findings of the previous study (Study1:I) with Amazon reviews mentioning an older adult, such as emphasizing easy access to digital technology and preference of using voice-controlled IPAs over a traditional computing device for accessing internet for older adults who do not use a computing device frequently. Although IPAs have the potential to assist memory (found in Study 1: I and Study 2) by setting reminders, timers, calendars and maintaining lists, we identified design implications and scope of further research to enhance these devices for better supporting memory needs of older adults (Study 2). Challenges of discoverability of voice commands (Study 1:II) and dependency on a paired computing device (Study 1: I-II) which surfaced in the previous study, was reflected again in Study 2 with older adults having limited technology experience. The other sparsely reported accessibility challenges from Study 1:I (Amazon reviews analysis) which include device timing out before completion of voice command and memory requirement of remembering the specific keywords to use the IPA was also observed in Study 2. This highlights how addressing these accessibility issues can be widely beneficial. 90 Contributions of this study are: 1) characterizing how home-based, voice- controlled IPAs are used by older adults who are not everyday users of a smartphone, computer or tablet, 2) identifying accessibility challenges, 3) deriving a set of design implications to enhance IPAs for memory support. 5.2. Reflection and Implications for Design 5.2.1. Smartphone vs. home-based IPAs. Home-based IPAs offer different affordances than smartphone IPAs (e.g., Siri), and, at least at the time of study, offered greater functionality with the ability to connect to smart home technology. The largely positive findings from our studies contrast work with smartphone-based IPAs, where relatively younger participants without disabilities considered the IPAs to be “entertaining / gimmicky” [42]. This difference may reflect the differences of home-based devices compared to a purely mobile, smartphone option and/or the preferences of the specific user groups studied. Further work should explore these possibilities. 5.2.2. Design opportunities Discoverability. Discoverability of voice commands is a long-standing problem with voice interaction [77]. This issue arose in both Study 1 (II-with visually impaired users of IPAs) and Study 2, where users found it difficult to discover different applications, advanced commands and troubleshoot. Adaptive and contextualized learning may enhance learnability and discoverability in voice interfaces [17,25], and has been recently used for voice-based interaction for users with motor impairments 91 [17].Typically, however, the voice input is paired with visual output. Supporting voice- based contextual help by using simple voice commands can be help solve this problem (e.g., a user can ask the device for help or for additional information when needed). Richer voice-based applications. A related issue is the limited nature of voice-based interaction on IPAs which surfaced in Study 1:II (IPA users with visual impairments) and Study 2. This includes dependency on a paired computing device (e.g., smartphone, computer or tablet), limited functionality of voice-based apps as compared to mobile or desktop counterparts and limited support of complex voice commands (e.g., string commands for multi-layered tasks such as reminder to play news at specific time). While a number of users (e.g., sighted, high to moderately tech savvy) may not mind switching to a visual interface (e.g., smartphone or computer) for in-depth tasks or use advanced supported platforms such as IFTTT to perform complex actions, understanding how to better support rich interaction through a voice-only interface is important for accessibility. While existing auditory interface interaction techniques should prove useful (e.g., Spearcons [72]), new advances are needed to support complex information access. Memory support. Participants in Study 2 and some users with memory loss in Study 1:I (Amazon review analysis) sometimes encountered difficulties in remembering the voice commands (specific keywords). Adaptive interaction may address this problem, for example, by learning a user’s usage patterns to efficiently prompt actions. Although we observed broad use of the IPAs for aiding memory: setting reminders, maintaining lists, and other memory-related tasks, these devices can be further enhanced to support memory. 92 Collaborative reminders. For users with some memory loss, who might be benefited by reminders but forget to set them, providing access to other people (e.g., family members, doctors) to set and check reminders on the user’s device can be useful to further assist memory. Collaborative platforms. Along with setting reminders using the IPAs alone, collaborating with other relevant platforms such as social media (e.g., Facebook) for sending reminders of birthday can be used to enhance the capabilities of IPAs to support memory. IPAs as storage unit. The ability to store memos and later ask the IPA to search and return a specific memo using simple voice commands would be helpful in aiding memory. Customization. In both Study 1 and Study 2 we found instances where customization would be useful. Customizing the “device timing out” time will be useful for older adults who speak slowly. The ability to change the device voice to male was also desired by one participant in Study 2. Customizing the alarm and timer sounds was desired by some participants in Study 2, preferring to hear Alexa talking to them for the alarm over the traditional alarm sounds. Based on our findings, we suggest the hardware designers of IPAs should focus on improving the accessibility of the device ecosystem such as accessibility of the IPA app (e.g., Alexa app) on paired computing device (Study 1:I-II), set up process (Study 1:II), provide additional speech output settings (Study 1:I), improve the sensing capabilities of the device microphone (Study1:I) and improve the accessibility of the physical design of the device (Study 1:II), e.g., the low contrast labels on buttons. The 93 software developers of the IPAs should address the issues due to discoverability of device features (Study 1: II and Study 2) and memory requirement of voice commands (Study 1: I and Study 2) by exploring adaptive and contextual learning, along with reducing the dependency on paired computing device (Study 1 and Study 2) and allowing customization of device timing out time (Study 1: I and Study 2). The developers of specific IPA apps (e.g., “skills” on Alexa), can improve the accessibility of their apps by supporting feature rich voice-only applications to reduce the use of paired computing device, along with supporting complex voice commands and wide guessability of voice commands. 5.3. Limitations The overall limitation of this thesis is the broad range of population it tries to gain insights from. Although the first part of Study 1—analysis of Amazon Echo reviews provided a broader understanding of the accessibility benefits and barriers of using IPAs, the second part of Study 1—interviews with visually impaired users and Study 2—with older adults having limited technology experience, are largely targeted at specific user groups only. This thesis does not provide an in-depth understanding of IPA usage by other sub populations—people with motor, speech or cognitive impairments and older adults in general. Study 1: I—Analysis of Amazon reviews and Study 2—field deployment of Amazon Echo Dots, largely focus on the specific IPA released by Amazon and hence the findings from both these studies are likely to be biased towards Amazon IPAs. All data from Amazon reviews, interviews and diary studies is self-reported and cannot be independently verified. This can potentially include personal bias of the respondent due 94 to the selective memory since the respondent might not remember the exact experience that occurred at some time in the past. The diary study (in Study 2) conducted on a daily basis somewhat helped to reduce the latter issue, however, Study 2 lacked due to a small sample size. In study 1, part I—analysis of Amazon reviews, although we included only verified reviews because they are more credible than otherwise [1], but there is still the possibility of misleading reviews (e.g., ads for third-party features). Additionally, the remote interview method with visually impaired participants at times created difficulty in maintaining rapport with respondents, whereas in-person interviews may have yielded more in-depth responses. 5.4. Scope of Future Work As a formative HCI research, this thesis enumerates various avenues for future work. Future work on improving the accessibility of voice-controlled IPAs should focus on discoverability of voice commands (to support the memory requirement of remembering specific keywords) and support rich voice-only applications (to reduce the dependency on paired computing device and increase level of complex tasks on IPAs by voice interaction). IPAs may be a promising platform for extending existing work on memory support (e.g., prompting systems [15]) and to explore new possibilities for more explicitly supporting independent living for users with memory loss. From findings of Study 2, we suggest design implications as potential areas of future work. Unexpected uses of IPAs in speech therapy and learning support was reported in Study 1:I—Analysis of Amazon reviews. Future work should explore how IPAs are 95 used by people with speech impairments, understand the extent to which IPAs are able to support speech therapy and identify the design opportunities to better support speech therapy. Similarly, for learning support, it is important to understand how people with Dyslexia use these devices and what further opportunities emerge to better support IPAs for enhancing learning. IPAs provide an easy integration with smart home technology. However, presently the smart home technology is relatively new, and findings of this thesis is a preliminary understanding of home automation usage (actual/perceived). For future work, it will be important to revisit adoption rates in a few years to assess how adoption is changing. 5.5. Conclusion With the increasing adoption of voice-controlled conversational interfaces and home- based IPAs, understanding the accessibility aspect of this technology is critical. This thesis presented three exploratory studies investigating the use of home-based IPAs by these user groups. In the first study, to understand current use by people with disabilities, we analyzed 346 Amazon Echo reviews that mentioned a user with a disability and interviewed 16 visually impaired participants who owned a home-based IPA. Analysis of Amazon reviews showed that users with a range of disabilities are using the Amazon Echo, including for unexpected cases such as speech therapy and support for caregivers. Accessibility issues due to speech input and output, along with memory requirement of voice commands was also reported. Interviews with visually impaired users provided a more in-depth analysis of one specific group—users who are blind or visually impaired—with findings reflecting the initial study as well as 96 emphasizing the efficiency of the devices for a variety of tasks, difficulties with discovering new functionality, desire for richer voice-only applications and issues with the device ecosystem. Study 2 with older adults who do not use a computing device frequently, demonstrated the potential of a voice-based interface for providing easy access to digital technology and online information. However, accessibility challenges related to speech input (e.g., device timing out), memory requirement of voice commands, discoverability of device features and dependency on paired computing device (e.g., smartphone, tablet or computer) still exist. We propose design suggestions and areas for future work, such as exploring the use of IPAs for speech therapy, memory aids and designing of feature rich and purely voice-based interfaces and apps. 97 Appendices Appendix A. Study 1-II: Interviews with Visually Impaired IPA users 1. Interview Protocol Hello, my name is [name]. Thank you for participating in our study. We are studying the use of intelligent personal assistants like Amazon Echo and Google Home by people with disabilities. During the interview, I will first ask you some basic demographic questions followed by questions on Amazon Echo/Google Home in general and how you use these devices in your daily life. Have you read the consent form which I emailed to you? So, you own ____ (Echo/GH), right? As you are already using an Echo/GH, you may be aware of its capabilities. In case there are some common features that you’re unaware of, I will start by playing a video of Amazon Echo which shows how it can play you the news, create shopping list, tell you the weather and more. https://www.youtube.com/watch?v=hR3Re5YjokM These devices can also connect to home automation technology. Now I am going to play a video where a man uses his Google Home to control the house temperature, smart TVs, home security system, garage doors, lights, and blinds of window by simple voice commands. https://www.youtube.com/watch?v=EFOzYgTz35I So, both the videos gave a brief introduction about these devices and how they can help with daily activities like creating a shopping list, checking weather, traffic, setting a timer, listening to news, playing a music, and controlling house lights, house temperature, or a home security system among other features. Let’s begin with some background questions. Background questions: 1. Age 2. Gender 3. Do you have any motor impairments? If yes: Please describe your motor impairments. - Age of onset? - Is there a diagnosed medical condition associated with your motor impairments? - Do you use a mobility device (e.g. wheelchair, walker, cane)? If so, what? 98 4. Do you have any visual impairments? If yes: Please describe your level of vision. - Age of onset - Is there a diagnosed medical condition associated with your visual impairments? 5. Do you have any hearing loss? If so, please describe. - Age of onset 6. Tech usage: Do you own a smartphone or tablet? If Yes, what? For each: How often do you use it? a. Every day b. Once every few days c. Once a week d. Less than once a week Introduction 1. How long have you had AE/GH? 2. How many Echo/GH devices do you have? 3. Do you have your AE/GH connected to home automation technology? a. If Yes: What household devices do you have connected to Echo/GH? b. For how long have you used it with home automation technology? 4. Overall, what do you think about AE/GH? 5. How many people other than you live in your house, including adults and kids? Which of them use AE/GH? Usage: In General 1. How often do you use your Amazon Echo/GH? a. Multiple times a day b. Once a day c. Once every few days d. Once a week e. Less than once a week 2. What do you use it for? Follow up: a. What about online information tasks, like managing To-do lists, checking the weather or news, checking your calendar, looking up information, playing voice-based games, and so on? b. (So earlier you mentioned about having connected your AE/GH with Home automation devices like…) What about those? How and when do you use Echo/GH to control those devices? 99 Initial Motivation, Expectations and Overall Experience 1. What was your motivation behind buying Amazon Echo or Google Home? 2. What were your expectations for [Echo/GH] before using it? 3. When you first got it, how did you experiment with using it? What did you try doing? 4. Are there any differences in how you use it now compared to the first few weeks? 5. Now that you’ve had it for a while, do you think it’s useful or not? If no: Why? For which tasks or activities do you find AE/GH most helpful? Why? How did you do those tasks before? [Beyond what you’ve already told me,] is there anything that you use Echo/GH for that you can’t do or have difficulty doing without AE/GH? Please explain. How did you do “it” before? Device Location 1. Where are/is your Echo/GH device(s) placed in your home? 2. Why did you choose “that/those location(s)”? Usage: Most Recent 1. Within the last 24 hours [few days / week], specifically, can you describe how you’ve used it? Challenges and Concerns 1. What do you think are the limitations of this technology? 2. Have you encountered any challenges in using it? This could be anything from setting it up to whether it understands when you speak to it. a. Can you elaborate “it” please. - What have you done to overcome “it”? Device Setup and smart home Ecosystem 1. When you first got AE/GH, who setup the device? Could you do it yourself or did you take help from someone? Did you/they face any difficulty while setting it up? 2. What about the setup of home automation devices? Speech Recognition / Understanding 1. Do you ever have problems with AE/GH not being able to understand your speech? If yes, please describe. (example) 2. How about problems with AE/GH understanding your speech but not knowing how to respond or responding in the wrong way? Please describe. (example) Privacy and Social Aspects Let’s talk about some privacy and security aspects of Echo/GH. 1. Do you have any privacy concerns about using AE/GH? If yes, please describe. 100 2. How about specifically to do with being overheard by others in your household or by visitors? If yes, please describe. 3. How about to do with audio clips of your voice being uploaded to Amazon/Google’s servers? 4. How about with visitors being able to control AE/GH? 5. [If yes to any] Have any of these concerns affected how you use the device? Home Automation You had mentioned that you have/haven’t connected your Echo/GH to Home automation technology and you use it for____. Let’s talk a little more about that and smart home technology in general. If yes, 1. Are there any other home automation features that exist for AE/GH or more generally (e.g., emergency calling, health monitoring, fall detection etc….) that you think would be useful for you? No: why not? Yes: So, why don’t you use them 2. Beyond existing home automation features, are there any other capabilities you would want to see in an ideal future smart home? Imagine anything that you want. It doesn’t have to be technically feasible now. Follow up: How do you think those would be useful to you? If not connected to Home automation 1. Is there any reason why you haven’t connected AE/GH with home automation technology? 2. If given the opportunity, would you want to connect AE/GH to any household devices, lights, and so on so that you can control them with speech? If so, which ones and why? 3. Do you see any challenges in connecting home automation technology with AE/GH? 4. Beyond existing home automation features, are there any other capabilities you would want to see in an ideal future smart home? Imagine anything that you want. It doesn’t have to be technically feasible now. Follow up: How do you think those would be useful to you? Interface Design and Overall Suggestions 1. Do you think it would be useful to control AE/GH at times by means other than voice? For example, by directly touching it, using your smartphone or a wearable device like a smartwatch? If yes, please describe how/why. 2. [For people who have at least low vision (who are not legally blind)] Do you think it would be useful for AE/GH to be able to show you a visual display at times? If yes, please describe how/why. 101 3. Do you have any other suggestions to improve this technology for you? Please describe (how/why). This is the end of the study. Thank you for participating. We appreciate your time and input. You should have just received an Amazon gift card via email. Can you please check and confirm that if you have received it? 2. Interview Codebook 1. Comparison between different personal assistant technology (Google home, SIRI, Google Now etc.) 2. Motivation of buying the device a) Perceived utility/usefulness of the device b) No motivation: Got as gift c) Curiosity to check the experience d) Other e) Heard from other people about the device f) Perceived ease of doing things 3.Device usage a) Specific activities performed b) Most recent usage 4. Expectations before using the device a) Enhanced ability of performing various tasks: AE/GH will make it easy to do tasks which weren’t easy previously b) Other c) Capability of performing basic tasks like listening music, setting timers, reminders, setting alarm, listening news etc. d) Capability to perform a variety of tasks (well informed about the device capabilities before purchasing it) e) Didn’t have any expectations 5. Initial experimentation with the device a) Explore the device- explore various skill, try to find more information about the device and its capabilities b) Home automation c) Other d) Tried the basic things- playing music, news,radio, setting timer, alarm, reminder, asking questions etc. 6. Differences in use between now and first few weeks a) Using more features and skills b) Using it for lesser activities now c) Using it more often. 102 d) Using it lesser than before e) Other 7. Finds device useful a) Activities for which the device is useful b) Reason for finding device useful 8. Mentions device location a) Bedroom b) Living room c) Kitchen d) Personal rooms e) Office/study f) Family room g) Other 9. Interface preference a) Would prefer to control the device by means other than voice, at times b) Wouldn't prefer to control the device by means other than voice 10. Home Automation: NO a) Reason for not using home automation b) Perceived challenges/concerns of home automation use c) Home appliances which they would want to be automated 11. Interesting thoughts about the device 12. Privacy a) Privacy concerns affecting device usage Yes: Privacy concerns affected device usage NO: Privacy concerns haven't affected device usage b) Privacy issues mentioned 13. Speech recognition and processing a) Problems with understanding the speech b) Accidental triggering of the device c) Other d) Device is not knowledgeable enough to handle natural conversations e) Speech recognition positive or neutral 14. Mentions limitation/challenges/concerns/suggestions a) Inaccessible smartphone app (not voiceover compatible) b) Cost c) Not portable d) Other e) Paired smartphone/computer 103 f) Issues with information finding g) Other challenges 15. Ideal smart home vision 16. Mentions general positive about this technology. 17. Other desired capabilities 18. Device setup 19. Security issue mentioned 20. Home Automation: YES a) Appliances used currently b) Other home appliances/devices which they would want automated 104 Appendix B. Study 2 1. Demographic and background questionnaire 1. Age: 2. Gender: 3. What is the highest degree or level of school you have completed? a) Less than a high school diploma b) High school degree or equivalent (e.g. GED) c) Some college, no degree d) Associate degree (e.g. AA, AS) e) Bachelor’s degree (e.g. BA, BS) f) Master’s degree (e.g. MA, MS, MEd) g) Professional degree (e.g. MD, DDS, DVM) h) Doctorate (e.g. PhD, EdD) 4. What is your current employment status? a) Retired b) Employed full time (40 or more hours per week) c) Employed part time (up to 39 hours per week) d) Unemployed and currently looking for work e) Unemployed and not currently looking for work f) Self-employed g) Unable to work 5. Do you use the internet or email, at least occasionally? a) Yes b) No If yes, which devices do you use in general? (Please select all that apply) I. Cellphone II. A tablet computer like an iPad, Samsung Galaxy Tab, Microsoft Surface Pro, or Amazon Fire. III. A handheld device made primarily for e-book reading, such as a Nook or Kindle e-reader but NOT the Amazon Fire IV. Other mobile handheld device V. Desktop or laptop computer 6. Overall, how confident do you feel using computers, smartphones, or other electronic devices to do the things you need to do online? (select one option) Do you feel a) Very confident b) Somewhat confident 105 c) Only a little confident d) Not at all confident 7. Overall, what do you think of technology’s impact on society? a) Mostly positive b) Neutral or mixed effects c) Mostly negative [Note- We also included the OPQOL Brief questionnaire: ] 2. Interview one questionnaire Introduction Hello, my name is [name]. Thank you for participating in our study. The purpose of this study is to gain an understanding of how you use voice controlled intelligent personal assistants, like Amazon Echo and Google Home in everyday life. Consent Form Before we begin, I have a consent form for you to sign. In case you haven't read the consent form, you can read it now or I can read it to you and we will discuss any questions you may have before signing the form. If at any point you feel that you no longer want to continue with the study, just let me know. Please sign the form. (After signing) Here is a copy for you to keep. Background questionnaire Thank you. Let’s begin with some background questions (background questionnaire). Can you please fill this questionnaire? Follow up on frequency of use of the devices mentioned in the form- You mentioned using ___for accessing internet. How often do you use it? e. Multiple times a day f. Once a day g. Once every few days h. Once a week i. Less than once a week Physical abilities Let’s talk about your physical abilities. o Do you have any vision difficulties that affect your daily activities? If yes: o Do you use any visual aids o How it affects your daily activities o Do you have any difficulties with motor control? If yes: a) Please describe your motor impairments b) Do you use a mobility aid (e.g. wheelchair, walker, cane)? If so, what? And since when do you use it? 106 o Do you have any hearing loss? If yes: o Please describe your level of hearing. o Do you have any difficulty remembering every day activities? If yes: o Please describe any memories difficulties. o Do you use any memory aids such as paper lists, digital reminders, digital calendars, and so on for everyday tasks? What do you use? Okay, now let’s move to the interview. Questions: 1) How did you become interested in being a part of this research? 2) Have you heard of the Amazon Echo or Google Home before? [if yes, can you tell me what do you know about it?] Give the description: These are devices that you can interact with by voice, instead of typing or through a screen. They can help you with a number of tasks. It can be your personal assistant and help you set timer, maintain a to-do list, maintain a calendar, play music, play radio, give weather update, give traffic update, create and maintain shopping list and much more, all by simple voice commands. [Show the video on Amazon Echo to everyone] VIDEO LINK 3) What do you think of devices like the Amazon Echo? 4) Do you think you might like to use a device like this? a) Why/Why not? 5) Do you know people who are using a device like this? [probe for: who these people are (peers? Children/grandchildren?)] 6) What do you think about being able to communicate with the device by voice? [probe for: differences from traditional tech in terms of accessibility, cognitive requirements] i) What do you think of using these devices in terms of difference from traditional tech such as smartphones or desktop computer which have a visual interface? ii) So, in the video, we saw how the device responds only to the wake up word Alexa. What do you think about it in terms of remembering the required commands for using it? 7) Do you think the device might be useful for you? a) Can you give an example of what you think it might be useful for? b) Is there anything you are currently not doing that you think you might be able to do with the device? Transition Now, let’s move on to setting up the Echo Dot device. I’m going to set it up for you, following these instructions here in this manual. You probably won’t have to do this yourself at any point, but you can always call me for help if you do. The number is here in this sheet. After I get it set up, we will start playing around a little with the Echo Dot. Connect device to power outlet, introduce the dot and the tablet, set up the device following the instructions on the instruction sheet. 107 Here is a list of things that you can do with your Echo Dot. This is a copy for you to try out. Remember, this is just a few examples of what you can do with this device. You can ask it much more questions than this. However, it might not be able to answer everything that you ask. Now let’s use the device a bit and see what you think about it. “Alexa.” [pause] This blue light ring indicates that the device is hearing you. It listens to anything you say after the wake-up word “Alexa”. Volume controls- You can increase or decrease the volume by using these buttons. Microphone mute- Press this to turn off the microphone. The device won’t hear you if the microphone is turned off. Press this button again to turn on the microphone. If you want to explore more, you can have a look at this sheet, and let me know if you have any questions. Shall we proceed? Alarm: Let’s set an alarm for 5pm today. “Alexa, set an alarm for 5 pm today.” Now let’s delete the alarm, “Alexa, delete the 5 pm alarm.” Reminder and timer: Now, let’s set a reminder for the device to remind us calling someone: “Alexa, set a reminder at 8pm to call Alisha tomorrow.” Similarly, let’s set a timer to check the oven 10 mins. “Alexa, set a timer for 10 mins.” 8) What do you think of these features? 9) How do you normally do these activities? Shopping list: Now let’s create a shopping list and add some items, say, milk and eggs to it. “Alexa, create a shopping list.” “Alexa, add milk and eggs to my shopping list.” Now let’s add potatoes to the list. “Alexa add potatoes to my shopping list.” Now let’s check what’s on your shopping list. “Alexa, what is on my shopping list?” To-do list: Similarly you can also create a to-do list and ask Alexa to read the items on your to-do list. 10) What do you think of these features? 11) How do you normally do these activities? Entertainment, humor, joke Now let’s ask the device to play a song. “Alexa, play the piano man.” Okay. Let’s ask her to stop playing it. “Alexa, stop.” Alexa can also tell you a joke or a story for humor. Let’s ask the device to tell a joke. “Alexa, tell a joke.” 12) What do you think about this? 108 13) What do you normally do for entertainment? Device feature: Information finding Now, let’s try finding some information on the internet using the device. “Alexa, what restaurants are near me?” “Alexa, how hot is the sun?” 14) What do you think of this feature- finding information using the device? 15) How do you generally find information from the internet? Device feature: Chat Now, let's try chatting with the device. To improve these devices, university competitions were organized where universities send their teams. Now, let’s chat with one of the winner teams. “Alexa, let’s chat”. Okay. Let’s ask her to stop. “Alexa, stop.” 16) What do you think about this? 17) Have you ever used technology for chatting with someone or with some device like this? 18) What do you think about this idea of talking to a device like this? Overall: Voice interaction 19) Overall, what do you think of being able to use this device with voice? 20) Are there any challenges you can foresee in using the device? What are they? i) Challenges similar to using any other tech: Do you think you might encounter challenges similar to using any other technology? ii) Challenges specific due to voice interface: Do you think you might experience any difficulty due to the voice interaction? iii) Challenges due to setup: Do you think you might encounter challenges in the setup? iv) Information finding: And how about finding information on how to use the device? Do you think there might be any challenges in that? Thank you for your time. We have reached the end of the study. Is there anything else you want to share or ask? 3. Interview two questionnaire Questions 1. How was your experience of using the device in this week? 2. Did you find out anything new about the device this week? If so, what? How did you find it out? 3. Where did you keep the device, mostly, this week? Why? a. Did you move it around at all? 4. Did you find the device useful? a. For which activities did you find the device useful? 109 b. Could the device help you with any task that previously you couldn’t do or had difficulty doing? 5. Did you enjoy using the device? Can you please give me an example of such a situation? a. Did you find the device entertaining? Please describe. 6. Did you find any new skills this week? Can you tell me more about the skills? [ask detail description of each skill and the experience.] 7. So, [X- interesting challenges or limitations pointed out by the participant in the daily telephone call] was a challenge that you experienced and mentioned over the phone call. Can you describe that? How did you overcome that? a. Apart from that, did you face any other difficulty? b. (Or, if no challenge mentioned in weekly diary study) Can you describe any challenges you encountered while using the device this week? c. Do you have any suggestions for making it better? 8. Did anyone apart from you use the device this week? If so, please describe who, when, and for what. 9. Overall, did you like talking to the device? Can you please describe a specific conversation which you enjoyed? Introduction to new skills (Week 2) So, as you have seen, your Echo Dot can do a variety of stuff for you. Now, let’s look at some other things that it can do. The developers call these ‘skills’ of Alexa. You can ask Alexa about recipes, stock prices, play relaxation music, play games etc. We have enabled a few skills for you which are listed in this paper. Let’s try one skill from each the category listed in this paper and see what you think about it. Game- Akinator Let’s start with a game, say, Akinator- the second one under the games category. “Alexa, play Akinator.”..Let’s keep R2-D2 from Star Wars as our character and see if it can guess what we are thinking. [Play the game] Similarly, there are other games –Jeopardy and Magic door for you try. • What do you think about playing games using this device? • Do you play any games? If yes, which games do you play and how? This day in history Now, let’s try this skill- This day in history. “Alexa, what happened today in history.” Now, you tell me a date and let’s ask Alexa what happened on that day. [Participant answer] “Alexa, what happened on ___ in history?” • What do you think of this? Relaxation- 110 Now let’s try one of the skills from the relaxation category, say, Sleep sounds. “Alexa, open sleep sounds.” [Select and pay 2 sounds from the list read out by Alexa.] • What do you think about it? Food and Drinks Now let’s try AllRecipes from the food and drinks category. “Alexa, ask All recipes for a lasagna recipe.” • What do you think of that? News Apart from the default news settings as in last week, there are a few more news skills e.g., CNN and Fox news and if you enable these, news from these channels will be read out in your flash briefings, when you say: “Alexa, what’s the news?” • What do you think about this feature of listening news on this device? Did you try this feature during the past week? [ask only if not mentioned explicitly in any of the interviews or calls before] • Earlier, how did you follow news? • How does hearing news on Echo dot compare to the previously used methods? Let’s see what news skills you want to enable. Similar to news, there are many other categories of skills. Let’s look which skills you want to explore. [Show them various skills category, and enable the skills they choose] 4. Interview three questionnaire 1. How was your experience of using the device in this week? 2. Did you find out anything new about the device this week? If so, what? How did you find it out? 3. Where did you keep the device, mostly, this week? Why? a. Did you move it around at all? 4. Did you find the device useful? a. For which activities did you find the device useful? b. Could the device help you with any task that previously you couldn’t do or had difficulty doing? 5. Did you enjoy using the device? Can you please give me an example of such a situation? a. Did you find the device entertaining? Please describe. 6. Did you find any new skills this week? Can you tell me more about the skills? [ask detail description of each skill and the experience.] 111 7. So, [X- interesting challenges or limitations pointed out by the participant in the daily telephone call] was a challenge that you experienced and mentioned over the phone call. Can you describe that? How did you overcome that? a. Apart from that, did you face any other difficulty? b. (Or, if no challenge mentioned in weekly diary study) Can you describe any challenges you encountered while using the device this week? c. Do you have any suggestions for making it better? 8. Did anyone apart from you use the device this week? If so, please describe who, when, and for what. 9. Overall, did you like talking to the device? Can you please describe a specific conversation which you enjoyed? 5. Interview four questionnaire 1. What are your overall thoughts about this technology? [probe for: usefulness, enjoyment using, willingness to use] 2. For which tasks or activities do you find AE/GH most useful? Why? a. How did you do those tasks before? 3. Challenges: Can you describe the challenges that you encountered using the system? a. Were you able to overcome these challenges? If so, how? b. Were there things you couldn’t do that you wanted to do? 4. Perception of digital technology: Do you think your perception of the device has changed over time? [probe for: changes in ideas of usefulness, willingness to use, confidence of using digital technology] 5. Comparison with other technology: What do you think of a voice-based technology like Echo Dot as compared to traditional digital technology devices such as smartphones or tablets? Personally, what would you prefer and why? a. Has this device affected your use of other digital technology devices? [IF earlier mentioned about using smartphones or similar devices] 6. Memory aid: What do you think about these devices in terms of assisting memory? a. During the three-week period, did you encounter situations where reminders, timers, alarms etc. were helpful for you? b. Where there situations when the device didn’t do a good job of or couldn’t support in assisting memory? Please describe. c. How does Echo compare to the previous memory aids that you used (if any mentioned in the background interview)? 112 7. Entertainment: How was your experience of using the Echo for entertainment? a. How does it compare to your previously used entertainment devices? 8. Information finding: What do you think of using Echo for finding information on the internet by using voice commands? a. How does Echo compare to the other devices used by you for accessing internet? 9. Skills: What do you think of the various skills of Alexa? How was your experience using them? a. Did you find any skills particularly enjoyable? b. Did you find any skills useful? c. Was any skill, particularly, frustrating to use? d. Are there any skills which you wanted to use but was not available? 10. Companion: During the three-week period, was there any instance when you enjoyed having a conversation with the device? Can you share your experience of a specific such time? 11. Suggestions: Do you have any suggestions to what we could do to improve the device? 12. Tab usage: How was your experience of using the Fire tablet? Apart from using the Alexa app, did you try anything else with the tablet? If yes, what did you try doing with the tablet? 13. Smart home: These devices can also connect to smart home devices like lights, blinds, switches, thermostat, television, etc. and you can control all these by simple voice commands. For e.g., you can change the temperature by saying: “Alexa, set the temperature to 72 degrees” or “Alexa, close the blinds”, for closing the blinds of your window. a. What do you think about using your Alexa to control your home appliances? b. Are there any appliances you would want to connect? Why? 14. Were you ever uncomfortable while speaking to the device because there were people around you? Can you please describe a situation when that happened? 15. Privacy: Do you have any privacy or security concerns about using this device? If yes, please describe. [If yes] Have any of these concerns affected how you use the device? 13. Is there anything else you want to share about your Echo Dot or Fire tablet? Study end This is the end of our three-week study, thank you so much for your time and inputs. The Echo Dot and Fire tablet are yours. Can you please sign this receipt saying you have received the study devices as a compensation? 113 6. Interview codebook 1. Change in perception with time (usefulness, willingness and confidence of use) 2. Desired features/limitations/challenges 3. Device location 4. Device usage- Echo (includes) Activities used for [not necessary, will make a note at the end] Confidence of using the device Device interaction (voice-based interaction) Finds device enjoyable Finds device entertaining Finds device useful (includes how they did that before) Getting used to the device New things found about device or willingness to find new things Thoughts about visual screen Willingness to use 5. Echo affecting use of other technology at home 6. Entertainment a. Current entertainment practices/devices b. Improvement over current method/device for entertainment c. If not used, reason for not using d. Things they actually do with the device e. Things they would want to do with the device for entertainment (mentioned in beginning) 7. Forgetting the capabilities of the device/tablet (e.g., forgetting things which was discussed before) 8. Improvement over current technology/currently used method a. Accessibility b. Comparison with current tech (e.g., more confident of using technology compared to other tech) c. Interaction/Interface- easy than computer etc. d. Other 9. Independence (doing activities for which they were dependent on someone or could not do at all) 10. Information Finding a. Actual usage--what info they find using it 114 b. Current (i.e., before using Echo) information finding ways (also includes the problems encountered currently while finding info) c. Improvement over previous information finding methods d. Other e. Possible improvements f. Things people want to find out using this device g. Thoughts-- educational, encyclopedia etc. 11. Initial thoughts about the device a. Device interaction b. Device will be useful c. Difference from traditional tech d. Possible challenges e. Other 12. Memory aid a. Current (i.e., before using Echo) used memory aids and current situation b. Improvement over previous memory aids/methods c. Reason if not used d. Things people want to use it for supporting memory in beginning or think it would be helpful for e. Things they actually use it for to support memory f. What can be done to make it a better memory aid/where it cannot support memory 13. Other 14. Other people - social ecosystem of tech usage a. Older adults bragging or showing this device to others b. Others depending/thinking of asking them for information c. Speaking in front of other people d. Other 15. Overall thoughts about tech 16. Talking to device as person/Personification a. Companion b. Conversation c. Embodiment 17. Preference for using existing/previously used tech or methods 18. Preference for visual interface 19. Previous knowledge about this technology 115 20. Privacy 21. Skills a. Food and drinks b. Games (includes previous games played) c. Info finding- This day in history, Kayak etc. d. News e. Other enabled skills f. Relaxation 22. Smart home 23. Tablet use 24. Tech usage (includes) Eagerness to learn about technology Feeling of not knowing much about technology Is scared / fearful of messing things while using technology Negative effect of tech on society Not interested in trying new technology Other Takes help from others while using tech Trust issues with technology 116 Bibliography 1. Amr Alanwar, Bharathan Balaji, Tian Yuan, Shuo Yang, and Mani Srivastava. 2017. EchoSafe : Sonar-based Verifiable Interaction with Intelligent Digital Agents. In Proceedings of the 1st ACM Workshop on the Internet of Safe Things, SafeThings’17, 38–43. 2. Eric T. Anderson and Duncan I. Simester. 2014. Reviews Without a Purchase: Low Ratings, Loyal Customers, and Deception. Journal of Marketing Research 51, 3: 249–269. https://doi.org/10.1509/jmr.13.0209 3. Lisa Anthony, Yoojin Kim, and Leah Findlater. 2013. Analyzing User- Generated YouTube Videos to Understand Touchscreen Use by People with Motor Impairments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), 1223–1232. https://doi.org/10.1145/2470654.2466158 4. Vikas Ashok, Yevgen Borodin, Svetlana Stoyanchev, Yuri Puzis, and I. V. Ramakrishnan. 2014. Wizard-of-Oz evaluation of speech-driven web browsing interface for people with vision impairments. In Proceedings of the 11th Web for All Conference (W4A ’14), 1–9. https://doi.org/10.1145/2596695.2596699 5. Shiri Azenkot and Nicole B. Lee. 2013. Exploring the use of speech input by blind people on mobile devices. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’13), 1–8. https://doi.org/10.1145/2513383.2513440 6. James P. Barber and Kelley K. Walczak. 2009. Conscience and Critic : Peer Debriefing Strategies in Grounded Theory Research. In Paper presented at the 117 Annual Meeting of the American Educational Research Association, San Diego, California, April 13-17, 2009. 7. Sara Basson, Peter G. Fairweather, and Vicki L. Hanson. 2007. Speech recognition and alternative interfaces for older users. Interactions - Designing for seniors: innovations for graying times 14 Issue 4, 26–29. https://doi.org/10.1145/1273961.1273980 8. Ann Bowling, Matthew Hankins, Gill Windle, Claudio Bilotta, and Robert Grant. 2013. A short measure of quality of life in older age: The performance of the brief Older People’s Quality of Life questionnaire (OPQOL-brief). Archives of Gerontology and Geriatrics 56, 1: 181–187. https://doi.org/10.1016/j.archger.2012.08.012 9. Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Pyschology 3, 2: 77–101. 10. Robin Brewer, Raymundo Cornejo Garcia, Tedmond Schwaba, Darren Gergle, and Anne Marie Piper. 2016. Exploring Traditional Phones as an E-Mail Interface for Older Adults. ACM Transactions on Accessible Computing 8, 2: 1–20. https://doi.org/10.1145/2839303 11. Robin N Brewer and Anne Marie Piper. 2017. xPress : Rethinking Design for Aging and Accessibility through a Voice-based Online Blogging Community. Proceedings of the ACM on Human-Computer Interaction 1, CSCW, November 2017 Article No. 26. https://doi.org/10.1145/3139354 12. A J Bernheim Brush, Bongshin Lee, Ratul Mahajan, Sharad Agarwal, Stefan Saroiu, and Colin Dixon. 2011. Home Automation in the Wild: Challenges and 118 Opportunities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11), 2115–2124. https://doi.org/10.1145/1978942.1979249 13. Erin Buehler, Stacy Branham, Abdullah Ali, Jeremy J. Chang, Megan Kelly Hofmann, Amy Hurst, and Shaun K. Kane. 2015. Sharing is Caring: Assistive Technology Designs on Thingiverse. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15), 525–534. https://doi.org/10.1145/2702123.2702525 14. Zoraida Callejas and Ramón López-Cózar. 2009. Designing smart home interfaces for the elderly. ACM SIGACCESS Accessibility and Computing, 10– 16. https://doi.org/10.1145/1651259.1651261 15. Clare Carroll, Catherine Chiodo, Xin Adena Lin, Meg Nidever, and Jayanth Prathipati. 2017. Robin : Enabling Independence For Individuals With Cognitive Disabilities Using Voice Assistive Technology. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI ’17), 46–53. https://doi.org/10.1145/3027063.3049266 16. Shomir Chaudhuri, Thai Le, Cathy White, Hilaire Thompson, and George Demiris. 2013. Examining health information-seeking behaviors of older adults. CIN - Computers Informatics Nursing 31, 11: 547–553. https://doi.org/10.1097/01.NCN.0000432131.92020.42 17. Eric Corbett and Astrid Weber. 2016. What can I say? Addressing User Experience Challenges of a Mobile Voice User Interface for Accessibility. In Proceedings of the 18th International Conference on Human-Computer 119 Interaction with Mobile Devices and Services (MobileHCI ’16), 72–82. https://doi.org/10.1145/2935334.2935386 18. George Demiris, Marilyn J Rantz, Myra A Aud, Karen D Marek, Harry W Tyrer, Marjorie Skubic, and Ali A Hussam. 2004. Older adults’ attitudes towards and perceptions of “smart home” technologies: a pilot study. Medical Informatics and the Internet in Medicine 29, 2: 87–94. https://doi.org/10.1080/14639230410001684387 19. Jan Derboven, Jonathan Huyghe, and Dirk De Grooff. 2014. Designing voice interaction for people with physical and speech impairments. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI ’14), 217–226. https://doi.org/10.1145/2639189.2639252 20. Mari Carmen Domingo. 2012. An overview of the Internet of Things for people with disabilities. Journal of Network and Computer Applications 35, 2: 584–596. https://doi.org/10.1016/j.jnca.2011.10.015 21. Stefania Druga, Randi Williams, Cynthia Breazeal, and Mitchel Resnick. 2017. “Hey Google is it OK if I eat you ?” Initial Explorations in Child-Agent Interaction. In Proceedings of the 2017 Conference on Interaction Design and Children (IDC ’17), 595–600. https://doi.org/10.1145/3078072.3084330 22. Aarthi Easwara Moorthy and Kim-Phuong L Vu. 2015. Privacy concerns for use of Voice Activated Personal Assistant in the public space. International Journal of Human-Computer Interaction 31, 4: 307–335. https://doi.org/10.1080/10447318.2014.986642 120 23. Nelson. Edewor, Osaze Patrick. Ijiekhuamhen, and Uche P. Emeka-Ukwu. 2016. Elderly people and their information needs. Library Philosophy and Practice 2016, 1. 24. Susan Koch Fager. 2017. Speech Recognition as a Practice Tool for Dysarthria. Seminars in Speech and Language 38, 3: 220–228. https://doi.org/10.1055/s-0037-1602841 25. Anushay Furqan, Chelsea Myers, and Jichen Zhu. 2017. Learnability through Adaptive Discovery Tools in Voice User Interfaces. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI’17), 1617–1623. https://doi.org/10.1145/3027063.3053166 26. Piedad Garrido, Francisco J. Martinez, and Christian Guetl. 2010. Adding semantic web knowledge to Intelligent Personal Assistant Agents. In CEUR Workshop Proceedings. 27. KAREN GRACE-MARTIN. An Easy Way to Reverse Code Scale items. Retrieved from http://www.theanalysisfactor.com/easy-reverse-code/ 28. Stacey Gray. 2016. Always on: Privacy implications of microphone-enabled devices. April: 10. Retrieved from https://fpf.org/wp- content/uploads/2016/04/FPF_Always_On_WP.pdf 29. Eric Griffith. 2017. The Best Amazon Alexa Skills. Retrieved December 6, 2017 from https://www.pcmag.com/article/352136/the-best-amazon-alexa- skills 30. Matt Hanson and Jon Porter. 2018. The best Alexa skills and commands: The ultimate Amazon Echo tips and tricks. Retrieved December 6, 2017 from 121 https://www.techradar.com/how-to/the-best-amazon-echo-alexa-skills-and- how-to-install-them 31. Susumu Harada, Jacob O. Wobbrock, and James a. Landay. 2007. Voicedraw: A Hands-Free Voice-Driven Drawing Application for People with Motor Impairments. In Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility (ASSETS ’07), 27. https://doi.org/10.1145/1296843.1296850 32. Susumu Harada, Jacob O. Wobbrock, Jonathan Malkin, Jeff A. Bilmes, and James A. Landay. 2009. Longitudinal study of people learning to use continuous voice-based cursor control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), 347–356. https://doi.org/10.1145/1518701.1518757 33. Panu Harmo, Tapio Taipalus, Jere Knuuttila, José Vallet, and Aarne Halme. 2005. Needs and solutions - Home automation and service robots for the elderly and disabled. In 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ’05), 2721–2726. https://doi.org/10.1109/IROS.2005.1545387 34. Mark S. Hawley. 2002. Speech recognition as an input to electronic assistive technology. British Journal of Occupational Therapy 65, 15–20. 35. Julia Himmelsbach, Markus Garschall, Sebastian Egger, Susanne Steffek, and Manfred Tscheligi. 2015. Enabling accessibility through multimodality? In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia - MUM ’15, 195–199. https://doi.org/10.1145/2836041.2836060 122 36. Daniel J. Hruschka, Deborah Schwartz, Daphne Cobb St.John, Erin Picone- Decaro, Richard A. Jenkins, and James W. Carey. 2004. Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research. Field Methods 16, 3: 307–331. https://doi.org/10.1177/1525822X04266540 37. Shaun K Kane, Meredith Ringel Morris, Ann Paradiso, and Jon Campbell. 2017. “At times avuncular and cantankerous, with the reflexes of a mongoose”: Understanding Self-Expression through Augmentative and Alternative Communication Devices. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’17), 1166–1179. https://doi.org/10.1145/2998181.2998284 38. Sumi Kim and Heekyung Choi. 2014. Still Hungry for Information: Information Seeking Behavior of Senior Citizens in South Korea. iConference 2014 Proceedings. https://doi.org/10.9776/14300 39. Bret Kinsella. 2017. Gartner Predicts 75% of US Households will Have Smart Speakers by 2020. Voicebot, 1–8. Retrieved from https://www.voicebot.ai/2017/04/14/gartner-predicts-75-us-households-will- smart-speakers-2020/ 40. Matthew L. Lee and Anind K. Dey. 2008. Lifelogging memory appliance for people with episodic memory impairment. In Proceedings of the 10th international conference on Ubiquitous computing - UbiComp ’08, 44. https://doi.org/10.1145/1409635.1409643 41. Irene Lopatovska and Harriet Williams. 2018. Personification of the Amazon Alexa. Proceedings of the 2018 Conference on Human Information 123 Interaction&Retrieval - CHIIR ’18: 265–268. https://doi.org/10.1145/3176349.3176868 42. Ewa Luger and Abigail Sellen. 2016. “Like Having a Really bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 5286–5297. https://doi.org/10.1145/2858036.2858288 43. Michal Luria, Guy Hoffman, and Oren Zuckerman. 2017. Comparing Social Robot, Screen and Voice Interfaces for Smart-Home Control. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 580–592. https://doi.org/10.1145/3025453.3025786 44. Bill Manaris, Valanne Macgyvers, and Michail Lagoudakis. 2002. A Listening Keyboard for Users with Motor Impairments — A Usability Study ∗. International Journal of Speech Technology 5, 4: 371–388. 45. TAYLOR MARTIN. 2016. 50 most useful Alexa skills. Retrieved December 6, 2017 from https://www.cnet.com/how-to/amazon-echo-most-useful-alexa- skills/ 46. Alex Mihailidis, Amy Cockburn, Catherine Longley, and Jennifer Boger. 2008. The acceptability of home monitoring technology among community-dwelling older adults and baby boomers. Assistive technology 20, 1: 1–12. https://doi.org/10.1080/10400435.2008.10131927 47. P. Milhorat, S. Schlogl, G. Chollet, J. Boudy, A. Esposito, and G. Pelosi. 2014. Building the next generation of personal digital assistants. In 2014 1st International Conference on Advanced Technologies for Signal and Image 124 Processing, ATSIP 2014, 458–463. https://doi.org/10.1109/ATSIP.2014.6834655 48. Maia Naftali and Leah Findlater. 2014. Accessibility in context: understanding the truly mobile experience of smartphone users with motor impairments. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility (ASSETS ’14), 209–216. https://doi.org/10.1145/2661334.2661372 49. Stacey F. Nagata and Mark A. Neerincx. 2004. Interaction design concepts for a mobile personal assistant. Proceedings of the conference on Dutch directions in HCI - 65: 9. https://doi.org/10.1145/1005220.1005232 50. Priya Narasimhan, Rajeev Gandhi, and Dan Rossi. 2009. Smartphone-based assistive technologies for the blind. Proceedings of the 2009 international conference on Compilers, architecture, and synthesis for embedded systems: 223–232. https://doi.org/10.1145/1629395.1629427 51. NPR. 2018. The Smart Audio Report. Retrieved from https://www.nationalpublicmedia.com/smart-audio-report/ 52. Gregor Pačnik, Karl Benkič, and Bojan Brečko. 2005. Voice operated intelligent wheelchair - VOIC. In IEEE International Symposium on Industrial Electronics (IEEE ISIE ’05), 1221–1226. https://doi.org/10.1109/ISIE.2005.1529099 53. Rebecca Palmer, Pam Enderby, and Mark Hawley. 2007. Addressing the needs of speakers with longstanding dysarthria: computerized and traditional therapy compared. International Journal of Language & Communication Disorders 42, 125 Sup1: 61–67. https://doi.org/10.1080/13682820601173296 54. George R Peterson-karlan. 2011. Technology to Support Writing by Students with Learning and Academic Disabilities: Recent Research Trends and Findings. Assistive Technology Outcomes and Benefits 7, 1: 39–62. 55. Sam Pinney and James Laird. 2017. Best ABest Alexa Skills: 17 essential skills for your new Amazon Echo speaker. Retrieved December 6, 2017 from http://www.trustedreviews.com/news/alexa-skills-amazon-echo-2946222 56. François Portet, Michel Vacher, Caroline Golanski, Camille Roux, and Brigitte Meillon. 2013. Design and evaluation of a smart home voice interface for the elderly: Acceptability and objection aspects. Personal and Ubiquitous Computing 17, 1: 127–144. https://doi.org/10.1007/s00779-011-0470-5 57. Amanda Purington, Jessie G. Taft, Shruti Sannon, Natalya N. Bazarova, and Samuel Hardman Taylor. 2017. “Alexa is my new BFF”: Social Roles, User Satisfaction, and Personification. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17), 2853–2859. https://doi.org/10.1145/3027063.3053246 58. Frank Rudzicz, Rosalie Wang, Momotaz Begum, and Alex Mihailidis. 2015. Speech Interaction with Personal Assistive Robots Supporting Aging at Home for Individuals with Alzheimer’s Disease. ACM Transactions on Accessible Computing (TACCESS) 7, 2: 1–22. https://doi.org/10.1145/2744206 59. Joao Santos, Joel J. P. C. Rodrigues, Joao Casal, Kashif Saleem, and Victor Denisov. 2016. Intelligent Personal Assistants Based on Internet of Things Approaches. IEEE Systems Journal PP, 99: 1–10. 126 https://doi.org/10.1109/JSYST.2016.2555292 60. Daisuke Sato, Masatomo Kobayashi, Hironobu Takagi, Chieko Asakawa, and Jiro Tanaka. 2011. How Voice Augmentation Supports Elderly Web Users. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, 155–162. https://doi.org/10.1145/2049536.2049565 61. S. Schlögl, G. Chollet, M. Garschall, M. Tscheligi, and G. Legouverneur. 2013. Exploring voice user interfaces for seniors. Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments - PETRA ’13: 1–2. https://doi.org/10.1145/2504335.2504391 62. Ben Shneiderman. 2000. The limits of speech recognition. Communications of the ACM 43, 9: 63–65. https://doi.org/10.1145/348941.348990 63. Richard C. Simpson and Simon P. Levine. 2002. Voice control of a powered wheelchair. IEEE Transactions on Neural Systems and Rehabilitation Engineering 10, 2: 122–125. https://doi.org/10.1109/TNSRE.2002.1031981 64. Christoph Stahl and Pascal Laub. 2017. Maintaining multiple sclerosis patients’ quality of life – a case study on environment control assistance in a smart home. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’17), 83–86. https://doi.org/10.1145/3056540.3064943 65. Susan Stanziano. 2016. Information seeking behavior of older adults. Serials Librarian 71, 3–4: 221–230. https://doi.org/10.1080/0361526X.2016.1230532 66. Greg Sterling. 2018. Survey_ Smart speaker ownership steals time from 127 smartphones, TV, radio. Retrieved from https://searchengineland.com/survey- smart-speaker-ownership-steals-time-smartphones-tv-radio-289916 67. Versha Thakur. 2016. People ’ s perception of Smart Home devices in their Homes & the factors which influence this perception. State University of New York (SUNY) Polytechnic Institute at Utica/Rome. 68. Matthieu Tixier and Myriam Lewkowicz. 2015. Looking for Respite and Support : Technological Opportunities for Spousal Caregivers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15), 1155–1158. https://doi.org/10.1145/2702123.2702563 69. Michel Vacher, Benjamin Lecouteux, Dan Istrate, Thierry Joubert, François Portet, Mohamed Sehili, and Pedro Chahuara. 2013. Experimental Evaluation of Speech Recognition Technologies for Voice-based Home Automation Control in a Smart Home. In 4th Workshop on Speech and Language Processing for Assistive Technologies, 99–105. 70. Eleftheria Vaportzis, Maria Giatsi Clausen, and Alan J. Gow. 2017. Older adults perceptions of technology and barriers to interacting with tablet computers: A focus group study. Frontiers in Psychology 8, OCT. https://doi.org/10.3389/fpsyg.2017.01687 71. Amber Wagner, Ramaraju Rudraraju, Srinivasa Datla, Avishek Banerjee, Mandar Sudame, and Jeff Gray. 2012. Programming by voice: a hands-free approach for motorically challenged children. In Proceedings of the 2012 ACM annual conference extended abstracts on Human Factors in Computing Systems Extended Abstracts (CHI EA ’12), 2087–2092. 128 https://doi.org/10.1145/2212776.2223757 72. Bruce N Walker, Jeffrey Lindsay, Amanda Nance, Yoko Nakano, K Palladino, Tilman Dingler, and Myounghoon Jeon. 2011. Spearcons ( Speech-Based Earcons ) Improve Navigation Performance in Advanced Auditory Menus. Human Factors 55, 1: 157–182. https://doi.org/10.1177/0018720812450587 73. Krzysztof Willman. 2017. 20 Best Alexa Skills for Amazon Echo. Retrieved December 6, 2017 from https://turbofuture.com/consumer-electronics/The-20- Best-Amazon-Echo-Skills-in-the-Alexa-App 74. Maria Klara Wolters, Fiona Kelly, and Jonathan Kilgour. 2016. Designing a spoken dialogue interface to an intelligent cognitive assistant for people with dementia. Health informatics journal 22, 4: 854–866. https://doi.org/10.1177/1460458215593329 75. Ya-Huei Wu, Souad Damnée, Hélène Kerhervé, Caitlin Ware, and Anne- Sophie Rigaud. 2015. Bridging the digital divide in older adults: a study from an initiative to inform older adults about new technologies. Clinical interventions in aging 10: 193–200. https://doi.org/10.2147/CIA.S72399 76. Linda Wulf, Markus Garschall, Julia Himmelsbach, and Manfred Tscheligi. 2014. Hands free-Care free: Elderly people taking advantage of speech-only interaction. In Proceedings of the NordiCHI 2014: The 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, 203–206. https://doi.org/10.1145/2639189.2639251 77. Nicole Yankelovich. 1996. How Do Users Know What to Say? Interactions 3, 32–43. https://doi.org/10.1145/242485.242500 129 78. Hanlu Ye, Meethu Malu, Uran Oh, and Leah Findlater. 2014. Current and Future Mobile and Wearable Device Use by People With Visual Impairments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), 3123–3132. https://doi.org/10.1145/2556288.2557085 79. Eric Zeng, Shrirang Mare, and Franziska Roesner. 2017. End User Security & Privacy Concerns with Smart Homes. In Symposium on Usable Privacy and Security (SOUPS ’17). 80. Amazon Echo. Retrieved from https://en.wikipedia.org/wiki/Amazon_Echo 81. VoiceOver for Mac: VoiceOver overview. Retrieved September 14, 2017 from https://support.apple.com/kb/PH22549?locale=en_US 82. JAWS Screen Reader - Best in Class. Retrieved September 14, 2017 from http://www.freedomscientific.com/Products/Blindness/JAWS 83. Dragon NaturallySpeaking - world’s best-selling speech recognition software | Nuance. Retrieved September 14, 2017 from https://www.nuance.com/dragon.html 130