ABSTRACT Title of Dissertation: Designing Multimodal Touchscreen Interactions For Accessible Data Visualization Supporting Blind Users Venkata Sai Pramod Chundury Doctor of Philosophy, 2025 Dissertation Directed by: Dr. Jonathan Lazar Professor, College of Information Data visualization can be a democratizing force for providing advanced data analysis tools and capabilities to everyday users. However, data visualization also creates barriers for blind and low-vision (BLV) individuals, a fact that has long been recognized in the accessibility research community. Assistive technologies such as tactile graphics, data sonification (using audio to convey data), and refreshable tactile displays (RTDs) can be utilized to lower the accessibility barriers to data visualization. Yet it is only recently that visualization research has realized this fact. Since this realization, there have been various efforts from the visualization community, such as data-centric alt-text, accessible tables, and richer screen reader experiences. However, the visualization community, which arguably is best poised to tackle these challenges, has so far only scratched the surface of creating rich human-data interactions for blind individuals. Commercial touchscreen devices such as smartphones and tablets now have in-built ac- cessibility features, and are thus increasingly being adopted by blind individuals. These devices are also well-suited to support direct data manipulation through touch interactions. I adopted a mixed-methods approach to design multimodal (audio and haptic) chart representations and interactions that leverage such touchscreen devices by conducting four studies. The first study involved semi-structured interviews with Ten Orientation and Mobility (O&M) experts who train BLV individuals in non-visual navigation and spatial understanding. The goal was to derive design principles for effective non-visual data interaction. Findings em- phasized the usefulness of crossmodal sensory substitution (CMSS)—a strategy where tactile interactions are paired with sonification to enhance spatial awareness. Participants highlighted that BLV individuals have diverse preferences for sensory modalities, necessitating personalized multimodal experiences that cater to different skill levels and cognitive strategies. These insights informed the design of an accessible data visualization system. The second study explored the lived experiences of BLV professionals in data-related fields through a two-step online survey. Responses from BLV individuals engaged in data analysis re- vealed persistent accessibility barriers at multiple stages of the data workflow—including data loading, transformation, analysis, and visualization authoring. Despite expertise in programming (e.g., Python, R, and SAS) and GUI-based tools (e.g., Excel), participants reported substantial reliance on assistive technologies, often requiring sighted colleagues’ assistance to interpret vi- sualizations. These findings highlight the need for “born accessible” tools that allow independent and efficient data exploration without requiring external support. The third study introduces TactualPlot, a multimodal data interaction system that leverages CMSS principles to enable blind users to explore data through touch and sound on touchscreen devices. TactualPlot was developed through an iterative participatory design process involving a blind collaborator who provided feedback on early prototypes. The system supports scatter- plots, bar charts, line graphs, and pie charts, allowing users to explore data through multi-finger touch gestures combined with audio cues and spatial feedback. Unlike traditional sonification approaches, TactualPlot employs direct touch interactions (similar to tactile exploration) to guide users through high-level data trends before enabling deeper exploration. The final study presents an empirical evaluation comparing TactualPlot to other accessi- bility solutions, including screen readers (Olli) and refreshable tactile displays (Monarch). Ten blind participants, recruited from blind individuals working in data-intensive fields, performed data analysis tasks of varying complexity across multiple visualization types. The study assessed task correctness, completion times, and user preferences, revealing that hybrid approaches com- bining touch and sound were preferred over uni-modal (audio-only or tactile-only) solutions. Novel multi-line braille displays such as the Monarch offer features that can combine both touch- screen interactions and haptic feedback. To better understand how blind individuals can create charts for and use RTDs in the future, I also conducted a 3-hour long co-design session with a blind participant, providing insights into how blind users conceptualize and create tactile-based charts. This dissertation contributes to accessible data visualization research by demonstrating the effectiveness of multimodal (touch-audio) interactions and highlighting new design opportunities for refreshable tactile displays. The findings provide practical guidelines for creating “born ac- cessible” data tools for BLV individuals in data-intensive fields. By integrating touch, sound, and personalized interaction techniques, this work helps with the creation of next-generation acces- sible visualization systems, empowering BLV individuals to engage with data independently and effectively. DESIGNING MULTIMODAL TOUCHSCREEN INTERACTIONS FOR ACCESSIBLE DATA VISUALIZATION SUPPORTING BLIND USERS by Venkata Sai Pramod Chundury Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2025 Advisory Committee: Dr. Jonathan Lazar, Chair/Advisor Dr. Niklas Elmqvist, Co-chair Dr. Ramani Duraiswami, Dean’s Representative Dr. Stephanie Valencia Dr. Zhicheng Liu © Copyright by Venkata Sai Pramod Chundury 2025 Acknowledgments This dissertation marks the end of a long and challenging journey for me. During my time as a doctoral student, I have had the opportunity to learn from and collaborate with many inspiring scholars. My advisors, Dr. Jonathan Lazar and Dr. Niklas Elmqvist, have consistently supported me and my ideas and have shaped me into a more confident researcher. Together, they made sure that I did not fly “into the sun”—and when I inevitably flew a little too close, they guided me back with patience, and the right mix of support and advice. They supported me through every professional challenge I faced during my PhD, and I am deeply grateful for that. I would like to thank them for all the opportunities they provided and for sharing their knowledge, wisdom, jokes, memes, and more along the way. I would also like to thank my committee members—Dr. Ramani Duraiswami, Dr. Stephanie Valencia, and Dr. Zhicheng Liu—for their time, thoughtful insights, and feedback on the dissertation. Additionally, I would like to thank Dr. Amanda Lazar for her feedback on my integrative paper and dissertation proposal. I will always be grateful to Dr. Marshini Chetty for giving me my first research opportunity during my master’s. That experience sparked my interest in research, and set me on the path that led to this dissertation. The Accessible Visualization Research Team has been an incredible source of support throughout my research journey. They have helped me ask better questions, scope my projects more thoughtfully, design stronger studies, and—most importantly—ground my work in the lived experiences and needs of the Blind community. I have truly enjoyed working with this passionate group of research collaborators: Yasmin Reyazuddin, Biswaksen Patnaik, Christine Tang, Naimul ii Hoque, Urja Thakkar, Enric Jiao, and Dr. J. Bern Jordan. I am especially grateful to Yasmin Reyazuddin, who graciously offered her time, expertise, and perspective for all of my research studies. Her insights and lived experiences brought depth and meaning to this work, and I am deeply thankful for her support and collaboration. I have also been fortunate to be part of the vibrant Human-Computer Interaction Lab (HCIL) at the University of Maryland. The lab has been an intellectually rich and welcoming space, and I have benefited greatly from conversations with and feedback from its talented students, faculty, and staff. I want to thank my family and friends for their constant support over the years. My mom, dad, and brother have been my pillars of strength, always offering a safety net and encouraging me to pursue my goals. Their love, patience, and belief in me have carried me through this journey. I would like to dedicate this work to my family. My tribe of friends has been the source of my sanity during the challenging times, and I am thankful to them: Abhilash, Ajay, Akhil, Akshita, Alisha, Biswaksen, Sriram, Tameem, Teja, Utkarsh, Venkat, Vikram, Vinod, and Vishnu. A special thanks to Akshita and Vishnu for provid- ing early feedback on my dissertation. I am thankful to the following people for being my harsh- est critics and biggest supporters—without whom I could not have stayed motivated through the highs and lows of this journey: Mukil, Prashanth, Sophie, Victoria, and Vijay. My dissertation work was partly supported by grant IIS-2211628 from the U.S. National Science Foundation, Villum Investigator grant VL-54492 from Villum Fonden, and a grant from the Maryland State Department of Education (MSDE). I’m deeply grateful to all my study par- ticipants who shared their time and feedback. I also thank the National Federation of the Blind (NFB) for their support with participant recruitment, and the American Printing House (APH) for loaning us the Monarch device for the comparison study. iii Table of Contents Preface ii Acknowledgements ii Table of Contents iv List of Tables vii List of Figures viii List of Abbreviations xi Chapter 1: Introduction 1 1.1 Research Overview and Approach . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Dissertation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 2: Related Work 11 2.1 Mental and Spatial Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Sensory and Multimodal Substitution for Blind Users . . . . . . . . . . . . . . . 12 2.3 Sound Perception and Sonification . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.4 Chart Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5 Sonifying Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.6 Touching Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.7 Multimodal Data Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Chapter 3: Understanding Sensory Substitution 22 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2.1 Orientation and Mobility (O&M) Training . . . . . . . . . . . . . . . . . 25 3.2.2 Study Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2.4 Data Collection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . 28 3.3 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.1 Perceiving Space as a Blind Individual . . . . . . . . . . . . . . . . . . . 31 3.3.2 Interacting with Space as a Blind Individual . . . . . . . . . . . . . . . . 38 3.3.3 Effectiveness, Information Access, and Usage . . . . . . . . . . . . . . . 39 3.3.4 Towards Accessible Charts: Needs and Challenges . . . . . . . . . . . . 43 iv 3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.4.1 Design Considerations for Accessible Visualization . . . . . . . . . . . . 48 3.4.2 Example: Accessible Bar and Pie charts . . . . . . . . . . . . . . . . . . 52 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Chapter 4: Understanding the Visualization and Analytics Needs of Blind and Low- Vision Professionals 55 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.2 Survey Methodology and Findings . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.2.1 Data Analysis Goals, Types, and Tools . . . . . . . . . . . . . . . . . . . 58 4.2.2 Collaboration Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2.3 Authoring Data Visualizations . . . . . . . . . . . . . . . . . . . . . . . 61 4.2.4 Accessible Visualization Practices and Challenges . . . . . . . . . . . . . 62 4.3 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Chapter 5: Design space for Crossmodal Sensory Substitution 66 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.2 Design Space: Crossmodal Substitution . . . . . . . . . . . . . . . . . . . . . . 68 5.3 The TactualPlot Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 5.3.1 Spatial Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 5.3.2 Interaction Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.3 Continuously Touching Data . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.4 Data Sonification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.5 Edge Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.3.6 Zooming and Details-on-Demand . . . . . . . . . . . . . . . . . . . . . 76 5.3.7 Beyond Scatterplots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.4 Formative Design Assessment: TactualPlot . . . . . . . . . . . . . . . . . . . . . 77 5.4.1 Design Probes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.4.2 Participatory Design Sessions . . . . . . . . . . . . . . . . . . . . . . . 79 5.4.3 Dataset and Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 5.4.4 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 5.4.5 Design Session 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.4.6 Design Session 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.4.7 Design Session 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.5 Design Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.5.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.5.2 Method and Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 5.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.6.1 Improving crossmodal sensory substitution . . . . . . . . . . . . . . . . 93 5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Chapter 6: Sound, Touch, or the full Monty: a comparison study of multimodal acces- sibility 96 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 6.2 Design Space: Multimodal Assistive Technologies for Representing Data . . . . 97 v 6.2.1 Definition: Multimodal Data Access . . . . . . . . . . . . . . . . . . . . 98 6.2.2 TactualPlot: A Crossmodal Data Access Technique . . . . . . . . . . . . 101 6.2.3 Olli: screen reader accessibility solution that uses audio only . . . . . . . 112 6.2.4 Monarch: multi-line refreshable braille display that uses touch only . . . 119 6.3 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.3.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.3.3 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 6.3.4 Dataset, charts and tasks . . . . . . . . . . . . . . . . . . . . . . . . . . 128 6.3.5 Experimental Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.3.6 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 6.3.7 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6.3.8 Data Metrics and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6.4.1 Task Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.4.2 Task Completion Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 6.4.3 Subjective Ratings: NASA TLX . . . . . . . . . . . . . . . . . . . . . . 140 6.4.4 Qualitative Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 6.5.1 Sound, touch, or the Full Monty? . . . . . . . . . . . . . . . . . . . . . . 147 6.5.2 Towards multimodal chart accessibility . . . . . . . . . . . . . . . . . . 148 6.6 Co-designing tactile graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 6.6.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 6.6.2 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 6.7 Design Guidelines for Authoring Accessible Multimodal Visualizations . . . . . 163 6.7.1 Perceivable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 6.7.2 Operable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 6.7.3 Understandable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 6.7.4 Robust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 6.8 Limitations and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Chapter 7: Conclusion and Future work 170 7.1 Key Findings and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 170 7.1.1 Understanding sensory substitution and data analysis practices of blind individuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 7.1.2 TactualPlot: Crossmodal Sensory Substitution . . . . . . . . . . . . . . . 171 7.1.3 Comparative Study of Modalities . . . . . . . . . . . . . . . . . . . . . . 172 7.1.4 Co-designing Tactile Graphics . . . . . . . . . . . . . . . . . . . . . . . 172 7.1.5 Design Guidelines for Accessible Multimodal Visualizations . . . . . . . 172 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Bibliography 175 vi List of Tables 3.1 Demographics and O&M experience of the 10 participants. Abbreviations: M—Male; F—Female; NB—Non-binary. . . . . . . . . . . . . . . . . . . . . . 28 5.1 Task types. List of task types and corresponding question structures for our user study. Each trial corresponded to a given task sub-type. . . . . . . . . . . . . . . 81 5.2 Performance comparison. Comparing Tactile Graphics and TactualPlot for the Numerosity (N) task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.3 Expert review participants. Demographics and experience of the participants. . 91 6.1 Comparison of Different Chart Characteristics and Interaction Modes . . . 100 6.2 Participant demographics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 6.3 Technology and data analysis experience. . . . . . . . . . . . . . . . . . . . . . 127 6.4 Comparison of mean task correctness for a combination of Device Types (DT) across Visual Representations (VR) and Task Types (TT). Task correct- ness values are reported as percentages, and we also show the no. of correct tasks for each condition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 6.5 Comparison of mean completion times for a combination of Device Types (DT) across Visual Representations (VR) and Task Types (TT). The comple- tion time is measured in seconds (s). . . . . . . . . . . . . . . . . . . . . . . . . 139 vii List of Figures 3.1 Navigating outdoor space using non-visual senses. Blind individuals build mental maps (visual thinking) by sensing environmental sounds, and use their cane for performing echolocation and perceiving haptic feedback; in other words, a combination of sound and touch towards non-visual sensemaking of spatial con- cepts. Our work explores how these capabilities can be used for accessible data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.1 Designing the TactualPlot technique. (A) Illustration of the TactualPlot tech- nique with its continuous and discrete touch actions and regions. (B) Our imple- mentation of the technique that was evaluated in formative design session 3 and design review sessions. (C) Tactile graphics aligned to scale, and overlaid on an Apple iPad to collect touch information. . . . . . . . . . . . . . . . . . . . . . 70 5.2 Tactile graphics. Scatterplots implemented using American Thermoform swell touch paper to match the dimension of the D3.js visualization in the TactualPlot system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.3 Design Sessions 1 and 2. Our Blind collaborator interacting with a low-fidelity tactile graphic using both hands and multiple fingers. . . . . . . . . . . . . . . . 82 5.4 Design Session 3. Our Blind collaborator interacting with the final version of our TactualPlot system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.5 Lateral scanning. Left: Initial one finger exploration that starts at the origin, and moves laterally across the screen. Right: Final touch trail at the end of the trial. . 85 5.6 Two-finger lateral scanning. Left: Exploration started with two fingers being placed on the origin Right: Final touch trail where all the data points have been sampled. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5.7 Two-finger vertical scanning. Left: Exploration started with two fingers being placed on the origin and moved vertically and then towards the right. Right: All the data points have been sampled, and 3 unexplored regions. . . . . . . . . . . 87 5.8 Comparison of touch behavior. Touch interaction logs for TactualPlot and tac- tile graphics (TG). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 6.1 TactualPlot interactions. An overview of the TactualPlot system’s interactive techniques for line charts. Panels (A–F) illustrate key gestures such as dragging to continuously sonify data, tapping to trigger verbalization of data values, and multi-finger actions for filtering and zooming. These interactions enable blind users to explore and interpret visual data through a combination of touch and auditory feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 viii 6.2 TactualPlot—Pie Charts. Users can drag their finger along pie segments to sonify proportional values and tapping actions that verbalize labels, with auditory cues (earcons) signaling transitions between slices for improved spatial aware- ness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 6.3 TactualPlot—Bar Charts. Users can drag across bar segments to receive soni- fied representations of quantitative values—where pitch maps to bar height; and tap to obtain precise numerical feedback. . . . . . . . . . . . . . . . . . . . . . . 109 6.4 TactualPlot—Scatterplots. Continuous finger dragging triggers real-time au- ditory feedback based on point density and distribution, while tapping provides verbal summaries of the data within a given region, facilitating multi-series com- parisons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 6.5 Olli interactions. This figure has been adapted from Zong et al. [1] A chart is broken down into levels—from an overall summary to axis encodings, cate- gorical intervals, and individual data points; enabling structured, audio-guided exploration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 6.6 Olli—Pie Charts. Panels (A–C) show the progression from a top-level summary to detailed descriptions of individual pie slices, supporting an screen reader ex- ploration of categorical proportions. . . . . . . . . . . . . . . . . . . . . . . . . 117 6.7 Olli—Bar charts. Panels (A–C) illustrate a hierarchical approach—from chart summary to axis intervals and detailed data points—facilitating navigation through both simple and stacked bar charts using auditory feedback. . . . . . . . . . . . 117 6.8 Olli—line charts. Panels (A–C) detail the breakdown from a high-level summary to specific X and Y-axis intervals, allowing users to drill down into trends and compare multiple data series. . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.9 Olli—Scatterplots. Panels (A–C) guide the user from an overall chart summary to detailed intervals that capture the distribution and density of data points across different categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 6.10 Monarch Interactions. Panels (A–B) highlight key features such as the Braille keyboard, tactile display, and navigation buttons that enable blind users to access both static text and graphical content through touch. . . . . . . . . . . . . . . . 119 6.11 Monarch—Pie Charts. Panels (A-C) show a pie chart adapted for the Monarch device, where each segment is annotated with an external Braille label connected by a guiding line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 6.12 Monarch—Bar Charts. Panels (A-C) show a simple and stacked bar chart. The tick marks and in-cell labels allow users to identify quantitative differences and categorical groupings through tactile exploration. . . . . . . . . . . . . . . . . . 122 6.13 Monarch—Line Charts. Panels (A–C) show a single and multi-series line charts created for the Monarch. Panel B shows that for multi-series line charts the design embeds different line types (dashed vs. dots) for each data series. . . . . . . . . 123 6.14 Monarch—Scatterplots. Panels (A-C) show the tactile rendering of scatter plots where different shapes (such as circles, squares, and triangles) encode categorical values. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.15 Analysis of correctness (accuracy) across the experimental factors. Results from an analysis of 95% CIs after bootstrapping (R = 1000). . . . . . . . . . . . 133 ix 6.16 Analysis of correctness (accuracy) across a combination of experimental fac- tors. Results from an analysis of 95% CIs after bootstrapping (R = 1000). . . . . 136 6.17 Analysis of mean task completion time across the experimental factors. Re- sults from an analysis of 95% CIs after bootstrapping (R = 1000) . . . . . . . . . 138 6.18 Analysis of mean task completion times. Comparison across a combination of experimental factors—results from an analysis of 95% CIs after bootstrapping (R = 1000) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.19 NASA TLX Ratings. Comparison of mental demand, physical demand, tempo- ral demand, performance, effort, and frustration on a 20-point scale between the three device types. A lower score indicates a better rating. . . . . . . . . . . . . . 141 6.20 Co-design study setup. Panel A and C illustrate the participant’s workstation where charts are iteratively created using a personal laptop and the Monarch. Panel B displays the tactile chart in a visual form in an external monitor connected to the Monarch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.21 Sequence of design iterations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 x List of Abbreviations ADSR Attack Decay Release Sustain AI Artificial Intelligence AISP Audio Information Seeking Principles API Application Programming Interface BLV Blind and Low-Vision CSV Comma-Separated Values GUI Graphical User Interface HCI Human-Computer Interaction HTML Hypertext Markup Language HVAC Heating, Ventilation, and Air Conditioning ICAD International Community for Auditory Display IRB Institutional Review Board JSON JavaScript Object Notation LLM Large Language Model ML Machine Learning NASA TLX NASA Task Load Index NFB National Federation of the Blind NOMC National Orientation and Mobility Certification O&M Orientation and Mobility OCR Optical Character Recognition PDF Portable Document Format PNG Portable Network Graphics RTD Refreshable Tactile Display SCSS Simultaneous Crossmodal Sensory Substitution SDCT Structured Discovery Cane Travel SQL Structured Query Language STEM Science, Technology, Engineering, and Mathematics TTS Text-to-Speech TVSI Teachers of Students with Visual Impairments UI User Interface UX User Experience WCAG Web Content Accessibility Guidelines xi Chapter 1: Introduction Visualizations are largely inaccessible to individuals who are Blind or have visual impair- ments. Screen readers, the method Blind people most commonly use to transform on-screen text to speech, generally cannot parse pixel visualizations, and few web-based visualizations provide sufficient textual descriptions or the underlying datasets [2]. People with visual impairments are a large population of potential data visualization users. In 2015, globally there were 253 million people with visual impairments, out of whom 26 million were Blind, and this number is estimated to reach around 703 million by the year 2050 [3]. This is also not just a sociotechnical problem, but a potentially legal one; for example, in the United States, Section 508 of the Rehabilitation Act requires that all federal government websites be accessible for people with disabilities, and the Americans with Disabilities Act similarly requires accessibility for most websites of public accommodations [4]. Blind people1 navigate a 3D world of space and objects, and are therefore equally capable of understanding spatial layouts as sighted individuals. However, despite contin- uous advances in visualization research, little effort is devoted to accessibility. While our focus is on Blind individuals, inaccessible visualization practices not only affect Blind people, but also those with other impairments such as motor or cognitive impairments [2, 5]. The visualization community must work to lower barriers for Blind individuals by focus- 1While people-first language is preferred by most people with disabilities, much of the blindness community prefers the use of the term “Blind individuals or Blind people”. In this dissertation, I use both the terms interchange- ably. 1 ing on accessible visualization and data analysis. Additionally, collaboration between sighted and Blind individuals in data-rich social contexts such as work and school might still be challenging due to technology that places sighted and Blind individuals in two different silos. Particularly in STEM fields, spatial and graphical reasoning are invaluable skills that are taught and used by individuals for problem-solving and data analysis. Accessible data visualizations can improve data literacy among Blind individuals, equipping them with practical knowledge of data repre- sentation, interpretation, and analysis. This empowers them to actively engage in data-driven discussions, make informed decisions, and advocate for their interests in our data-centric world. One of the goals of this dissertation is to follow a unified approach to lowering accessibility bar- riers to minimize the need for separate or alternate representations of data visualizations—for Blind and sighted individuals. While there are differences in the needs of Blind and sighted indi- viduals, I believe that fostering collaboration between people with different abilities is essential in educational and professional contexts. Accessibility technologies for Blind users tend to employ a method called sensory substi- tution [6] by conveying data using other senses such as hearing and touch, and even smell [7, 8] instead of vision. Efforts to render visualizations accessible for Blind users have been explored by research communities such as human-computer interaction (HCI), accessibility, and sonification for specific contexts. Of the two most common sensory substitutes—sound and touch—sound is by far the easiest to deploy since it does not require any specialized hardware. While there exist many examples of sonification and auralization [9] (the use of non-verbal and verbal sound), these efforts primarily involve the Blind community only as users or testers, and not as full-fledged in- formants or design partners [10, 11] in the development team. Data visualizations can be made accessible through techniques such as data-centric alt-text [1] or figure captions, screen-reader 2 explorable tables and lists [12], and rich screen reader experiences [13]. Blind individuals can use sensory substitution techniques, such as tactile charts and data sonification, to improve chart accessibility. They also utilize various assistive technologies for non-visual sensemaking, such as screen readers, refreshable Braille displays, and tactile media. The sense of touch is critical for most Blind individuals. Bereft of full use of their vi- sion, Blind people often use their hands to explore unfamiliar objects or their white cane to learn about the world surrounding them. Building on this idea, data physicalization [14] creates tan- gible representations of data to enable a Blind person to feel rather than see the physical shape of a bar chart [15], 3D landscape [16], or node-link graphs [17]. However, data physicaliza- tions that are cheap and accessible, such as thermoform paper and 3D printing, are static and require significant time to produce. More advanced data physicalization techniques that enable interactive feedback—such as shape-changing displays [18], haptic touch displays [19], and re- freshable Braille displays—rely on specialized technology that is expensive and therefore not readily available to an often underemployed Blind audience with limited purchasing power. Fur- thermore, the accessibility field is rife with “silver bullet” technologies that have been abandoned by over-optimistic inventors [2], making investing in experimental devices both costly and risky. While these sophisticated technologies may make data accessible for Blind individuals, adoption at scale is not guaranteed because of the aforementioned limitations. What is readily available, however, are touchscreen devices. Modern smartphones have had a near-revolutionary impact on Blind individuals, putting screen readers into virtually everyone’s pocket [20]. While still costly, smartphones have significant utility for both personal and profes- sional use and thus enjoys widespread adoption—more than 46% [21]—among Blind people, and the number is likely higher for gainfully employed knowledge workers. Touchscreens provide a 3 tangible and interactive interface, allowing Blind users to engage with data in a more immersive manner suited for data exploration. Through tactile exploration and haptic feedback, individuals can perceive patterns, trends, and relationships inherent in data visualizations. This multimodal experience facilitates a deeper understanding of complex information, enabling Blind users to grasp nuances, detect anomalies, and extract meaningful insights that may otherwise be elusive through traditional non-interactive methods, or through text and other narratives. As opposed to keyboard input in desktops and laptops, touchscreens are better suited for direct manipula- tion [22] of data, while still demonstrating comparable computational power. The overarching goal of this dissertation is: How can we design a method for perceiving and interacting with complex datasets on touchscreen devices, allowing Blind users to explore data through multimodal (audio and haptic) feedback—leveraging the same touch-based strategies they use for tactile exploration? 1.1 Research Overview and Approach To design accessible data visualization touchscreen experiences, I adopt a mixed-methods approach to design multimodal (audio and haptic) touchscreen chart representations and interac- tions that could potentially be utilized in touchscreen devices such as smartphones and tablets. Overall, the following high-level research questions are addressed in this dissertation: RQ1: How do Blind individuals perceive and retain sound and touch-based information so that spatial layouts such as charts can be effectively translated into non-visual senses, i.e., sensory substitution? RQ2: How can we effectively employ sensory substitution to support multiple levels (i.e., 4 overview, zoom, filter) of visualization-related tasks on touchscreen devices? RQ3: How can we integrate touchscreen screen reader and multimodal data visualization interactions to support data analysis of multidimensional data and multiple chart types? 1.2 Dissertation Overview CHAPTER 1. In this chapter, I have described the motivation, problem statement, and research questions for the dissertation. I also provide an overview of the dissertation,list the publications from my research, and conclude with a thesis statement. CHAPTER 2. I discuss the relevant background literature on mental and spatial mapping; sen- sory substitution for Blind users; sound perception and sonification; non-visual chart access through sonification, tactile interaction, and multimodal systems. CHAPTER 3. To answer RQ1, I conducted a qualitative study that explored the potential of ac- cessible visualization design by understanding how Blind individuals perceive the world around them using non-visual senses, i.e., sensory substitution. This knowledge enables the design of more efficient audio representations of data that leverage the mechanisms that Blind people al- ready use in their everyday life. My goal was to broadly understand (1) how Blind individuals perceive and retain sound and touch-based information so that visual layouts such as charts can be effectively translated into non-visual senses, and (2) what this means for accessible visualization design so that future tools are robust enough to support multiple levels of visualization-related tasks [23]. To this end, I conducted semi-structured interviews with 10 Orientation and Mobil- ity (O&M) instructors (experts), all of them Blind, to understand how they teach their students 5 (Blind individuals) to navigate physical space using the sense of sound and touch. This particular population was selected because O&M instructors not only have significant personal expertise in leveraging non-visual senses to navigate the physical world everyday life, but also the knowledge and experience of teaching these skills to others. As part of the interviews, I also conducted discussions with these instructors on how to use sound and touch to convey data, and to translate visualizations. I found that Blind individuals who undergo O&M training engage in experiential learning [24]—e.g., hands-on learning followed by reflection—to calibrate their minds to map- ping sounds and tactile feedback to real-world aspects such as size, distance, angles, and position. Individuals actively use echolocation and environmental sounds to make sense of physical space, and the importance of sensory integration (e.g., combining audio and tactile feedback) in under- standing it. Based on these findings, I derived design implications for accessible visualization design, with a focus on audio—speech and non-speech, and tactile representations, feedback and interactions. And finally, I discuss the importance of training and usability of tools for accessible visualization for Blind individuals. Role: My contribution to the research study includes 1) designing the interview script based on the research questions 2) conducting the interviews and the thematic analysis, and 3) authoring the research manuscript. Publication: Chundury, P., Patnaik, B., Reyazuddin, Y., Tang, C., Lazar, J., and Elmqvist, N. (2021). Towards understanding sensory substitution for accessible visualization: An interview study. IEEE transactions on visualization and computer graphics, 28(1), 1084-1094. DOI: 10.1109/TVCG.2021.3114829. 6 https://doi.org/10.1109/TVCG.2021.3114829 CHAPTER 4. I explored the current challenges with accessibility and data analysis of Blind and low-vision (BLV) individuals. I conducted a survey to better understand the user context of Blind individuals working professionally with data and visualizations. I found that Blind and low-vision professionals had challenges with creating accessible charts that can be used by both sighted and Blind individuals. Blind individuals often depended on their sighted colleagues to refine charts; and were limited to using tools (e.g., Microsoft Excel, Python) that were used as part of their current data analysis workflow. Additionally, accessibility issues with tools and processes such as data loading and transformation can lead to further challenges with accessible data visualization during chart creation and communication. Role: My contribution to the research study includes: 1) collaborating with a research team member to design the survey 2) analysis of the survey data, and 3) authoring the research manuscript. Publication: Chundury, P., Thakkar, U., Reyazuddin, Y., Jordan, J. B., Elmqvist, N., and Lazar, J. (2024, October). Understanding the Visualization and Analytics Needs of Blind and Low- Vision Professionals. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 1-5). DOI: 10.1145/3663548.3688496 CHAPTER 5. To answer RQ2, I defined a design space for simultaneous crossmodal sensory substitution (SCSS). We call this approach crossmodal substitution because it replaces the ex- pected haptic feedback from such touch-based sensing with another sensory stimulus; in this case sound. Crossmodal substitution draws on sensory substitution [6, 25], a common technique from assistive technologies where stimulus for one sensory system is replaced with stimulus for 7 https://doi.org/10.1145/3663548.3688496 another, such as a screen reader verbalizing written text on a screen with synthesized speech. A special case of multimodal substitution [26, 27] because of the use of multiple sensory channels, the approach explicitly “crosses” touch input with sound output. I achieved this by producing au- dio rather than haptic output in response to touch exploration. To explore and validate this idea, I proposed and implemented TACTUALPLOT, a crossmodal substitution technique that enables a person to “touch data” spatialized in two dimensions on a tablet touch screen, similar to a visual 2D scatterplot. With TactualPlot, a Blind user can use their fingertips to explore the shape of the data, receiving a dynamically changing audio tone in response that conveys the density under the user’s touch using pitch and verbalization. Unlike many other accessible data representations, TactualPlot was designed for scalability, with thousands of data points in mind. TactualPlot was designed using a user-centered approach by engaging my collaborator, who is Blind, in a series of formative design sessions. Our observations yielded several design revisions, including the need for multiple touch interaction and axes manipulation. Our participatory design sessions helped us understand that it is important to develop a touch-based, non-visual graphical percep- tion technique that could be generalized beyond scatterplots. In-depth expert reviews with two Blind professionals helped assess the utility of TactualPlot, and—by extension—the validity of the crossmodal substitution concept. Role: My contribution to the research study includes: 1) designing and implementing the fea- tures of TactualPlot on the iPad 2) conducting the design sessions with our collaborator, 4) per- forming the data analysis of the task results and analyzing the touch logs, and 5) authoring the research manuscript. 8 Publication: Chundury, P., Reyazuddin, Y., Jordan, J. B., Lazar, J., and Elmqvist, N. (2023). TactualPlot: spatializing data as sound using sensory substitution for touchscreen accessibility. IEEE Transactions on Visualization and Computer Graphics, 30(1), 836-846. DOI: 10.1109/TVCG.2023.3326937. CHAPTER 6. To answer RQ3 and thereby support data analysis of multidimensional data and multiple chart types, I expanding and quantitatively evaluated the crossmodal sensory substitu- tion design space. I compared TactualPlot to other devices that used only one modality—sound (Olli [12]) and touch (Monarch [28]) to isolate differences between modalities for visualization tasks. I compared task performance with bar charts, pie charts, line charts, and scatterplots. The results indicated that TactualPlot generally led to better accuracy. The Monarch often resulted in lower completion times. Qualitative insights from Blind participants offered perspectives on the use of sound versus touch in data interpretation. To better understand tactile chart authoring for refreshable tactile displays (RTDs), I conducted a co-design session with a Blind participant using a pair analytics approach to explore financial data. Finally, I discuss the need for stan- dards and validated audio-visual scales; and explain the implications of the findings from the two studies for expanding established web content accessibility guidelines for multimodal data visualizations. Role: My contribution to the research study includes: 1) designing and implementing the charts for TactualPlot, Olli, and the Monarch 2) conducting the pilot sessions and the experiment, 3) performing the quantitative and qualitative analyses, 4) conducting the co-design session and analyzing the data from the session, and 5) authoring the research findings and discussion. 9 https://doi.org/10.1109/TVCG.2023.3326937 CHAPTER 7. In this chapter, I summarize the contributions from my mixed-methods approach, and propose future work for designing multimodal interactions for accessible data visualization supporting Blind users. This dissertation addresses the critical challenge of making data visualizations accessible to blind individuals by investigating multimodal touchscreen interactions. Through a mixed- methods approach—including interviews with orientation and mobility experts, a survey with blind and low-vision professions, the design and evaluation of the TactualPlot system, a compar- ative study of accessibility modalities (screen readers, sonification, and braille), and a co-design session with a refreshable tactile display—my work demonstrates the potential of multimodal sensory substitution for non-visual data exploration. The research contributes empirical insights, and practical design guidelines aligned with accessibility principles to inform the development of inclusive data visualization tools. 10 Chapter 2: Related Work Much work has been done across communities such as HCI, Accessibility, ICAD, and cognition to make data and charts accessible. We believe that many of these solutions could make specific chart types or data accessible. As newer and more complex innovation in the visualization community grows, there is a need within the visualization community for frameworks or models to guide researchers and technologists to make their innovations accessible. Below we discuss prior work across research disciplines in the use of sensory substitution to aid in data analysis and visualization. 2.1 Mental and Spatial Mapping Mental maps are cognitive constructs that are used to understand and explain the environ- ment around a person to support spatial thinking and discussions [29, 30]. These mental maps are stored as schematic representations [31], usually based on rectangular grid structures [32], and contains information related to objects, spatial relations between objects, landmarks, inter- sections, and route descriptions [33]. Research shows that Blind and sighted people construct spatial maps in similar ways [34], and that Blind individuals use a combination of sensory cues such as auditory, tactile, movement, and proprioception to perceive, store and recall spatial concepts [35–37]. Research shows that 11 the “visual” cortex of the Blind is activated to process other sensory modalities [38]. More recently, Hersh [7] conducted interviews with 300 Blind and visually impaired individuals about perceiving spatial layouts, and identified that these individuals used their hearing, touch, and a combination of both to perceive space. Orientation and Mobility (O&M) training has been studied in the literature primarily to inform technology development to support O&M training [7, 8]. O&M training teaches per- ceptual and conceptual abilities to tackle indoor and outdoor navigation tasks. In fact, O&M skills have been shown to be transferable between virtual environments to the real-world and vice-versa [39, 40]. This partly motivated our focus on O&M instructors in our study. 2.2 Sensory and Multimodal Substitution for Blind Users Sensory substitution refers to the use of one sense to provide information normally provided by another sense [25], and is a common approach for assistive technology in accessibility [6]. Au- diobooks are a prime example of assistive technology using sensory substitution; originally in- vented to help Blind individuals enjoy reading books, their widespread adoption also with sighted individuals showcases a common phenomenon in accessibility: the “curb-cut effect,” where im- provements for one population of users end up benefiting many. Multimodal substitution, on the other hand, involves combining different sensory modalities to create a richer experience [26,27]. For instance, combining touch, sound, and smell can provide a more complete representation of an environment for a Blind user. Sensory and multimodal substitution are commonly used also for accessible visualization [20]. This is primarily done through sound and tactile feedback; we describe these research efforts in detail below. However, researchers have also explored the 12 potential of using smell to make visualization accessible for Blind individuals [41]. 2.3 Sound Perception and Sonification Sonification is the use of non-speech sounds to convey data or information where data are mapped to sound parameters to generate sound [9]. Auralization is the process of modeling and simulating the experience of sound in a virtual space [42]. Audification [43], a type of sonification is the technique of directly mapping all data values one-to-one continuously to audio-samples. The aforementioned data mapping techniques have been extensively used by the International Community of Auditory Display (ICAD). In a typical auditory display, one may have multiple auditory dimensions (e.g., frequency or loudness of a tone) with each dimension bearing a light information load (data), or relatively more number of auditory dimensions, with each dimension bearing a considerably higher information load. The information being represented using sounds range from a low-resolution equivalent of images [44, 45]; to system functionality and actions– auditory icons and earcons [46,47]; and abstract data such as temperature and pressure [48]. Brit- tel [49] presents a review of literature on sonification in conveying geospatial data by mappings to non-speech sound dimensions such as frequency and timbre, and to temporal characteristics such as duration and time. Sonification using spatial sounds—2D or 3D, where the position of the sound source is modeled, helps Blind and partially sighted users explore spatial layouts such as virtual city maps [50]. Nasir and Roberts [51] present a comprehensive review on spatial sound and sonifi- cation techniques that are beneficial in conveying spatial as well as non-spatial data (such as pie charts). The authors state that the complete potential of spatial 3D audio is yet to be explored. Du- 13 raiswami et al. [42] present techniques for creating virtual auditory spaces to aid acoustic source localization. Geronazzo et al. [52] present a spatial sonification system that enables audio-haptic exploration of virtual maps, and show that a 3D spatial audio and tactile combination outperforms just tactile feedback and tactile feedback with 2D audio. Sound-based substitution has the advantage of being easily available in professional set- tings to Blind individuals through personal computing and audio devices such as computers, smartphones, and headphones. However, there are disadvantages such as requiring extensive training [44], and varying auditory perception of individuals. Humans find it hard to distin- guish different levels of a sound dimension (e.g., multiple frequency levels) and are better able to distinguish between different dimensions (e.g., between two tones with fewer segments across frequency and loudness) [53]. Walker et al. [48] experimentally compared sound mapping en- sembles created by sound designers to be “Intuitive”, “Okay”, “Bad” and “Random”; and the “Random” ensemble resulted in the highest task accuracy over “Intuitive” or “Okay“ as one would expect. Hence, it is crucial to empirically test the auditory display system with the in- tended user to define the most efficient mappings. We seek to better understand such variability in sound perception through interviews, and experimentally testing our sound design. 2.4 Chart Accessibility Visualizations are by definition visual in nature, so making them accessible to a Blind au- dience is a significant challenge with facets, many of them social rather than just technical in nature [2]. Visualization has long mostly ignored the fact that visual representations are not accessible for Blind or low vision (BLV) individuals [54], but this is now changing [54–58]. 14 However, designing accessible visualization is fraught with complexity, both in terms of tech- nical challenges (e.g., how to represent complex and large-scale data primarily using sound and touch) as well as social (adoption, maintenance, and training) and economical (high cost to an often underemployed user population) barriers [2]. Here we review the literature on accessible visualization, focusing mostly on the aforementioned technical challenges. Non-verbal sound (sonification) and speech (auralization) have been used in place of visual representations. Early solutions used musical cues to convey shapes and graphs, but required visually impaired users to have musical knowledge [59, 60]. While not directly related to visualization, audio and hap- tic feedback have been used to help people with visual impairments understand the structure of web pages representing buttons, links, and search features [61,62]. NASA researchers developed MATHTRAX [63], an interactive graphing software that sonifies mathematical data, functions, and equations for Blind and low-vision students. Visualization-specific audio translations such as sonification of 2D data tables [64], line charts [65], shapes [66], bar charts [67], pie charts [68], and network structures [69] have mapped audio notes to data values as effective non-visual chart equivalents. The aforementioned solutions were experimentally evaluated with Blind individ- uals; with sighted or blindfolded individuals oftentimes included for comparison. While these chart specific solutions are effective, issues with training and usability due to lack of familiarity with audio dimensions still persisted. Sonification of axes and labels improve point estimation by providing contextual references [70]—leading to improved graphical perception. Tactile representations are effective in representing data and in translating visual represen- tations such as maps [71] and bar charts [72] into touch-perceivable equivalents. Guinness et al. [73] used miniature robots to convey data, and found that target acquisition was easier using tactile feedback as compared to sound. The sense of smell could also potentially be used as 15 a complementary modality towards making visualization accessible for Blind individuals [41]. Zhao et al. [74] created a tool that used both sound and speech to enable visually impaired users to explore maps and several other statistical data graphics. Brown et al. [75] propose audio representation guidelines for graphs and tables, and Zhao and colleagues [74, 76] propose au- dio information seeking principles (AISP) for abstract data. Computational methods have been used to extract semantic information that can subsequently be sonified or read aloud [77], from charts in applications such as accessible floor plans [78], to metadata added to charts generated in R [79]. However, these solutions work only for charts authored in specific tools. Multimodal solutions using a combination of sound and touch have been shown to be more effective than us- ing single modalities, and have inherently focused on chart translation, interaction, and authoring as well [80–84]. Web chart accessibility focuses on screen reader integration leading to solutions that integrate naturally into Blind individuals’ technology ecosystem [5, 67, 84]. In our work, we focus on bridging these disciplines, confirming results from and adding to past work, and broadly focus on sensory substitution for spatial understanding by interviewing experts from the Blind community. Given the understandable reluctance among the Blind community to adopt untested and poorly maintained technology [2], one strategy has been to target low-hanging fruit. Screen readers may be one such opportunity. Alternate texts (alt-texts) are machine readable descriptions associated with images on the web, and are commonly verbalized by screen readers to help Blind individuals access image content via screen readers. Jung et al. [85] propose a comprehensive set of guidelines for writing alternative text descriptions for visualizations to cater to the diverse needs of Blind individuals. However, the adoption of alt-text on the internet is poor even for regular images [86], let alone chart images. To deal with these situations, Al-Zaidy and Giles 16 presented an algorithm that uses computer vision and OCR techniques to automatically extract data from bar charts with an accuracy of over 90% [87]. Similarly, Choi et al. [88] proposed an approach to reverse-engineer rasterized several types of charts to make the data accessible for Blind users. Another approach is to design visualizations to play nice with a screen reader. Zong et al. [1] worked with Blind collaborators to design visualizations whose structure, navigation, and descriptive content are optimized for rich screen reader experiences. The VoxLens [89] integrates with the screen reader to convey a multimodal approach to visualization accessibility, providing voice commands, summarizing data, and sonifying details on demand. Most recently, Thompson et al. [13] worked over a period of five months to develop a chart accessibility engine combining a screen reader, data sonification, and descriptive content generation for web-based charts. 2.5 Sonifying Data In a state-of-art report written for the U.S. National Science Foundation in 1997, Kramer et al. [90] define data sonification as “the use of nonspeech audio to convey information” (p. 4), further qualifying it as “the transformation of data relations into perceived relations in an acous- tic signal for the purposes of facilitating communication or interpretation” (p. 4). Sonification is a form of an auditory display [91]; other forms (not necessarily mutually exclusive with sonifi- cation) include audification (directly converting large-scale data to the audible domain, auditory icons (short and self-contained sounds representing discrete events), and verbalization (synthe- sized speech conveying data). The ICAD community has been the epicenter of auditory display research for the last 30 17 years, but not all auditory displays are concerned with conveying data; some approaches are mostly artistic in natures, whereas others focusing on conveying realistic soundscapes rather than abstract data. Nevertheless, many sonification efforts can be applicable for such abstract datasets; the Sonification Handbook [9] surveys the state of the art in the field. One of the early sonification approaches was the iSonic system [92], where spatialized ge- ographic data is conveyed using sound to support a Blind user navigating and querying a map us- ing physical key mappings. Our work is heavily inspired by iSonic, but draws on the widespread adoption of smartphones where the touchscreen becomes the equivalent to the physical keys. The web-based chart library Highcharts has recently begun distributing an accessibility tool called the Sonification Studio [93], which enables robust and flexible data exploration using sound. Wang et al. [94] performed a study to rank audio channels in the sonification of data, confirming that pitch is optimal for encode data, but that tappings and length can be effective for specific tasks or data types. Finally, Hoque et al. [95] present a study on how the use of natural sounds can enable blending multiple data channels in parallel for increasing the sensory bandwidth of the sonification. Holloway et al. [96] use both sonification and speech to make infographics accessi- ble. While we use sonification in our approach, we do not sonify the data directly, but rather we sonify the spatial attribute of data when visualized in a scatterplot. 2.6 Touching Data Standard touch-based assistive technology [6] include the ubiquitous white cane, Braille text that can be read using the fingertips, and tactile maps [96], whereas Braille keyboards enable Blind individuals to generate text. However, for representing data, the options are limited. Static 18 tactile graphics are mostly made through 3D printing, thermal printing (where lines and shapes are raised when heat is applied), or embossing (which press designs into paper). Digital tactile representations have the benefit of being able to refresh dynamically, but are often specialized or costly, or both. Nevertheless, such refreshable technologies have been proven effective in conveying data from visual representations into touch-perceivable equivalents, as evidenced by studies on maps [96] and bar charts [15]. Guinness et al. [73] found that using miniature robots to convey data through tactile feedback was more effective for target acquisition than using sound. Such data physicalization [14], shape-changing displays [18], and haptic touch displays [19] could well present workable solutions. However, with the exception of the long- awaited Dynamic Tactile Device (DTD)1 being developed by HumanWare and American Printing House for the Blind, most of these advanced devices are costly research prototypes, and thus are not widely available to the general Blind population. Another option may be the commercially available ultrahaptics display [97], which generates mid-air tactile sensations using ultrasound, but the device still provides a fairly low resolution. The nearly ubiquitous smartphone [21] may be a better solution since it incorporates both audio output and a touch surface. A recent paper explores the use of touch for exploring 2D visu- alizations to yield sonified 3D sound output [98]. However, their evaluation uses six blindfolded sighted participants, which is questionable and not an ecologically valid approach [54]. While our work is based on a similar idea, our approach to multi-touch interaction using a sampling region is more robust and was iteratively developed in a participatory design process with a Blind collaborator and tested with Blind experts. 1https://www.aph.org/dtd-fact-sheet/ 19 https://www.aph.org/dtd-fact-sheet/ 2.7 Multimodal Data Access Recognizing the strengths and limitations of individual modalities, researchers have in- creasingly turned to multimodal approaches that combine different sensory inputs to create more comprehensive and accessible data representations. These hybrid solutions aim to leverage the complementary nature of different senses to enhance overall data comprehension and explo- ration. The concept of sensory substitution, where information typically acquired through one sense is conveyed through another, forms the foundation of many multimodal approaches [6,20]. Deroy and Auvray provide a comprehensive perspective on sensory substitution, exploring how information can be effectively communicated through alternative sensory channels [26]. Kramer et al. [90] reported that sonification requires users to interpret abstract acoustic cues—such as changes in pitch or timbre—that can impose a high cognitive load if the mapping between the data and sound is not intuitive. Building on this, Nanay introduces the concept of multimodal mental imagery, suggesting that our mental representations of data can integrate information from multiple senses simultaneously [27]. One promising approach is the combination of sonification and tactile feedback. Nikitenko and Gillis explored the potential of combining touch and sound for data exploration on mobile devices [98]. Their work demonstrates how the integration of tactile interaction with sonifi- cation can create more intuitive and engaging data exploration experiences. Our TactualPlot system [99], and ChartA11y [100] are examples of multimodal approaches, combining sound and touch to represent data. Such hybrid systems have the potential to overcome some of the limitations of single-modality solutions. For instance, while tactile displays may excel at con- veying spatial relationships, sonification could complement this by providing quick overviews or 20 highlighting temporal patterns in the data. Researchers have also explored the potential of other sensory modalities in data repre- sentation. Patnaik et al. [101] investigated the use of olfactory display for data communication, proposing “information olfactation” as a novel approach to convey data through scent. While still in its early stages, this work highlights the potential for engaging additional senses in multimodal data representations. This adaptability is particularly valuable in professional settings, where BLV individuals may need to work with a wide variety of data types and complexities. However, care must be taken to ensure that the different modalities complement rather than interfere with each other, and that the cognitive load of integrating multiple sensory inputs does not become overwhelming for the user. Additionally, as we [20] note in our interview study (chapter 3), it’s important to consider the practical aspects of using these technologies in professional settings, where compatibility with existing tools and workflows is crucial. More recently, researchers con- ducted a wizard-of-oz study on the use of refreshable tactile displays (RTDs) to make accessible charts, and explored how data and charts can be combined with the speech modality—both for interaction and for verbalization. Reinders et al. [102] conducted a systematic review of touch- based accessibility and identified the need for more comparison studies of presentation technique (sensory modalities) for a wide variety of charts. In chapter 6, we use speech (verbalization) to read out labels and data values, and additionally compare the effectiveness of: sound—textual descriptions that use the Olli [12]; touch—tactile graphics on the Monarch [28]; and an audio- touch— sonified charts that support touch interaction using the TactualPlot [99] system. We explore differences across four chart types: pie charts, bar charts, line charts, and scatterplots. 21 Chapter 3: Understanding Sensory Substitution 3.1 Introduction Visualizations are largely inaccessible to individuals who are blind or have visual impair- ments. Screen readers, the method blind people most commonly use to transform on-screen text to speech, generally cannot parse pixel visualizations, and few web-based visualizations provide sufficient textual descriptions or the underlying datasets [2]. People with visual impairments are a large population of potential data visualization users. In 2015, globally there were 253 million people with visual impairments, out of whom 26 million were blind, and this number is estimated to reach around 703 million by the year 2050 [3]. This is also not just a sociotechnical problem, but a potentially legal one; for example, in the United States, Section 508 of the Rehabilitation Act requires that all federal government websites be accessible for people with disabilities, and the Americans with Disabilities Act similarly requires accessibility for most websites of public accommodations [4]. Blind people1 navigate a 3D world of space and objects, and are there- fore equally capable of understanding spatial layouts as sighted individuals. However, despite continuous advances in visualization research, little effort is devoted to accessibility. While our focus is on blind individuals, inaccessible visualization practices not only affect blind people, but 1While people-first language is preferred by most people with disabilities, much of the blindness community prefers the use of the term “blind people.” In this text, we use both approaches interchangeably. 22 also those with other impairments such as motor or cognitive impairments [2,5]. We believe that the visualization community must work to lower barriers for blind individuals by focusing on accessible visualization and data analysis. Accessibility technologies for blind users tend to employ a method called sensory substi- tution [6] by conveying data using other senses such as hearing and touch, and even smell [7, 8] instead of vision. Efforts to render visualizations accessible for blind users have been explored by research communities such as human-computer interaction (HCI), accessibility, and sonification for specific contexts. Of the two most common sensory substitutes—sound and touch—sound is by far the easiest to deploy since it does not require any specialized hardware. While there exist many examples of sonification and auralization [9] (the use of non-verbal and verbal sound), these efforts primarily involve the blind community only as users or testers, and not as full-fledged in- formants or design partners [10, 11] in the development team. In this chapter, we seek to add to the growing body of literature on the potential of ac- cessible visualization design by understanding how blind individuals perceive the world around them using non-visual senses. Such knowledge would enable the design of more efficient audio representations of data that leverage the mechanisms that blind people already use in their every- day life. Our goal is to broadly understand (1) how blind individuals perceive and retain sound and touch-based information so that visual layouts such as charts can be effectively translated into non-visual senses, and (2) what this means for accessible visualization design so that future tools are robust enough to support multiple levels of visualization-related tasks [23]. To this end, we conducted semi-structured interviews with 10 Orientation and Mobility (O&M) instructors (experts), all of them blind, to understand how they teach their students (blind individuals) to navigate physical space using the sense of sound and touch. We chose this particular population 23 because these O&M instructors not only have significant personal expertise in leveraging non- visual senses to navigate the physical world everyday life, but also the knowledge and experience of teaching these skills to others. As part of the interviews, we also conducted discussions with these instructors on how to use sound and touch to convey data, and to translate visualizations. We found that blind individuals who undergo O&M training engage in experiential learning [24]— e.g., hands-on learning followed by reflection—to calibrate their minds to mapping sounds and tactile feedback to real-world aspects such as size, distance, angles, and position. We also learned how individuals actively use echolocation and environmental sounds to make sense of physical space, and the importance of sensory integration (e.g., combining audio and tactile feedback) in understanding it. Based on these findings, we derive design implications for accessible visualiza- tion design, with a focus on audio—speech and non-speech, and tactile representations, feedback and interactions. We also discuss the importance of training and usability of tools for accessible visualization for blind individuals. The contributions of this chapter are the following: (1) results from semi-structured inter- views with 10 O&M experts that convey how blind individuals perceive spatial concepts using sound and touch; (2) design implications for accessible visualization design with the idea of sen- sory integration of sound and touch; and (3) a design space on accessible visualization for blind individuals. 3.2 Methodology To better understand how Blind individuals learn to use sound and touch to perceive and navigate physical space, we conducted semi-structured interviews with 10 Blind Orientation and 24 Mobility (O&M) instructors. Here we first provide background on O&M training, present our study rationale, and then describe our data collection and analysis process. 3.2.1 Orientation and Mobility (O&M) Training Blind individuals enroll in Orientation & Mobility (O&M) training to learn to become in- dependent travelers. As part of their O&M training, individuals are taught to use environmental cues to construct mental maps of the space around them. Orientation and Mobility instructors teach Blind individuals—“clients” or “students”—to travel both indoors and outdoors, and to increasingly rely less on the visual sense. It is often assumed that Blind individuals are a ho- mogeneous user group, but research has shown that the attitudes, needs, and behavior of persons who are Blind vary greatly. In addition to O&M or Cane Travel, Blind individuals are also able to enroll in programs such as Braille learning, Technology, Job Readiness, and Wood Shop Training. Orientation and mobility experts receive National Orientation and Mobility Certification (NOMC), a certification that is offered by the National Blindness Professional Certification Board (NBPCB). Certified trainers teach under the Structured Discovery Cane Travel (SDCT) model; one that focuses on individuals acquiring non-visual travel skills through experiential learning based on personal experiences. The instructors teach concepts such as cane grips, men- tal mapping, environmental cues; and problem solving [7, 8]. Structured Discovery Cane Travel is one among two primary O&M training models; the other one—Sequential Learning (SL)—is a medical model for rehabilitation that was designed in the 1940s for World War II veterans and did not allow Blind individuals to become teachers. 25 Cane tip: echolocation and haptic feedback Soundscape Environmental sounds: car and construction Cardinal directions Mental map Figure 3.1: Navigating outdoor space using non-visual senses. Blind individuals build men- tal maps (visual thinking) by sensing environmental sounds, and use their cane for performing echolocation and perceiving haptic feedback; in other words, a combination of sound and touch towards non-visual sensemaking of spatial concepts. Our work explores how these capabilities can be used for accessible data. 3.2.2 Study Rationale While there are potentially many user groups to interview in order to understand sensory substitution mechanism, we chose Blind O&M instructors because they (1) have significant lived experience of using non-visual senses in perceiving space, as well as (2) are competent at teaching these skills to others, and have thus spent a significant amount of time retrospectively thinking about the skills. O&M training is particularly relevant because these skills have been shown to transfer to other contexts and settings in prior work [39, 40]. More specifically, visualizations such as maps, scatterplots, bar charts, and graphs rely on visual semantics such as shapes, size, color, position, labels, and axes to convey data to sighted individuals. Speaking to O&M instructors is a reliable way to explore how to translate a visu- alization’s visual semantics into non-visual modalities based on how Blind individuals perceive visual semantics of space. Compared to Blind sonification or tactile graphic designers who may use their own experiences, Blind instructors have a broader view from their training and certifi- cation to teach other Blind individuals. Prior work also shows that the intuition of sonification 26 designers may not lead to the best data-to-sound mappings [48]. Blind individuals are also taught mathematics and graphing primarily using tactile graphics such as embossed or Braille charts by Teachers of Students with Visual impairments (TSVI) [103, 104]. TSVI focus more on tac- tile representations, while O&M instructors focus more on teaching spatial understanding from sound and touch. Tactile charts have several limitations [103] such as cost, information over- load, and longer production time. While out of the scope of this work, we do think interviews with TSVI could lead to interesting insights on aspects such as chart authoring, and collaboration in classroom settings. Ultimately, we hope to apply these sensory substitution insights towards accessible visualization design. Additionally, we believe that comprehending fundamental chart concepts such as reference frames, estimating distances, understanding angles, and other visual variables [105] are similar to understanding and visualization navigation layouts and routes in the real world. 3.2.3 Participants We recruited 10 Blind O&M instructors to learn about their personal and professional perspectives on how Blind individuals use sound to create mental maps, understand their sur- roundings, and navigate in a physical space. We focused on SDCT instructors because the self- confidence levels of SDCT students are higher than those from SL training. Table 3.1 provides an overview of the participants. Two of our participants were also itinerant trainers who would visit their students at their preferred locations, while most of our participants conducted classes at institutions. Participants taught students as young as 5 years and as old as 70 years; and also students with other disabilities. OM4 was also a certified TSVI. 27 Table 3.1: Demographics and O&M experience of the 10 participants. Abbreviations: M— Male; F—Female; NB—Non-binary. ID Age Gender Education O&M Experience OM1 30 M High school 3 - 5 years OM2 32 M Master’s degree 5 - 7 years OM3 55 M Associate degree More than 7 years OM4 57 F Ph.D. More than 7 years OM5 66 M High school More than 7 years OM6 62 M Master’s degree More than 7 years OM7 36 NB Master’s degree More than 7 years OM8 25 M Master’s degree 3 - 5 years OM9 27 NB Master’s degree 6 months - 1 year OM10 30 M Master’s degree 3 - 5 years Overall, the participants had a strong expertise in teaching their students to travel using non-visual skills. Participants were recruited through mailing lists associated with the National Federation of the Blind (NFB). The study was approved by our university’s Institutional Review Board, as well as the Research Advisory Council of the NFB. 3.2.4 Data Collection and Analysis We conducted semi-structured interviews via video and audio conferencing on the internet to collect our data. We then transcribed and analyzed the resulting data using thematic analysis. 3.2.4.1 Semi-structured Interviews Each semi-structured, audio-recorded interview was scheduled for 60 minutes, and participants— as rehabilitation experts—were compensated with a $100 Amazon gift card. Broadly, the research goal was to understand the design space to help design and build tools that support data analysis 28 through sound and touch representations and interactions. This could mean making data repre- sentations such as visualizations or charts more accessible, or finding new ways of representing data in using sound—both speech, and non-speech. To understand O&M training procedures, we asked participants how they trained Blind individuals to rely on sound for creating mental maps of the environment as well their use of auditory interfaces in activities of daily living. Secondly, we asked the participants how Blind individuals perceive and infer different aspects and properties of sound, such as loudness, position and direction, pitch, repetition, moving vs. static sounds, and verbal sounds (speech). Finally, to brainstorm about the idea of translating a virtual and visual layout (a visualization) into an audio representation, we introduced our design idea to foster discussion and receive feedback. Participants were introduced to the idea of a web-based interface that allows users to upload a chart image; the tool will then extract text labels, data, and other semantic information from charts—for example, bar charts, scatterplots, maps, and line charts. Next, the tool will translate the visual elements and data into audio representations by simulating spatial audio [106]. Users can interact with the sounds using their keyboard or touch screen. The O&M instructors were not data visualization experts, but this part of the session guided participants to discuss their familiarity with audio and tactile feedback. 3.2.4.2 Transcription The audio recordings of the 10 interviews—10 hours and 38 minutes, were transcribed using an online service—Rev [107]. On average, each interview lasted 64 minutes. 29 3.2.4.3 Analysis We used thematic analysis [108] to open code the transcripts. We started by randomly se- lecting two transcripts to be open-coded by two researchers. In other words, we each separately tagged text from the transcripts with multiple codes (see below examples) to add semantic struc- ture to our data. After open coding, the two researchers discussed and merged the codes to create an initial codebook. The merging process included a discussion of rephrasing codes, adding miss- ing codes, removing codes that resulted in a codebook that was agreed upon by both researchers to improve reliability of our results. Next, one researcher coded the remaining transcripts using the initial codebook, and added codes as they emerged. Some examples of the codes that were used are: “O&M concepts”, “Indoor navigation”, “Residual vision use”, “Multiple sounds”,“Technology use in O&M”, “Embodied Cognition”, “Prior chart knowledge”, “Sound Isolation strategies and challenges”, and “Sound Mapping and Inference.” As new codes emerged, we returned to older transcripts to apply the new codes. Overall, 89 unique codes were used, and 258 excerpts were extracted from the coding process. Our codes are included in the supplementary material. Results from our analysis have also been reviewed by one of the co-authors who is Blind, and has long experience in information, data, and knowledge work. 3.3 Findings In this section, we present three main themes that broadly describe (1) Orientation and Mobility concepts that reveal how Blind individuals perceive elements and layouts of space using non-visual senses, (2) how individuals interact with space using sound and touch, and (3) the 30 challenges of using non-visual sense in using visualization. We highlight insights derived from this process as follows: Insight #1. Orientation and Mobility training highlights many audio and touch affordances that may be useful for creating accessible representations of data and visualizations. The color coding for these boxes signify whether they arise from perception, interaction, effectiveness, challenges, or design guidelines. We discuss the implications of our insights in detail in section 3.4, but briefly explain how our insights relate to visualization in this section. 3.3.1 Perceiving Space as a Blind Individual O&M training does not follow a prescriptive approach. Instead, it generally encourages Socratic questioning to help students associate their own meaning to various environmental cues. When asked specifically about different properties of sound, all participants mentioned that while certain properties can have an objective meaning, it is very hard to prescribe a particular threshold when considering the magnitude of these properties. For example, while loudness of sound might increase as someone walks closer to a sound source, estimating precise distances is still challenging. Additionally, participants noted that there is a lot of variability among students in terms of perceiving different sounds. Here we describe specific instances of using different properties of sound and touch in the non-visual sensemaking of physical space. Participants described the white cane as the primary tool used in the orientation and mobil- ity process. The non-folding version is recommended by the NFB, and is the primary one used in the O&M training process. Other canes, such as folding ones, are also used by Blind individuals, but are not recommended because of the limited haptic feedback that they offered compared to 31 the non-folding cane. Such haptic feedback is critical; the cane was described as an “extension of the self,” and cane techniques help students actively interact with the environment to receive sensory information. For example, participants described “shorelining” as a technique used by students to understand their position on the sidewalk by walking in “parallel” using the grass or the edge of the sidewalk as a “reference.” Insight #2. Many white cane interaction techniques help Blind individuals receive haptic feedback while traveling. Additionally, a sweeping motion of the cane on the ground can also convey tactile infor- mation to the students based on the continuous haptic feedback that the hollow stem of the cane transfers to the individual’s hands. OM4 explained how texture changes in the aisles of a grocery store are perceived during indoor navigation: “There’s a texture change when you’re in front of the grocery aisle or the refrigerated section, or even the fruit and vegetable sections. Because, that’s a little bit rougher, the texture is a little rough.” By leveraging the familiar metaphor of actively probing to understand a surface, tactile solutions could strongly couple interaction and representation to improve accessibility. Participants indicated that the sounds, especially distinct sounds, produced by certain ob- jects in the environment, such as lawn mowers, HVAC (heating, ventilation, and air conditioning) systems, streetcar, and traffic sounds, provide many clues about the environment around them. These ambient sounds, when mapped with concepts that students have stored in their memory, assist in the process of navigation and understanding the shape or layout of the surrounds, es- pecially when walking into unfamiliar routes or buildings. OM6 explains: “I think there’s also another thing which is the knowledge that a person has accumulated over the years. When I walk 32 into a high-rise building, there’s really only a couple possible places that the elevators could be located. And so I’m able to use that knowledge to help me figure out where I need to go to find the elevator.”. Chart literacy can be challenging for sighted individuals, and is even more for Blind individuals if they are unable to perceive the concepts used in highly customized be- spoke charts—i.e., non-standard charts. The importance of concept building indicates a need to effectively represent geometric units used in charts through non-visual senses. Insight #3. Sounds known to individuals from lived experiences help build spatial aware- ness. Many participants mentioned that the loudness is usually associated with the distance be- tween the student and the sound source. For example, OM7 mentioned that they talk to their students about loudness, where a change in loudness indicates changing distance between the two reference points—the individual and the sound source: “But I’ll ask them, “The sound that you hear, does it seem quiet or does it seem louder to you?” [...] And if they say,“Oh yeah, it seems quieter.” Then I might say, “Why does it seem quieter?” [...] And if they honestly cannot make that association, then what I will do is have them walk toward the sound and ask them what’s happening to the sound as they approach it”. One participant also indicated that students perceive bigger objects, such as a truck, to be louder than smaller objects, such as a car, indicating that the loudness may also be associated with size. Insight #4. Loudness is commonly used for perceiving spatial distance, and is also associ- ated with size of objects producing the sound. Not many participants mentioned the pitch of a sound being directly mapped to a partic- 33 ular physical property, but indicated that higher pitched sounds were easy to isolate from other environmental sounds. The pitch of the sound was associated with familiar concepts such as the sound of a car engine: “If a car is idling, I will point out to the student that when the car is shifted into the reverse, the pitch of the engine goes lower because the engine is laboring and that’s something to be aware of because that car could come backing out and you don’t want to be in the way.” (OM6) Insight #5. Pitch is used to recognize known objects and their state; higher pitched sounds are generally easier to isolate from other environmental sounds. OM7 mentioned that their students were cognizant of the “sound space” around them, and interestingly described certain aspects using visual space. When students are immersed in audi- tory environments, multiple sounds, with their different dimensions occupy the auditory capacity. OM7 describes the visual space in front of the students, indicating that sounds are mapped to vi- sual space: “They should tell me that the sound not only gets louder, but it takes up more of the sound space. It takes up more space in front of them. Like a fountain in the distance, it will sound quieter, but it won’t take up quite as much of the stereo space in front of them, in the same way that a distant fountain won’t take up as much visual space.” Insight #6. Individuals map visual space by sampling the various sounds of the environment and interpreting changes in sound dimensions such as loudness. Participants also mentioned that the sequence of sounds and the duration of a particular sound were important features to interpret to understand sound. Similar sounding tones when played sequentially can also be distracting: “So as long as there’s a clear enough duration in 34 between, like if you’re going to play a sound, you want there to be a gap in between.[...] So if they just get distracted for a second, they might miss that there was two clicks instead of one.” (OM8) Insight #7. The sequential nature of sound dictates that sampling frequency and duration be optimized to improve accuracy of spatial awareness. Participants perceived the absence of echoes or sounds, and associated the presence of large objects at a certain distance causing sound to be blocked and creating “sound shadows”— another visual concept. OM6 describes this phenomenon as follows: “An individual that has well-developed listening skills can hear those echoes off of telephone poles and even sometimes off of sign poles, depending on conditions. The person can also hear sound shadows from objects in the environment where they’re blocking out the sound of maybe a car passing by. They hear a moment where it’s blocking out the sound.” Insight #8. The absence of sound—sound shadows—could be interpreted as being caused by intersecting objects in space. The soundscape, i.e., the sound space around the individuals, often consists of multiple sound sources at different positions; some louder than others, and at different pitches. Participants mentioned that one of the important skills that students learn is “sound isolation.” Based on the problem or task at hand, individuals needed to focus their attention on certain sounds, and oftentimes these relevant and necessary sounds would be occluded by environmental sounds. Prior work on effectiveness of data-to-sound mapping recommends empirical assessment based on end users and the data task [48]. Our findings about loudness, pitch, and lack of sound 35 being mapped to spatial or geometric properties could be a starting point for accessible sound representations since they are based on Blind individuals’ mental models. The idea of sound shadows has not been explored in prior work, and could be an interesting design material for audio charts in the future. Insight #9. Individuals isolate and focus their attention on specific sounds in a soundscape based on the task at hand. Participants also mentioned that some important sounds such as the “buttons at the (road crossing) intersection” sounded very similar to environmental sounds such as the sounds of “birds chirping.” The ability to isolate different sounds when needed was a skill that depended on the the person because “people have (had) different levels of hearing ability, and then being able to discriminate [...] used judgement on the cues that they’re getting.” (OM3) Participants mostly agreed that familiarity of sounds make sound isolation much easier. Participants also describe how triangulation is used to establish their position with more con- fidence: “You usually have to have more than just one source of information to make a solid deduction as to where you are. You might need to feel the direction of the sun at that particular time of day, hear where the sound is coming from and maybe some other sound off in the distance, maybe traffic off in the distance somewhere tells you.” (OM6) Insight #10. Familiarity as well as dissimilarity of sounds in a soundscape help an individual to switch their hearing focus. Participants mentioned that “sound localization” is part of the concepts covered during training, and is a fundamental concept that “can be taught or is a learning curve”. However, 36 there are variations among students’ efficiency in localizing sound. This could be because of students’ hearing impairments, when present, that causes “bilateral imbalance in their hearing.” (OM3) Participants indicated that static sounds can be used as landmarks or reference points. This indicates that sound localization, while being a fundamental human feature, can be “highly dependent on a person’s ability to discern an angle of sound.” Whether a sound is static or moving, also influences how much an individual can sample and associate meaning: “[...] when you have a static sound you can use that specific sound for orientation purposes, right? Say that sound’s been occurring at that location for the last several minutes, so why don’t we use that as a point of reference as opposed to a moving sound?” (OM1) Prior work shows that sonification of chart axes adds context to audio charts [70]. Since static sounds are useful as references, accessible design could consider metronomes or always- present sounds to convey chart boundaries, legends, or even scales of the axes. Insight #11. Estimating positions of and distances between sound sources varies greatly between individuals based on hearing and spatial awareness, with static sounds being most helpful. For sighted individuals viewing a chart, their eyes move rapidly to perform actions similar to auditory actions such as filtering, scanning, differentiating, etc. But with hearing support- ing lower information bandwidth, careful accessibility design may be needed to translate charts showing a large amount of data. 37 3.3.2 Interacting with Space as a Blind Individual Overall, participants indicated that the white cane helped in interacting with the environ- ment as an “extension of their body” and acts like an “antenna.” Participants noted that canes with a metal tip encourages the use of “auditory information” (OM4), whereas folding canes tend to use a plastic tip that does not provide the same auditory feedback (OM4). The cane helps produce a crisp sound that bounces off objects around the students. The echo that is produced conveys different aspects of the environment such as distance from buildings, number of objects, and wide versus narrow spaces. The duration of the echo also conveys meaning to the students. OM8 explains echolocation as follow: “So an example is when they’re in a parking lot looking for a building, I would have them tap at various parts of the ground in front of them, they might tap at nine o’clock first, then 12 o’clock and three o’clock, based on the time it takes for the echo to come back to them, then that will tell them whether there is a building in that particular direction or not. [...] If the building is within 15, 20, 25 feet away from them where the echo would return, then they’ll quickly hear echo back, and the length of the echo varies depending on how far the object is.” Insight #12. Blind individuals often use echolocation to understand space around them. In general, participants mentioned that their students visualize objects and space using their own body as reference. For example, participants described the positions and direction of objects using terms such as “eye-level”, “front”, “back”, “left side”, “right side” and “above.” This indicates that, considering 3D space, perspective appears to be from the point-of-view of the individual’s body or as extension of the body in the form of the white cane. All participants 38 mentioned the use of “cardinal directions” by their students during travel. Many participants also described the use of the sun’s position by perceiving the direction of heat to gain an understanding of both time of the day as well as understanding the cardinal direction they were traveling. Insight #13. Blind individuals often interact with sound from a perspective that is relative to different parts of their body. Our finding on egocentric sound perception suggests that future work on accessibility could explore egocentric perspectives for both sound and touch-based solutions. While this has been explored in pie chart sonification with sighted individuals [68], our findings highlight the value in exploration with Blind individuals. 3