ABSTRACT Title of Dissertation: DESIGNING AND EVALUATING NEXT- GENERATION THERMOGRAPHIC SYSTEMS TO SUPPORT RESIDENTIAL ENERGY AUDITS Matthew Louis Mauriello Doctor of Philosophy, 2018 Dissertation directed by: Professor Jon E. Froehlich Department of Computer Science Buildings account for 41% of primary energy consumption in the United States—more than any other sector—and contribute to an increasing portion of carbon dioxide emissions (33% in 1980 vs. 40% in 2009). To help address this problem, the U.S. Department of Energy recommends conducting energy audits to identify sources of inefficiencies that contribute to rising energy use. One effective technique used during energy audits is thermography. Thermographic-based energy auditing activities involve the use of thermal cameras to identify, diagnose, and document energy efficiency issues in the built environment that are visible as anomalous patterns of electromagnetic radiation. These patterns may indicate locations of air leakages, areas of missing insulation, or moisture issues in the built environment. Sensor improvements and falling costs have increased the popularity of this auditing technique, but its effectiveness is often mediated by the training and experience of the auditor. Moreover, given the increasing availability of commodity thermal cameras and the potential for pervasive thermographic scanning in the built environment, there is a surprising lack of understanding about people’s perceptions of this sensing technology and the challenges encountered by an increasingly diverse population of end-users. Finally, there are few specialized tools and methods to support the auditing activities of end-users. To help address these issues, my work focuses on three areas: (i) formative studies to understand and characterize current building thermography practices, benefits, and challenges, (ii) human-centered explorations into the role of automation and the potential of pervasive thermographic scanning in the built environment, and (iii) evaluations of novel, interactive building thermography systems. This dissertation presents a set of studies that qualitatively characterizes building thermography practitioners, explores prototypes of novel thermographic systems at varying fidelity, and synthesizes findings from several field deployments. This dissertation contributes to the fields of sustainability, computer science, and HCI through: (i) characterizations of the end-users of thermography, (ii) critical feedback on proposed automated thermographic solutions, (iii) the design and evaluation of a novel longitudinal thermography system designed to augment the data collection and analysis activities of end-users, and (iv) design recommendations for future thermographic systems. DESIGNING AND EVALUATING NEXT-GENERATION THERMOGRAPHIC SYSTEMS TO SUPPORT RESIDENTIAL ENERGY AUDITS by Matthew Louis Mauriello Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2018 Advisory Committee: Professor Jon E. Froehlich, Chair/Advisor Professor David Jacobs Professor Andrea Wiggins Professor Niklas Elmqvist Professor Michelle Mazurek © Copyright by Matthew Louis Mauriello 2018 Dedication To Brenna for her unwavering support that cannot be adequately described nor repaid. To my family and friends for their constant encouragement. And, to coffee. ii Acknowledgements This work would not be possible without the encouragement and support of my advisor, Dr. Jon E. Froehlich. Under his direction, I have been able to pursue my lifelong passion for learning while exploring how computing can be applied to solving challenges in our daily lives and, more broadly, facing society at large. Moreover, I have learned much about the kind of researcher and scholar I want to be in the future. I also want to acknowledge Dr. Andrea Wiggins for her high-quality feedback on many aspects of the work contained within these pages. Additionally, I would like to thank the other members of my committee—Dr. David Jacobs, Dr. Niklas Elmqvist, and Dr. Michelle Mazurek—whose questions, insights, and suggestions pushed me to think a little more deeply about this work and its implications. I would also like to acknowledge the work and contributions of my undergraduate and high school mentees as well as the volunteers in the studies themselves who, in the end, probably taught me far more than I taught them. It also would have been impossible to have completed this work without the support of the communities of people who make up the Makeability Lab, the Human-Computer Interaction Lab, and the Department of Computer Science at the University of Maryland. Similarly, I would like to thank my advisors at SUNY at Albany who encouraged this endeavor from my earliest days as a student and my family for instilling in me a drive to learn and wholeheartedly pursue my goals. Finally, I honestly do not know where I would be without the social support I received from my graduate student peers: Brenna, Kotaro, Cody, Sana, Aleks, Megan, Leyla, Christoph, Zamira, Ioana, Kris, Zahra, Mike, Lee, and so many others. Thank you. iii Table of Contents Dedication ............................................................................................................... ii  Acknowledgements ................................................................................................ iii  Table of Contents ................................................................................................... iv  List of Tables ......................................................................................................... vii  List of Figures ....................................................................................................... viii  List of Abbreviations .............................................................................................. xi  Chapter 1 ................................................................................................................ 1  Introduction .................................................................................................................. 1  1.1  Dissertation Research Approach and Overview .................................................. 4  1.2  Summary of Contributions .................................................................................. 8  1.3  Dissertation Roadmap ....................................................................................... 10  Chapter 2 .............................................................................................................. 11  Background and Related Work .................................................................................... 11  2.1  Background ....................................................................................................... 11  2.2  Related Work..................................................................................................... 19  2.3  Summary ........................................................................................................... 28  Chapter 3 .............................................................................................................. 29  Characterizing Novice Thermography .......................................................................... 29  3.1  Introduction ...................................................................................................... 30  3.2  Method .............................................................................................................. 32  3.3  Findings from Video Analysis ............................................................................ 39  3.4  Findings from Online Survey ............................................................................. 47  3.5  Discussion .......................................................................................................... 51  3.6  Conclusion ......................................................................................................... 56  iv Chapter 4 .............................................................................................................. 57  Novice Thermographic Energy Auditing ....................................................................... 57  4.1  Introduction ...................................................................................................... 58  4.2  Method .............................................................................................................. 59  4.3  Results ............................................................................................................... 65  4.4  Discussion .......................................................................................................... 76  4.5  Conclusion ......................................................................................................... 79  Chapter 5 .............................................................................................................. 81  Professional Thermography & Automation .................................................................. 81  5.1  Introduction ...................................................................................................... 82  5.2  Study 1: Interview Study and Design Probes .................................................... 84  5.3  Study 2: Observational Case Study ................................................................... 98  5.4  Discussion ........................................................................................................ 100  5.5  Conclusions ..................................................................................................... 104  Chapter 6 ............................................................................................................ 105  Development and Testing of a Preliminary Temporal Thermography Sensor System . 105  6.1  Introduction .................................................................................................... 106  6.2  System Design ................................................................................................. 107  6.3  Usability Study ................................................................................................. 110  6.4  Usability Study Results .................................................................................... 110  6.5  Field Deployment Study  ................................................................................. 111  6.6  Field Deployment Results ................................................................................ 112  6.7  Conclusion ....................................................................................................... 112  v Chapter 7 ............................................................................................................ 114  Longitudinal Thermal Camera Sensor System Design, Validation, and Evaluations ..... 114  7.1  Introduction .................................................................................................... 115  7.2  Temporal Thermography Sensor System Design ............................................ 117  7.3  Study 1: Technical Evaluation ......................................................................... 129  7.4  Study 2: Residential Homeowner Deployment Study ..................................... 136  7.5  Study 3: Professional Energy Auditor Design Study ........................................ 153  7.6  Discussion ........................................................................................................ 164  7.7  Conclusion ....................................................................................................... 172  Chapter 8 ............................................................................................................. 173  Conclusion ................................................................................................................. 173  8.1   Summary of Contributions .............................................................................. 175  8.2  Future Work .................................................................................................... 179  8.3  Final Remarks .................................................................................................. 186  Appendices ......................................................................................................... 187  Bibliography ........................................................................................................ 254  vi List of Tables Table 3.1: The search terms and training data codebook used to assemble our study dataset throughout SMIDGen’s four steps. Table 3.2: Topic and sub-topic codes applied to analyze the content of “everyday use” videos. Table 3.3. The categorical results from our coding process sorted by frequency. Categories were exclusive (i.e., videos were coded into a single category). Table 3.4. Breakdown of comments on YouTube thermography videos. Table 4.1: A summary of the novice field-study participant’s demographic information. Table 4.2: The pre-study survey asked participants how concerned they were about climate change and energy efficiency in specific contexts of their daily lives. Table 4.3: Weekly mission descriptions were sent to participants via email along. Lightweight feedback about the previous week was also provided. Table 4.4: An overview of participant behavior and survey responses. Average time spent was calculated by adding the total minutes spent across all data collection sessions in a given week based on participant’s self-report data. For Likert questions, we used a 7-point scale ordered from strongly disagree (1) to strongly agree (7); 4 was neutral. We report median (Mdn=X) and standard deviation (SD=X). For mission difficulty, higher is easier. Table 5.1: Study 1 participant (professional auditor) demographics. Table 7.1: Material class distribution after re-labeling classes with few examples to “Other”. Table 7.2: Material classification results on test set. Table 7.3: Results from the full 48-hour temporal thermographic measurement campaign compared to notional values and THM method, with intermediate calculations from the two 12-hour overnight segments demonstrating increased accuracy with longer data collection periods. Table 7.4: Participant demographic information for the field study deployment of the temporal thermographic sensor system with homeowners. Table 7.5. Participants’ use of the sensor kit to analyze and uncover issues in their first capture sessions. Results of using the temporal sensor system conflicted with three participants expectations. Table 7.6: Professional energy auditor participant demographic information. vii List of Figures Figure 1.1: Thermal cameras come in many form factors including handheld models (left) and more recently released smartphone-based thermal camera attachment (right). Figure 1.2: A visual overview of the three threads of research presented in this dissertation. Figure 2.1: Commodity thermal cameras for smartphones and small electronics projects. From left to right: FLIR Breakout Board, Seek Thermal Imaging Camera for iOS, FLIR One iOS Attachment Gen II, FLIR One iOS Attachment Gen I. Figure 2.2: Thermographers use thermal imaging software like FLIR Tools [40] to surface thermal data captured by their cameras. Figure 2.3: Examples of automated thermography from literature Figure 2.4: Fox et al. [45] uses a small multiples method to look for evidence of previous repairs and the effects these have had on building performance. Figure 2.5: Danese et al. [28] monitors a wall for several hours (after sunset) and notices differences in cooling, indicating areas that have been patched or are degrading. Figure 3.1: A montage video of thermal camera observations from YouTube video V79 showing (a) electrical power lines, (b) a woman with a jacket outside in the cold, (c) hot coffee with cream, and (d) hand washing with cold water. Figure 3.2: The images above portray a typical video from each coded category and the category’s percent representation in the overall dataset (N=675). Figure 3.3: Illustrative examples described throughout our findings: (a) exploring whether thermal cameras see through water in V90, (b) comparing a Raspberry Pi’s internal temperature sensor to a handheld thermal camera reading during a stress test in V801, (c) an exterior home inspection in V351, and (d) describing how convection in hot coffee causes cells to be visible in thermal images in V154. Figure 3.4: Survey participant’s planned (pre-purchase) and actual (post-purchase) thermal camera activities in terms of percentage of total respondents (N=79). Figure 4.1: (a) A close-up of thermal camera application on iPhone5s. (b) A pilot participant collecting photos. (c) A thermal camera attached to an iPhone 5s. (d) A close-up of the standard FLIR One camera application that ships with the attachment. Figure 4.2: A participant describes an air leakage issue found while auditing his workplace during the post-study debrief interview (left). A close-up of the actual thermal image being discussed (right). viii Figure 4.3: Examples of the image contexts, subjects, and non-mission photos as well as the percentage of the dataset that includes these features. Figure 4.4: Example imagery from participant investigations: (a) an insulation issue in the roof of a residential home, (b) observing power consumption of computer equipment in an office, (c) gathering evidence of insufficient winterization procedure of window air conditioning units in a university building, and (d) documenting the need to repair weather stripping around an emergency exit door at a community theater. Figure 4.5: During the community mission participants explored utility infrastructure (left); one participant explored a makerspace (right). Figure 5.1: We developed five automated thermography scenarios inspired by the research literature (e.g., [14,30,85,102]) to help elicit reactions to envisioned automated solutions. Above, a screen capture from our unmanned aerial vehicle (UAV) design probe. Figure 5.2: Screen captures from the UAV-based thermography video design probe (Scenario 4). Figure 5.3: Screen excerpts from our interactive mid-fidelity prototype built in Axure. (a) Users input a rough thermography route for the UAV. (b) The analysis software automatically identifies anomalous thermal signatures (red brackets) on a 3D-reconstruction. (c) The auditor can zoom in and see how this area of the building has changed over time (every few months). (d) A co-located temporal graph of the estimated surface temperature is also provided. Figure 6.1: Common energy efficiency issues visible to a thermal camera. Figure 6.2: Our prototype for a temporal thermography sensor system used in these studies to augment building energy audits. Figure 6.3: The sensor kit provides a webserver allowing users to set schedules and view results, which are fed into the visualization tool. Figure 6.4: The single-image mode of the interactive visualization with temporal slider. Figure 6.5: The multi-image temporal mode of the interactive visualization. Figure 6.6 (Left to right): Our sensor kit was placed in the center of the film archive, analysis was performed on large sections of walls and windows by the auditor, and the visualization tool provided views of the data. Figure 7.1: The iterated longitudinal sensor system (left) and report (middle), and a sample of data collected by homeowners in the field-study (right). Figure 7.2: The final version of our Tinkercad CAD model before exporting for 3D printing (left) and a completed sensor system with a FLIR One, version 2, attached (right). The sensors are constantly streaming data to the touchscreen display which is in its idle mode, displaying data, recording state, and other status messages (e.g., Wi-Fi connectivity, storage space remaining, etc.) ix Figure 7.3: Calibration Procedure. After receiving the inferred emissivity map a box appears over the QR code or in the middle of the screen if the QR code was not found (left); by dragging the white box users can update the position and by dragging the small circle users can resize it. When the user accepts the QR code position using on screen buttons (not shown) the distance to the wall and the position of the high-emissivity foil is inferred (middle). A second box appears on screen to the left of the QR code (middle); the user can move the second white box to select the region they are interested in scanning (right). When users select ‘Done’ the calibration is complete. Figure 7.4: Example infographic report (left); zoomed in example of thermographic analysis (right). For each metric in the report there is an explanatory section that presents a summary of the automated results, suggestions for interpreting them, and recommendations with links to source materials. In the case of the thermal results, a warning about potential environmental conditions that could negatively impact results is also presented. Figure 7.5: Alternate displays of the automated online report: small multiples view of all the data collected (left) and the compare snapshots view that enables viewing two different data collections simultaneously (right). Figure 7.6: Deployment setup for the validation experiments. The sensor system in the process of being configured (left) to analyze the area framed by the thermocouples (middle), and destructive testing to verify composition of wall assembly (right). Figure 7.7: Data from homeowner inspections using smartphone attachments in mission one. Figure 7.8: Data from homeowner inspections using temporal thermography sensor system in mission two. Figure 8.1: Multi-unit scenario (presented to auditors in Chapter 7, Study 3) where several of our sensors are deployed to increase the simultaneous coverage area represented by the red arcs paired with each wireless sensor unit. x List of Abbreviations CHI Conference on Human Factors in Computing Systems DIY Do-It-Yourself DOE Department of Energy FLIR Forward Looking Infrared HBI Human-Building Interaction HCI Human-Computer Interaction HFM Heat-Flow Meter Method HVAC Heating, Ventilation, and Air-Conditioning ISO International Organization for Standardization IoT Internet of Things LEED Lead in Energy and Environmental Design OSN Online Social Network PCP Parallel Coordinate Plot PTEM Physical-Technical-Economic Model ROI Region of Interest SHCI Sustainable Human-Computer Interaction THM Thermometric Method xi Chapter 1 Introduction The building sector accounts for 41% of primary energy consumption in the US, more than any other sector, and contributes an increasing portion of total carbon dioxide emissions—40% in 2009 compared to 33% in 1980 [88]. One reason for these high emissions is building age. Residential buildings, for example, constitute 95% of all buildings in the US and are on average over 50 years old [155]. Most of these buildings were constructed using energy inefficient designs and their materials have degraded over time. To address these issues, renovations and retrofits of existing building stock has become a pressing need. The US Department of Energy (DOE), for example, has set a goal of reducing housing energy use by up to 70% before 2020 through the use of innovative new technologies and upgrades to existing building stock [117,148]. In response, energy auditing has seen a resurgence of interest [82,123]. Energy audits identify building inefficiencies through strategies such as walk-through inspections, on-site measurements, health and safety checks, blower door tests, visual inspections, and computer simulations [142]. Though time intensive, the DOE recommends home energy audits because of their impact on reducing energy use (e.g., 5-30% reductions in monthly utility bills) and improving housing stock (e.g., structural safety) [147]. Energy audits are also becoming part of Lead in Energy and Environmental Design (LEED) 1   Figure 1.1: Thermal cameras come in many form factors including handheld models (left) and more  recently released smartphone‐based thermal camera attachment (right); images courtesy of the DOE  website [146].  building efficiency certification programs [84,145] and municipal ordinances [8] with costs and repairs subsidized through government assistance programs [38,141,149,153]. While energy auditors employ numerous tools and techniques to assess the built environment, improvements to handheld infrared sensors and falling costs have resulted in the increasing use of thermography during walk-through inspections (Figure 1.1) [9,20,44,92]. Thermography is a data collection and visual analytics technique that uses hand-held thermal cameras to help detect, diagnose, and document energy issues that are visible as anomalous patterns of electromagnetic radiation. These patterns may indicate locations of air leakages, areas of missing insulation, or moisture issues in the built environment [109,122]. Not only do professional energy auditors use thermography to identify sources of energy inefficiencies, prior work has shown that when homeowners review thermal imagery from their energy audits, it positively influences retrofit decisions and conservation behaviors [66,122]. With respect to increasing the overall energy efficiency of the built environment, both outcomes are desirable. Recent releases of thermal camera attachments for smartphones (Figure 1.1, right) have encouraged consumer adoption of thermographic technology [158,159]. 2 Thermal cameras are being marketed toward general consumers for a wide range of uses, including Do-It-Yourself (DIY) energy audits, art and electronics projects, and outdoor recreation (e.g., see FLIR Systems’ marketing materials [162]). New consumer-facing smartphone applications support these activities, including applications for thermographic energy auditing [48]. Further, the first smartphones with fully integrated thermal cameras have been released [164] and low-cost thermal cameras are increasingly popular within maker communities for electronics and home-sensing projects [133,134]. Beyond individual use, new commercial methods for semi-automatic and automatic thermographic scanning of the built environment, including residential housing, are on the rise [11,103]. While still early, these trends foreshadow a future in which thermal cameras are ubiquitous—integrated into commodity electronics and consumer services. Despite the increasing availability of thermal cameras and their utility in energy auditing, the practice of thermography remains a laborious activity requiring training and experience [109]. There is also a surprising lack of understanding about people’s perceptions of this sensing technology or the challenges encountered by the increasingly diverse population of end-users. Moreover, there are few specialized tools or methods to support end-users—be they novices or professionally trained thermal camera users—in conducting thermography during energy auditing activities. To address these issues, we ask: How might automated and temporal thermography be incorporated into energy auditing practices? This dissertation focuses on three areas: (i) formative studies to understand and characterize current building thermography practices, benefits, and challenges among professional and novice thermographers, (ii) human-centered explorations into the role of automation and the 3 potential of pervasive thermographic scanning in the built environment, and (iii) advancing the state-of-the-art through the development and testing of new interactive building thermography systems. Within this dissertation, we present a set of studies that characterize thermography practitioners, explores prototypes of novel thermographic systems, and offers findings from field deployments. Our contributions to the fields of sustainability, computer science, building science, and human-computer interaction (HCI) include: (i) characterizations of end-users of thermography, (ii) critical feedback on potential automated thermographic solutions, (iii) a novel longitudinal thermography system designed to augment data collection and analysis activities performed by human auditors, and (iv) design recommendations for future thermographic systems. 1.1 Dissertation Research Approach and Overview This dissertation describes three threads of research (Figure 1.2). In the first thread, we characterize end-user behaviors, advance understanding of current practices in thermographic energy auditing, and identify what roles HCI may play in this domain, following methodological approaches common in sustainable HCI (e.g., [40,69,157]). This thread culminates in a set of design recommendations for future thermographic applications. In the next thread, we explore the technical knowledge and experiences of professional energy auditors and ask them to evaluate potential automated solutions to scale thermographic data collection and analysis using design probes. Building on the outcomes of these studies, in the final thread of research we describe the design and evaluation of an easy-to-deploy, longitudinal thermographic sensor system and accompanying reporting tools designed for residential building energy audits. 4 Figure 1.2: A visual overview of the three threads of research presented in this dissertation. Research  Thread 1 (left to right): YouTube analysis of novice uses of thermal cameras, a novice investigating a  building envelope during a field study, sample of novice collected data. Research Thread 2 (left to right):  Design probe of UAV collected thermal data, medium fidelity design probe of a data analysis interface,  observation of a residential energy audit. Research Thread 3 (left to right): Iterations of a longitudinal  thermographic sensor system, iterations of interactive end‐user reporting and analysis tools, sample  data collected during a field study.  5 1.1.1 Studies of Novices Thermal Camera Use and Energy Audits Commodity thermal cameras have only recently become available to the public [47,159]; consequently, little is known about how these end-users approach thermography, what challenges they encounter, or what benefits they perceive. To better understand these end-users, we designed and conducted two formative studies. In the first study, we qualitatively examined a dataset of 1000 YouTube videos showcasing non-professional, everyday uses of thermography. Our findings suggest thermal cameras are effectively used by novices to improve energy efficiency and our contributions include a characterization of common novice uses of thermal cameras. In the second study, we recruited 10 novice participants—persons with no previous experience using thermal cameras—for a four-week field study of end-user behavior that focused more specifically on building thermography and energy auditing activities. We examined key challenges participants encountered when collecting and interpreting thermal imagery during DIY energy audits, explored participants’ attitudes surrounding the technology and its application, and developed a set of design recommendations for supporting novice use of thermography. Findings from these studies influenced the design decisions in the remainder of this dissertation research and should similarly be helpful to researchers and applications designers developing tools for supporting thermographic assessment activities by end-users. 1.1.2 Studies of Professional Energy Auditing and Thermographic Automation Work in automated approaches to thermography has also grown markedly over the past few years. These approaches attempt to alleviate issues around how time consuming and challenging thermographic auditing can be, leveraging the disciplines of computer 6 science, robotics, environmental engineering, and civil engineering. To assist with thermographic data collection and analysis, researchers have primarily explored approaches for automatically transforming thermal images into high fidelity 3D representations of buildings [64,70,94,95,119] and deploying robots as a means of large- scale data acquisition [17,37,41,96,107,125]. However, there have been few user-studies nor any other types of human-centered evaluations of these solutions. Within this domain, we investigated current professional energy auditing practices, highlighting the role of thermography therein, while critically examining potential automated solutions to thermographic data collection and analysis. To accomplish this, we conducted two studies: a semi-structured interview study with 10 professional energy auditors, which each included five design probes, and an observational case study of an on-site residential building energy audit. The five design probes were based on research literature on automated thermography and were used to provoke and ground discussion. Findings provided insight into current professional procedures, challenges, and perceptions of thermographic automation, while the observation helped contextualize these findings. 1.1.3 Development and Deployment of a Longitudinal Thermographic Sensor System Thermographic sensors and home automation technologies are becoming increasingly popular, providing data about utility use, thermal comfort, and management of building resources (e.g., lighting, HVAC) [63,65]. Toward the goal of supporting the detection of structural degradation in building envelopes and providing energy auditors of varying skill with insights about key environmental and health and safety issues, we explore 7 augmenting residential energy audits and home automation systems with an easy-to- deploy, temporal thermographic sensor system. We used a two-phase design approach featuring several user studies to design this thermographic sensor system. In the first phase, we developed an initial sensor system and an interactive visualization tool, which we evaluated through a short deployment in a university building with a graduate student energy auditor and through a lab-based usability study with four graduate students with novice thermography experience. We then refined this system based on the feedback, improving the data collection system’s ability to be easily deployed and redeveloping our approach to the data visualization— instead using an automated, lightly interactive infographic. We then conducted three studies with this iterated system: a technical evaluation of the sensor system to determine its accuracy, in-home end-user deployments with homeowners, and semi-structured interviews including design probes with professional energy auditors. Taken together, findings from these studies highlight (i) the effectiveness of temporal thermography to assist end-users with gauging the severity of energy efficiency issues and (ii) the potential for new auditor-client interactions. Contributions from this work include the design of a novel temporal thermographic sensor system designed to support residential energy audits, a summary of the benefits and challenges users perceive about such systems, and design recommendations for supporting the needs of novice and professional energy auditors with future temporal thermographic sensor systems. 1.2 Summary of Contributions Contributions from Research Thread 1: Studies of Novices’ Thermal Camera Use and their Use of Thermography During Energy Audits 8  A characterization of non-professional, novice end-users of thermography with a focus on their DIY energy auditing practices.  An identification of key design recommendations for future thermographic systems and applications designed to support novice use. Contributions from Research Thread 2: Professional Energy Auditors Practices and Perspectives on Potential Automated Approaches to Thermography  A characterization of professional end-users of thermography and the role of thermal cameras in professional energy auditing.  A critical examination of recently proposed automated and semi-automated solutions to thermographic data collection and analysis in the built environment.  An identification of key design recommendations for future thermographic systems and applications designed to support professional use. Contributions from Research Thread 3: Development and Deployment of a Temporal Thermographic Sensor System  The design, development, and evaluation of a novel, temporal thermographic sensor system that can be used effectively by novice and professional energy auditors to collect and analyze thermography data in residential buildings.  A summary of the benefits and challenges associated with such systems.  An identification of key design recommendations for future temporal thermographic systems that support in-home use by novice and professional energy auditors. 9 1.3 Dissertation Roadmap Chapter 2 provides background on energy auditing and building thermography while also situating this dissertation research within the existing bodies of work in sustainable HCI, automated thermography, and temporal thermography. Chapters 3 and 4 contribute to the first thread of this dissertation research by describing non-professional, novice end- users of thermography and exploring their current use of existing consumer thermal cameras and thermographic energy auditing practices. Chapter 5 adds the perspective of professional building energy auditors, the second thread of this dissertation research, describing their use of thermography during energy audits and examining proposed automated thermographic systems. The third and final thread of this dissertation research is addressed in Chapters 6 and 7, which describe the design, development, and evaluation of an easy-to-deploy, longitudinal thermographic sensor system. Finally, Chapter 8 concludes this dissertation by synthesizing key findings across all three research threads, reviewing the contributions of this dissertation, and putting forth potential avenues for future work. 10 Chapter 2 Background and Related Work This chapter presents background information and discusses related work most relevant to this dissertation. The first section positions this dissertation within the literature on Sustainable HCI and then provides background information on relevant practices and technologies including: building energy audits, thermal cameras, thermography use in the built environment, and thermographic software for energy auditing. We then survey research related to the questions posed in this dissertation, including: (i) automated thermographic data collection and 3D reconstruction, (ii) quantitative and temporal thermographic analysis, and (iii) temporal thermographic visualizations. Finally, we describe an unexpected gap in thermographic research: a lack of end-user studies with professional and novice energy auditors. 2.1 Background In this section, we describe Sustainable HCI, how building energy audits are performed, the functionality of thermal cameras, the use of thermography in energy audits, and offer an overview of current thermographic data analysis and report-generation tools employed by building energy auditors. 11 2.1.1 Sustainable HCI Since its emergence at CHI in 2007 [15], a large portion of Sustainable HCI (SHCI) literature—the area of HCI research in which this dissertation is centered—has focused on curbing CO2 emissions through the design of eco-feedback [58] and persuasive [51] technologies (see surveys [21,42,89]). Work in this area frequently focuses on monitoring resource consumption (e.g., electricity [4], water [59]) or promoting sustainable practices that can influence emission rates (e.g., use of public transportation [57], recycling [30]). Looking specifically at home energy consumption, research has shown that technology- based feedback interventions can reduce energy consumption by 4-12% [43]. As Gardner and Stern note [62], these interventions place a disproportionate focus on curtailment behaviors, which involve forming new routines to reduce environmental impact (e.g., turning off lights when leaving a room), rather than on promoting one-time behaviors, such as upgrading a home’s insulation, which provide a lasting impact and can be far more significant to improving efficiency. Two recent “call to action” articles focusing on SHCI similarly outline limitations of the field and articulate paths forward [101,132], including the needs to draw from and study work outside of HCI, to pursue practical as well as fanciful research, and to address broader topics (e.g., electrical infrastructure vs. using energy-saving light bulbs). In this dissertation, we provide the first HCI-based examination of: (i) the everyday practices and views on thermographic energy auditing which can uniquely aid one-time performance upgrades in the built environment, (ii) the potential disconnects between the technology-driven research in automated thermography and the complexities, nuances, and practical demands of performing manual energy audits in the 12 field, and (iii) how elements from building science, computer vision, information visualization, and home automation research can be combined to enable new opportunities for improving energy efficiency and influencing attitudes about building maintenance practices which may potentially lead to new forms of human-building interactions (HBI) [5,106]. 2.1.2 Building Energy Audits As noted in the Chapter 1, energy audits to identify building inefficiencies are becoming increasingly common. For example, recent legislation mandates building audits every 5- 10 years in some cities [8]. Energy audits are conducted across building types (e.g., offices, industrial plants, multi-family residences) to investigate whether they are operating efficiently and to identify sources of inefficiencies in underperforming buildings [147]. Most energy audit programs extend from a Physical-Technical- Economic Model (PTEM) of energy consumption, which posits that technology improvements are the main drivers of energy efficiency [80]; the dynamics of human behavior within a building are not usually considered [81]. Our work focuses largely on residential energy audits, which are often incentivized by energy utilities and are increasingly being performed by private companies, government contractors, and even by the general public in the form of DIY energy audits. These audits often involve a range of evaluations from blower door tests1 1 A blower door is a powerful fan mounted on an exterior door that lowers indoor air pressure. This causes outside air to flow through unsealed cracks and openings. These air leaks appear as conspicuous streaks with the infrared camera [44]. 13 [44] to thermography [146]. Commonly, residential building audits begin with a walkthrough inspection to collect information about a building’s construction, on-site appliances, and environmental comfort. Health and safety components, such as testing to ensure large appliances are venting properly and not negatively impacting indoor air quality, are also typically associated with residential audits. This information is usually combined with the building’s historical data (e.g., utility bills) to make recommendations for improvement. In some cases, these data are entered into software tools that predict the expected financial return of efficiency recommendations. Finally, client-facing reports are produced outlining the efficiency recommendations. Despite the resurgence of interest in energy auditing and increasing number of energy auditing subsidy programs, the annual participation rate by homeowners in such programs is estimated to be about 3.2% per year (with even fewer homeowners actually performing subsequent renovations and retrofits) [13,82,123]. One potential reason for these trends is that many homeowners are not aware of what tools or services are available to help them combat household energy efficiency issues—assuming they are even aware that problems exist [123]. Within this sphere, the goals of this dissertation include (i) exploring thermography’s role in energy audits, (ii) understanding thermography’s influence on attitudinal and other barriers, beyond cost, that prevent homeowners from engaging in energy audits, (iii) developing tools that empower end-users and promote positive perspectives on energy auditing and building maintenance, and (iv) generating design recommendations that will enable technologists to design tools that support broader, appropriate, and effective use. 14 Figure 2.1: Commodity thermal cameras for smartphones and small electronics projects. From left to  right: FLIR Breakout Board, Seek Thermal Imaging Camera for iOS, FLIR One iOS Attachment Gen II,  FLIR One iOS Attachment Gen I.  2.1.3 Thermal Cameras Thermal camera technology became commercially available in the 1960s [50], though it was expensive, bulky, and intended for professional use. Today, thermal cameras are relatively inexpensive and readily available. FLIR Systems Inc™ and Seek Thermal™, for example, each sell consumer thermal camera attachments for smartphones at major retailers (Figure 2.1). Thermal cameras are marketed for a broad range of applications, including observing wildlife, rescue operations, electrical inspections, energy audits, medicine, and small electronics projects (see [162]). Thermal cameras work by measuring the electromagnetic radiation emitted by all objects with temperatures above absolute zero [60]; this data is typically converted into a common temperature scale (e.g., Celsius) when displayed on an end-user’s device. When displaying or saving an image or Thermogram, thermal data is combined with images from a conventional, co-located camera to provide context (i.e., object outlines) to the data during in-situ and retrospective analysis. Thermography is thus a data 15 collection and visual analytics technique used to generate insights from typically invisible phenomena that influence heat transfer in unexpected and potentially problematic ways. Unlike traditional photography, thermography requires end-users to account for factors that impact the accuracy of thermal measurements. For example, every material has a specific emissivity—a ratio reflecting how well an object emits heat compared to a perfect emitter. Users must therefore calibrate thermal cameras for each material being measured (e.g., glass, drywall) to acquire accurate temperature readings. Moreover, materials such as metals and glass may reflect infrared radiation from their surroundings (i.e., surrounding reflectivity), which further complicates the measurement and interpretation of thermal imagery. Finally, environmental factors including ambient temperature and relative humidity can negatively impact thermographic scans. Because of these complexities, professional energy auditors are typically expected to complete a thermography certification program (e.g., medical practitioners [127], building inspectors [109]) before operating thermal cameras or presenting findings within their practice. In this dissertation, particularly Chapters 3 – 5, we explore the benefits and challenges both novices and professionals face in using these tools during building energy audits. 2.1.4 Thermography Use in the Built Environment Energy auditors use thermography in the built environment to measure surface temperatures of walls, roofs, ceilings, floors, and other parts of a building’s envelope—the physical separator between the conditioned interior of a building and the unconditioned environment outside it. This enables auditors to detect heat loss, air leakage, moisture buildup, and locate hidden infrastructure (e.g., hot water pipes) [20,92]. Before surveying 16 a building, the thermographer must assess environmental conditions such as weather, wind, HVAC operations, and the direction/intensity of the sun as each of these factors can affect or prevent proper scans. For example, the International Organization for Standardization (ISO) requires a minimum temperature differential of 10°C between the interior and exterior temperatures of a building to properly detect thermal irregularities in a building’s envelope [83,150]. In addition, blower door tests are commonly used in conjunction with thermography to increase air flow between the building envelope and the outdoors which helps to highlight air leakage issues [44,146]. While the DOE recommends thermographic-based energy audits [146], criticisms of the practice include that it remains a qualitative method subject to the expertise of the auditor and lacks special software tools, algorithms, and audit guidelines [150]. This subjectivity has led to calls for developing quantitative methods and tools for collecting and analyzing thermal data as well as attempt to standardize the output of such initiatives [116]. This need, in part, motivates this dissertation; but before we describe our research toward this need, we first provide an overview of current software tools that assist energy auditors with employing thermography. 2.1.5 Thermographic Software for Energy Auditing Software applications designed for thermographic data collection primarily rely on direct user manipulation [130] and operate in a manner similar to photographic cameras; however, the applications tend not to provide interactive support for data collection (e.g., like facial recognition might on a photographic camera [152]) or training to help less experienced users collect useful thermographic data. Moreover, analysis of thermographic data, beyond a qualitative scan, is generally done retrospectively using a 17 separate software tool, making it difficult to go back and collect additional data if the need arises. The most commonly used capture and analysis tools are made by thermal camera manufacturers (e.g., FLIR Tools™ [49]) and enable energy auditors to manually annotate pixels (or regions of pixels) in thermal images with temperature data, correct parameters that influence the accuracy of the temperature measurements being displayed (e.g., emissivity, humidity, surrounding reflectivity), and improve visual contrast to make it easier to gather insights (i.e., a process called thermal tuning [160]); see Figure 2.2. A skilled energy auditor can use these tools to collect and analyze thermal imagery. However, the focus of most of these tools (e.g., FLIR Tools [49]) is on streamlining professional reporting, not on supporting auditors in analysis or decision making. In Chapters 3 – 5, we conduct studies that elicit feedback on such tools and Figure 2.2: Thermographers use thermal imaging software like FLIR Tools [49] to surface thermal data  captured by their cameras. In this image, the thermographer has applied a pixel label, an area label, tuned the image using the thermal tuning slider, and adjusted the palette colors that communicates the thermal differences in the non‐labeled portions. Metadata and details about the labels are available in the right panel; image courtesy of FLIR Systems.  18 develop design recommendations for future software applications that support both novice and professional energy auditors. Later, we leverage these design recommendations in designing our own tools to support residential energy auditing (Chapters 6 and 7). 2.2 Related Work Having provided background on topics relating to this dissertation, we now turn to surveying related research on: (i) automated thermographic data collection and 3D reconstruction, (ii) quantitative and temporal thermographic analysis, and (iii) temporal thermographic visualizations. Finally, we describe an unexpected gap in thermographic research: a lack of end-user studies with professional or novice energy auditors. 2.2.1 Automated Thermographic Data Collection and 3D Reconstruction Most thermography research focuses on automating thermographic data collection. Common automation approaches include employing robots and vehicles to scale up data collection (e.g., [17,37,41,96,103,107,125]) and often have the goal of transforming the collected thermal images into 3D-reconstructions of buildings (e.g., [64,70,94,95,107,119]); see Figure 2.3. The emphasis on using 2D data to create 3D reconstructions stem from the limitations of 2D thermal images: (i) they do not include geometry and spatial relationships, which are important for interpreting thermal imagery [70,94]; (ii) they are unordered, messy, and difficult to organize [64,70]; (iii) and they require time-consuming and labor intensive post-hoc analysis [64,70,119]. 19 Figure  2.3:  Examples  of  automated  thermography  from  literature  (from  left‐to‐right):  UAV‐based  thermography [96]; a textured 3D façade model [125]; 3D thermographic reconstruction of a kitchen [70]; a UAV equipped with a thermal camera [125]; the Irma3D indoor thermal mapper robot [17].  Thermographic 3D reconstruction is cast as a solution to these problems and as a means of enabling better modeling that should reduce auditor error and subjectivity [64,94]. However, substantial amounts of data are required for the 3D-models, whether they are built either by computational photography methods—e.g., structure-from-motion (SFM) [64,70,107]—or through the use of precise range scanners such as LiDAR, which are texture-mapped with thermal images [95,112,119]. Thus, researchers are increasingly using automated and semi-automated robots for data collection, including ground-based rovers for indoor thermography (e.g., [17,37]) and UAVs for outdoor thermography (e.g., [41,96,107,125]). The robots are equipped with a suite of sensors such as thermal and optical cameras, laser scanners, and GPS. These “massive data acquisition” systems [96] are considered advantageous because they reduce manual labor, can survey otherwise inaccessible areas of buildings (e.g., high floors, rooftops), and collect more precise data. They may also enable or facilitate new types of analyses (e.g., surveying and comparing thermal performance from large numbers of buildings [103], creating datasets that can be used in temporal analyses [35,116]). Given the technical complexity of collecting thermal data and rendering 3D models, most research thus far has focused on technology evaluations (e.g., verifying the accuracy of geometric models [70]) rather than user studies. Indeed, we could find no 20 prior work that explored the auditor perspective of these emergent methods, that attempted to elicit user feedback to early models/designs, or that tried to demonstrate that 3D reconstructions enabled auditors to better detect building defects or energy inefficiencies compared with their 2D counterparts. In Chapters 5 and 7, we begin to address these gaps by eliciting feedback from professional auditors on design probes inspired by the automated thermography literature and our own work, respectively. 2.2.2 Quantitative and Temporal Thermographic Analysis Two common criticisms of thermography in energy auditing is that it is subjective and inaccurate. Thus, there have been several proposed quantitative methods for improving thermographic assessments of a building envelope by measuring the rate of heat transfer through it, which is typically referred to as its U-value or (in the US) its R-value2 [54,64,71,75,99,120]. An early procedure for performing these calculations was put forward by Madding [99], which relies on ensuring that environmental conditions for thermographic scanning are met and that various environmental measurements are collected to solve Equation 1: (1) 2 In the US, insulation sold by manufactures is rated on an R-value scale, whereas using U-value scale is most common in the world, including in related literature; these measures are reciprocally related once unit conversions have been accounted for (e.g., metric vs imperial units). 21 Here, the auditor needs to know the temperature difference between the wall surface and inside air (ΔTa), between wall surface and surrounding reflectivity (ΔTr), and between inside air and outside air temperatures (ΔTio). Additionally, the auditor needs to estimate the emissivity (ε) of the surface material under measurement, measure and cube the average temperature of the surface area (Tm3), and rely on empirically validated constants such as the Stefan-Boltzmann constant (σ). By solving this equation, Madding was able to quantitatively estimate the performance of a stud frame constructed wall within a 12% deviation from its known performance value under winter weather conditions. However, Madding notes that the limitation of this approach includes that two measurements taken some time apart will likely differ due to changes in environmental parameters (e.g., exterior temperature rising) and increased procedural complexity. Moreover, it is rare that precise performance values of a building envelope would be known a priori when trying to apply this technique to aging building stock. To address these issues, several temporal methods of analysis with varying modifications on Madding’s formula have been proposed [6,34,52,99,113–115]. These methods typically require similar data to be collected over time with the resulting data being averaged, which researchers have shown to be repeatable and at least as accurate as Madding’s formulation. To acquire ground truth data and demonstrate that these methods could be applied to building envelopes of unknown construction, performance calculations based on temporal thermography are frequently compared to the Heat-Flow Meter (HFM) method [23,83] and, more recently, the THM method [14]. Both methods use contact sensors to measure surface temperatures directly over several days. Nardi et al. [114] performed a comparative analysis of these methods and found the formulation 22 put forward by Albatici et al. [6] formulations to be among the most accurate. The authors of these studies suggest that these temporal thermographic methods are reliable enough to be used more broadly as a general measuring technique for energy auditing applications, though the technique has not been applied outside of research environments (e.g., in actively lived-in residential homes). The primary advantage of thermographic assessments over direct surface contact measurements is that they can be performed over several hours instead of several days. Despite this benefit, numerous limitations remain. The complexity of setup procedures, the numerous pieces of sensor equipment necessary to collect the data, and the multi- hour timeframe required to acquire data each hinder integration with current energy auditing practices [56]. Moreover, calibrating thermal cameras for quantitative assessment is not commonly done by auditors in the field due to their reliance on qualitative scanning techniques, personal experience, and potential for error associated with estimating calibration parameters like the emissivity of surface materials. To address this latter point and better align this analysis approach with current practices of energy auditors, it may be possible to borrow from computational methods to partially automate and scaffold these procedures. Recent work in material recognition [12,79,98] could be helpful in allowing thermographic systems to automatically assess an image and infer material properties about the measurement area—specifically, the emissivity of materials in the images. In 2015, Bell et al. [12] released several classification models that infer materials contained in regions of images (e.g., wood, metal). These models could be adapted to infer emissivity values used to partially calibrate thermal cameras for thermographic scans automatically, which would not only facilitate reducing 23 the complexity of the calibration process but could also improve accuracy and reduce operator error in other thermographic assessment activities. In Chapters 6 and 7 we propose an easy-to-deploy, longitudinal thermographic sensor system designed to augment residential energy audits. To reduce the complexity of setting up and configuring our system for temporal scanning, we use a collection of off-the-shelf sensors (housed in a single custom enclosure) to collect the required data and rely on computational methods to automatically infer emissivity values of the materials in captured images easing setup procedures and reducing the potential for operator error. Finally, the system uses the collected data to analyze regions of interest specified by the user—comparing the performance of the envelope to regional building codes using an if-then rule-based strategy [138] for recommendation generation (e.g., repair insulation, improve thermal comfort). 2.2.3 Temporal Visualizations of Thermographic Data Advances in automated thermographic data collection promises to enable new forms of analysis. To date, using this type of data for diagnostic purposes is rare because it remains difficult to collect, analyze, and visualize in meaningful ways. Consequently, most visualizations of this data are often non-interactive and rely largely on an individual’s qualitative assessment rather a quantitative analysis. A common method for visualizing temporal thermography data is through the use of small multiples [143], which some in the building science community have utilized to gain qualitative insights about degradation [35], construction practices [56], and 24 Figure 2.4: Fox et al.[56] uses a small multiples method to look for evidence of previous repairs and the effects these have had on building performance.  thermal comfort [116]. For example, software tools similar to those discussed in Section 2.3 were used by researchers in [56] to go through thermal imagery collected via a temporal scan (i.e., approximately 144 images, collected 30 minutes apart over the course of three days). They manually identified patterns and gained insights about the building being surveyed and later constructed the visualization shown in Figure 2.4 that reveals a construction defect in a structural element of an 18th century building. Another promising visualization for temporal thermographic data was put forward by researchers in the historical building preservation community [35], who collected data over a period of 3-hour and visualized the change in surface temperature using a parallel coordinate plot. The plot was part of an interactive visualization tool which the researchers used to understand the degradation of a building’s envelope/facade (Figure 2.5) by applying common visual analytics techniques (i.e., data filtering, data abstraction [131]). 25 Figure 2.5: Danese et al. [35] monitors a wall for several hours (after sunset) and notices differences in cooling, indicating areas that have been patched or are degrading. The average temperature change  is displayed in the top left, the top right is a SOM representation of this same data, while the bottom two graphs represent temperature change between data collection periods (one photo every hour).  While the insights that these visualizations provide are promising, it is unclear if energy auditors working in the field would be able or inclined to use these techniques to gain insights about the building they are surveying; it is also unclear what benefits and challenges might arise. In Chapter 5, we discuss such opportunities and issues with professional energy auditors through our design probes. In Chapters 6 and 7, we further explore the use of these temporal thermographic visualizations in the field. 2.2.4 End-Users of Thermography Studies of energy auditing tend to focus largely on the potential environmental benefits [10] and/or on the building-owner perspectives [80,81]. For example, in a large-scale study of homeowner experiences, Ingle et al. [82] found that physical, face-to-face discussion with auditors was critically important to successful audit (“the most informative part of the whole process”, p.13) and that the use of thermography was “particularly compelling” (p.16) because it made invisible energy flows and leakage problems more 26 tangible. This latter point is crucial toward motivating end-users to make changes. Similarly, in an experimental study of 87 homes Goodhew et al. [66] found that those households who saw thermal imagery from their audits were nearly five times more likely to install retrofits. Thus, thermography is not just a measurement approach: it is a way of effectively communicating findings to building owners that motivates action. Despite this influence, studies of end-users of thermography are rare (i.e., the energy auditors who use thermography). One exception is a study by Palmer et al. [123], who surveyed 459 professional auditors and explored common audit practices, shared challenges, and the degree to which homeowners took action on efficiency recommendations. Though thermography was not their primary focus, they found that 63% of the auditors they surveyed used thermography “fairly often” or “always.” Among those energy auditors who did not employ thermography during audits, the primary impediment was equipment cost. With regard to novice thermographers, who now have increased access to thermal cameras via smartphones, there has been no examination of how they approach using this technology or what challenges they face when interpreting thermal imagery. Due to the subjective nature of thermographic energy auditing it is unclear whether novices can perform thermographic energy audits, especially in the absence of tools designed specifically with them in mind. The user studies presented in this dissertation (across Chapters 3 – 7) offer a complementary, qualitative perspective on energy auditing research with a specific emphasis on thermography. The design recommendations generated offer potential solutions to concerns raised by auditors about current thermography practices as well as proposed automated solutions for scaling data collection and analysis. 27 2.3 Summary This chapter has provided background on the practices and technology that underly this dissertation—including how building energy audits are performed, how thermal cameras operate, the use of thermography in the built environment, and existing thermographic software for energy auditing—as well as related work across the areas of automated thermographic data collection, quantitative and temporal thermographic analysis, and temporal thermographic visualizations. Within each of these sections we have described how the work in this dissertation either builds on and extends or fills overt gaps in the literature, toward the overall goal of illustrating how automated and temporal thermography can be incorporated into energy auditing practices. 28 Chapter 3 Characterizing Novice Thermography   Figure 3.1: A montage  video of  thermal  camera observations  from YouTube video V79 showing  (a) electrical power lines, (b) a woman with a jacket outside in the cold, (c) hot coffee with cream, and (d) hand washing with cold water.  This chapter begins our first thread of research. Here, we describe a formative study of novice users of thermal cameras through a qualitative analysis of 1,000 YouTube videos depicting everyday use. Our findings provide: (i) a high-level characterization of common thermographic use cases and extend discussions surrounding the challenges novice encounter, (ii) evidence that thermographic energy auditing by these users can have an impact on building energy efficiency, and (iii) initial insights into the design of future thermographic systems for energy auditing and other consumer-oriented applications. This chapter has been adapted from a paper to appear at MobileHCI 2018 [108]. 29 3.1 Introduction From its faux use in movies like 1987’s Predator [31] to its recent artistic use by the rock band 30 Seconds to Mars [24], thermal imaging has long captured public interest. Until recently, thermographic technologies—which capture and display patterns of heat from infrared emissions—were bulky, prohibitively expensive, and intended for professional use [109]. Today, low-cost thermal cameras are widely available for smartphones either as mobile attachments (e.g., FLIR One™ [158]) or built-in to the phone (e.g., CAT S60 [164]). Small, inexpensive thermal sensors are also widespread on “maker” electronic sites (e.g., Sparkfun’s FLiR Dev Kit [133]). Software development kits, interactive tutorials, and online communities have grown commensurately to share thermographic knowledge and create novel applications. Thus, what was once considered an expensive, expert technology is quickly becoming ubiquitous with a growing, diverse userbase. Despite these developments there have been no investigations into commodity thermal camera use and adoption patterns, which this study seeks to address. Published at MobileHCI 2018 [108], the research questions examined in this work are exploratory, intended to advance understanding of thermographic end-user behavior, and include: What activities do non-professional users of mobile and handheld thermal cameras engage in and why? What level of understanding about the technology is demonstrated? How might these observations inform the design of future thermographic technologies? To address these questions, we collected and qualitatively analyzed 1,000 thermographic videos from YouTube. Our research methods were informed by previous work [7,16,22,77], which combined structured manual search with qualitative coding to acquire and analyze large datasets of user behavior from Online Social Networks (OSNs) 30 (e.g., Twitter, YouTube, Thingiverse). These types of studies are successful at reaching user populations that are difficult to access directly and often provide insights into users’ natural technology interactions [7,22,77]. However, qualitatively analyzing data from OSNs can nonetheless be challenging: query results can be large (i.e., in the thousands or more) and noisy [144]. To mitigate these challenges, past studies have examined a single query and downsampled the results [16] or conducted multiple queries using a systematic search strategy on highly specific topics [7,22]. In our work, we combined manual search methods with semi-automated techniques common in information retrieval to extend the dataset and manage volume before conducting an analysis of the video content itself. In addition to video analysis, we also assessed YouTube-user- provided questions about thermography and their answers through an analysis of each video’s comment section. Finally, to complement this video and comment analysis, we invited content creators to complete an online survey about their thermal camera use, motivations, and experiences posting on YouTube. Results show content creators were eager to learn about and test the limitations of their thermal cameras as well as their practice of thermography while engaging in a myriad of activities. Consumers primarily used mobile thermal camera attachments, initially purchased for purposeful activities but are later used for entertainment and exploration. Content creators often engaged in uploading informal exploration videos (Figure 1)—those that depicted their observations and play—as well as videos that focused on three areas: (i) building audits and urban observations, (ii) small electronic and software projects, and (iii) outdoor recreation and agricultural uses. Moreover, the most important aspect with respect to the rest of this dissertation are findings that point toward 31 non-professional, novice thermography use having a positive impact on energy efficiency in the building sector by helping to motivate people to make retrofit decisions (e.g., repair building insulation) and to support claims about issues that are usually not directly observable (e.g., verify results of an insulation repair). This work’s primary contributions include the first study of “in the wild” data depicting everyday uses of commodity thermographic technology by non-professionals and a characterization of common novice uses of thermal cameras with a further focus on novice building thermography. Additionally, a secondary contribution is the extension of methods used by recent qualitative studies of data from OSNs [7,16,22,77] through the use of a hybrid manual+computational approach to dataset generation. We conclude this study with discussions of this approach as well as novice “in the wild” uses of thermal cameras, the challenges and misconceptions they encounter, and implications for the design of future thermographic systems and tools. 3.2 Method Similar to previous qualitative studies of user-generated content on OSNs [7,22,77,121], this study was conducted in three stages: first, we generated a dataset containing OSN data relevant to the target domain—in this case, videos featuring novice use of thermal cameras. Second, we qualitatively analyzed video content along multiple dimensions. Finally, we conducted an online survey soliciting additional information from content creators (i.e., the persons who posted the YouTube videos). 32 Step Terms Step 1: infrared, lepton, thermal, thermal camera, thermal image, thermal imaging, Initial Keywords thermography Step 2: breast thermography, flir lepton, flir one, flir thermal, imaging camera, infrared Expanded Keywords camera, infrared thermography, night vision, seek thermal, thermal imager Step 3: everyday use, product review, news coverage, unboxing, professional demo, Iterated Codebook advertisement, off topic Table  3.1:  The  search  terms  and  training  data  codebook  used  to  assemble  our  study  dataset  throughout SMIDGen’s four steps.   3.2.1 Dataset Generation We generated the dataset using SMIDGen (Scalable, Mixed-Initiative Dataset Generation) [105], a hybrid manual + computational approach to collecting large amounts of relevant, OSN-sourced data. SMIDGen has four steps: (i) manually exploring an OSN to generate an initial set of keywords, queries, and data, (ii) computationally expanding these queries to increase domain/topic coverage, (iii) mixed-initiative data labeling and training to construct automated models, and (iv) applying these models at scale to generate a large, diverse, but still relevant, final dataset. Step 1: Creating an Initial Dataset. In July of 2017, we searched YouTube for the quoted string “thermal camera” alone and in combination with keywords representing common thermographic applications (e.g., “building”, “medical”). We then manually assessed the search results to construct a list of general thermography-related search terms (Table 3.1). Next, we queried these terms via the YouTube Data API (v3) to create an initial dataset. Following Anthony et al. [7], we extracted the first 200 YouTube results for each term and stored the resulting video URL and metadata (title, description, view counts, etc.). In all, the search results contained 1,400 videos, which was reduced to 1,092 after removing duplicate videos. 33 Step 2: Automatically Expanding the Dataset. To identify keywords that YouTube content creators commonly used to describe their videos in addition to the keywords generated, we applied two standard query expansion algorithms: word co-occurrence and Kullback-Leibler Divergence [93]. After applying these algorithms to the 1,092 videos’ titles and descriptions in our initial dataset, we merged the top ten keywords from each method and identified 13 new, unique search terms [27]. We then queried each new keyword alone and in combination with the initial keywords extracting the top 200 videos in each query (similar to [7]) to capture videos the initial search may have missed. This process generated an expanded dataset of 6,790 unique, potentially relevant videos. Step 3: Mixed-Initiative Analysis and Modeling. Keyword-based queries are imprecise, thus a subset of these 6,790 videos are expected to be irrelevant to the thermography domain. Even within the thermography domain, specific types of videos were off-topic for the research questions (e.g., product reviews or unboxing videos don’t portray everyday use of this technology). Manually filtering thousands of videos for relevance (i.e., thermal camera use) and topic identification (e.g., everyday use) is time- and labor- intensive. To accelerate these tasks, we used a mixed-initiative approach that employed classification algorithms to learn what constitutes relevant and topical videos. To create training data for these classification algorithms, two research assistants iteratively coded the initial dataset from Step 1 using the traditional coding process in [18,78]. They began with a modified codebook from [16], which offered high-level codes typifying smartphone use videos on YouTube (Table 3.1). Video titles, descriptions, and the content were used as input. Each video was labeled with a single category and Cohen’s kappa was used to calculate inter-rater reliability (IRR). After three rounds of coding, 34 each on 200 randomly selected videos, average IRR across codes was 0.69 (SD=0.09), considered good agreement [151]. The research assistants then divided and coded the remaining Step 1 data (N=1,092). This data was then used to train a machine learning classifier to complete the relevance and topic filtering tasks. To convert YouTube videos into a training samples, we featurized the videos by converting their titles and descriptions into a bag-of-words model and re-weighting terms using term-frequency, inverse-document-frequency (TF- IDF) to reduce the weights of common keywords, as is standard in information retrieval research [27]. Following an evaluation of several classification algorithms (see [105] for an in depth description of this process), we selected a Random Forests model to identify domain relevance (e.g., is the video about thermal camera use) and the Logistic Regression model to identify specific sub-topics (e.g., everyday use). Using 10-fold cross- validation, the accuracy of the relevance and topic classifiers were 0.91 and 0.73, respectively. The topic classifier’s lower accuracy is to be expected since the semantic similarity between in- and out-domain videos is likely much lower than in-domain videos of different topics (e.g., an irrelevant video about gaming likely has fewer words in common with a thermography video than a video about unboxing a thermal camera has with a video about using that camera to observe heat loss in a home). Furthermore, to avoid accidental omission of “everyday use” videos, we chose to prioritize recall over precision to obtain potentially more diverse data from the topic classifier. As researchers would review all videos classified as “everyday use” and could remove off-topic videos at that stage, this prioritization does not impact the results. 35 Topic Codes Sub-Topic Codes Content Areas Building and Urban Environments, Health and Wellness, Paranormal (N=10) Investigations, Electronics and Software Projects, Recreational Outdoor Activities and Agriculture, Informal Exploration, Pollution Activism, Vehicles, Research, Security and Emergency Services Misconceptions See Through Objects, Measure Air Temperature, Measure Gases, Faux Filters, Faux Thermal (N=6) Imagers, Camera Operation Issues Comments Containing Q/A Content Questions, Technical Specifications, Follow-up Request, Other (N=4) Table 3.2: Topic and sub‐topic codes applied to analyze the content of “everyday use” videos.   Step 4: Applying Classifiers and Final Dataset. Finally, we applied these classifiers in sequence—relevance filtering then topic identification—to the unlabeled data from Step 2. Research assistants manually validated the output of 200 randomly sampled videos from each classifier, finding the relevance and topic accuracy to be consistent with the F1 scores. The final labeled dataset included 1,686 videos from 772 human-labeled videos and 914 machine-labeled videos. From this final dataset we randomly sampled 1,000 videos for further content analysis. 3.2.2 Qualitative Analysis of “Everyday Use” Video Content We qualitatively analyzed the 1,000 sampled videos to investigate the research questions regarding how and why people use thermal cameras. We coded the videos using a combination of inductive and deductive codes by using the video titles, descriptions, content, and comments. Non-everyday use videos were coded as “off-topic” and no further action was taken. The codebook (Table 3.2) included 16 dimensions across two topics: content areas (e.g., outdoor recreation, agriculture) and misconceptions (e.g., thermal cameras can see through walls). Videos containing questions (e.g., in the video description, in the comment feeds) were further analyzed across 4 dimensions (Table 3.2) describing the question content. When determining in what activities non- 36 professional users most often engaged, we coded each video for its primary content (i.e., the activity that took up at least 80% of video’s duration). We randomly selected and coded 20% of the data (200 videos), achieving an initial IRR of 0.68 using Cohen’s kappa [151]. After resolving disagreements and clarifying the codebook, we coded a new, randomly selected 20% sample of the data and achieved an average IRR of 0.75 (SD=0.27). After resolving differences, the remaining 600 videos were divided and coded independently. Ultimately 67.5% (675/1,000) of the videos in the dataset depicted everyday use, the rest being thermography videos with other focuses (e.g., marketing, professional services). Findings will focus on the content, misconceptions, and community responses around these 675 videos. Comment Feed Analysis. We performed an additional analysis of the 209 (20.9%) videos that contained questions in either the video description or posted in the comment feed. For each video, we reviewed questions asked within the top 20 “most popular” comments. Questions from content categories accounting for ≥10% of the dataset (165 questions, Table 3.3) were coded into four categories:  Content questions about the video’s subject matter (e.g., “Aren't hornets cold blooded?”)  Technical specification questions about the devices being used or the process of making the video (e.g., “What kind of camera did you use?”)  Follow-up requests to make more videos on the same or different topics (e.g., “Can you do a video on the heat emitted by cellphone usage?”)  Other questions (e.g., “What is this music playing?”) 37 Dataset Average Median Contains Q&A in Categories (N=675) Duration (SD) Views Misconceptions Comment Informal Exploration 46.5% 2.28 9.8% 27.7% (314) (5.11) 507 (31/314) (87/314) Outdoor Recreation & 16.1% 3.24 0.9% 34.8% Agriculture (109) (7.50) 807 (38/109) (38/109) Electronic or Software 11.9% 3.03 1.2% 28.7% Project (80) (4.70) 368 (1/80) (23/80) Buildings and Urban 11.1% 3.06 4.0% 24.0% Observations (75) (4.11) 351 (3/75) (18/75) Vehicles 6.5% 1.90 822 0.0% 27.2% (44) (2.48) (0/44) (12/44) Paranormal 2.8% 4.30 10.5% 63.1% Investigations (19) (4.25) 2327 (2/19) (12/19) Emergency 2.1% 1.09 7.14% 28.5% Applications (14) (1.05) 637 (1/14) (4/14) Health and Wellness 1.8% 5.19 0.0% 0.3% (12) (7.49) 2116 (0/12) (4/12) Research 0.9% 1.02 (6) (0.80) 385 0.0% 16.6 (0/6) (1/6) Pollution Activism 0.3% 0.34 103 0.0% 0.0% (2) (0.03) (0/2) (0/2) Table 3.3:  The categorical results from our coding process sorted by frequency. Categories were  exclusive (i.e., videos were coded into a single category).  Moreover, for each question we recorded whether an answer was posted and, if so, who the respondent was: the original poster of the video, other YouTube users, or both. We analyzed the correctness of responses related to thermographic misconceptions but not for more general discussions (e.g., camera costs, background music titles). 3.2.3 Online Survey To complement the qualitative video analysis, we surveyed YouTube content creators with videos in the dataset. The online survey asked about demographic information, reasons for owning a thermal camera, usage patterns, motivations for posting videos online, and perceived benefits from engaging with the YouTube community. As this dissertation is focused on the role of thermal cameras in energy auditing, we also asked what impact, if any, thermal cameras had on building improvements or energy usage. 38 Figure 3.2:  The  images above portray a  typical video  from each coded category and the category’s percent representation in the overall dataset (N=675).  The survey included 5pt-Likert questions, check-all-that-apply questions, and open- ended, short-response questions. We contacted all unique content creators (N=1,023) in the final everyday use dataset generated in Step 4 of the dataset generation process using YouTube’s direct message feature and a pre-scripted macro. Participants who completed the survey and opted to voluntarily disclose contact information were entered in a raffle for one of two $20 Amazon gift cards. In all, 78 participants (7.6%) completed the survey, which had an average completion time of approximately 8 minutes. 3.3 Findings from Video Analysis We report on the most common everyday uses of thermal cameras shown in YouTube videos (n=675), when misconceptions occurred, and the information users exchanged in question and answer discourses. Overall, we found four primary uses of thermal cameras in practice and a knowledgeable base of users who respond to questions and provided information. Quotes from content creators—transcribed or from video descriptions—as well as commenters are attributed using a ‘V’ followed by the video number. 39 Figure 3.3:  Illustrative examples described  throughout our  findings:  (a)  exploring whether  thermal  cameras see through water in V90, (b) comparing a Raspberry Pi’s internal temperature sensor to a  handheld thermal camera reading during a stress test in V801, (c) an exterior home inspection in V351,  and (d) describing how convection in hot coffee causes cells to be visible in thermal images in V154.  3.3.1 Common Thermal Camera Usage Activities The most common thermographic videos focused on informal exploration (46.5%), outdoor recreation and agriculture (16.1%), electronic and software projects (11.9%), as well as building energy audits and urban observations (11.1%). Less frequent categories (<10% of the dataset) included vehicles, paranormal investigations, emergency applications, and health and wellness (Table 3.3; Figure 3.2). The average video duration was 2.7 minutes (SD=5.3 min), and most covered a single thermal observation (e.g., coffee brewing). Below we expand upon the four most common thermal camera uses. Informal Explorations. Nearly half of all everyday use videos (46.5%, 314/675) were informal explorations (314/675). Many (19.1%, 60/314) of these videos focused on how an individual phenomenon appeared in infrared (e.g., nostril temperature when breathing, setting a ping pong ball on fire, thermal handprints on different surfaces, running water in sinks or over a person’s hands). While the subject matter was very diverse, some of the most common observations within this category included household pets (9.9%, 31/314), filming the user’s face (8.9%, 28/314), coffee cups and brewers (8.2%, 26/314), running water in sinks and bathtubs (4.4%), and children (2.2%, 7/314). 40 Other interesting, but less common, subject matter included crushing objects in a hydraulic press and looking at the heat dispersion, throwing liquid nitrogen down a hallway, recording the effects of incendiary devices (e.g., model rockets, fireworks), and observing the extrusion process of 3D printers. Some content creators chose to create montage videos (14.6%, 46/314) to call attention to the diverse phenomena they investigated with their thermal cameras (e.g., a user filming a coffee pot, then looking at an electrical appliance, then an insulation problem in the home; Figure 1). These montages occasionally featured short segments related to other content areas (e.g., wildlife, electronics), but still emphasized exploration. Another common type of informal exploration was testing the technical limits of the thermal camera (11.8%, 37/314) by, for example, walking away from the camera to test its detection range and clarity. These videos typically explored how well a thermal camera could distinguish objects at various distances as well as the properties of different materials (e.g., reflectivity of glass). For example, one video asked, “Can a Thermal Camera See Through water?” (V90, Figure 3.3a): “I’m going to dip my hand down into the aquarium, right into the water on the top, and let’s see what happens. I’m going to calibrate the camera first and dip my hand in the water. (Dips hand in aquarium.) Yeah, the surface of the water really reflects the heat away. But we can actually see my hand is heating the very surface of the water. […] So yeah, the thermal camera doesn’t see through water very well, but it is sensitive enough that you can actually see my hand warming up the water. Pretty cool.” (V90) 41 Videos investigating if or how well a thermal camera could “see in the dark” were also relatively common (12.8%, 40/314). Some experiments had targeted applications, such as parents attempting to observe whether their children were sleeping without turning on the lights or a father mounting his thermal camera to a UAV to find a child’s lost headband in a backyard at night. Outdoor Recreation and Agriculture. Outdoor recreation and agriculture was the second most common type of video (109/675; 16.1%). This included passively observing farm animals and wildlife (42.2%, 46/109) and hunting (e.g., deer, boar) (22.9%, 25/109). For example, the creator of V668 stated: “I see many birds while hiking with the thermal imager at night. Most are sleeping, some are nocturnal.” Other activities included walking dogs (9.2% 10/109), cloud watching (8.3%, 9/109), and beekeeping (10.0%, 11/109). Electronic and Software Projects. Electronic and software projects was the third most common (11.9%, 80/675). Most often these videos were styled more as time-lapses of how electronic devices managed heat (38.8%, 31/80)—either heating up, cooling down, or ventilating heat during operation. In V801 (Figure 3.3b), for example, one content creator compared a Raspberry Pi’s internal temperature sensor to a thermal camera reading during a stress test: “The temperature spikes up quite quickly and you'll notice when it hits the 80C mark it starts to throttle the speed. [However,] the temperature outside on the chip is significantly higher as you can see.” (V801) Videos in this category also showed users specifically diagnosing issues (22.5%, 18/80) such as a missing component on a printed circuit board: “Now that we have a 42 thermal camera we can see that the card quickly detects that there is no heatsink and [it] throttles itself to prevent damage” (V572). Finally, a few videos (18.9%, 15/80) demonstrated using thermal cameras as a sensor for a software project. Notable examples included detecting and using thermal input for an interactive table. Building Energy Audits & Urban Observations. Finally, building energy audits and urban observations comprised 11.1% (75/675) of the everyday use dataset. During home inspections, users either performed a general walkthrough of their home or focused solely on a problem area. They investigated large appliances (18.9%, 14/75) (e.g., as in V199 of a faulty radiator), hidden structures (14.7%, 11/75) (e.g., wall studs, insulation issues), electrical panels (10.7%, 8/75), air leakage around a window or door (10.7%, 8/75), and moisture issues (2.7%, 2/75). General urban observations (e.g., train yards, people walking on city streets) made up 10.7% (8/75) of videos in this category. Some users (13.3%) seemed to be knowledgeable about how environmental factors may influence their inspections. For example, the user in V548 stated: “I’m out here early for a reason, this wall catches all the afternoon sun.” implying that later scans would be problematic because solar loading would impact measurement accuracy. Similarly, in V351 the user described the importance of temperature differentials for proper energy audits of building envelopes (Figure 3.3c) [84]: “I used my new Seek Thermal camera […] to look at the exterior of my house when it was -19C outside. You can see the heat loss of my foundation, the front door, and my 20+ year old single pane windows.” (V351) 43 Other Video Categories. The remaining six categories each accounted for 10% (0.2- 6.5%) of our dataset and are briefly summarized here. For vehicles (6.5%, 44/675), videos included passive observation of vehicles in motion, actively diagnosing component issues (e.g., defective heating coils in a steering wheel), or engines heating up. Paranormal investigation videos (2.8%, 19/675) showed users exploring ghost sightings, tracking UFOs, and looking for Bigfoot. Health videos (1.8%, 12/675) focused on the potential diagnostic properties of thermography, such as checking body temperature or detecting cancerous growths near the skin’s surface. Finally, two videos focused on gaseous output from energy production facilities and were coded as pollution activism (0.3%, 2/675). 3.3.2 Misconceptions We found four types of misconceptions about thermography and three types of technical misconceptions, which were present in 5.3% (N=36) of the videos, to be common. For each video we reviewed the comment thread to determine whether the misconception was corrected by another member of the community. The most common thermography misconception (31.4%), which was likely satirical, suggested that consumer thermal cameras could image flatulence. These videos were strongly rebuked by commenters who described the inability of standard thermal cameras to observe gases. The second most common misconception (19.4%) was that thermal cameras could directly measure ambient air temperatures by viewing the effects of hot/cold air on a surface or imaging condensation (e.g., a person heavily exhaling in the cold and imaging the moisture vapor). Again, in all cases, this misconception was corrected in the comments section. Third, 13.8% of videos claimed that thermal cameras could “see through” clothing or walls; however, thermal cameras can only measure 44 surface temperature. For instance, the “see through” effect of clothing does not actually show a naked person, but instead highlights areas where body heat transfers through layers of clothing differently—which, perhaps, is a type of “see through” behavior in colloquial terms. Fourth, 11.1% of videos exhibited confusion about IR reflection when imaging glass or other surfaces. Again, all these misconceptions were typically corrected by other YouTube users in the video’s comment section. Misconceptions about what constituted thermal imagery or devices also existed: 13.8% of videos were made with faux thermal photo filters and 5.8% described homemade “near-infrared” thermal imaging devices that were made by modifying cameras (to remove infrared light filters). The latter was most likely a misnomer rather than an explicit misconception but could promote the concerns mentioned in [140]. Finally, a few videos (5.5%) demonstrated general confusion about the camera’s features (e.g., why were the camera’s conventional and thermal images misaligned). 3.3.3 YouTube Comment Threads To understand the types of discussions that occur around thermal videos posted to YouTube, we coded all 675 videos for whether they contained question-and-answer discourse—see Table 3.4. Below, we focus on the 165 videos that had Q/A comment threads across the top four video categories. Across these videos, we found a total of 365 unique questions, including about: technical specifications (41.9%), content (29.9%), other (19.5%), and follow-up requests (8.8%,). For example, a typical technical specification Q/A comes from V359: 45 Who Responded Question Number Number Type Asked Answered Original Other Poster Poster Both Technical 41.9% 53.6% 75.6% 12.2% 12.2% Specification (153/365) (82/153) (62/82) (10/82) (10/82) Content 29.9% 58.7% 65.6% 12.5% 25.0% (109/365) (64/109) (42/64) (8/64) (16/64) Other 19.5% 71.8% 52.9% 21.6% 22.5% (71/365) (51/71) (27/51) (11/51) (12/51) Follow-Up Request 8.8% 50.0% 62.5% 18.9% 18.9% (32/365) (16/32) (10/16) (3/16) (3/16) Table 3.4: Breakdown of comments on YouTube thermography videos.  Commenter: “Any way to calibrate the sensor? That would remove the “noise curtain” Response: “I think with the proper software, this would be more than possible, no idea if you can calibrate the sensor to the exact temperature, but there must be a way to remove the noise, especially at low delta-T, where it occurs most […] Convenient thing there is a free SDK to Therm-App owners.” (V359) For content, a YouTube commenter asked about the bubbling surface of a coffee cup (V154, Figure 3.3d): Commenter: “what is [does] this mean????” Response: “This is what happens in every cup of coffee. [...] This video demonstrates a phenomenon of convection into the water, i.e. interfusion of more cold layers on the water surface and more hot layers in the deep of the water. As a result, we can observe cells on the water surface in infrared frequency band.” (V154) While more than half of all questions were answered (58.4%), questions categorized as other—which tended to be less specific to the video (e.g., “what song is 46 this?”)—received markedly more responses than other question types (71.8%, Table 3.4). Across all questions, the original content creator was most likely to respond (Table 3.4): Commenter: “Can you do a video showing the sky. I can't find any videos showing the sky. I'm a sky watcher and am thinking of getting a thermal device.” Response: “Thermal isn't really good for skywatching unless you are looking at clouds. Water vapor tends to show, and it [is] generally very cold. Almost always black compared with terrestrial objects other than clouds or aircraft” (V79). Despite this activity, over half of the questions (58.4%) asked across the 165 videos remained unanswered due to low interaction with the community (e.g., no comments). 3.4 Findings from Online Survey To complement the video analyses and to better understand thermal camera use and motivations for sharing on YouTube, we invited content creators to complete an online survey. We contacted 1,023 unique YouTube users across the final video dataset and received 79 completed surveys (a response rate of 7.7%). As our focus is on novice use, we report on those 48 respondents who stated that they do not use a thermal camera professionally. Participants are identified by “P” and their survey number (e.g., the 13th 47 Pre-Purchase Post-Purchase Other Energy Auditing Outdoor Recreation Nighttime Navigation Security Culinary Agriculture 0% 20% 40% 60% 80% Figure 3.4: Survey participant’s planned (pre‐purchase) and actual (post‐purchase) thermal camera  activities in terms of percentage of total respondents (N=79).  survey respondent is identified as P13). Some percentages do not add to 100 due to the check-all-that-apply questions. Demographics. All survey respondents were male (100%, 48). The average age was 39.3 years (Mdn=39, SD=11.0, range=20-68). Most respondents held an advanced degree (60.2%) or had completed vocational certification programs (10.4%); all others had a high school diploma (29.1%). Respondents mostly reported technical professions, including: various kinds of engineers (29.1%), information technology specialists (22.9%), and security professionals (10.4%). There were also a few teachers (8.3%), students (4.1%), and other professions (e.g., martial arts instructor). Most participants rated themselves as being concerned about climate change (Mdn=4, M=3.4, SD=3.5) on a 5-pt Likert scale (with ‘5’ being “extremely concerned”), which we used as a proxy for assessing eco- consciousness as a primary use of thermography is energy auditing. Thermal Camera Use. Most respondents used thermal camera smartphone attachments (52.0%)—specifically the FLIR One (33.3%) and the Seek Thermal Compact (18.7%)— or handheld thermal cameras (15.5%). Others used the CAT s60 smartphone with a built-in thermal camera (4.0%), the Lepton module for Raspberry Pi (2.0%), and the 48 Tau640 for UAVs (2.0%). When asked why they initially acquired a thermal camera (Figure 3.3), almost half (45.8%) reported purchasing for energy auditing, followed by wildlife observation and outdoor recreation (33.3%), nighttime navigation (22.9%), security (14.5%), culinary (4.2%), and agriculture (2.0%). Respondents (60.4%) also reported purchasing their camera for “other” activities, including: for curiosity or fun, electronics testing, ghost hunting, and flying UAVs. When asked about actual post- purchase uses, responses for security increased (+8.3%) as well as energy auditing (+6.2%), culinary (+6.2%) and outdoor recreation (+2.0%) activities. However, nighttime navigation and agriculture use both fell (-4.1.0% and -2.0%, respectively). Additionally, the quantity of “other” uses also fell (-14.5%), but new uses from the write-in responses emerged (e.g., pest control, monitoring 3D printers) and some respondents offered reasons why they discontinued use. As P39 described, the thermal camera was “…not as good for wildlife observations as I would have thought.” To get a sense of how often respondents used their thermal cameras for these activities, we asked them to rate their use on a 5-pt Likert scale ordered daily to never. Most reported using their cameras monthly (39.5%) followed by semi-annually (25%), weekly (18.7%), then daily (12.5%). Experience with YouTube. Most respondents commented that their reason for sharing videos on YouTube was to educate or share with the YouTube community (45.8%). As P79 said, “[I post] for views and science”. Other reasons included for fun (22.9%) or to show friends and family (8.3%) while the remaining (23%) provided non-descript or unclear responses (e.g., “because I can”). Many seemed to find the content of their videos fascinating, stating they shared their videos and images “to show things you can never see 49 without a thermal camera” (P32). Half our survey respondents (50.0%) reported interacting with other users on YouTube including engaging in commenting, receiving requests for follow-up videos, and providing feedback—which is consistent with our earlier comment analysis. Most participants at least somewhat agreed (58.3%) that the feedback they received on YouTube was valuable or personally beneficial, almost a third (29.1%) were neutral, and three (6.25%) disagreed. General Thoughts on Thermography. Overall, most respondents (97.9%) agreed that their thermal camera was a useful tool and half (47.9%) strongly agreed. Almost all participants (95.8%) agreed that their camera was helpful in discovering new things about the world around them and ~half (47.9%) strongly agreed. Similarly, most participants (95.8%) agreed that they would continue to use their thermal camera in the future and half (50.0%) strongly agreed. Finally, 85.4% expected to continue sharing their thermal content on social media. Building Thermography. While 45.8% of respondents mentioned energy auditing as a specific motivation for purchasing a thermal camera, a higher percentage (52.0%) reported using their device in this way after purchase. Participants who used their camera for building thermography inspected a wide variety of building types, from single-family homes (85.7%) and multi-unit dwellings (28.6%) to commercial buildings (14.3%) and schools (8.6%). Inspection tasks included: observing air leakage (71.4%), insulation checks (71.4%), electrical issues (57.1%), moisture inspections (40.0%), or locating hidden structures (34.2%) such as hot water pipes or wall studs. 50 When asked about why they performed thermographic inspections of buildings, most respondents (86.9%) cited saving on utility bills, energy conservation (e.g., finding leaks, supporting winterization efforts) or both, while the rest (17.1%) cited curiosity. A few participants (5.7%) reported using the camera to provide supplement claims against landlords or home improvement companies. For example, as one respondent explained: “I had new windows installed that appeared to be leaking air. A home inspection was $450, a thermal imager was $300 and given that I know how it works it was an easy choice. [The] window installer had to do warranty work that they didn't initially agree with.” (P3) Overall, most respondents were positive about the outcomes of their building thermography activities. Based on their inspections, more than half (60% or 21 respondents) reported making decisions to pursue renovations or retrofits. All agreed that these building improvements directly resulted in saving money on utility bills and almost a third (28.6%) strongly agreed. Fewer agreed (71.4%) that these renovations or retrofits led to improvements in the building’s thermal comfort. Most (71.4%) did not agree that engaging in building thermography had resulted in any new conservation behaviors, but those that agreed believed that these behavior changes had led to both energy savings and improvements in thermal comfort. 3.5 Discussion Through a mixed-methods approach of analyzing OSN video data, comments, and an online survey with content creators, this work advances understanding of non- professionals’ uses and conceptions of thermography. We investigated what activities 51 non-professional users of thermal cameras pursue “in the wild” as well as how well they understand this technology. In particular, the study shows that novice users are able to develop the skills necessary to use and explore with this technology. In contrast to [110], while there were indications of misuse and misconceptions, these were rare and were typically corrected by other members of the YouTube community. Below, we reflect on major findings, present design recommendations, and discuss the study methodology as well as key limitations. 3.5.1 Novice Uses of Thermography Much like previous work investigating technology use via OSNs [7,16,22,77,121], we found that user-generated videos offered an otherwise inaccessible window into user behavior of an emerging technology. In particular, novice users expressed positive attitudes toward thermal cameras and performed diverse activities ranging from imaging pets and beverages to investigating electrical failures and home improvements (i.e., need for or success of a repair). Thermal cameras provided not just a new avenue to explore the world but also, in some cases, supplied important information that helped users diagnose problems and support decision making. For example, 60% of survey respondents performed home renovations based on their self-diagnostics. Videos also showed users utilizing thermography as a visual aid during electrical and agricultural inspections. Contributing to the YouTube Thermography Community. This work also offers insights into why these users chose to post videos and engage with YouTube. We found that users engaged in rich dialogues about thermal camera use and limitations through 52 YouTube videos and comment feeds. Survey and comment analysis revealed both intrinsic and extrinsic motivations to participate in the online community similar to [100,118]. Content creators reported posting videos to help showcase a particular thermal camera application, to explore a specific phenomenon, and/or to help teach others. Users reported enjoying sharing content and believed that this content would attract viewers. As P79 summarized, he shared videos “for views and science.” 3.5.2 Novices Understanding of Thermography In our study, we found misuses of thermal cameras (e.g., attempting to observe gases) as well as misinterpretations (e.g., using surface temperature as a direct proxy for air temperature). However, these were less frequent than expected—comprising only 5.3% of our dataset. Moreover, we found that some content creators demonstrated a sophisticated level of understanding (e.g., describing thermal reflectivity of a material or the need to calibrate for emissivity). Nevertheless, overcoming these challenges will be critical to helping users avoid the negative consequences of incorrectly interpreting thermal data as there can be tangible costs to such misinterpretations. For example, a misdiagnosis could lead to investing in needless repairs or, conversely, a missed opportunity for improvement in the building and electronics contexts. Anticipating a Shift in User-base and Understanding. Admittedly, the users in the dataset likely represent the most interested non-professional thermal camera users, who may be more confident in their activities and interpretations than the general population (e.g., novice thermographers not on YouTube). As the user population shifts from those having made a conscientious decision to purchase thermal cameras to a population with 53 a less purposeful acquisition (e.g., smartphones that include thermal sensors [164]) users may have a less vested interest in learning about the technological constraints of thermal cameras. Such non-expert, non-invested users may be more likely to encounter challenges and misconceptions. To support a future where novices have easier access to thermal camera technology, future applications and services should consider how to support users in learning thermography best practices. 3.5.3 Implications for Design To better support non-professional thermal camera users in collecting and analyzing thermography data, we offer several implications for the design of future thermographic systems and tools that will address the challenges identified in the findings of this study. Provide Contextually Relevant Information. Future applications should suggest appropriate uses of thermography within different contexts (e.g., the potential value of time lapse video in assessing heat and power management in electronics) and offer information related to common interests (e.g., why the surface of hot liquids such as coffee display patterns). Such dynamic context awareness can improve thermographic systems [2] and help users learn to use the technology properly. While the YouTube community supports this informally through online videos, learning and application are likely to improve by integrating this information directly in the thermal user interface via interactive onboarding within the mobile applications that smartphone camera attachments (and integrated cameras) rely on. Encourage Exploration. While thermal camera users initially purchased devices for purposeful activities (e.g., wildlife tracking, energy audits), users often ended up exploring 54 a wider range of uses out of curiosity. Encouraging exploration would empower users to take full advantage of this sensing technology in diverse ways provided data is correctly collected (i.e., with respect to the application domain). This practice could have further benefits such as contributing to citizen science efforts by leveraging interest in wildlife tracking to simultaneously create new sources of data for environmental and conservation purposes (e.g., locating bird nesting sites [61], monitoring honeybee colonies [87]). Anticipate and Prevent Misconceptions. Advances in integrating thermographic data with machine learning and computer vision technology [37,165] could help combat misconceptions, misinformation, and misuse by aiding users in analysis and making the limitations of thermography more understandable. For example, automatically detecting the presence of glass windows or ceramic bathroom tiles in an image could bring up information about the reflectivity of these materials. To accomplish this goal, it will be important to continue studying thermography users and communities to identify common pitfalls and determine when in-situ assistance is applicable and desired. Enable Social Supports. This work provides shows that thermography users enjoy and learn from social interactions, here, in an online community. As with previous work emphasizing the impact of social supports in online communities [126,128], this work suggests that providing online social supports for thermal camera users could promote users’ enjoyment, technical understanding, and proper use. 3.5.4 Limitations In addition to previously described limitations, each method in our mixed-methods study—video content analysis, comment analysis, and the online survey—has limitations. 55 The video analysis is limited to the YouTube community and those users with the ability and inclination to upload videos. Survey findings are also limited by a similar self- selection bias, unverifiable participant claims (e.g., energy savings), and, as all survey respondents were male, a gender skew (similar to [109]). Finally, within the YouTube comment analysis, answer accuracy was only evaluated in relation to misconceptions or misinformation. These limitations suggest that our results represent only the most confident of novice users and likely excludes those who may not have discovered anything of interest or used the technology in a way that left them feeling generally unhappy with their application of thermography. Thus, this work should not be viewed as a comprehensive view of the novice user experience and needs to be considered in the context of the other studies presented in this dissertation. 3.6 Conclusion This work presents the first qualitative, human-centered inquiry into “in the wild” use of thermal cameras by non-professionals. Using a mixed-method approach, we analyzed 1000 YouTube videos, analyzed the question and answer discourses within video comments, and further surveyed the content creators to characterize end user-behavior and motivations. Results indicate that non-professional users apply thermography widely: activities ranged from investigating domestic objects to focused investigations of buildings and electronics. This study found that users investigated technological limitations and, largely correctly interpreted their data. The characterization of novice users and common thermographic use cases extends discussions surrounding novices uses and the challenges novices encounter which have implications for the design of future thermographic systems and tools. 56 Chapter 4 Novice Thermographic Energy Auditing Figure 4.1: (a) A close‐up of thermal camera application on iPhone5s. (b) A pilot participant collecting  photos. (c) A thermal camera attached to an iPhone 5s. (d) A close‐up of the standard FLIR One camera  application that ships with the attachment.  This chapter further examines novice thermal camera use. We recruited 10 participants for a four-week field study of end-user behavior exploring novice approaches to semi- structured thermographic energy auditing tasks. We analyze thermographic imagery captured by participants as well as weekly surveys and post-study debrief interviews. Our findings suggest that while novice users perceived thermal cameras as useful in identifying energy-efficiency issues in buildings, they struggled with interpretation and confidence. We characterize how novices perform thermographic-based energy auditing activities, synthesize key challenges, and discuss implications for design. This chapter has been adapted from a paper published at CHI 2017 [104]. 57 4.1 Introduction Having provided this overview of thermal camera use by non-professional users, we now turn toward work that more specifically explores thermographic building energy auditing activities by novice users. Published at CHI 2017 [110], we conducted the first qualitative field study of thermal camera use by novice users in the context of building energy auditing. Our research questions were, again, exploratory and included: How do novice users of thermal cameras assess the built environment? What attributes of the built environment do they focus on, learn about, and discover? What challenges do they encounter? What benefits do they perceive? To explore these questions, we recruited 10 novice participants to take part in a four-week field study of smartphone-based thermal camera usage. Findings from this study further suggest that novice users with minimal training can effectively use thermal cameras to document energy-efficiency issues in buildings and to find previously unknown problems. Participants also reported a general heightened awareness of electrical energy use and a greater likelihood of engaging in energy conservation practices (complementing findings of [66,122]). However, participants had difficulty gauging the severity of the issues they encountered making it difficult to determine the impact of energy-efficiency improvements. In the discussion of this work, we (i) synthesize key challenges novices experience when collecting and interpreting thermal imagery for during energy audits, (ii) describe barriers to novice thermographic energy auditing, and (iii) discuss design implications for both Sustainable HCI and public auditing of the built environment. 58 ID Age Gender Education Profession iPhone Concern Average P1 22 Female Bachelor’s Public Affairs Specialist 6 Climate Change 6.5 (SD=0.8, Mdn=7.0) P2 25 Female Bachelor’s Graduate Student 6 Home 5.3 P3 30 Male Master’s Graduate Student 5s (SD=1.2, Mdn=5.5) P4 58 Female Doctorate Research Scientist 5s Community 5.2 (SD=1.5, Mdn=5.5) P5 31 Female Master’s Higher Education 6s Workplace 4.8 Professional (SD=1.5, Mdn=4.5) P6 56 Male Master’s Government Scientist 5 Table 4.2:  The pre‐study  survey asked  P7 28 Male Master’s User Experience Designer 6s participants how concerned they were  P8 53 Male Master’s Marketing Coordinator 5 about  climate  change  and  energy efficiency  in  specific  contexts  of  their  P9 34 Female High School Education Coordinator 6 daily lives.  P10 40 Male Master’s Educator 6 Table 4.1: A summary of the novice field‐study participant’s  demographic information.  4.2 Method The four-week field study was scheduled to take place during the winter months of 2015. Each of the 10 participants was provided a FLIR One thermal camera attachment (Figure 4.1a-b) for their personal smartphone and told to explore freely throughout the study period. To help guide their auditing activities, participants were also asked to complete weekly thermographic “missions” (adapted from the prompting method in [129]). Missions were included to scaffold and motivate data collection across a range of use- contexts: home, work, and two public spaces. Prior work informed the study design [109] as did earlier pilot studies [106] where we found that missions helped structure auditing activities and helped participants to think more broadly about locations to capture thermal imagery. To help me understand their activities, participants answered an online questionnaire and uploaded their thermograms weekly. At the end of the study, participants were debriefed via a semi-structured interview and compensated with $100 for their time and any expenses incurred by their participation. 59 4.2.1 Equipment The FLIR One thermal camera attachment was used in this experiment as it is widely available—sold at Apple Stores and online—and fits a wide range of iPhone models. As shown in Figure 4.1c, the thermal camera attaches to the iPhone’s Lightning port. For this study participants also used the FLIR One thermal camera application, which looks and largely functions like a conventional camera application with a “Take Photo” button in the bottom center and a list of image capture options above it (Figure 4.1d). The display updates in pseudo-real-time and photos can be taken at any time, but the camera works best in a stable position. The user can change how the camera colorizes the thermal data (via the “Change Palette” button). In the example shown, the “Iron” palette is used which displays colder regions of the image in shades of purple and warmer regions in shades of orange. The icons on the top menubar allow users to change measurement settings, display a temperature measurement tool (i.e., averaging the temperature between a superimposed crosshairs), and see when the camera is calibrating. 4.2.2 Participants We recruited 10 participants (5 female) from the general population using local mailing lists and community message boards (Table 4.1). Our recruitment ad specified that we were interested in studying the use of smartphone-based thermal cameras for energy auditing applications and asked potential participants to complete a short eligibility questionnaire. Participants were enrolled on a first-come, first-served basis after screening for adults (ages 18+) and compatible smartphones. 60 To collect demographic information and understand attitudes toward environmental sustainability, enrolled participants completed a short, pre-study questionnaire. The survey revealed that participants were, in general, eco-conscious and concerned about the environment. Using 7-point Likert scales ordered from very unconcerned (1) to very concerned (7), participants reported being very concerned about climate change, concerned about the energy efficiency of their homes and their local community, but less concerned about their workplace—see Table 4.2. Additionally, half (N=5) reported regularly engaging in conservation behaviors (e.g., turning off lights) and making minor efficiency modifications in their homes (e.g., upgrading light fixtures). Some (3) reported making large efficiency improvements (e.g., installing solar panels). A few (2) reported making minor changes to solve winter comfort issues (e.g., sealing drafty windows with plastic). Participants reported no previous experience with thermal cameras; however, a few (3) previously had professional energy audits of their homes; two included thermography. Week Mission Home Investigate your home with your thermal camera for signs of energy inefficiencies and comfort issues; collect at least 25 photos that highlight aspects of your investigation. Workplace Investigate your workplace to help inform new policies on energy conservation and comfort; collect at least 25 photos that highlight aspects of your investigation. As if you were a building inspector, investigate a commercial location (e.g., a café) for Commercial potential issues based on your previous experience; collect at least 25 photos that highlight aspects of your investigation. Community As if you were a municipal inspector, investigate your local downtown or community area; collect at least 25 photos that highlight aspects of your investigation. Table  4.3: Weekly  mission  descriptions  were  sent  to  participants  via  email  along.  Lightweight  feedback about the previous week was also provided.  61 4.2.3 Procedure Introductory briefings were held in the Human-Computer Interaction Lab or in a local café, depending on participant preference. Upon arrival, we discussed the study plan, obtained consent, provided the thermal camera and accessories (e.g., manufacturer’s documentation), and reviewed a 4-page custom training document (see Appendices). The document was synthesized from thermal smartphone applications [48], how-to guides from manufacturers [163], and DOE materials [146,147] by a research team member with a professional thermography certification; it covered key elements of a successful thermographic investigations. Participants were encouraged to freely explore objects, their environment, or anything that struck their interest with their thermal cameras. To help structure and motivate their explorations, we also provided them with weekly energy-themed missions. The missions ranged from home inspections to community explorations; see Table 4.3. All participants received the missions in the same order. At the end of each week, participants uploaded their photos and completed an online questionnaire about their mission experience. Figure 4.2: A participant describes an air leakage issue found while auditing his workplace during the post‐study debrief interview (left). A close‐up of the actual thermal image being discussed (right).  62 At the end of week four, participants completed an in-person, semi-structured interview with a photo-elicitation component [28]. During the photo-elicitation, participants used their thermogram collection as a visual aid to help recall and describe experiences (Figure 4.2). Except where we had marked a photo for discussion, participants chose which photos to discuss. After the photo-elicitation, participants described their experiences over the four-week study, including discussions about how current thermal cameras could be used by non-professionals and improved to better support their use. 4.2.4 Data and Analysis Images and interviews were qualitatively coded. Counts and descriptive statistics were calculated for survey data. Images. In total, participants took 1,991 thermographic images; however, 83 of these images (4.2%) were invalid because either the thermal camera was calibrating when the image was captured or the image was indecipherable (e.g., a thumb blocking the camera lens). To determine what participants were taking pictures of, the remaining 1,908 images were analyzed through an iterative coding method using both inductive and deductive codes [19,78]. Multiple codes could be applied to the same image. We first selected and coded a random participant’s image dataset (total images=139). The initial codebook was composed of a list of expected objects and contexts (e.g., window, outdoor) and a miscellaneous code that allowed researchers to tag unforeseen yet significant elements within the images (e.g., pet). Two researchers independently coded each image. Cohen’s 63 Kappa (κ) was used to measure inter-rater reliability (IRR). IRR on the first iteration of the codebook was κ=0.57 (SD=0.23) suggesting it required iteration [151]. The two researchers met, resolved disagreements, and updated the codebook accordingly. We again coded a second, randomly selected participant’s image collection and achieved an IRR of κ=0.80 (SD=0.20) with codes ranging from strong to near perfect agreement. Our final codebook included 19 codes grouped into four categories: subjects (e.g., electrical device), context (e.g., indoor), biologic (e.g., animal), and misc. (e.g., clutter). The remaining images were then split between us and coded independently. The final codebook is included in the Appendices. Weekly Surveys. The weekly surveys captured feedback on each mission such as: a description of what participants found during their assessment activities and recommendations, if any, that they might have to improve building performance. The surveys also asked for procedural details such as the date and duration of their audit activities. Finally, participants filled Likert-scale questions about their experience using the thermal camera. The survey took approximately 30 minutes to complete. Debrief Interviews. The semi-structured interview sessions lasted an average of 75 minutes (SD=18.2). Interviews were audio recorded and professionally transcribed. Similar to the image analysis, we pursued an iterative coding approach using a mixture of inductive and deductive codes. We explored the interview transcript of a randomly selected participant using an early codebook developed based on research literature, our study protocol, and discussions amongst the research team. The final codebook included 12 codes grouped into three categories: experiential (e.g., exploratory behavior), design 64 Weekly Image Avg. Images Avg. Time Avg. Thermal Thermal Camera Mission Totals Per Spent # Audit Avg. Mission Difficulty Camera Helped Helped w/ Participant (mins) Sessions w/ Learning Identification 572 57.2 34.9 1.9 5.3 5.9 5.4 Home (SD=52.27) (SD=15.02) (SD=1.25, (SD=1.19, (SD=0.66, Mdn=6.0) Mdn=6.0) Mdn=5.5) 405 40.5 32.0 2.0 4.4 5.4 5.2 Workplace (SD=18.02) (SD=14.59) (SD=1.26, (SD=0.32, (SD=0.32, Mdn=4.5) Mdn=6.0) Mdn=5.5) 415 41.5 28.7 1.7 4.2 6.3 5.9 Commercial (SD=9.72) (SD=16.77) (SD=1.39, (SD=0.67, (SD=1.19, Mdn=3.5) Mdn=6.0) Mdn=6.0) 516 51.6 29.7 2.1 4.5 5.5 5.0 Community (SD=26.73) (SD=13.69) (SD=1.26, (SD=0.84, (SD=1.05, Mdn=4.5) Mdn=5.5) Mdn=5.0) Table  4.4:  An  overview  of  participant  behavior  and  survey  responses.  Average  time  spent  was calculated by adding the total minutes spent across all data collection sessions in a given week based on participant’s self‐report data. For Likert questions, we used a 7‐point scale ordered from strongly disagree (1) to strongly agree (7); 4 was neutral. We report median (Mdn=X) and standard deviation (SD=X). For mission difficulty, higher is easier.  ideas & challenges (e.g., design idea), and broader impact (e.g., potential benefit). The unit of analysis was the response to a single question or image. IRR on the first iteration of the codebook was κ=0.51 (SD=0.21). Again, we met and resolved disagreements. This was repeated with randomly selected transcripts three times achieving an overall IRR of 0.87 (SD=0.08); remaining transcripts were split and coded. Again, the final codebook is included in the Appendices. 4.3 Results We first provide an overview of the field study activities. Next, we review each mission based on the weekly survey responses and captured images. After presenting the field study results, we address our research questions through thematic analysis of the entire corpus of study data. Finally, we present participant design considerations for future thermographic tools. Participant quotes are attributed using a ‘P’ followed by their identification number (e.g., P1). 65 4.3.1 Overview of the Four Auditing Missions To characterize participant activities during the missions, we examined: what participants took pictures of, how much time participants spent performing their auditing activities, and the perceived utility of the thermal camera. For the latter, participants reported how helpful they felt the thermal camera was for learning about and identifying energy-related issues during audits. Table 4 presents specific details for each mission, which we summarize next. Data Collected. Participants took 47.7 photos per mission, most commonly containing walls (71.6% of images), windows (30.3%), doors (24.4%), and electrical devices (23.7%). Participants concentrated on interior inspections (64.2 %) rather than outdoors. See Figure 4.3 for examples. Time Spent. Participants typically spent 1.2 hours completing each mission, which was often divided across multiple days (usually 2). Participants reported spending 30 minutes Figure  4.3:  Examples  of  the  image  contexts,  subjects,  and  non‐mission  photos  as  well  as  the  percentage of the dataset that includes these features.  66 (a) (b) (c) (d) Figure 4.4: Example  imagery from participant  investigations:  (a) an  insulation  issue  in the roof of a  residential home, (b) observing power consumption of computer equipment in an office, (c) gathering  evidence  of  insufficient  winterization  procedure  of  window  air  conditioning  units  in  a  university building, and (d) documenting the need to repair weather stripping around an emergency exit door at a community theater.  capturing thermal imagery and another 30 minutes on reporting (i.e., completing the weekly survey). The remaining time was spent planning (i.e., what building to audit) and uploading imagery to the research team. Thermal Camera Utility. Overall, the thermal camera was deemed helpful in identifying and learning about potential problems in buildings, particularly for the first three missions (Home, Workplace, and Commercial). 4.3.2 Overview of the Four Auditing Missions In each of the four missions, participants were asked to explore a different location. Here, we briefly describe results from each mission before discussing pervasive themes. Home Mission. In this mission, participants investigated their homes looking for potential energy inefficiency issues. Half of the participants (5) investigated single-family homes, three investigated town homes, and the remaining two investigated apartment units. In the post-mission survey, all participants (10) reported checking for window, door, and insulation issues. Most participants (8) started with pre-existing comfort issues (e.g., rooms that were not adequately heated or cooled). A few (3) explored electrical 67 appliances (e.g., dryer) due to a safety concern. Additionally, a few (2) investigated a friend’s home. Based on their auditing activities, several participants (4) concluded that the windows in their homes needed minor repairs (e.g., improved air sealing), a few (3) reported insulation issues, one was motivated to contact an electrician, and the rest (2) did not report finding any issues. As a positive example, in the post-mission survey, P7 reported exploring a pre-existing thermal comfort problem and that the thermograms made him “very confident about missing insulation issues, especially in the ceilings of that room” (Figure 4.4a). Thus, the participant decided, “I would like to share this image with my landlord,” to see if this issue could be addressed. Workplace Mission. In the second mission, participants explored energy use in their workplaces. Most participants (7) investigated office buildings, two investigated university buildings, and another investigated a local grocery market. Like the Home Mission, all participants (10) reported looking for leaky windows, doors, and noted interest in the heat signatures produced by electronics. Two participants did not report finding any energy efficiency issues. Three reported finding leaky windows and doors, and five reported finding electronic devices using phantom energy (Figure 4.4b): “I was stunned to realize that my monitor doesn't completely turn off when it goes to sleep. It was unused for the weekend but still appeared hot. So, I turned it off when I went to lunch and when I came back and it was indeed cooler.” (P4) As in the Home Mission, two participants used thermal comfort as motivation to explore their workspace. For example, due to this mission being conducted in the winter 68   Figure  4.5:  During  the  community  mission  participants  explored  utility  infrastructure  (left);  one  participant explored a makerspace (right).  season, P5 noted that many offices in her building were cold and that she used the thermal camera to confirm her suspicion: “I found that most of the ceiling vents were colder which leads me to believe they might still be pumping out cool air.” Thus, P5 concluded that her workplace’s air conditioning settings might need to be adjusted. P10 described a similar shared concern about how drafty his workplace became because of insufficient air conditioning unit winterization procedures and used his thermal camera to investigate (Figure 4.4c). Based on his imagery, he concluded: “The situation with the window A/C units is absurd. Honestly, they should be removed in the fall and reinstalled in the spring since it is so hard to insulate them and they are only needed during the summer. Having that much air getting through in the summer is also a problem, we just don't realize it and continue running the units.” (P10) At the time of the debrief interview, P10 reported that he was considering sending the imagery to his facilities management to help evaluate the problem. Commercial Mission. The third mission asked participants to explore a commercial building. Participants investigated a wide range of establishments from restaurants to hardware stores. Seven participants did not report finding any evidence of potential 69 efficiency issues. Unlike the previous missions, participants were not able to use their knowledge and experience of a place to guide their explorations (e.g., where cold drafts were located). Most participants (9) investigated equipment such as storage, food preparation, and serving areas found in commercial cafés or markets. One participant found potential evidence of moisture damage in a restaurant. Two participants reported finding evidence of leaky windows and doors. For example, P8 investigated a community theater and reported finding air leakage issues prompting a discussion with the operators (Figure 4.4d): “The theatre underwent went a major renovation in 2014-15 where it was closed for several months. …In speaking with the operator, she indicated that although there were all new exteriors doors and windows on the main level, the upstairs office windows and fire doors were original.” (P8) P8 reported sharing his thermal photographs with the operators, who planned to send the images to city officials to show the need for further repairs. Community Mission. The final mission was open ended; participants were asked to investigate their local community, which they mostly did outdoors. Nine participants did not report finding any issues but did describe finding and learning about utility infrastructure in their community such as water lines and electrical equipment (Figure 4.5 left). P4 additionally explored a local makerspace and reported (Figure 4.5 right): “The makerspace was a treasure trove: clear differences between new and old windows, where patches of the walls were made (cold sources), evidence of water damage (confirmed by renter of the space), and old pipes creating cold spots on the walls and ceiling.” (P4) 70 Summary. During the first three missions, participants primarily investigated buildings for missing insulation, air leakage issues, and understanding phantom energy use. In the home, participants seemed comfortable drawing conclusions about the need for repairs. In each mission, a small number of participants indicated that they wanted to address discovered problems by contacting a landlord, an electrician, or building owners/operators. In the final mission, participants used their cameras to explore their community and, most did not find concerning issues; however, they mostly took exterior pictures of buildings. 4.3.3 Major Themes Across Missions While the previous section characterized participant behavior on a per-mission basis, we now turn toward describing themes that emerged across the four-week study, including: how participants collected and interpreted thermal imagery, what they learned, and what influenced their ability to act on their findings. Data Collection. Rather than following any specific plan or procedure as an expert auditor might do [137], all participants (10) described their investigations as random walks through the interior of buildings. Participants occasionally followed their interior walk with another around the building’s exterior, and participants who were aware of pre- existing issues tended to start in those areas. This was especially true during the home and workplace missions. With no pre-existing issues in mind, participants described their activities as exploratory, often using the camera as an augmented reality lens into otherwise invisible energy flows. As P1 said: 71 “I was mostly just looking through the lens of the camera. I wasn't looking at my surroundings and then putting a camera up. I was holding the camera up and taking photos.” (P1) When looking through the live view of the thermal camera, if participants discovered what they perceived as an anomalous heat signature, they would then take two-to-three images from various positions or angles to ensure adequate capture. Even if they did not find anything of interest, participants would still take one or two wide-angle photographs to help them record areas that they investigated. Due to the time it took to attach the camera and load the thermal app, most participants did not report taking many photos outside of the mission scenarios. Interpretation. When asked about interpreting thermal imagery, participants described how they appraised an image and things that made this task challenging. To determine if an anomalous heat signature was an issue, all (10) participants described looking for areas of extreme contrast in the images. Participants believed they could readily identify air leakages around windows and doors as well as the heat signatures from electronics; however, participants also described capturing imagery that they did not understand such as the cause of a warm spot on a wall that did not have any obvious source. While participants were not always able to describe what made interpreting a thermal signature difficult, most participants (8) attributed difficulties to the presence of confounding objects (e.g., heating elements), materials (e.g., metals), and other environmental factors (e.g., sunlight). For example, referring to an image P3 said: "This is all glass, so it's reflective. It's not clear to me if it's really that much warmer on the inside of this building than the outside." (P3) 72 All (10) participants said that at times they lacked confidence in their ability to draw appropriate conclusions from the thermal images. Most participants (6) found it difficult to determine the severity of issues they encountered and the potential impact repairs would have on the efficiency of the building. As P2 described, “I don't know how much [the issue] really affects the energy use of my apartment.” Additionally, half (5) of the participants suggested that a lack of information about a building (e.g., age) and/or its construction (e.g., type or rating of insulation used by the builders) limited their ability to draw confident conclusions. Knowledge Gains. Through their use of the thermal camera, all participants (10) reported learning to identify hidden structures or common issues in the built environment such as hot water pipes or leaky windows. Many participants (6) also stated that they learned about how materials had different conductive or reflective properties. P3 said: “I certainly learned about the thermal reflectance of common surfaces, that’s something that I had not known before.” Awareness of Energy Efficiency. In the debrief interviews, seven participants described how their perspective on the way buildings are used and maintained had changed. We classified these perspectives into two categories, related to energy consumption (5) and building maintenance (2). participants frequently mentioned that seeing the easily recognizable thermal signatures from electronic devices forced them to consider electrical use and conservation. For example, P10 found that thermal images were a helpful reminder to turn off devices that are left on standby (and consuming phantom energy [124]): 73 “It’s one of those things that I’m aware of in theory: when you leave things plugged in there is still some energy use but seeing it like this reminds me about it.” (P10) However, a few participants also pointed out that there are many “always on” devices that do not have a convenient way to manage their energy consumption (reaffirming [26]), including their internet routers at home or the phone systems common in office environments. Two participants noted that their perspective on building maintenance had changed. P6, for example, had come to believe that inspections and building efficiency maintenance should be an ongoing practice, like with a car: “It's one of these things you've got to keep working at to incrementally find, you know, I can do something more efficiently here, turn this off more, or fix that problem.” (P6) Perceived Value of Thermography. All (10) participants perceived value in having a tool that helped them investigate potential energy-related problems in buildings. Most (8) suggested that thermal imagery could provide supporting evidence to building owners and or others in charge of building maintenance. For example, P3 stated “I’ve been meaning to contact my landlord with these images and say, look, there seems to be a clear issue here that I think you should address.” Two participants suggested thermography might be useful for community related improvements. As P2 described: “It would be interesting to go and do this in the local high school and see if it's built well, that we're not wasting energy and resources that we could be using for something else... I feel like if there are ways that we could save on energy by repairing things, then that would be beneficial.” (P2) 74 Locus of Control. Two key issues were raised about making energy improvement decisions: lack of control and apathy. Some participants (4) who rented or lived in housing cooperatives were concerned that if they found evidence of a problem that they would not be in a position to make retrofit decisions. As P5 stated, “If I took a picture that showed an insulation issue, I don’t necessarily think the owner would get on top of fixing it.” In missions outside the home, one participant expressed that it was not clear who they should talk to if they discovered an issue. In response to performing a mission in a local café, P2 asked: “If I find an issue, who am I going to tell and are they really going to care? My biggest concern is what if something is wrong and they don’t want to do anything about it.” (P2) While locus of control issues are non-trivial, especially in residential buildings where asymmetric power relationships may exist with landlords (e.g., [21,136]), thermal cameras may play a unique advocacy role for tenants to highlight otherwise difficult-to- observe problems or provide continued evidence of an unresolved issue. 4.3.4 Participant Design Recommendations At the end of the debrief interview, participants were asked for suggestions to improve thermographic data collection and analysis practices. Participants discussed support for automation, privacy, and general usability improvements. Automated Assistance. Similar to our findings with professional auditors [109], most participants (8) suggested adding “intelligent” mechanisms that would help them collect and analyze thermographic data. For example, participants wanted the live camera view 75 to automatically identify anomalous thermal signatures as well as provide an estimate of problem severity and the amount of money saved if addressed. P9 summarized: “You want to make sure that you are in a very energy efficient area, so that you’re not spending too much money. Does making a change really help save energy costs? These are things I am interested in learning.” (P9) Privacy. While three participants had no concerns, half of participants (5) indicated they would adopt their normal digital photograph sharing practices for the thermograms. Two participants who had investigated the homes of others during the study considered those thermograms to be potentially sensitive and felt that they would need to ask for permission to share. P3 summarized: “All the photos from Missions 2, 3, and 4, I have no problem sharing. The ones from my friend's house I wouldn't want to share period; it's not my house to share. The ones from my house I'd be fine sharing online.” (P3) Usability. Most participants (9) wanted the thermal camera to be fully integrated with their smartphones due to the perceived tediousness of retrieving and connecting the attachment. Participants speculated that this change would make them more likely to perform explorative activities. 4.4 Discussion As the first qualitative, human-centered inquiry into novice approaches to smartphone- based thermographic energy auditing, the findings in this study demonstrate that novice users with minimal training can use thermal cameras to detect potential energy efficiency issues in the built environment; however, they lacked confidence in correctly interpreting 76 thermographic imagery and understanding the severity of problems they identified. Furthermore, our findings described: (i) how novice users collect and interpret thermal imagery, (ii) challenges that impede their auditing activities, and (iii) design considerations that could guide the development of future thermographic systems. Below, we reflect on our findings, suggest future work, and discuss limitations. 4.4.1 Reflection on the Method: Mission Structure In this study, we asked novices to freely explore their environment using a thermal camera as well as complete structured weekly missions (adapted from [129]). While the mission structure may have prompted certain behaviors that would otherwise not have been observed, they also allowed participants to explore different scenarios, motivated data collection, and helped keep participants engaged over the four-weeks. We believe that these methods enabled us to extract meaningful data and that they would be appropriate for studying similar technologies in the future within specific use scenarios like this one. 4.4.2 Barriers to Novice Thermographic Energy Auditing While novice users perceived value in their use of thermal cameras, they also highlighted several potential barriers to utilization of this data, which we discuss here. Knowledge and Experience. Future systems designed for novice use will need to consider how to assist them with performing thermographic inspections and interpreting thermal imagery. As noted by [109], professional thermographers suggested that knowledge of building materials, construction practices, and thermographic measurement procedures (e.g., ISO standards) are critical to performing a good thermographic scan. Future 77 applications could provide the needed scaffolding during data collection activities (e.g., via on-screen prompts). Tools that support novice analysis of thermographic data could help generate recommendations with assistance from automation, social networks, or professionals; this might help reduce the experiential gaps between thermographers. Decision Making. With the emergence of low-cost thermography tools, end-users will likely play an increasingly active role in energy auditing activities. Participants observed that thermal cameras were useful for detecting problems (e.g., air leakage around windows or doors) and, as others have noted [109,123], to perceive energy use in buildings. However, participants also expressed concern about not always knowing what to do with the information they obtained from their audits. Particularly in cases where users have the locus of control necessary to implement changes, it will be important to understand how to bridge the gap between information and action (e.g., through actionable recommendations) [73]. Future, more longitudinal work should investigate how likely novice auditors are to implement their self-generated recommendations, particularly in the home, and if energy efficiency improvements are achieved. Locus of Control. It is important to consider the limits of a user’s ability to effect change outside of their immediate locus of control (or use-contexts [96]). The barriers to effecting change expressed by our participants are consistent with the findings of other researchers who examined the role of social factors in energy consumption and building maintenance [26,28]. Unless the end-user is the owner or operator of the building, it may be difficult for them to enact change—particularly structural upgrades like improving insulation or the purchase of energy-efficient appliances. However, as building energy 78 efficiency is increasingly as a priority [8,84,145], authorities may give more credence to issues with sensor-based evidence such as that from a thermal camera. Future work should investigate how to assist end-users with verifying their sensor-based recommendations and advocating for having issues addressed. 4.4.3 Limitations Our study had four primary limitations, which should be addressed in future work. First, our participants were eco-conscious and highly educated, which may have influenced their perceptions and interpretations of thermography as well as their willingness to suggest taking actions. However, the participants also likely represent early adopters making their feedback and experiences valuable. Second, as participants were involved in a semi-structured study, findings may not translate to general, unguided use of these tools. Third, while a trained thermographer reviewed participant data, we did not attempt to systematically verify or study the accuracy of participant diagnoses based on their thermal images. Finally, some participants discussed making retrofit decisions or conversing with building operators (e.g., landlords) based on their thermographic findings; however, follow-ups were not part of this studies procedure, so it is not known what (if any) actions took place. 4.5 Conclusion This study contributes the first qualitative investigation of novice approaches to smartphone-based thermographic energy auditing. Through a four-week field study of end-user behavior, we assessed the efficacy of novice thermographic energy auditing activities across different use-contexts. Findings indicate that participants perceived 79 thermal cameras as effective diagnostic tools and suggests that novice imagery could have an impact on improving energy efficiency in the built environment. Through semi- structured interviews, we identified important challenges and potential benefits of engaging novices in thermographic energy auditing. These findings have implications for both the design of future thermographic tools and for Sustainable HCI researchers working in energy efficiency. Emerging, low-cost thermal cameras have the potential to broadly impact the way we interact with and understand our built environment—from residential homes to commercial buildings [5]. 80 Chapter 5 Professional Thermography & Automation   Figure  5.1:  We  developed  five  automated  thermography  design  probes  inspired  by  the  research  literature to help elicit reactions to envisioned automated thermography solutions, such as the above  unmanned aerial vehicle (UAV) design probe.   This chapter begins our second thread of research. Here, we focus on professional auditors and explore their perspectives on thermography and reactions to emerging automation. We present results from two studies: a semi-structured interview with 10 professional energy auditors, which included five automated thermography design probes, and an observational case study of a residential audit. We report on common perspectives, concerns, and benefits related to thermography and summarize reactions to our automated scenarios. Our findings have implications for thermography tool designers as well as researchers working in robotics, computer science, and engineering. This chapter has been adapted from a paper published at CHI 2015 [109]. 81 5.1 Introduction As noted previously, professional energy auditing has seen a resurgence of interest [82,123]. Professional energy auditors help identify building inefficiencies through walk- through inspections, on-site measurements, and computer simulations [142]. The DOE recommends home energy audits because of their impact on reducing energy usage (e.g., 5-30% reductions in monthly utility bills) and increasing structural safety [147]. With recent improvements in handheld sensor technology and falling costs, auditors are increasingly using thermography—infrared (IR) scanning with thermal cameras—to detect thermal defects and air leakage [9,20,92,146]. Work in automated thermography has also grown markedly in the past few years, encompassing disciplines from computer science and robotics to environmental and civil engineering. Researchers have primarily explored technical approaches for automatically transforming thermal images into higher fidelity 3D representations of buildings [64,70,94,95,119] and employing robots for data collection [17,37,41,96,107,125]. However, user studies of these tools have not been performed. And while some work exists on examining client reactions to thermography in general (e.g., [66,82]), perceptions of thermography use from the professional auditor’s perspective has received little attention. As the primary users of thermography (in the energy auditing context), this perspective is important both to the design of current thermal scanners and analysis software as well as to this growing area of automated thermography. In this work, we investigate current energy auditing practices and the role of thermography therein. we also critically assess the potential for automated thermographic methods. Our research questions include: How is thermography currently being used by 82 professional energy auditors? What benefits and drawback do auditors identify when envisioning the use of robotics for thermographic data collection? What are the implications for the design of these automated thermography tools? To address these questions, we conducted two studies: a semi-structured interview study with 10 professional energy auditors that included five design probes, and an observational case study of one on-site residential audit. For the design probes, we developed five scenarios of automated thermography based on the research literature— e.g., indoor robotic thermography [17,37] and large-scale urban thermography using unmanned aerial vehicles (UAVs) [41,96,107,125]. The scenarios were designed to provoke and ground discussion and critically assess how automated thermography may be used in the future. The interviews provide insight into current auditing procedures, the benefits and challenges of thermography, and reactions to the design probes, while the observation helps contextualize findings and further emphasizes the complexities of energy auditing. Inspired by the recent call to action from within HCI [132] to better understand practical efforts towards sustainability and to question the (over)promise of purely technological solutions, this chapter contributes the first human-centered investigation of thermographic automation. Our contributions include: (i) an assessment of energy auditing and thermography’s role therein through semi-structured interviews and a complementary observational study; (ii) a critical examination of the potential for emerging automated thermographic solutions through the use of five custom a design probes; and (iii) a set of reflections and guidelines to help inform the design of future energy auditing and thermographic tools. As interdisciplinary work, our findings have 83 implications not just for the design of emerging thermographic tools but also those research communities focused specifically on automated methods and human robotic interaction, which span computer science, building science, and civil engineering. 5.2 Study 1: Interview Study and Design Probes To investigate the role of thermography in energy auditing and to elicit feedback about emerging automated methods, we conducted a two-part study with 10 professional auditors: a semi-structured interview and presentation design probes based on the automated thermography literature. To help contextualize findings from these activities, we also conducted an observational case study of a residential energy audit. 5.2.1 Automated Thermography Design Probes The design probes included five scenarios using three different mediums: (i) three written narrative scenarios (~150 words) of increasing complexity that described thermographic 3D reconstruction and robotic data collection, (ii) a short video mockup of a UAV performing a thermal audit, and (iii) an interactive medium-fidelity prototype demonstrating how automation control and analysis software of a thermographic UAV may work in the future. Each probe was inspired by recent work in automated thermography and was designed to provoke discussion, ground conversation, and elicit feedback. The probes used 2nd-person narration to help our participants envision the scenarios. The full probes are included in the Appendices and are summarized below. 84 Scenario 1 (Text): Residential Audit. The first text probe described a residential audit using a UAV as follows: As you arrive at a home, meet with the client, and assess the home’s interior, a UAV collects exterior thermographic data and builds a 3D thermal model of the building in real-time. You investigate the 3D model (all building sides and the roof) via an interactive application on your tablet/smartphone. You can also browse anomalous thermal signatures, which can be shared with your client. The UAV automatically returns to a docking station on your vehicle after completing its scans. Scenario 2 (Text): Automated Audit of a Large Campus. The second probe positioned the participant as a facilities manager at a large campus site such as a university or government facility with many buildings. You are responsible for a small fleet of thermography UAVs. The UAVs fly around semi-autonomously collecting thermal data about each building on your campus. When abnormalities are detected, the UAVs are programmed to more closely examine these areas and provide high resolution reports of potential problems. The UAVs reduce labor costs compared with manual assessments, can investigate otherwise inaccessible areas of buildings (e.g., high exterior floors), and enable historical reports showing thermal performance over time. Scenario 3 (Text): Large-scale Urban Audit. The final text probe had the participant work as a government employee in charge of analyzing the energy efficiency of a large urban center with skyscrapers, office buildings, and other structures. You have real-time access to utility usage for each building as well as indoor and outdoor thermographic robots. The ground-based robots are permanently deployed at the larger buildings (e.g., skyscrapers) and communicate with the UAVs to provide interior/exterior thermal scans. As with Scenario 2, the UAVs function semi-autonomously and special software compares utility usage with thermal data over time. 85 Figure 5.2: Screen captures from the UAV‐based thermography video design probe (Scenario 4).   Scenario 4 (Video): UAV-based Thermography. The ~41 second video probe showed a semi-autonomous UAV collecting thermographic data of a campus building and performing real-time analysis. We created the video using a Parrot AR Drone 2.0 UAV, which is equipped with a 720p 30fps optical video camera. Adobe After Effects™ was used to create the robot’s interface and to apply a “thermal filter” to the video stream (Figures 5.1 and 5.2). Scenario 5 (Medium-Fidelity Prototype): UAV Control and Analysis Interface. For the final probe, we presented a medium-fidelity interactive software prototype that scaffolded the participant through the process of establishing a new survey project, including: scheduling a semi-autonomous UAV data collection flight and analyzing the collected data both spatially and temporally. This analysis procedure included the automated generation of a 3D model with a thermal overlay, an overview of interactive features, an automated point-of-interest analysis, and a comparison of historical data. The prototype was created in Axure™ using a combination of hand-drawn sketches and built-in widgets (Figure 5.3). For consistency, a researcher operated the prototype for all participants. 86 Figure 5.3: Screen excerpts from our interactive mid‐fidelity prototype built in Axure. (a) Users input a rough thermography route for the UAV. (b) The analysis software automatically identifies anomalous thermal signatures (red brackets) on a 3D‐reconstruction. (c) The auditor can zoom in and see how this area of the building has changed over time (every few months). (d) A co‐located temporal graph of the  estimated surface temperature is also provided.   In summary, the probes depicted a range of automated thermography scenarios, which varied in technological complexity, autonomy, and scope. Each scenario emphasized at least one new data collection approach along with some new analysis not currently possible with handheld IR cameras (e.g., the ability to see thermal signatures change over extended periods of time). 5.2.2 Methods Here, we describe our participants and study procedures. Participants. We recruited ten professional energy auditors (8 current, 2 former; 1 female) through email lists, word-of-mouth, and social media from across the US. Our ID Employer Auditor Experience (Yrs.) Age Gender Thermography Training P1 Former 1 25 Male Training Course P2 Private 20 61 Male Level 3 P3 Former 6 30 Male College Course P4 Private 11 57 Male Level 2 P5 Private 6 41 Male Level 2 P6 Private 7 51 Male Level 1 P7 Government 2 36 Female None P8 Private 4 64 Male Level 2 P9 Private 3 30 Male None P10 Government 7 53 Male Training Course Table 5.1: Study 1 participant (professional auditor) demographics.  87 recruitment materials specified that participants needed professional experience using thermal cameras for building energy audits. Our participants ranged in age (M=44.8 years old; SD=14.2), audit experience (M=6.7 years; SD=5.5), and location—six states were represented in total (Table 5.1). All participants reported the same number of years performing energy audits as performing thermography with the exception of P6 (7 years energy auditing, 4 with thermography). For thermography training, five participants received certification training through professional organizations (e.g., Infrared Training Center, The Snell Group, and similar organizations), two participants received on-the- job training through a company sponsored program, and one had taken a college course. Two reported no official training. To enable geographic diversity, half the interviews were conducted remotely via Skype with screen sharing to view the design probes. For the co- located interviews, participants read printouts of the text scenarios and used a researcher’s laptop for the video and mid-fi prototype. Procedure. Each session lasted an average of 93 minutes (SD=19.47) and included a background questionnaire, semi-structured interview, and design probes. The semi- structured approach allowed us to dynamically pursue themes we had not identified a priori. All participants were asked a similar set of questions, but new topics emerged in accordance with a participant’s background, skills, and experience. The design probes immediately followed the interviews. Participants were asked to “think aloud” and evaluate each scenario. The researcher provided guidance at the start of the video and throughout the mid-fi prototype. Our objective was to identify aspects of the probe that participants were interested in and uncover concerns. At the completion of the session, participants were compensated $20. 88 Data and Analysis. The sessions were audio recorded, transcribed, and coded for themes of interest. As exploratory work, we pursued an iterative analysis approach using a mixture of inductive and deductive codes [19,78]. We created two codebooks—one for each part of the study—which initially derived from the research literature, our study protocol, and post-interview discussions amongst the research team. The unit of analysis was an answer (or stream of answers) for a specific topic in Part One and a full reaction to each probe in Part Two. Our iterative coding approach was similar for both. Part One Analysis. Codes included: views on thermography (e.g., misconceptions, procedures, automation), impact (e.g., uses, benefits, findings), and challenges (e.g., application, clients, interpretation). A random transcript was selected and coded by two researchers. To calculate inter-rater reliability (IRR), we used Krippendorff’s alpha (α=0.64; SD=0.43; total disagreements=7 out of 120 decisions). Krippendorff’s [90] suggests that scores below alpha < 0.667 be discarded or recoded. In our case, 4 of the 10 codes were < 0.667. The two researchers met, resolved all 7 disagreements, and updated the codebook accordingly. Both researchers then independently coded a second random interview, establishing IRR (α=0.85, SD=0.24). Finally, the first researcher coded the remaining interviews. Part Two Analysis. For Part Two, we started with 11 codes including: types of use (e.g., traditional thermography, alternative applications), interests (e.g., automation, data, features), concerns (e.g., technical feasibility, data quality) and reactions to scenarios (e.g., positive, neutral, and negative). IRR was established after two iterations (final α=0.80; SD=0.27). However, the neutral reaction code was difficult to reach agreement on and 89 was removed before the 2nd code pass. Again, disagreements were resolved through consensus. The remaining eight probe transcripts were coded by a single researcher. 5.2.3 Findings Here, we present frequent patterns and emergent themes from our analysis. Interview Findings on Existing Practices We summarize five themes related to the current practices, concerns, and desires of thermography practitioners. Though our interviews asked about general auditing practices, our focus here is on thermography. Required Knowledge. Our participants highlighted the expertise needed to assess thermographic data, including: an understanding of building materials and construction (6 participants), an understanding of the physics of heat and airflow (5), applied training and experience (4), and an awareness of environmental conditions (3). As P2 states: “the thing that is absolutely the most critical is to understand how heat behaves and interacts with different materials.” Participants emphasized that simply pointing a thermal camera at a structure was insufficient: “you have to keep the environment in mind or else you’re going to make a judgment call and it’s going to be wrong” (P7). Thermography Benefits. Despite the admitted complexity, all participants (10) expressed that thermography provided at least some benefit to the audit process. Reaffirming prior work (e.g., [66,82]), 7 participants thought that thermography was useful as a communication tool for interacting with clients: e.g., P1 reflected, “how do you explain convective heat flow? If you have an image you can go and look… sometimes it’s tough 90 in words.” In addition, as a form of non-destructive testing, thermal cameras allow auditors to assess areas that are hidden or difficult to access. P10 gave a poignant example from an audit where he had trouble believing a client’s complaint: “But we gave her the camera, [and] she went right into the bedroom …directly over her bed between the two ceiling joists was about a four-foot strip with no insulation.” (P10) Participants also found thermography useful as a diagnostic (6) or verification tool (5). For example, P6 said, “I use [thermography] as a screening tool to then target the areas that you want to focus on.” For verification, thermography was used to confirm a suspicion or to check that a retrofit (e.g., new insulation) was completed correctly. P7 stated, “…you can survey large areas very quickly and… it should help you target areas to further investigate.” Client Interactions. Participants emphasized that an energy audit is a social process as much as it is a technical one. Most participants felt that client interactions were crucial to a successful audit (9), including information gathering at the audit’s onset, understanding client perceptions and motivations, and establishing trust. Some clients were wary that auditors were attempting to sell them retrofit materials, as described by P10: “[the auditor is] just telling me that so he can sell me a new HVAC system”. This attitude, P5 conjectured, “…stems from people selling [thermography] as a silver bullet. You’ve got people that say it can do more than it does.” To overcome these issues, energy auditors strategically include the client in the audit process, asking residents to identify problems with them: 91 “…give the customer the thermal camera. And have them look around and have them say ‘oh my look at that’, ‘what's that’, which is very engaging and opens them up to a discussion about the dynamic of what's happening in the house, or the wall, or the attic.” (P10) In terms of client motivations for scheduling an audit, comfort was the most frequently mentioned. Cost savings and environmental concern were also mentioned, though less common (e.g., P8 recalled only one household who was concerned with their “carbon footprint” over 4 years). Thermography Challenges. All participants expressed concerns about thermography, including the difficulty of interpretation (8), untrained or undereducated practitioners (7), and equipment sensitivity (6). Interpretation was viewed as difficult because of the influence of confounding factors such as weather, shading, nearby buildings, and building materials. Given these complexities, thermography was characterized as a highly subjective process (similar to [150]), as captured by P2: “The reality is that you can have three guys with the same camera, looking at the same thing, and have three totally different reports.” (P2) Weather could also be frustrating because of the required interior-to-exterior temperature differential: “unless there's a really big temperature swing you're not going to see much with the thermal camera” (P1). However, high end equipment has the potential to mitigate some weather conditions, as stated by P5: “if you’ve got the right camera the time of year [or day] really doesn’t matter.” While having adequate 92 equipment is important, participants emphasized that having a better camera only helps to a degree; it will not make practitioners better interpreters. Desire for Automation. Before moving into the design probes (so as to limit bias), we asked our participants how they might automate an energy audit with or without thermography. Eight participants expressed interest in automation including: data collection (6), assessment (5), and report generation (2). For data collection, participants mentioned thermal cameras mounted to cars that survey neighborhoods quickly to identify locations with “visual thermal patterns that may be indicative of energy issues” (P4). Similarly, P10 suggested a thermographic overlay in Google Earth that would allow inspection of entire areas and identify “building stock that is inefficient.” For automating assessment, three participants mentioned 3D reconstruction, two mentioned better energy models or simulations, and two mentioned reducing or eliminating subjectivity. For example, P9 thought a dream tool would be a thermography report that “could interface with a 3D model of the [audited] home.” P4 thought automation should eliminate subjectivity: “make it independent of the auditor… my interpretation should not be different from yours.” For report automation, participants mentioned efficiency and reducing the tedium of manual preparation, P6 states: “The biggest problem in dealing with the volume of work is creating reports.” Still, some participants expressed concerns with automation, such as P2, “how do you get the software to understand what the [auditor] otherwise understands.” Part One Summary of Existing Practices. Our findings reaffirm and extend past explorations of energy audits (e.g., [66,82,123]). Thermal tools should be designed both 93 for expert users (the auditors) and for client interactions. In terms of automation, our participants were most interested in automating data collection and assessment followed by report generation. However, these automated solutions should remain visually oriented to facilitate client interactions and will need to address the same challenges that manual approaches have to overcome (e.g., establishing temperature differentials). Findings from Individual Design Probes We first summarize overall reactions to the design probes before describing common themes, suggestions, and concerns. Overall Reactions. Our design probes elicited mixed reactions. Though most (9) reacted positively to the mid-fi prototype (Scenario 5) and to the multi-building and urban scenarios (Scenarios 2 and 3), only 2 participants found value in the UAV-based residential audit (Scenario 1) and reactions were equally split to the video (Scenario 4). P5 reacted negatively to all scenarios, feeling that it would be hard to acquire “actionable data” and expressing concerns for data quality: “doing an exterior flyby is not going to be a replacement for an actual audit of a building.” He was most positive about automating interior scans. Scenario 1. Most participants (7) reacted negatively to the UAV-based residential audit, expressing doubt that meaningful data could be acquired from exterior scans without, for example, blower door tests as well as concerns for cost and data overload. For the two participants that reacted positively, they mentioned its ability to examine inaccessible places, save time, and generate 3D models. 94 Scenario 2. In contrast, 9 participants reacted positively to the UAV-based multi-building scenario, largely because of opportunities such as tracking degradation over time and examining inaccessible areas and equipment (e.g., HVAC). Still, participants expressed concerns about cost and the need for the system to have more information on building materials and construction for proper analysis. Scenario 3. Similar to Scenario 2, most participants (9) were positive about the large-scale urban monitoring system, including the connections between thermography and utility data, the automatic anomaly detection, and “push” notifications. Participants also mentioned that this system could be used to check on LEED certified buildings that are supposed to be performing efficiently. Primary concerns included handling reflective surfaces and the “heat island effect” (where built structures like pavement cause increased ambient temperatures). Scenario 4. Half of the participants (5) reacted positively to the video probe of a UAV surveying a campus building. Identified benefits included the ability to reach inaccessible areas (“terrific for large buildings,” P6), and as a tool for performing rapid preliminary investigations. Concerns included feasibility, the need for more information than is available from an exterior scan, and the autonomy of the UAV (how it was controlled). Scenario 5. Finally, most participants (9) reacted positively to the mid-fi prototype, citing its ability to provide geometrical (3D) model data, historical analysis, and automation scheduling. Participants suggested that the software tool should incorporate energy analysis from metering, information about building construction, and combined interior/exterior views. 95 Themes Across Design Probes Automated Data Collection. Most participants (9) agreed that there was the potential to save time and money with automated data collection “You can get the UAV to film a whole side of the building at once and then you can zoom in on the sections you want to see.” (P8). However, there was general recognition that simply performing thermography was not sufficient—more data was necessary such as utility usage, weather, and information on building materials. Still, most participants thought UAV-based or other automated methods would be sufficient for preliminary analysis—though P5 thought it would create too many false positives. Historical Analysis. Most participants (9) mentioned the benefits of historical analysis, which are really only feasible via automated data collection due to labor/time costs. As P7 highlights: “…If you said, ‘Hey, for four months, we've had this. Let's look and see how it could be fixed.’ I like that idea.” Typically, thermal scans do not include temporality (i.e., the ability to look back in time and observe changes). Model Generation. A majority of participants (6) saw value in automatically generating 3D building models with accurate geometry because it increases the utility of the collected data, enables faster analysis, and the resulting geometry could be exported into other tools: “You spend a lot of time building this model, just measuring the outside of the house, counting the windows and the doors, and looking around. Then, you take that data load it into your modeling program… this would streamline that.” (P10) 96 Automatic Anomaly Detection. While most participants (8) accepted the “anomaly detection” in our scenarios without comment, 2 provided critical feedback related to the complications of filtering out noise, removing false positives, and the difficulty of interpreting the data: “How do you get rid of the noise and have actionable data so that you save labor? …I think you’re going to expend a vast amount of labor in chasing down false positives.” (P5) Data Quality. Half (5) of our participants raised concerns about data quality including the feasibility of using automated exterior scans to acquire useful thermal data across environmental conditions (e.g., weather, sun direction). P8 questioned whether exterior scans could yield meaningful data at all: “I don’t see this as being very useful at this point primarily because the use that I've been able to make of [external] thermography is limited.” (P8) Data Overload. Three participants expressed concerns about data overload: “I don't see the value at this initial moment …there’s some new generation tools but it’s still just too much data” (P6). Others thought the 3D reconstructions would allow for better organization of the data leading to better interpretations. Feasibility. Feasibility concerns included technological viability, robustness, and cost. Robustness and maintenance costs were potential barriers to adoption: “I don't know that many fiscal managers would be able to justify the system” (P3). Additionally, some participants (4) raised concerns about the need to have control over the environment 97 because, “you have to set up a pressure difference to be able to identify air infiltration… a UAV can’t do that” (P1). Fear and Privacy. Though only mentioned by three participants, there was reasonable concern about how robotic thermography may frighten people or impinge on privacy: “If [people are] in the building, they’d feel a little bit frightened” (P3). P7 mentioned that UAVs may collect unintentional data: “though you're focused on your clients' residence, you're going to get some of the neighbors” (P7). Part Two Summary of Design Probes. Our findings highlight important concerns with automated solutions described in the literature but which have previously not been discussed or acknowledged such as issues of data quality, data overload, technical feasibility, privacy, and problems of overreliance on automated exterior scans. Still, participants were positive with the general idea of automation especially 3D reconstruction, historical/temporal analysis, anomaly detection, and the potential for time savings. 5.3 Study 2: Observational Case Study To help contextualize Study 1 findings we also conducted an observational case study of a residential energy audit. 5.3.1 Method We recruited a senior energy auditor from the Maryland Energy Administration’s list of certified practitioners. The participant was male, age 50 and had 5 years of energy auditing experience. Informal thermography training was provided by his employer. For 98 the observation, the auditor selected an appointment he considered a “typical audit” and received client permission for our presence. The audit took place in a mid-sized home and lasted ~100 minutes. A researcher shadowed the auditor, taking field notes and photographs. Due to weather conditions, thermography was not used; however, the auditor spoke about how/why he would ordinarily use thermography. Following the audit, the participant completed a 30-minute debrief and was compensated $20. We thematically analyzed field notes from the observation and debrief session [19] looking for patterns that confirmed, extended, or differed from Study 1. 5.3.1 Findings Here, we present my observational findings around three areas: procedure, using thermography, and challenges. Audit Procedure. The auditor said that he generally spends 2 hours in the field, plus 4- 5 hours for report generation and follow-up confirming Study 1’s finding that report generation is effortful and time consuming. The audit procedure included meeting the client, establishing rapport, and determining reasons for the audit. The client joined the auditor for an initial walkthrough, which the auditor later explained was critical to enhancing client understanding and buy-in. During the walkthrough, the auditor took pictures of areas of interest and performed both a combustion test (e.g., checking appliances) and a blower door test. Here, the auditor indicated he would normally use his thermal camera. Finally, the auditor met with the client to explain findings and suggested changes explained in terms of cost savings. The next day, the auditor created and sent his report to the client using in-house software. 99 Thermography. Though thermography was not used, the auditor did not think thermography would have altered his overall efficiency recommendations to his client. He described using thermography for confirmation, client communication, and to help work crews perform retrofits. Again, the visual nature of thermography was key to “help[ing] explain complex things.” He described a client base motivated by utility bills: “Many people expect the bill to be wrong, not to have an issue in the home.” The thermographic images helped overcome that misconception. Primary Challenges. The auditor described two technical challenges: establishing proper conditions for thermography and the effort required to generate a report. 5.4 Discussion As the first qualitative, human-centered inquiry into both conventional and emerging thermographic processes and tools, our findings help reveal challenges, highlight energy audits as a social-technical process, and inform future work. Below, we reflect on our findings, provide design considerations, and discuss limitations and future work. 5.4.1 Conventional Thermography Auditors were generally positive about the role of thermography in energy auditing, particularly as a communication and diagnostic tool—but stressed that it required skill and expertise to use. Here, we focus on three aspects of conventional thermography that have implications for design and future research. Social Process. As in [82], energy auditing was portrayed as a social process as much as a technical one. Auditors emphasized the importance of understanding their client’s 100 needs and reasons for a home assessment, gaining trust and credibility, and being able to explain identified problems and their implications. To help establish trust and communicate findings, auditors allowed clients to operate their thermal cameras. This “role reversal” places increased emphasis on the thermal camera while deemphasizing the interpretative role of the auditor. In this way, the thermal camera becomes a sort of “inscription device” [97] that translates the complex or the contested into material fact but potentially obscures the full complexities of the instrument, the techniques required for proper use, and the skills necessary for interpretation. To support this social process and role reversal, how can future tools be designed to accommodate both expert and novice users (clients)? How can tools better support auditor-client interaction, both in real-time during the audit as well as post-hoc in the report generation process? Subjectivity. Though thermography relies on sophisticated technology, the interpretation of its output is fundamentally subjective. Our participants desired greater objectivity in how to apply and interpret thermography but barriers included a lack of universal standards, varying levels of training in the auditing community, poor guidelines, and the inherent complexity of the task (echoing [150]). Participants with higher levels of training in our study (Level 2 or 3) felt that they had superior interpretative skills than those without. However, more work is needed to study how training and experience affects interpretation, how interpretations vary across thermographers for the same audit site (extending [74]), how these differences manifest in energy efficiency recommendations, and how tools can be better designed to aid analysis and reduce subjectivity. 101 Ethical Concerns. Subjectivity is also a concern in a transactional context where thermography is used not only to identify problems but also to make a sale. As noted in our findings, some clients are skeptical of auditor motives, particularly when the auditor works for a home improvement company (as 3 did in our study). In these cases, auditors may consciously or unconsciously be biased in their interpretations. If future thermographic tools can reduce subjectivity, ethical concerns may be mitigated. 5.4.2 Automated Thermography Our five design probes allowed us to explore thermographer reactions to various automation scenarios, including indoor and outdoor robotic data collection, 3D reconstruction, automatic anomaly detection, and advanced temporal analyses. We discuss challenges, privacy and policy implications, and a call-to-action. Challenges. Though 9 of 10 participants reacted positively to one or more design probe(s), our findings surfaced important concerns regarding data quality, data overload, fear and privacy, and technical feasibility—none of which have been studied in the automation literature. For automated data collection, in particular, our auditors were concerned with the lack of environmental control compared with manual audits (e.g., to establish pressure differentials), how to manage this large amount of data, and general data quality issues (e.g., filtering). However, most were interested in how this “big data” may transform and enable new analyses (e.g., historical comparisons). For 3D reconstructions, our auditors noted that thermal data alone, though useful, is insufficient—better models would include information about building materials, weather conditions during the scan, utility data, and even occupant behaviors. 102 Privacy and Policy. Though mentioned by only a few participants, the use of remote, automated data collection has privacy and policy implications. For example, if buildings can easily be scanned at scale, how may this change the way governments assess and regulate building energy efficiency and/or award and monitor “green” certifications (e.g., LEED)? New business models are emerging (e.g.,[103]) based on automated thermography that sell exterior scan data and analyses to utility companies to help determine which houses “leak the most energy” and target energy-efficiency programs. Because exterior thermal scans can be performed remotely (e.g., from the street or air), should a building’s thermal profile be considered public data? Can building owners opt- out of scans? Moving Forward. As a pursuit framed purely as a technical challenge, the automation literature has been, unsurprisingly, focused on engineering. However, our findings further highlight thermography as a socio-technical problem where the interplay between auditor, client, and thermal camera plays a crucial role (e.g., in building trust, communicating results). Future automation work should consider existing thermographic practices and engage in human-centered design with both auditors and clients to improve and validate their tools. As others have argued, the Sustainable HCI community needs to be more engaged in these emerging areas, especially those that are not necessarily consumer-facing. Thermography is a growing area that will likely become more popular as governmental institutions increasingly recommend thermographic-based energy audits and thermal devices become more prevalent. 103 5.4.3 Limitations There are four primary limitations to this work. First, we interviewed professional energy auditors who specialize in residential buildings. Reported practices and reactions to the design probes may differ from those of commercial and industrial energy auditors. Second, our design probes emphasized UAV-based exterior data collection, anomaly detection, historical analysis, and 3D reconstruction. Future work should examine other parts of the automation pipeline (e.g., indoor robotic, data collection, report generation). Third, our study method relied on self-report data, complemented by a single energy audit observation (without thermography). Longer-term ethnographic fieldwork of energy auditors may yield new insights. Finally, we acknowledge the potential dichotomy in asking professional auditors about scenarios that could be perceived as replacing or undercutting their jobs; however, none made such comments. Instead, auditors expressed interest in automation for its potential to increase their efficiency, enable new types of analyses, improve building models/simulations, and allow for greater coverage. 5.5 Conclusions This paper contributes the first human-centered investigation of thermographic automation. Through semi-structured interviews and a complementary observational case study, we assessed energy auditing practices and thermography’s role therein. Through five design probes, we critically examined emerging automated thermographic solutions and identified important challenges/concerns. Our findings have implications not just for the design of emerging thermographic tools but also for researchers focused on automation and human robotic interaction. 104 Chapter 6 Development and Testing of a Preliminary Temporal Thermography Sensor System Figure 6.1: Common energy efficiency issues visible to a thermal camera.  In this chapter, we begin our final thread of research. We present a novel sensor system for collecting and analyzing temporal thermography data, which reduces manual labor and enables new types of thermographic analyses for energy auditing applications. To gather initial user reactions and better understand the potential of this temporal data, we conducted a small field study deployment supporting an energy audit in a university building and a usability pilot study with four graduate students with previous experience using a thermal camera and. We describe initial results, drawbacks, and enumerate directions forward for this emerging area. This chapter has been adapted from a poster paper published at Ubicomp 2017 [109]. 105 6.1 Introduction There have been three main approaches to addressing the impact of buildings on the environment in the ubicomp community: (i) behavior change research (see survey [58]), (ii) building sensors that monitor energy related characteristics (e.g., [68]), and (iii) interactive visual analysis tools (e.g., [32]). Here, we focus on a combination of (ii) and (iii) by exploring new methods and tools to support the growing community of professional and novice energy auditors who inspect buildings to estimate their energy efficiency and generate improvement recommendations [109,110,117]. Energy auditors investigate buildings using a variety of techniques including thermography where an infrared thermal camera is used to scan for anomalous heat signatures, which may indicate insulation problems, air leakage locations, or other issues with a building’s envelope (Figure 1). Thermal imagery is also an effective visual communication aid used to describe problems to building owners [109]. However, collecting thermal imagery can be laborious and, if environmental conditions are incorrect, misleading or ineffective [109]. Compounding this problem, energy auditors must also adjust measurement parameters (e.g., emissivity) that impact measurement accuracy [55]. Enabling auditors to analyze a sequence of images of the same location over time (i.e., temporal thermograms) is one method which may mitigate these issues and provide new insights [55]. However, widely available tools (e.g., consumer thermal cameras) do not support this use case well. In this chapter, we introduce a temporal thermography system that consists of: (i) a novel sensor system mounted on a servo motor to periodically collect panoramic thermograms paired with humidity, temperature, and motion sensor data and (ii) a 106  Figure 6.2: Our prototype  for a  Figure 6.3: The sensor system provides a webserver allowing users temporal  thermography  sensor  to  set  schedules  and  view  results,  which  are  fed  into  the system used in these studies to  visualization tool. Computer and Person Icons © Tinashe Mugayi and Sharma, augment building energy audits.  respectively, of the Noun Project:  https://thenounproject.com/  corresponding interactive visual analytics tool for viewing and analyzing this temporal data. Through a pilot usability study and a small field deployment, we begin to examine the utility of temporal thermograms and reactions to such tools. This work is a first step toward exploring: What value and insights, if any, does temporal thermography provide energy auditor? And, how might temporal thermography be incorporated into building energy audits? This work contributes to the growing area of using sensor systems, and similar Internet of Things (IoT) devices, for building analytics [46,76]. 6.2 System Design Here, we describe our preliminary sensor system which consists of two primary components: a longitudinal thermographic sensor system and an interactive temporal visualization tool. 6.2.1 Longitudinal Thermographic Sensor System Our sensor system consists of a custom 3D-printed enclosure that contains a FLIR One thermal camera, humidity, temperature, and motion sensors, and a Raspberry Pi for data processing (Figures 6.2, 6.3). The sensor system rests on a pan unit atop an 107 interchangeable mounting plate though in practice we use a standard tripod mount in our work. The system can be deployed in a location to collect data over days or weeks based on a user-specified schedule. Users specify a data collection schedule and access the results via a web application hosted on the Raspberry Pi. 6.2.2 Interactive Temporal Visualization Tool The multi-modal sensor data is viewable in a web application developed in JavaScript and Python (Figure 4). D3.js was used to develop two modes of interactive visualizations: A Single-Image Mode and a Temporal Mode. In both modes, users can make point and box selections to extract and display temperature data about the region of interest along with descriptive statistics (e.g., maximum, minimum, mean). Single-Image Mode. This mode was developed to provide auditors with a view of the data similar to FLIR Tools, a commercially available application for viewing and Figure 6.4: The single‐image mode of the interactive visualization with temporal slider.  108 Figure 6.5: The multi‐image temporal mode of the interactive visualization.  analyzing thermal images (e.g., using point and box selections). A single image from the dataset is displayed with a slider that allows the user to move through time (Figure 6.4). Temporal Mode. To more deeply explore comparing images over time, we created a “temporal mode” based on [36] (Figure 6.5). This mode is centered on a Parallel Coordinate Plot (PCP) of the temperature changes between the images, which visualizes temperature trends at each pixel location over time. Additionally, the sensor system’s measured internal and outdoor temperatures at the deployment location can be overlaid on the graph. Above the PCP are two user-selected images from the dataset, taken at separate times, controlled by the bottom slider. 109 6.3 Usability Study Participants were recruited for a usability pilot study via emails sent to a student listserv and enrolled on a first-come, first-served basis. Four University of Maryland graduate students participated in the study; three of whom had prior experience using thermal cameras for building energy auditing applications but no formal training. Sessions lasted approximately 40 minutes. Participants were asked to analyze two datasets previously collected with the sensor system using both the commercially available FLIR Tools software and the two modes of our visualization tool. The first dataset was used to train participants. The second dataset was from a test deployment—where the sensor system faced an exterior window of a second story building—and comprised the usability study (Figures 6.4 and 6.5). As participants used the tools to analyze the datasets they were asked to “think aloud.” Participants were then asked to describe their experience, what they learned, and how they might use the system in the future during a brief semi- structured interview. Session notes were thematically analyzed. Participant quotes are attributed using a ‘N’ for novice followed by their identification number (e.g., N1). 6.4 Usability Study Results All four participants stated that the visualization tool was easy-to-learn and allowed them to more easily notice temporal changes in thermal data compared to the widely available tool. All participants recognized the transient conditions caused by solar loading and reflections from surrounding buildings. Participants suggested that this type of information and visualization could be included in home automation systems. One participant said, “I’d like to connect this with my smart thermostat to compare the data 110 and see what impacts different settings have” (N3). Two participants discussed continually collecting this data for personal use and, when prompted, were not concerned about potential privacy implications. We also noted usability issues. The most pressing, as one participant said, was that “using the tool is easy if I know what I want to look at,” (N2) but that it could otherwise be unclear what to focus on. All participants pointed out that comparing images was difficult because the color scales were relative to the observed temperatures; they suggested normalizing the images and scales to help synchronize the displayed data. 6.5 Field Deployment Study Next, we deployed our sensor system to assist with an energy audit of a university building. The audit was being conducted by a Mechanical Engineering MS student on behalf of the Office of Facilities Management; he had some thermography experience, but no professional training. The sensor system was deployed in a room that staff had reported to be thermally unstable (Figure 5). The goals were to: (i) investigate whether recent changes to the room’s HVAC settings were properly regulating the conditions Figure 6.6 (Left to right): Our sensor system was placed in the center of the film archive, analysis was performed on large sections of walls and windows by the auditor, and the visualization tool provided views of the data.  111 since the room housed archival materials and (ii) check for any adverse effects caused by solar loading or structural degradation. The sensor system was scheduled to collect data in 30-minute intervals over three-day spans on two separate occasions, first during winter weather (i.e., cold, snow) and again during spring weather conditions (i.e., warm, sunny, clear). The participant reviewed the data using our analysis tool. Participant quotes are attributed using a ‘A’ for auditor followed by their identification number (e.g., A1). 6.6 Field Deployment Results The participant found no evidence of structural issues during the observation periods; all sensor data indicated stable environmental conditions that seemed to be invariant of external weather conditions. The PCP suggested there was some evidence of solar loading, but this was likely not significant. The participant commented, “The data supports the conclusions I made based on my models and makes me more confident in the recommendations that I’ll make going forward” (A1). Additionally, the participant was positive about the potential uses of the sensor kit itself, indicating that our system could be used to aid facilities management in other deployments. 6.7 Conclusion Through our usability pilot study, we found that the temporal data may make identifying certain transient environmental conditions easier, which would be useful for auditors of varying skill. However, inexperienced users will likely require more support before they can meaningfully interact with the system and extract insights. Moreover, through our case study deployment we explored augmenting traditional energy audits (e.g., those that rely on walkthrough inspections and building modeling) with temporal thermography 112 data. Future work will focus on: (i) integrating additional sensors (e.g., air quality) useful for auditing applications, (ii) implementing advanced signal processing and anomaly detection algorithms, (iii) exploring a wider range of visualizations that can be applied to temporal thermography and other collected data sources, (iv) assessing what insights, if any, professional and novice energy auditors derive from temporal thermographic data collection and analysis, and (v) investigating best practices for augmenting building energy audits, building automation systems, and smart homes. 113 Chapter 7 Longitudinal Thermal Camera Sensor System Design, Validation, and Evaluations Figure 7.1: The iterated longitudinal sensor system (left) and report (middle), and a sample of data  collected by homeowners in the field‐study (right).  In this chapter, we present the iterated longitudinal thermographic sensor system, incorporating quantitative analysis of temporal thermography data, a simplified reporting interface, and computational user supports. We then present three studies: a technical evaluation of the sensor system, in-home end-user deployments with 5 homeowners in 5 households, and semi-structured interviews including a presentation of design probes with 5 professional energy auditors. Our findings demonstrate that temporal thermography can assist end-users with gauging the severity of issues, and our system provides the possibility of new auditor-client interactions; from these findings, we derive design implications for future temporal thermographic systems and in-home sensing. This chapter is based on unpublished material. 114 7.1 Introduction Energy efficiency issues such as missing or degraded insulation are quite common in US residential buildings [117]. However, detecting these issues can be difficult as there is typically no visible indication of a problem on the surfaces of a building’s envelope. While energy audits are effective at locating insulation issues, professional services are not widely used [123] and, until recently, techniques that can reveal insulation issues were not easily applied by novices. Inspecting insulation in buildings can be done using destructive or non- destructive testing methods. Destructive testing involves directly inspecting insulation through a small amount of damage to a building’s envelope (e.g., drilling a hole), while non-destructive methods measure surface temperatures to estimate performance [92]. Unsurprisingly, non-destructive testing measures are often preferred by building owners. Non-destructive testing tools such as thermal cameras can help collect temperature data during walk-through inspections, insulation performance estimates are not typically calculated [109]. Instead, auditors tend to rely on rapid, subjective visual scans to locate problems and verify the impact of performance upgrades which can be an inaccurate. As described in Chapter 2, a promising approach for performing insulation assessments is temporal thermography [6,34,52,99,113–115]. Unlike in-situ scans, temporal methods collect and average data over time, improving assessment accuracy; however, there are many limitations, including: (i) needing multiple measurement devices, (ii) laborious setup procedures, (iii) equipment needing to remain in place and undisturbed for extended periods of time, and (iv) high-volumes of data requiring processing and analysis. As a result, it is unclear how such tools might be integrated into 115 current energy auditing practices or what benefits, if any, such analyses may provide over current methods. To address these issues, we present an easy-to-deploy longitudinal thermographic sensor system designed to support residential energy audits—a second generation system iterated from the designs described in Chapter 6. Our iterated system employs low-cost, off-the-shelf hardware and software to semi-automatically collect and analyze temporal thermographic data in the built environment. As work in applying temporal thermographic analysis to building energy auditing typically does not involve end-users (i.e., energy auditors or building owners) and focuses on unoccupied spaces instead of active residences, our research questions are exploratory and include: What might end- users learn from temporal thermographic analyses? How can temporal thermography be incorporated into current end-user’s energy auditing practices? How can such analyses influence end-user behaviors or perspectives? And, what implications are there for the design of future thermographic systems? To answer these questions, we conducted three studies: a technical evaluation of the sensor system, an in-home end-user deployment, and semi-structured interviews including a presentation of design probes with professional energy auditors. Findings from these studies highlight (i) the effectiveness of temporal thermography to assist end- users with gauging the severity of energy efficiency issues and (ii) the potential for new auditor-client interactions. Contributions from this work include the design of a novel temporal thermographic sensor system designed to support residential energy audits, a summary of benefits and challenges associated with such systems, and design 116 recommendations for future temporal thermographic systems that support in-home use by novice and professional energy auditors. 7.2 Temporal Thermography Sensor System Design As noted in Chapter 2, research into temporal thermography for building diagnostics promises to make performance issue analysis more accurate and to establish effective methods within building energy audits and modeling. Little attention, however, has been paid to how such approaches might be perceived by end-users (e.g., with regard to the practices of professional auditors or the daily lives of homeowners), performed in the field by human auditors, or incorporated into building sensing systems. As a result, the aim of our research is to (i) evaluate the use temporal thermographic scanning techniques to perform rapid inspections of potential insulation issues in the field, (ii) explore the potential benefits and challenges associated with this approach by end-users, and (iii) evaluate the potential for integration with current energy auditing practices. Our approach to this research focuses on the design and evaluation of an easy- to-deploy temporal thermographic sensor system for supporting longitudinal building energy audits by professional auditors and novices alike (Figure 7.1). The current version of our system consists of four core components: (i) a FLIR One smartphone-based thermal camera, (ii) a custom-built, portable docking station for the camera which provides additional sensing capabilities and semi-autonomous management of data collection, and (iii) a central server for analyzing data and preparing automated reports. After enumerating our design goals, we describe the data collection system, the server, their operation, and rationale for specific design decisions based on feedback from informal deployments and expert reviews. 117 7.2.1 Design Goals Our design goals for the system fall into five categories:  Easy-to-deploy/use: Both novice and professional end-users should be able to deploy the system in most buildings following a simple procedure so that the system fits into energy auditing practices easily lest it not be utilized.  Low-cost and use of off-the-shelf technologies: As a primary barrier with thermography and home automation technologies is cost, the system should take advantage of commodity sensors, hardware, and software wherever possible.  Fit with current energy auditing practices: The system should support the goals of a residential energy audit which include structural, thermal comfort, and health and safety elements; results from temporal scans should not greatly increase analysis time and should lead to a report that is holistic and informative.  Privacy preserving and minimal impact on occupants: The system and its use should minimize opportunities to capture data that participants may not wish to share and should have minimal impact on the building occupants’ routines.  Actionable recommendations: Assuming issues are identified as a result of performing a temporal scan, the system should provide recommendations which are accurate and actionable; recommendations should include both professional services and DIY alternatives where appropriate. 7.2.2 Data Collection System The docking station was designed to be mobile, easy-to-use, and to contain all the components needed to make temporal thermographic data collection easy for most 118 building thermography practitioners once their personal smartphone-based thermal camera is attached. The docking station itself consists of a custom-built 3D-printed enclosure containing a set of sensors commonly used in building management and data collection initiatives [46], a Raspberry Pi—that provides the computing resources, power distribution, and Internet connectivity—and a touchscreen interface for use during setup and calibration procedures. Here, we describe these elements in more detail. Enclosure Design. The enclosure was iteratively designed in Tinkercad3 and printed using a MakerBot4 5th Generation 3D printer (Figure 7.2). One challenge we encountered during pilot deployments was the dynamic layout of the buildings, with rooms of varying size, interior layouts that are rarely known a priori, and containing areas where placing equipment would inconvenience building occupants. For example, our previous design Air Quality FLIR One Motion Temp/Humidity Touchscreen Display     Figure 7.2: The final version of our Tinkercad CAD model before exporting for 3D printing (left) and  a completed sensor system with a FLIR One, version 2, attached (right). The sensors are constantly  streaming data to the touchscreen display which is in its idle mode, displaying data, recording state,  and other status messages (e.g., Wi‐Fi connectivity, storage space remaining, etc.)   3 Autodesk Tinkercad: https://www.tinkercad.com/ 4 Makerbot Replicator 5th Generation: https://www.makerbot.com/3d-printers/replicator/ 119 [104] required the sensor system to be placed on a tripod with a wide foot-print which made it difficult to deploy the system in rooms with tight spaces or in heavily trafficked areas where it could impede occupant mobility. Thus, each iteration on the enclosure and the system’s physical design tended toward increasing the ease of deployment in a home environment and minimizing the impact on occupants. The current enclosure is designed to be free standing on four legs, which allows the system to sit stably atop a table, shelf, or other piece of flat furniture. The enclosure is compact: the primary constraint on its minimum height and width being the dimensions of the touchscreen component. The enclosure is printed as one solid piece with two small support beams that affix to the touchscreen and snap into the rest of the enclosure. Though the enclosure is lightweight (~453 grams), it was printed with extra infill to increase resistance to minor bumps or nudges while deployed. However, using the system with a tripod offers additional versatility in deployment locations while maintaining stability. Therefore, a mounting plate is installed at the enclosure’s base. Sensing Hardware. The on-board sensors include temperature, humidity, air quality, GPS, and motion, which are commonly used in building assessments (e.g., [46,76]) and necessary for performance calculations. The indoor temperature and humidity are critical for correcting temperature measurements extracted from thermal imagery and supply basic thermal comfort metrics. The GPS sensor improves the sensor system’s accuracy by supplying a more exact position of the system when querying online data sources for external weather information as compared to IP-based lookups. To protect user privacy, a concern described by participants of our previous studies [109,110], the system uses a motion sensor to determine when a person or pet may have entered the 120 frame and filters out data and images from those periods of time. Finally, an air quality sensor was included to provide health and safety data, which complements professional audit assessments of air quality and ventilation issues. Computing Hardware. The enclosure contains two computing boards: a Raspberry Pi and an Arduino Uno. The Raspberry Pi runs the Android Things operating system (v6.0) and operates a custom-built application that manages: interactions with the user, data collection, data storage, power distribution, Wi-Fi connectivity, and the thermal camera. The Arduino board manages all other sensors and communicates their data to the Raspberry Pi over a serial connection which in turn stores the data in a local database and updates the user facing display. Given the relatively recent release of Android Things, this division of computing hardware allows us to take advantage of stable libraries for communicating with the onboard sensors while making assembly, maintenance, and troubleshooting of the system easier in practice. User Interface. We developed a custom android application to run on the touchscreen component and serve as the main point of interaction for users during data collection activities. The application provides simple menus for connecting the device to a building’s Wi-Fi network, calibrating the thermal camera, and scheduling data collection tasks. Users interact with the application through common touch gestures (e.g., swipes, taps) which allows them to quickly configure the system. When the user is not interacting with the screen, the application has a background mode that provides an ambient display of real-time sensor data and status messages; it also regulates screen brightness—reducing it over time and restoring it when interactions occur. 121 7.2.3 Operating Procedure Before beginning a temporal thermographic scan, the user must calibrate the system—a standard part of using any thermographic system. The parameters that are estimated during our calibration process are: distance to the measurement surface, surface emissivity, and background reflectivity; the user also provides a region of interest. To begin this process, the user places the system in an interior room facing perpendicular to the building envelope (i.e., an exterior wall’s surface) and as far back as possible without inconveniencing building occupants, which in a typical home is approximately 3-5 meters; during pilot deployments, we found that setting the system up at this distance also helps with context during later analysis and reporting activities. Next, the user affixes a calibration marker to the wall surface with painter’s tape. This custom- made calibration marker contains (i) an 11x9 sheet of paper with a QR code that enables the system to locate the marker and calculate the distance to the target surface (used later to extract accurate temperature information from the thermal images) and (ii) a high- emissivity sheet of tinfoil—crumpled then smoothed—that creates a baseline measurement for the system to use when it calculates surrounding reflectivity [135]. Once both components are placed, the user presses the calibration button on the main display and the parameter estimation process begins. This action tells the system to send a picture of the current scene to a central server which returns an inferred emissivity map that approximates the emissivity values at the patch level (process described below). Calibration results are typically better when the scene is well lit with few objects in the scene due to lower image complexity. Once the emissivity map is received, the system starts the rest of the calibration process. Figure 7.3 illustrates how users interact with the 122 Figure 7.3: Calibration Procedure. After receiving the inferred emissivity map a box appears over the  QR code or in the middle of the screen if the QR code was not found (left); by dragging the white  box users can update the position and by dragging the small circle users can resize it.  When the user  accepts the QR code position using on screen buttons (not shown) the distance to the wall and the  position of the high‐emissivity foil is inferred (middle). A second box appears on screen to the left of  the  QR  code  (middle);  the  user  can  move  the  second  white  box  to  select  the  region  they  are  interested in scanning (right). When users select ‘Done’ the calibration is complete.  system to adjust the fitting of the automatically identified QR code—which then automatically infers the location of the high-emissivity foil—and how they specify their region of interest. Calibration only needs to be updated if the system is moved or if the scene changes significantly (e.g., adding furniture). Once calibration is complete, a new menu becomes available that allows the user to specify a data collection schedule and begin their thermographic scans. Using default duration and capture-time parameters derived from literature [6,34,52,99,113–115], a user can launch a 12-hour time-lapse scan with thermograms captured every 15 minutes by providing a unique name for the data collection to the system. During data collection, the system sends data to the server periodically and the server sends back performance and other measurements to the phone’s local database. Once the session ends, users can upload their data to the server for further analysis and access automated reports from a standard web browser. At this time, the motion sensor data is used to filter out data that 123 could be potentially sensitive (e.g., a person walking in the frame) and stored weather data is used to filter out data could negatively influence the accuracy of the temporal analysis (e.g., rain forecast). 7.2.4 Server: Analysis API The backend server communicates with each sensor kit via an API to: run a computer vision module to create an emissivity map, calculate the thermal transmittance of the user-specified region of interest, and prepare data for a web interface where users can access their automatically generated reports. Here, we describe each of these elements in more detail. Emissivity Map. During calibration, the sensor system sends a thermogram to the server and requests an emissivity map; this generally happens once and only needs to be updated if the system is moved or the scene is changed. By extracting the photographic image contained within an incoming thermogram’s metadata, the server can infer the potential material classes in the scene by using the results of a neural network trained to recognize material classes (e.g., wood, glass) and used to classify individual regions in the image from a sliding window. However, constructing a dense map of emissivity values for every pixel in the 480 X 640-pixel image is computational expensive for the server and slows response time; to address this issue an 11x8 patch map of classes and emissivity values is created and returned along with the coordinates of the calibration target. Thermal Transmittance(U-Value/R-Value). Once calibrated, the data collection system sends requests for performance calculations once per minute. The requests contain the unit’s calibration and sensor data along with a thermogram. The server extracts the 124 thermographic data from the thermogram and estimates the thermal reflectivity by averaging thermal measurements collected from the foil side of the calibration target. This measurement of the reflectivity along with the other data in the request is used to correct the temperature data extracted from the region of interest in the thermogram. The thermal transmittance, or U-value, is then calculated based on the sensor data using the formulation put forward by Albatici et al. [6]. To provide users with a greater sense of control over their data, results are returned to be logged in the local database rather than stored remotely. Report Preparation. After a data collection session ends the user can choose to upload their data to the server and generate a report. Users are not required to upload their data and generate a report in case they are concerned that the data collection included private information; however, there is not currently a local substitute for this feature due to the lower processing power and other computational resource requirements not being easily available to the local hardware. Once the server has received the user’s data it runs a series of stored query routines to identify the highest quality data. These routines identify the periods of time where (i) the greatest difference between indoor and outdoor temperatures were observed for selecting the best images to display to the user, (ii) reviews recorded weather to provide warnings of issues that may impact results (e.g., precipitation events), and (iii) filters out data based on detected motion to ensure privacy and accuracy. Based on recommendations from [14,23,83,113], the server selects the best data and computes an average thermal transmittance value for the region of interest given by the user. 125 Figure 7.4: Example infographic report (left); zoomed in example of thermographic analysis (right).  For  each  metric  in  the  report  there  is  an  explanatory  section  that  presents  a  summary  of  the  automated results, suggestions for  interpreting them, and recommendations with  links to source  materials.  In the case of the thermal results, a warning about potential environmental conditions  that could negatively impact results is also presented.  7.2.5 Server: Reporting When a user uploads their data from a sensor system deployed in the field, they can access a list of automatically generated reports (i.e., one report for each successful data collection session) from our custom web portal. Reports are styled as lightly-interactive infographics (Figure 7.4, left) that pair simple visualizations (e.g., a graph of humidity measurements, a thermogram) with automatically generated analysis, recommendations, and other tips based on guidelines from national organizations for health (e.g., CDC [25]), building operation recommendations (e.g., ASHRAE [72]), and regional building code recommendations (e.g., Maryland Energy Administration [102]). The infographic content was designed to reflect the goals of a common residential energy audit and include analyses of building envelope performance, thermal comfort, and health and safety metrics [109]. 126 Automated Report. Our previous system, described in Chapter 6, relied on a highly interactive information visualization tool, but users indicated that it was difficult to extract insights from the temporal data using this tool [104]. Moreover, expert reviews of the system suggested that this approach might be difficult in practice due to high information visualization literacy requirements and likelihood of increased analysis time [33,45]. These factors motivated the decision to create an automated report in the form of a lightly interactive infographic that could summarize a data collection session quickly. The top of the reports offers viewers an at-a-glance overview of the (i) thermographic analysis, (ii) average measurements for the thermal comfort parameters (i.e., indoor temperature and relative humidity), and (iii) average measurements for the air quality parameters (i.e., CO2 and tVOC levels). Based on our early pilot studies, data is color coded uniformly throughout to quickly indicate whether there is an issue (i.e., red) or not (i.e., green or white depending on severity/confidence) and paired with non-threatening descriptors (e.g., “low”, “normal”, “high”) intended to raise awareness rather than alarm. Each metric in the overview is also a clickable hyperlink that rapidly navigates the user to the larger explanatory sections below. Each section below the overview describes one metric and includes two columns: (i) a visualization of the data on the left—typically an interactive line graph that displays more precise information with mouse-overs, and (ii) a textbox on the right that contains concise descriptions of the results, what the user should look for, and how the user should interpret the data. For the thermographic analysis, the average thermal transmittance, or U-Value, is converted to an R-value and compared to regional building codes [102] and this is paired with the “best” image captured during the data collection 127 period; a dropdown menu allows users to specify the type of region being analyzed (e.g., a basement wall) which updates the building code comparison accordingly. Our previous works [109,110] highlighted that one challenge energy auditors encounter in the field, particularly novice auditors, is identifying the severity of issues they find and knowing what to do about them. As a result, when an analysis suggests there might be a potential issue the report offers recommendations for addressing it and provides links to additional information should the user want to know more. The recommendations range from DIY solutions (e.g., hanging clothes to dry indoors to address low indoor humidity issues that impact thermal comfort) to suggesting professional assistance may be required (e.g., to help improve insulation performance). Potential cost saving based on industry averages (rather than direct calculations) are also provided for some recommendations.     Figure 7.5: Alternate displays of the automated online report: small multiples view of all the data  collected  (left)  and  the  compare  snapshots  view  that  enables  viewing  two  different  data  collections simultaneously (right).  128 Display Variants. There are two additional displays, accessible from the primary view, for users who desire further information: a display of all images collected (Figure 7.5, left) and a side-by-side display of two reports (Figure 7.5, right). As described in the previous section, the thermogram presented in the primary display is automatically selected based on temperature and weather criteria. However, as pilot participants wanted to view all of the images captured by the system, users may do so by selecting a small multiples view [143] of all the thermograms collected during the session. This display includes indicators of the average temperature differential for each hour on the right-hand side (i.e., indicating whether or not the differential has gone up or down from the previous hour). In the side-by-side display users can choose to view two reports from different data collection sessions simultaneously, which pilot participants found convenient when comparing different issues and recommendations (Figure 7.5, right). 7.3 Study 1: Technical Evaluation Having described our system in detail, we now describe two key areas of the system that demonstrate its feasibility for field deployments. First, we describe our process for inferring emissivity values. Then, we describe our test deployments. 7.3.1 Emissivity Detection Experiment To assess the feasibility of image classification techniques to assist with setup and calibration of our thermographic sensor system, we conducted a preliminary investigation into adapting pre-trained material recognition models to infer emissivity values that can be used in thermographic calculations from images provided by the on-board, low- resolution photographic camera of the FLIR One. 129 Early Experimenting and Setup. We started with an AlexNet [91] initialized with a model that was pre-trained on the 23 material classes of the MINC dataset (e.g., “Painted”, “Wood”, “Plastic”) [12]. The model was trained using Caffe [86], a deep learning framework, which we used to generate an 11x8 dense classification map to efficiently infer material classes for images received from deployed sensor systems by our backend server. By mapping each material class to an emissivity value, we were able to use the output in thermographic analyses. Our initial tests produced adequate results in controlled data collection settings, but it was unclear how well this approach would work in the field with end-user collected data. Test Dataset. To evaluate how this approach might work in the field, we selected the 571 thermograms captured by participants during the residential auditing mission described in Chapter 4 and extracted the photographic images. From the resulting set of images, we removed those that were too dark to be usable for classification (i.e., because they were captured outside at night or inside with little to no light) and, out of concern for privacy, those that contained people or pets. This left a total of 362 images in the dataset. We then labeled these images using the MINC classes as a codebook for each image patch corresponding to the dense classification map, producing 31,856 labeled image patches. Next, we assigned eight of the participants’ data to our training/evaluation set and the remaining to our test set, roughly corresponding to an 80:20 split. Class Train/Evaluation Test Percent of Dataset Painted 13,017 3,633 52.26% Fabric 2,543 173 8.52% Wood 1,858 248 6.61% Plastic 1,781 199 6.21% Glass 1,388 725 6.63% Table 7.1: Material class distribution after re‐labeling classes with few examples to “Other”.  130 Benchmarking and Classification Improvements. Comparing human labels to inferred labels in the training/evaluation set produced an overall accuracy of 93%; however, this result is misleading as most classes had few examples and the precision and recall metrics were poor. After reviewing the results and considering the distribution of data, we decided to reduce the number of classes to those that had more than 1,000 example image patches in our training/evaluation set, folding all remaining examples into the “Other” class; see final class distribution Table 7.1. Realistically, only the “Painted” class presented a real opportunity for improvement through re-training given the number of examples in the dataset. We ran each image patch through the neural network and extracted the output of ‘fc6’ layer to produce a descriptor containing 4096 features for each patch to train and evaluate on. We trained separate One-versus-All random forest classifiers for each class and applied them to the test via stacking [156]. As expected, the “Painted” class was the only one of the remaining classes to see noticeable improvement with accuracy going from 70% to 76% and improvements to both precision and—notably—recall (Table 7.2), while these same metrics are weaker in the case of classes with few examples. Given that the original pre-trained model’s average accuracy for the “Painted” class was approximately 84%, our results for this class may not improve much more; however, these results suggest that with more examples of the other material classes, we may be able to adapt the pre-trained model to our data—though this would require significantly more data collection in the field using our current approach. 131 Class Precision Recall F1 Accuracy Painted .80 (+.02) .79 (+.23) .80 (+.24) .76 (+.06) Fabric .10 (-.34) .16 (-.18) .12 (-.22) .93 (+.04) Wood .17 (-.07) .24 (+.08) .18 (+.07) .93 (+.02) Plastic .13 (-.08) .41 (+.10) .19 (-.12) .89 (+.02) Glass .26 (+.04) .18 (+.22) .21 (+.22) .85 (-.06) Table 7.2: Material classification results on test set with change from evaluation set.  Summary. Overall, classifying materials using low-resolution, real-world data from our FLIR One thermal cameras remains challenging. However, we were able to learn from those images that classified well and from those that classified poorly. Images that classified poorly tended to be captured in low-lighting, were high-complexity (i.e., containing numerous objects in the scene), and were often captured when participants were holding their phones at extreme angles. Each of these three factors seems to have negatively impacted classification accuracy on a per image basis and even made it difficult for humans to label. Conversely, well-lit, low-complexity images that were photographed perpendicularly to the floor tended to classify well. Moreover, as most of the classes in the MINC dataset are considered high-emissivity (above .9) with low reflectivity properties it is likely that misclassifications would not significantly impact the temporal analyses—unless encountering low-emissivity values like glass or metals [6]. Using these lessons, we updated our sensor system’s deployment procedures and instructions to participants to include suggestions for how to collect data (i.e., avoid the factors that negatively impacted classification and to focus on wall insulation) toward the goal of improving accuracy in the field during the future field studies. 132 Figure 7.6: Deployment setup for the validation experiments. The sensor system in the process of being configured (left) to analyze the area framed by the thermocouples (middle), and destructive testing to verify composisiton of wall assembly (right).  7.3.2 Temporal Analyses Experiment To assess the fitness of our system for the field deployments that follow, we conducted a short validation experiment in a controlled residential environment. Here, we compare results from our system and those calculated by the THM method [14] (i.e., a direct contact sensing method iterated from methods outlined in ISO 9869-1 [85]) and to notional values of a wall specimen. As is often the case in the literature [6,14,34,52,67,114], results from THM and our system need not agree but should be similar and deviations from notional values are expected to be approximately 10 - 15%. Wall Specimen. Measurements were performed in a residential apartment building in the DC metropolitan region of the US on a section of a north-east facing wall (Figure 7.6); due to the building being located within the bounds of a national forest the lower floors are highly shaded and receive little direct sunlight per day. Based on information provided by the building manager, our own observations, and interior destructive testing 133 (Figure 7.6, right), the wall assembly is estimated to be composed of layers of: brick (4.0”, R-0.88), insulate (0.5”, R-3.4), concreate (4”, R-0.60), an air/steel stud cavity (1.0”, R- 0.50), and an interior of finished gypsum board (0.5”, R-0.45). The overall resistance value of this wall assembly is estimated to be R-6.505. Equipment. Surface measurements were conducted using a custom-built data logger connected to two K-type thermocouples6 (Figure 7.6, middle). Thermographic measurements were taken from the interior using our sensor system with an attached second generation FLIR One thermal camera (Figure 7.6, left). Data Collection. Due to an unusually wet winter, the data collection session was started 12 hours after a precipitation event when local weather forecasts predicted clouds and/or overcast weather, a low chance of precipitation events over the following 48-hour period, and low wind speeds (< 8 m/s); sun, precipitation, and high wind speeds were avoided to reduce the potential of these conditions negatively influencing measurement accuracy. Thermocouples were affixed to the interior surfaces of the wall specimen (following procedures outlined in [14]) using painter’s tape. A thermal camera was used to assist with placement of the thermocouples helping to identify thermal bridges and possible surface level anomalies. Following our setup and calibration procedures, our system was placed on a tripod approximately 5m from the interior wall framing the calibration target; the area within the image that contained the thermocouples was input into the system as 5 Ekotrope R-Value Calculator: https://ekotrope.com/r-value-calculator/ 6 Adafruit K-Type Thermocouple: https://www.adafruit.com/product/270 134 the region of interest. The interior temperature was held in a steady state (~23°C) for 24 hours prior to and during data collection with no changes to HVAC set temperatures; the external conditions were subject to regional weather fluctuations. Data was recorded every minute by both measurement systems. Analysis. Using data from the thermocouples and our system, we analyzed the full 48- hour measurement campaign as well as overnight measurements from two 12-hour segments (i.e., 8pm – 8am) following recommendations from [14,23]. While temperature swings were larger over the course of the full 48-hours, the overnight temperatures were within recommended tolerances (e.g., not varying more than +/-5°C). Resistance values based on data from our system were calculated using the formulas and constants provided in Albatici et al. [6]. Results. Our analysis of the data highlights that the deviation from the notional values for the THM method and our system (IRT) are typically low (both around 3%) and tend toward agreement (Table 7.3); the first segment of data while higher, possibly a result of prior weather conditions, still provides a reasonable performance estimate. Additionally, our analysis suggests performance of the wall specimen is below minimum insulation recommendations for the local area (R-13) [102]. Though preliminary, results suggest our system should provide reasonable estimates of insulation performance in the field. Data Segment Notional THM (deviation) IRT (deviation) Average Temp. Delta Overnight 1 R-6.50 R-7.54 (16.00%) R-7.67 (18.00%) 27.47°C Overnight 2 R-6.50 R-6.67 (2.61%) R-6.29 (3.23%) 20.96°C Full Campaign R-6.50 R-6.30 (3.07%) R-6.39 (1.69%) 22.85°C Table  7.3:  Results  from  the  full  48‐hour  temporal  thermographic  measurement  campaign  compared to notional values and THM method, with intermediate calculations from the two 12‐ hour overnight segments demonstrating increased accuracy with longer data collection periods.  135 7.4 Study 2: Residential Homeowner Deployment Study To investigate actual end-user usage and perceptions of our temporal thermography sensor system, we conducted week-long, in-home field studies (modeled after our previous studies and pilot experiments [106,110]) with five participants in five different households during the early spring months of 2018. Each participant was provided with a study kit, which included: a FLIR One thermal camera attachment for their personal smartphone, one of our temporal thermography sensor systems (with a second FLIR One thermal camera pre-attached for convenience), calibration targets, painter’s tape, and a tripod. To guide their auditing activities, participants were asked to complete two thermographic “missions” (following the prompting method in [129]), the first mission asked participants to use the smartphone attachment and the second mission asked participants to use the sensor system (including setup and calibration). Between each activity participants completed an online questionnaire that asked about their experience and tracked changes in their attitudes. At the end of the week, participants were briefed via a semi-structured interview and were compensated $60. Approximately 45 days after completing the debrief interview, participants completed a final online questionnaire to determine lasting perceptions and whether any actions were taken. ID AGE Gender Location Home Type Education Profession P1 30 Male Maryland Single-family Bachelor’s Degree Music Licensing P2 41 Female Maryland Single-family Doctorate Professor P3 53 Male Maryland Single-family Bachelor’s Degree IT Professional P4 60 Male Washington, D.C. Single-family Master’s Degree Attorney P5 40 Prefer not to answer Virginia Low-rise condominium Doctorate Software Developer Table  7.4:  Participant  demographic  information  for  the  field  study  deployment  of  the  temporal  thermographic sensor system with homeowners.  136 7.4.1 Method Participants. We recruited 5 participants (3 male, 1 female, 1 preferred not to answer) from different households, who were on average 44.8 years old (SD=11.78, Mdn=41), using local mailing lists, list-serv, and social media (Table 7.4). Our recruitment ad specified that we were interested in studying new thermographic technologies for DIY energy auditing in residential homes with homeowners. We used an eligibility questionnaire to screen for home-owning adults (age 18+) with compatible smartphones who lived in the DC metropolitan area. Participants were enrolled on a first-come, first- served basis. To collect demographic information and attitudes toward environmental sustainability, participants completed a short, pre-study questionnaire. One participant was a university professor, two worked in the information technology field, and two worked in legal affairs. Formal education was high: two had doctoral degrees, two had bachelor’s degrees, and one had a master’s degree. Similar to our previous novice study [110], our participants were eco-conscious. On a 7-point Likert scale ordered very unconcerned (1) to very concerned (7), participants reported being concerned about climate change (M=6.40, SD=0.80, Mdn=7). Three had never engaged in energy auditing activities, one performed DIY energy audits bi-annually, and the last conducted monthly reviews of utility bills. Participants that audited more regularly made seasonal weatherization improvements (e.g. caulking air leaks), while those that did not engage in energy auditing activities cited uncertainty of how to begin or cost barriers. One participant had previously had a professional energy audit performed on their home. Two participants reported using a thermal camera previously, though not in connection to energy auditing activities. 137 Outside of smart thermostats, ownership of home automation and data collection devices was limited. Three participants used a smart thermostat to regulate heating and cooling (e.g., NEST) in their homes. One participant, in addition to engaging with NEST reports and utility bills, also tracked environmental metrics with common sensors (e.g., a thermostat displaying current indoor temperature and humidity) and kept some manual logs of this data. The remaining participants did not use any sensing technologies, nor did they track data about their homes outside of occasionally reviewing utility bills. Deployment Sites. Part of the demographic questionnaire asked about the deployment sites themselves (i.e., the participants’ homes), which were typical of those constructed in the DC metropolitan area. Three participants owned homes in suburban areas of Maryland, while the remaining two owned homes in urban areas of the District of Columbia and Virginia, respectively. With respect to evaluating insulation performance, regional building codes and recommendations are similar in these areas (e.g., recommendations for wall insulation being between R-13 and R-20 [102]). Participants’ homes were, on average, 54.2 years old (SD=21.63, Mdn=56) and they had owned their homes for the past 11 years (SD=8.60, Mdn=11). Three were single-family, wood and timber-framed homes with cavity insulation and finished drywall interiors; the last two were low-rise and high-rise condominiums which were similar in construction to the single-family homes, but steel framed with some areas of brick facing. Procedure. We held introductory briefings in the participant’s home at a time that was both convenient and that coincided with a week where weather conditions were predicted to be acceptable for thermographic scanning (e.g., low likelihood of precipitation). Upon 138 arrival, a researcher discussed the study plan, obtained consent, provided the participant with a study kit, and reviewed a set of training documents for both the thermal camera and the longitudinal sensor system (see Appendices). These documents were created by a research team member with a professional thermography certification and drew upon documentation from thermographic smartphone applications [48], how-to guides from manufacturers [163], and DOE materials [146,147]. After the introductory meeting, participants were encouraged to explore with their provided smartphone thermal cameras and to build familiarity before beginning to collect data for the study. To help structure and motivate data collection, we provided participants with two energy themed missions via email:  Mission One: Investigate your home with the thermal camera attachment for signs of energy inefficiencies and collect at least 25 photos.  Mission Two: Use the temporal sensor system and collect information about at least two areas that you are curious about and review this data online. After each mission, participants completed an online questionnaire covering topics such as their ability to locate issues, procedures, and attitudes toward the activities. At the end of the week, participants completed an in-person, semi-structured interview. Participants described their experiences with home maintenance and energy auditing prior to the study, reviewed the data that they collected during the study, and then discussed their perceptions of in-home sensing and any barriers to making changes. After completing the survey, participants were compensated $60 for their participation. Approximately 45 days later, participants completed a follow-up survey to determine lasting perceptions and whether any renovations were performed. 139 Data and Analysis. We calculated counts and descriptive statistics for the survey data and qualitatively coded the interviews; data from the activities was reviewed by a research team member with a professional thermography certification. Online Surveys. The surveys each took approximately 8 minutes to complete. They asked participants to review a recent utility bill alongside the data they collected (i.e., their photos and the automated reports generated by our system, respectively) and to report on various aspects of their experience. Additionally, the surveys covered: (i) procedural details such as the date and duration of their audit activities, (ii) a description of what participants found during their assessment activities and what recommendations, if any, they might have to improve building performance, and (iii) a series of Likert-scale questions about their experience, attitudes, and behaviors. The second activity survey also included a few open-ended questions that asked participants to briefly compare the two activities (smartphone vs. sensor kit). Debrief Interviews. The semi-structured interview sessions lasted an average of 54 minutes (SD=8.3). Interviews were audio recorded and professionally transcribed. Transcripts were analyzed through an iterative coding method using both inductive and deductive codes [19,78]. The initial codebook was based a codebook used in our prior study [110] and contained 12 codes grouped into three categories: experiential, design ideas & challenges, and broader impact; it was expanded to include codes for likes and dislikes for a total of 14 codes. Two researchers independently coded a randomly selected transcript. The unit of analysis was the response to a single question. Cohen’s Kappa (κ) was used to measure inter-rater reliability (IRR). IRR on the transcript was 140 κ=0.85 (SD=0.11) with codes ranging from strong to near perfect agreement [151]. Having achieved IRR, a single researcher coded the remaining transcripts. The final codebook is included in the Appendices. Follow-up Surveys. The follow-up survey took participants approximately 4 minutes to complete and asked a series of Likert-scale questions about their experience, attitudes, and behaviors; it was designed to ascertain any long-term impacts from participation. 7.4.2 Mission One Findings: Using Thermography Smartphone Attachments In mission one, participants used thermal camera smartphone attachments to inspect their homes. Here, we present an overview of the participant’s actions followed by their personal recommendations, their confidence in their recommendations, and their post- activity attitudes toward smartphone-based thermography. We report means (M), standard deviation (SD), and medians (Mdn) as appropriate. Participant quotes are attributed using a ‘N’ for novice, followed by an ‘S’ for survey response or ‘I’ for interview response, followed by their identification number (e.g., NS1). Overview of Activities. The five participants spent an average of 23 minutes (SD=5.7, Mdn=25) completing this mission. Survey responses reported that all participants looked for air leakages around windows and doors and attempted to discern problems with 141     Figure  7.7:  Data  from  homeowner  inspections  using  smartphone  attachments  in  mission  one.  Participants described inspecting common living areas (e.g., home office, left), looking for evidence  of air leakages (middle), and looking for evidence of insulation issues (right).  insulation by looking for strong differences in thermal signatures; two further reported actively looking for moisture damage. As NS4 wrote, “I was looking for anything out of the ordinary - places where cold might be getting in other than windows, like surrounding the windows, or irregularities in insulation patterns.” In the debrief interviews, participants reemphasized how this mission introduced them to the utility of thermal cameras in general and quickly enabled them to detect/inspect areas with potential issues that would later be subject to further analysis: “I found problems in [my home] office, which is where I did the scans with the sensor device. I knew it would be bad as it was formerly a sunporch that the previous owners had poorly refinished and I always wondered what it would look like with a thermal camera” (NI2; Figure 7.7, left). Participant Recommendations for Repairs. All five participants found evidence of air leakage and/or insulation issues in their surveys (Figure 7.7). For example, NS6 wrote: “Doors leak cold at the bottom more than other areas, some outlets appear to not be insulated, possible variation in insulation in the bathroom.” Two survey participants 142 uncovered phantom energy issues (i.e., devices consuming power while not being used). Most participants (4) suggested DIY fixes for issues they uncovered, such as the two participants who suggested resealing areas where they observed air leakages. In contrast, solutions for insulation issues were non-specific. Two participants described generally trying to “find a way” (NS3) to deal with these issues while one mentioned wanting to review their data with a professional. After reviewing their data, participants reported only being somewhat likely (M=5.40, SD=0.49, Mdn=5) to act on their recommendations. Confidence in Personal Assessments. When surveyed about confidence in their assessments on a 7-pt Likert scale (rated very unconfident to very confident), participants were only somewhat confident (M=5, SD=0.89, Mdn=5). More confident participants used thermal imagery for confirmatory purposes, such as NS4 who wrote, “I have the thermal readings to support my assertions.” Similar to previous studies [110], less confident participants noted it was challenging to determine if a photo revealed an actual issue and what the impact of fixing it might be. As NS2 wrote: “There are some very cold spots in the office, but it’s hard to tell if they are just because it's unheated or that there's some big gaps in the insulation.” Two participants reasserted their difficulty with interpreting thermograms during this mission in the subsequent interviews, such as NI5: “I don't think they were very interpretable on their own. The reticle with the temperature reading I think was particularly difficult to make sense of. So, I just ignored it and tried to frame the shot. I don't think I would say I had a thorough understanding, having used [the FLIR One], of what's happening in each of the pictures, although it did give me some questions to ask if I was consulting with an expert.” (NI5) 143 Post-Mission Attitudes Toward Smartphone-based Thermography. Participants found using the thermal camera to be easy (M=6.20, SD=1.17, Mdn=7), but similar to prior work [109,110], P5 notes that “taking photos was easy, but reading them and knowing what I am seeing is not as easy.” Two survey participants reported minor technical issues with connecting the camera to their phones and another mentioned that it was challenging to find times when the weather was suitable for thermographic scanning. All survey participants agreed or strongly agreed that the thermal camera was useful in terms of learning about their home (Mn=6.40, SD=0.80, Mdn=7) and agreed that the thermal camera was helpful in determining whether problems exist (Mn=5.80, SD=0.75, Mdn=6). Most (4) somewhat agreed that their thermal imagery was easy to interpret (Mn=5.60, SD=1.02, Mdn=6) and could be used to evaluate the need for improvements (Mn=5.80, SD=1.17, Mdn=6). Most (4) agreed that using the thermal camera had increased their interest in energy auditing in the home (M=6.40, SD=0.80, Mdn=7). 7.4.3 Mission Two Findings: Using the Temporal Thermography Sensor System In this mission, participants used our temporal sensor system to further investigate their homes. As before, we present an overview of the participant’s actions and their inspection results, followed by findings on issue discovery, the interactive report infographic, data privacy, participants’ personal confidence in their assessment activities, and their post- activity attitudes toward conducting temporal thermography with the sensor system. 144       Figure 7.8: Data from a homeowner single homeowner deployment using temporal thermography  sensor system in mission two. Data collection began at night (left two images; note that in the dark,  the RGB image can not provide contextual outlines to the thermal imagery) through early morning  (right  two  images);  fluctuations  in the top‐right corner reveals a potential  insulation  issue being  monitored by the system.   Overview of Activities. Participants all completed the two required 12-hour deployments of the system and spent an additional 16 minutes (SD=1.87, Mdn=15) reviewing their data via the online automated report. All survey participants reported that the sensor system helped them learn about and assess insulation performance. Participants that had previously noticed insulation issues (3) in mission 1 used the thermal sensor systems to test some of these areas, while the others chose to measure exterior wall insulation performance in primary living areas (e.g., dining room, office). In their interviews, all five participants offered positive sentiments about the data the system provided, particularly the holistic picture of their household provided by the summative report at the end of their data collection. As NI2 described, “Yeah. It kind of gave me a why, right? So, it's real cold here and this is below code. Here's some further information you can look at. That was super helpful, right? I can be like, I agree that this is a problem, and now it’s telling me something I can do" (NI2). 145 During the interviews, four participants described feeling a sense of engagement through the process of collecting and analyzing the longitudinal data; however, all participants reported desiring opportunities for the sensor system to offer more household coverage and, as a result, more data. Issue Discovery. While the thermal camera attachment was primarily described as being preferred to discover regions of interest (ROIs) rapidly, the sensor system was considered useful in determining whether or not interesting regions were, in fact, areas that contained issues and participants like this information was presented alongside other important environmental metrics (Table 7.5). As NI1 described, “I thought it was interesting that it tells you the wall insulation and the humidity, because we thought our humidity was on the lower side because we both get really dry, so we installed a whole house humidifier and it was good to know that it was, you know, good.” (NI1) One participant performed a general inspection of their home’s insulation with no specific ROIs previously found, and confirmed their home was performing efficiently. Three participants aimed the sensor kit at a suspected issue—two who identified the ROI with the thermal camera, one who was looking into performance claims made by their homeowner’s association about an insulation project. Of these participants, one discovered an issue that was severe, one discovered an issue that was less severe than anticipated, and the last was surprised to discover that there was no issue where one was expected. Additionally, NI4 uncovered an unforeseen insulation issue—not discernable with the thermal camera attachment alone—writing in their survey that “the R value is lower than I would've thought, especially in the living room which was upgraded 146 Participant ID Sensor Kit Aimed at Suspected Issue Issue was Found P1 No No P2 Yes Yes Less severe than anticipated P3 Yes Yes P4 No Yes P5 Yes No Based on intuition, not thermal camera mission Table  7.5.  Participants’  use  of  the  sensor  kit  to  analyze  and  uncover  issues  in  their  first  capture  sessions. Results of using the temporal sensor system conflicted with three participants expectations.  approximately 10 years ago.” As highlighted in these last three examples, temporal thermographic analysis conflicted with the participants’ expectations about insulation performance and surfaced unanticipated information. In addition to being able to perform insulation assessments, all survey participants found that having the additional data from the sensor system was interesting and potentially useful—particularly the air quality data. However, even given the confirmation of issues within their homes, homeowners may be reluctant to act on the data: “I'd say it's kind of too late once you get [the data] for the homeowner, unless you're about to do a renovation.” (NI4) Interactive Report Infographic. Four participants were positive about receiving the easy- to-read, automatically generated report across the surveys and interviews. As a direct result of the temporal data collection and analysis, all participants obtained new insights not revealed by their smartphone-based thermal camera use in mission one. The data in the report also helped participants learn about relevant building codes, thermal comfort, and air quality standards. As NI1 described “I learned what good levels for these [metrics] were, so that was helpful.” Most participants (4) liked the longitudinal data and additional depth the report provided in comparison to the thermograms they had previously collected. As NI3 summarized: 147 “I like the idea of having a report that I can refer to again afterward. You get that with pictures too, obviously. But the reporting aspect gives you more detail, […] the fact that you had the environmental and air quality readings gave you something more to look at.” (NI3) In contrast, NS5 thought the report lacked depth and utility, writing in his survey that “my reports were negative, I am not sure what else to glean from them.” Three interview participants envisioned using this data as a tool to communicate with professionals, as they thought they would be more prepared for discussing what updates may need to be made to their home. NI2, for example, appreciated having a report from personally collected data, as they didn’t trust professionals to be honest about the severity of issues: “If there's a big problem, that's the thing I want to fix, but I don't trust that some guy is coming in and not trying to sell me.” However, interview participants also desired more capabilities with regard to the report. All participants mentioned that evaluating temperature and humidity data was more nuanced than the system allowed. The system focused on thermal comfort (e.g., measurements staying within a certain range), but participants deliberately lowered temperatures at night to save on energy costs causing their overnight scans to suggest low thermal comfort in the home. Three participants wanted to customize the report to hide sensitive or personal data (e.g., before sharing with professionals, to remove any potentially embarrassing photographs the motion sensor may not have detected). While three participants appreciated the ability to visit links for more information, others felt it lacked depth—which was the reason one participant perceived the report more negatively: 148 “the time series weren't all that informative and it was unclear how to interpret them, the text summaries were more helpful, but I’d prefer it if I had a specific part list… and a better way to tag and compare things spatially.” (NI5) Data Privacy. Interviews indicated that deploying the in-home sensor systems brought up concerns over data privacy with four participants. These participants were okay with deploying the sensor system in their households so long as they had explicit control over the collected data and it wasn’t sent to external entities or corporations. NI2 summarized, “If it were not an internet connected thing, if it were just a local network thing that I use in my house, that would be fine, right? If information is going out, then I have a big problem with technology like that.” (NI2) While these participants indicated that the motion sensor helped to partially alleviate concerns about potentially embarrassing thermographic (and photographic) data being collected, they did not fully trust that such sensor-based filtering would be full proof. In stark contrast to these perspectives, NI4 described wanting to openly share data, compare their household to their neighborhood, and provide access to local policy makers. Personal Confidence in Assessment. Most participants (4) indicated that using the sensor system lent additional confidence to the earlier assessments from mission 1. One participant, NI3, noted she was not surprised by her results because she felt the issue was clear from the earlier thermal photos, but on reviewing her results she wrote “I didn't realize this area was so poor.” Most participants (4), however, remained only somewhat confident (M=5.00, SD=0.63, Mdn=5) that they would implement their recommendations. Participants with reports indicating there was an issue (3) tended to 149 be slightly more confident, like NS4 who wrote: “We have good information now, it will be a matter of cost/benefit/comfort analysis.” Conversely, participants with reports indicating there were no issues (2) were more neutral. As NS2 explained regarding his confidence score, “I have no recommendations.” Post-Mission Attitudes Toward Temporal Thermography with the Sensor System. Most participants (4) agreed or strongly agreed that the sensor system had helped them learn about their homes, one somewhat agreed (M=6.2, SD=0.75, Mdn=6). Two participants noted that the thermal comfort recommendations needed to be updated to better reflect their household schedule (e.g., lowering indoor temperatures in the evening for energy savings not impacting thermal comfort). Most (4) agreed or strongly agreed that the system was helpful to determining whether a problem existed, one neither agreed nor disagreed (M=5.60, SD=1.02, Mdn=6). Most (4) somewhat agreed that their collected data was easy to understand (M=5.80, SD=0.75, Mdn=6) and could be used to evaluate the need for improvements (M=5.80, SD=1.17, Mdn=6). Most (4) agreed that using the activity had increased their interest in auditing their homes (M=6.20, SD=0.75, Mdn=6). While positive about the temporal sensor system overall, most (4) participants nevertheless noted a software or hardware issue during the activity in their interviews. Participants found the sensor system was only somewhat easy (Mn=5.00, SD=1.41, Mdn=5) to use and indicated the increased difficulty was a result of setup being “a bit tricky” (P3) and, in particular, that booting the system up and waiting for the camera to connect was an issue. Lack of control over collection time was also reported as a frustration during interviews: the long data collection time was viewed as problematic by two participants, whereas one participant wanted to record data for longer consecutive 150 periods of time (though this is a limitation of the study and analysis procedure and not the system itself). Two participants also noted that the strength of their home Wi-Fi networks prevented them from being able to deploy the sensor where they would like. For example, NI4 stated “I originally was going to do it in the basement and look at that basement corner I identified with the thermal camera, and the Wi-Fi signal was just not strong enough. It was cutting out.” 7.4.4 Follow-up Survey Findings Approximately 45 days after the debrief interviews, we asked participants to complete a brief survey about whether or not they had taken any actions based on the data they had collected and if there was any lasting impact from their participation. Acting on Recommendations. Two participants reported acting on their recommendations for adding additional air sealing to window and door areas. One participant, who had not implemented recommendations, reported needing to wait for funds to be available to address the issues they found. The remaining two participants reported making no changes due to it being a low priority, but both wrote encouraging statements about their future intent. As NS2 explained: “It didn't seem super critical. However, I found some water damage on the outside lumber for the room with issues and if that requires some serious repair, I'll definitely incorporate some of the recommendations when we do that process.” (NS2) 151 Attitudes. All participants reported thinking more about energy efficiency issues in their home since their participation in the study had ended. As NS3, summarized “It has made me generally more aware of where there might be issues and why.” All participants reported thinking more often about insulation performance and air leakage issues. Most (4) reported thinking more often about thermal comfort issues in their homes. Two participants reported thinking more often about air quality issues. Finally, one participant reported an increased interest in looking into professional services to address issues that were uncovered. 7.4.5 Summary of Study 2 Findings Similar to our previous studies [110], participants investigated missing insulation and air leakages issues. However, in this study participants were provided with additional information about the severity of insulation issues through temporal thermography while also learning about building codes and other topics commonly associated with energy audits (e.g., air quality). As a result, participants concern about their use of thermography centered more on barriers to making changes than on whether or not change was necessary. Results from our following up with participants after their participation had ended suggest that there may be some lasting impact such as an increased awareness of the potential issues covered by the activities and increased knowledge of their potential causes; however, this did not necessarily translate into immediate action. Encouragingly, participants indicated that they may be more inclined to take actions in the future if cost barriers were removed or other renovations were being planned. 152 7.5 Study 3: Professional Energy Auditor Design Study To investigate integrating our sensor system into professional energy auditing and modeling activities, we conducted a two-part study (modeled after our previous studies [109]) with 5 professional energy auditors. Participants completed a semi-structured interview followed by a presentation of three design probes based on our technology. Participants were compensated $40. 7.5.1. Design Probe Descriptions The three design probes offered different scenarios using two different mediums: two written narrative scenarios (~250 words) of increasing complexity that described sensor networks (composed of nodes similar to our sensor system) being deployed at residential and urban scales, respectively, and an interactive demonstration with the sensor system itself (i.e., setup, calibration, etc.) including, with permission from previous participants, a review of some of the household data collected from Study 2. The written design probes used 2nd-person narration to help participants envision the scenarios and were designed to provoke discussion, ground conversation, and elicit feedback while the demonstration was intended to serve as a critical review of the previous study. The full probes are included in the Appendices and summarized below. Years of Experience ID Age Gender Sector Auditing With Thermography P1 35 Male Private 6 2 P2 29 Male Private 7 3 P3 28 Male Private 5 5 P4 32 Male Private 7 5 P5 49 Male Private 6 6 Table 7.6: Professional energy auditor participa1n5t 3d emographic information.  Scenario 1 (Text): Residential-scale Audit. The first text probe described a residential audit where a sensor network had been pre-installed prior to the auditor’s arrival. You have just arrived at a site to perform a residential energy audit. You proceed to greet the client, discuss the building in question, and assess the home. As part of this assessment, you download data and automated reports from the home’s performance monitoring system to your smartphone or tablet. The reports provide an overview of the home’s data in real-time, allow you to filter data by room, and view this across the lifetime of the home since the technology was installed. The data includes (i) inferred occupancy schedules, (ii) indoor climate measurements, (iii) thermographic analysis of the envelope, areas of potential water damage, and air leakages, (iii) air quality information, and (iv) local weather; photos and thermograms are also available. The client is familiar with the data and is looking for your recommendations to resolve comfort issues and improve energy efficiency. Scenario 2 (Live Demonstration): Multiple-Residential Audits. The second design probe walked participants through settings up the sensor system, described participant experiences in the study, and reviewed data from actual residential homes. Scenario 3 (Text): Urban-scale Audits. The second text probe described an urban scale audit where thermographic sensor networks were common in the built environment. You are asked to report on the energy efficiency of a large urban center with towering skyscrapers, metropolitan buildings, and a myriad of other constructions. You begin by downloading the raw data (i.e., utility usage, high definition photos, thermography data, etc.) and automated reports for the buildings in this urban area by accessing the remote network of sensors. These sensors are typically installed in new buildings at the time of construction, but others can be temporarily deployed as necessary. Like the previous scenario, the network continually monitors performance and degradation at both the individual building and neighborhood levels. A custom, interactive software interface allows you to review recently flagged anomalies along with historical data, which allows you to draft a report for stakeholders (e.g., property owners, green building agencies). 154 In summary, the text probes describe two possible scenarios that may be enabled by our system and the live demo demonstrates the feasibility of such systems. Each scenario built on the previous and emphasized a different way in which energy auditors may interact with stakeholders, described new data collection and analysis methods not widely available in the field, and ask participants to think about how integrating such system may impact energy auditing and modeling in the future. 7.5.2 Method Participants. We recruited five professional energy auditors (all male) in the DC metropolitan area through email lists, word-of-mouth, and social media posts. Our recruitment materials specified that participants needed professional experience using thermal cameras for building energy audits. Our participants ranged in age (M=34.6 years old; SD=8.5), audit experience (M=6.2 years; SD=0.8), and experience with thermography (M=4.2 years, SD=1.6); all were employed in the private sector (Table 7.6). All participants had received on-the-job training through company sponsored programs or workshops. Procedure. Each session lasted an average of 103 minutes (SD=26.3) and included a semi-structured interview and presentation of the three design probes. The semi- structured interview approach allows us to dynamically pursue themes we had not identified a priori. All participants were asked a similar set of questions, but new topics emerged in accordance with participant’s background, skills, and experience. The design probes immediately followed the interviews. Participants were asked to “think aloud” and evaluate each scenario or presentation. Our objective was to identify aspects of the design 155 probes that participants were interested in, uncover concerns, and identify how such technology might impact professional practices. Data and Analysis. The sessions were audio recorded, transcribed, and coded for themes. Similar to the previous study, we pursued an iterative analysis approach using a mixture of inductive and deductive codes [19,78]. We created two codebooks—one for each part of the study—which were initially derived from our codebooks used in our previous study [109], research literature, our study protocol, and post-interview discussions amongst the research team (see Appendices). For the semi-structured interviews, the 10 codes included views on thermography (e.g., procedures, automation), impact (e.g., uses, benefits, findings), and challenges (e.g., application, clients, interpretation). Two researchers independently coded a randomly selected transcript. The unit of analysis was the response to a single question. IRR on the transcript was κ=0.85 (SD=0.13) with codes ranging from strong to near perfect agreement [151]. The remaining transcripts were then coded by a single researcher. For the design probes, the 10 codes included: interests (e.g., automation, data, features), concerns (e.g., technical feasibility, data quality), and reactions to scenarios (e.g., positive, negative). Again, two researchers independently coded a randomly selected transcript. The unit of analysis was the response to a single question. IRR on the transcript was κ=0.89 (SD=0.13) with codes ranging from strong to near perfect agreement [151]. The remaining transcripts were then coded by a single researcher. 156 7.5.3 Interview Findings We discuss our interview findings with regard to the utilization of thermography by professional energy auditors, barriers to utilizing thermography during inspections, perceptions of potential new data sources (automated thermography and smart home data), and perceptions of homeowners completing DIY thermographic energy audits. Participant quotes are attributed using a ‘P’ for professionals followed by their identification number (e.g., P1). Utilization of Thermography by Professional Energy Auditors. All participants (5) considered thermography to be useful as a diagnostic tool, but felt it was especially useful as a tool for communicating with clients. Two participants, in fact, described encountering the same situation: After showing clients their readings from a blower door test during an audit that indicated air leakage issues, clients were hesitant to believe the results until they were shown a thermogram illustrating the problem. As P4 described: “It's good to convince people that what you're telling them is true. Because they're not going to believe, ‘Hey, I ran the blower door test and you're this leaky.’ They want to know where and the number just means the whole house is leaking somewhere.” (P4) These participants appreciated thermography’s ability to raise public awareness of energy issues and motivate change within homes because most (4) had chosen their career paths due to passions for energy efficiency, sustainability, and being eco-conscious. All participants were confident in the determinations they are able to make based off thermal imagery regarding air leakage and insulation issues. Moreover, three 157 participants pointed out that another utility of thermography is quality assurance, by visualizing changes in structure performance before and after performing renovations. Barriers to Utilizing Thermography During Inspections. Three participants suggested that they don’t get to use thermography as much as they would like or should. This resource was underutilized due to not having enough time on the job site and uncooperative weather. With regard to the aforementioned possibility of pre/post quality assurance scans, not having enough time on the job site, where even home access can be a time-consuming challenge (e.g., in multi-unit apartment buildings), was considered the primary issue. Additionally, two participants described challenges interpreting thermographic data with regard to detecting moisture issues. This starkly contrasted their confidence in their ability to use thermography to detect air leakage or insulation issues. Both participants described scenarios where they had thought they found moisture issues within a home but weren’t confident enough to report it. They felt they needed more training before making such an assertion to clients. As P1 described: “I don’t feel as comfortable diagnosing ... now, if it’s really obvious what it is then maybe, but if it’s a questionable moisture issue, personally, I am not as comfortable with diagnosing that. Just lack of training maybe.” (P1) Considering New Data Sources: Automated Thermography and Smart Home Data. All five participants thought having smart-home data that described household environmental and performance conditions would be valuable (e.g., thermostat data). 158 “Temperature – probably indoor and outdoor temperature. How often your unit is turning on and off. What it's being set to – so, homeowner behavior. It's like trusting an eyewitness, right? You can’t trust necessarily how accurate a homeowner's going to know their own behavior. So being able to see well, yeah, you've got all this condensation everywhere because you're setting it to 60 degrees at night, or your bills have been going up or down – or whatever. I'd very much like to have all that data, to be able to mess with it, analyze it, and see what's really affecting things.” (P3) Even so, one participant offered a caution: “[these are] data points and it comes down to the creativity of how you can use and apply that data to achieve a goal” (P5) and was not confident that more data would provide new insights. When interviewed about current initiatives within automated, large-scale thermography—specifically, using UAVs to collect rooftop imagery (e.g., [107,161]) and cars to collect images of the front-facades of buildings (e.g., [103])—four participants were interested in the prospects of these approaches. However, as with our previous work [109], all participants expressed the same concerns: getting enough building coverage (i.e., would thermal images from a drive-by front façade or roof image to be enough to make inferences?), questioning the ways the data was collected, and questioning how the data would be analyzed. It was because of these concerns that the fifth participant was doubtful about these large-scale, automated thermography approaches. Perceptions of Homeowners Completing DIY Thermographic Energy Audits. When prompted with the idea that homeowners could do DIY audits with thermal cameras, generate their own reports, and approach auditors with these, all participants were 159 receptive. Most (4) participants thought a report from the homeowner could potentially address two challenges they face. First, that scheduling audits throughout the day and the associated preparations are time-consuming and having more information up-front would help with knowing which areas to investigate first or which areas may need more attention. Second, that thermography data may be helpful in calibrating energy models. Despite these positive potential outcomes, 2 participants were simultaneously concerned that a homeowner’s use of thermal cameras for DIY energy auditing may lead them to focus on the wrong things (e.g., replacing windows, which may have a negligible impact on energy use). As summarized by P5: “In the sense that [thermal images] raise awareness, I think it’s good. A key hurdle to all energy efficiency programs is people being aware, if they don't care then you have a more difficult battle. But, people may misinterpret their thermal images and then be led down the wrong path if their home is better off than it appears or if there may be better solutions to problems than they’re aware of.” (P5) One participant reflected that building owners who are interested in improving energy efficiency or taking part in sustainability initiatives (e.g., such as installing solar panels) often don’t know what is involved and may become discouraged when they find out. 7.5.4 Design Probe Findings We begin this section with an overview of the energy auditor’s reactions to the design probes followed by an in-depth description of reactions to the individual design probes. 160 Overview of Reactions to the Design Probes. Overall, the first two of our design probes elicited positive reactions while the third, on urban-scale deployments, was viewed less favorably due to data overload concerns (similar to [109]) and it being outside typical audit practices. Major concerns across the design probes involved the appropriate placement of the sensor system(s) in residential buildings and validation of the measurements, but all participants were open to the idea of having access to a potentially helpful new source of data and the possibility of new client interactions. Design Probe 1 Findings. Most participants (4) reacted positively to the first design scenario, which depicted a built-in, multi-room, continuous, home-sensing system. These participants described how such a system would enable a number of new services and practices, such as remote auditing, quality assurance of retrofits, pre-screening locations, and making it easier to plan daily service routes ahead of time. All thought it would encourage building owners to reach out about services—be it to energy auditors or directly to contractors. Two participants described how such a system could also improve current practices: clients commonly are influenced by a social desirability bias—exaggerating their home maintenance practices (e.g., changing air filters on HVAC systems regularly)—and such an in-home system may offer more reliable data than homeowners themselves. Finally, one participant suggested that such systems may be beneficial in insurance claims. Participants also brought up a number of concerns relating to such a system. The primary concern, described by all participants, was the coverage areas and sensor placement. Unlike our novice participants who desired coverage by room, professionals also included crawl spaces and other uncommonly accessed points that are part of auditing procedures which homeowners may not normally consider as areas for 161 placement/installation. Even assuming good coverage, there remained a concern among participants (3) over whether the system would be installed correctly (e.g., installing sensors too close to a combustion source could result in inaccurate air quality measurements) and, therefore, whether the data would be reliable. Thereafter, participants (2) expressed concerns about the volume of data being collected, and how to make it useful. All five participants rejected the idea of offering a service where they would drop off and potentially deploy the equipment as a solution to these challenges due to the time it would require making multiple trips to the site. Design Probe 2 Findings. Every participant (5) was positive about the deployable sensor kit and accompanying automatically generated report they were presented with in the second design probe. With regard to the data the sensor kit offered, none were surprised that the participants in the novice deployments were interested in indoor air quality measurements: from the auditors’ personal experiences, many people are interested in these data when its presented, despite it not being strongly tied to energy efficiency program goals. Two participants, in fact, suggested the addition of a carbon monoxide sensor. Additionally, participants appreciated that a thermographic scan could determine the R-value of a wall, as—again—this could help calibrate their energy models (i.e., highlighting potential errors). One participant particularly liked being able to get an R- value without needing to know the wall assembly, as accurate information is not always available and few homeowners are comfortable with destructive testing. Despite the potential value of the sensor kit’s data, all participants also voiced concerns over data privacy and how to prevent unauthorized access to homeowners’ data. 162 Regarding the homeowner report examples, all (5) believed it could be useful for raising awareness of issues and the impact of environmental factors (e.g., humidity issues). Two participants asked about the language the report used to rate sensor readings (e.g., low/high vs. safe/unsafe), describing how they face similar challenges in their own reporting with regard to how to avoid scaring or potentially misleading clients. Similarly, one participant discussed how the report related to a broader issue in the field: reports don’t lead to action. The participant did not believe this system would resolve that issue. Participants offered suggestions to improve and expand the system toward the goals of obtaining more accurate data and improving its usability. After reviewing the example reports, one participant suggested the system implement more automation to scaffold homeowners on how to select regions of interest. The regions were too broad, and tighter selection of effected areas would improve the accuracy of the report. A second participant wanted to be able to set building codes for older buildings—particularly historical buildings—which would be unlikely to meet current building codes (the participant noted that this is also an issue in current modeling software). To make the report output more useable by professional auditors, two participants suggested it should feature an advanced “auditor view” containing direct access to raw data and options to export this data. Design Probe 3 Findings. Three participants were negative about the third design probe, which described sensor systems like ours being deployed at an urban scale. Their primary considerations were the generation of too much data and how such a program would not fit into current practice. As P1 described: 163 “This is operating at a different level than I’m used to dealing with, but its more or less replicating what the system does on an individual level, so I would say you might have some of the similar challenges only magnified.” (P1) While these participants did not believe such massive quantities of data would be necessarily helpful for energy auditors, two of these participants did propose that it could be helpful for policy makers. The remaining two participants were more neutral about the scenario, equating it to an eventuality of buildings having built-in smart technologies by default. Still, these participants described how such an “urban auditing” program would require new practices and procedures and were not sure what such a program would look like. 7.6 Discussion In this chapter, we presented three studies: a technical evaluation of a temporal thermography sensor system that logs environmental data and performs non-destructive insulation testing, live deployments of this system with homeowners, and semi-structured interviews featuring design probes with professional energy auditors regarding automated approaches to thermography—including those based on our sensor system. Here, we synthesize these findings with regard to (i) outlooks of homeowners and energy auditors on using temporal thermography in energy audits, (ii) homeowner agency, (iii) improving data interpretation and quality, (iv) motivating change, and (v) data privacy. We conclude by (vi) reflecting on our mission structure, (vii) providing recommendations for the design of future temporal thermography-based sensor systems and (viii) describing the limitations of the studies presented in this work. 164 Outlooks on Temporal Thermography in Energy Audits. Across both the studies with homeowners and professional energy auditors, we found participants believed the temporal thermography sensor system data would be valuable toward creating new products, services, and interactions. All participants envisioned using temporal thermography to achieve personal goals: using temporal thermography as a form of quality assurance for renovations, improving confidence in insights from thermographic data, having a trustworthy source of data on in-home practices (e.g., vs homeowner self- report), improving auditors’ ability to use thermography as they might like, and creating new forms of client-auditor interactions—where the client is able to initiate conversations with professional auditors with personal data. That each population described a similar potential for new client-auditor interactions offers a possibility to bring these two population’s goals into congruence. Homeowners wanted to address important issues but expressed concerns about professional auditors trying to push unnecessary products and services. This perception conflicts with professional auditors’ intrinsic motivation for pursuing their careers, including a passion for the environment and raising awareness of energy efficiency issues. The data from the temporal thermography sensor system repositions the starting point of discussions, emphasizing addressing issues to improve buildings rather than relying solely on one party to uncover them and present recommendations. Homeowner Agency. Even beyond offering new client-auditor interactions, homeowners discussed ways the temporal thermography sensor system improved their agency within their residence. While homeowners appreciated the rapid region of interest detection of the smartphone-based thermal camera, the depth of investigation offered by the temporal 165 sensor system improved their confidence in their observations and insights. The temporal sensor system offered homeowners the ability to make more informed decisions about what was or was not a problem and to make choices about whether or not to take action. Because self-collected data tends to be more meaningful and trusted than data presented by others [39], homeowners perceived an increased capacity to investigate and act—be it on their own, using results to facilitate conversations with professionals as previously mentioned, or, in the case of the participant who was also a landlord, as a way to investigate ill-described issues of renters (e.g., renters stating, “It’s cold in my bedroom” and wanting “it” fixed). Professional energy auditors offered strong support for the idea of empowering homeowners to collect their own meaningful, temporal thermographic data and to generate a report that would start a conversation between them, viewing the activity as a supplement to their practice rather than a replacement of it. This emphasis on thermography as a communication tool mirrors what was seen in our previous investigations [109]. However, professional energy auditors also described how such a new initiative would need to be instigated by homeowners, as none of the auditors were interested in managing the necessary equipment (e.g., renting it out). Improving Data Quality and Interpretation. While both professional auditors and homeowners described wanting more coverage of the homes than the temporal sensor system currently provided, professional auditors further described the specific coverage needs homeowners would need to comply with to ensure high data quality (e.g., sensor placement, improved ROI selection). Professional auditors also described their interest in having actual, historical client data as people’s perceptions are not always reliable 166 (corroborating results from related studies [29]). While not an issue raised by any participants, to ensure the continued trustworthiness of the temporal thermography sensor system, improving upon the validation measures such as those within Study 1 and the procedures that require user calibration before each use—which is not always done with current commodity thermal cameras in favor of qualitative scanning—will be critical toward obtaining high quality, reliable data for energy auditing purposes. As with previous studies [109], professional participants were concerned over the high volume of data temporal thermography would generate and if they would be able to utilize and interpret it. However, in contrast with the previous data visualization interface (Chapter 6), the automatically generated report helped address this potential issue of data overload by semi-automatically performing analysis and presenting it in an easily digestible format. The homeowners and professional auditors alike appreciated the ratings and recommendations offered in the reports—particularly the R-values presented as a result of temporal thermographic analysis which may help to reduce subjectivity when interpreting thermal data pertaining to insulation issues; however, auditors cautioned that such reports will need to be written carefully and potentially warn against misleading results. Motivating Change. Even with issues in the home being identified and increased homeowner agency, homeowners may be reluctant to act on this knowledge due to the overhead involved (e.g., financial, time, hassle [123]). The interviews with professional auditors added complexity to this issue, in that even interested homeowners face problems with unanticipated costs after having decided to pursue renovations or retrofits because they lack the expert knowledge to know what may be necessary (e.g., the need 167 to improve building infrastructure before installing rooftop solar panels). While the study was successful in improving homeowner awareness of issues, the 45-day follow-up survey revealed that participants had not yet taken serious actions on insulation improvements issues (though some had on air leakage issues). Thus, the role of thermography in DIY audits may somewhat contrast its role in professional auditing, where thermal imagery is a motivator for clients to pursue changes [109]. While improving awareness is a goal of both professional auditors and most urban energy program [38,141,149,153], we are left with the question of how to motivate change. This is a question that is common to many technological interventions within Sustainable HCI and energy literature [13,82,123]. Homeowners and professional auditors suggested that the answer within this domain may be in getting data about building stocks into the hands of policy makers who may be able to increase subsidies and other incentives for improvements. However, such an initiative may be met with reluctance given participants’ concerns over data privacy. Data Privacy. Concerns over data privacy were described in much the same manner across both participant studies: they were concerned with who would have access to these sensitive data. Moreover, none of the participants approved of current programs (e.g., [1,103]) that collected thermal images of their homes to solicit services to them. This “non-visible” thermal data was perceived as more private than regular street-view-like data (i.e., a photograph vs thermogram), and the marketing was perceived as a nuisance. Additionally, homeowners expressed two additional concerns over the temporality of the sensor system itself. Firstly, they were concerned that such data could offer unanticipated private information about their in-home behavior and habits. Secondly, they were 168 concerned over the content of the thermographic images, expressing a desire to curate any collected datasets to remove sensitive/embarrassing images. 7.6.1 Design Recommendations for Temporal Thermography Sensor Systems We offer eight design recommendations for future temporal thermography sensor system like ours that may be used for short- and long-term deployments in homes.  Encourage Exploration. In addition to examining regions of interest, temporal thermography sensor systems like ours have the ability to uncover unknown issues as seen in our in-home deployments. Encouraging exploration through systems that effectively scaffold this process may help with selecting effective deployment locations while potentially addressing professional auditors concerns over data quality and homeowners focusing on non-critical issues.  Selecting Regions of Interest (ROI). Additionally, such systems should offer further scaffolding to aid users with automatically selecting regions for analysis.  Integrating the System in Homes. To be easier to deploy or install into homes, such systems should: minimize the form factor so they can be deployed in hard to reach areas and incorporate stronger Wi-Fi connective hardware to be less dependent on proximity to wireless routers or signal extenders (e.g., for basements, crawl spaces, etc.). Offline modes for local data processing and management of weak Wi-Fi signals would also be advantageous. Finally, if successful, this integration may enable new opportunities for HBI [5] research. 169  Raw Data Access. The data from the automatically generated report should be complemented with a downloadable link to the raw data, which can be used in standard energy modeling software (e.g., possibly in BIM-like formats [116]).  Data Overload. Systems should be careful not to overwhelm users with data; temporal analysis worked well as a backend, automatic process with overviews of the output being presented to users.  Alerts and Messaging. Reports should use non-threatening language and consider using push notifications for issues that arise during long-term deployments (e.g., such as with a sensor installed permanently into a building) which would improve versatility of the system.  Customizable Reports. To motivate sharing and having conversations with professionals, reports should have customizable filtering (e.g., specific periods), editing (e.g., removal of embarrassing information), and secure sharing features (e.g., passwords that are added to sensitive documents). Additionally, enabling support for specific end-users goals (i.e., thermal comfort versus energy savings) is also essential.  Short-term Deployments. Having an in-home camera always recording was unsettling to homeowners, despite a motion sensor filtering out data. Shorter- term deployments (e.g., 3 hours) may offer benefits so long as environmental conditions are met throughout the chosen time period. 170 7.6.2 Mission Structure The missions in Study 2 were designed to rapidly surface potential problems with insulation in homes and to evaluate if the introduction of temporal data collection and analysis could aid end-users in determining the severity of issues as well as the need for repairs. More specifically, the first activity with the smartphone-based thermal camera attachments helped acclimate participants to the use and limitations of thermography and the second activity allowed participants to extract additional insights from the temporal sensor system. Our aim was not to directly compare these two activities, but to explore how they might complement each other and improve the overall experience of using thermography in the home through the common experience of using docking stations designed for mobile products. Moreover, direct comparison would be difficult due to the small scale of our study and the likelihood that participants were influenced by the order of the activities. The combination of these two activities makes participants’ feedback about potential future uses for thermographic technology valuable because they were able to compare the technology individually and the experience overall. The mission structure, as with previous studies [110], does limit our results as they would likely be different if the study were structured another way and other scenarios, such as using the sensor system first or in lieu of the smartphone-based thermal camera, are certainly possible. However, we believed these scenarios would not be congruent with the way docking stations are used in the home and would be limited in their effectiveness. While such scenarios could be explored, longitudinal studies with semi-permanently installed systems or differing form factors are likely more promising avenues for future research. 171 7.6.3 Limitations In addition to the limitations described within the findings (e.g., participants being unable to set up the sensor kit where they wanted due to weak in-home Wi-Fi signals) and discussion (e.g., mission structure), we acknowledge several additional limitations to this study. Firstly, the sample size in studies 2 and 3 were small, with 5 participants each. There was also a gender skew in study 3, which was performed exclusively with male energy auditors (consistent with the field demographics and previous research [109]). Additionally, following up with participants after 45-days may not have allowed enough time for action, especially considering the significant expense noted by homeowners. 7.7 Conclusion In this chapter, we first presented an easy-to-deploy longitudinal thermographic sensor system that was paired with an automatically generated, interactive report. We then present three studies: a technical evaluation of the sensor system, in-home end-user deployments with 5 homeowners in 5 households, and semi-structured interviews including a presentation of design probes—including our sensor system—with 5 professional energy auditors. Our findings suggest that temporal thermography can assist people with gauging the severity of issues, may provide new auditor-client interactions, and may improve homeowner agency as well. While we observed some long-term benefits such as increased awareness, motivating change and maintaining homeowner privacy are areas future work should further explore. From these findings, we offered eight recommendations for the design of future temporal thermographic and in-home sensing systems useful to researchers and application designers working in related areas. 172 Chapter 8 Conclusion The purpose of this dissertation has been to: (i) understand and characterize current building thermography practices, benefits, and challenges among both professional and novice thermographers, (ii) conduct human-centered explorations into the role of automation and the potential of pervasive thermographic scanning in the built environment, and (iii) advance the state-of-the-art for interactive systems to perform building thermography. In this chapter, we summarize the completed threads of research, review the contributions of this dissertation, discuss limitations of the work, and put forth avenues for future research. We addressed the goals of this dissertation through three threads of research. In the first research thread, we explored novices’ thermal camera use and their practice of performing thermographic energy audits through two studies. In Chapter 3 we presented a study characterizing novice uses of thermal cameras broadly through an examination of 1,000 YouTube videos, complemented by an online survey of the videos’ content creators. Findings characterized consumers’ many uses of thermal cameras, notably suggesting that they can be effectively used by novices to improve energy efficiency. In Chapter 4 we presented a four-week field study of end-user behavior with novice thermal camera users who investigated the built environment; we explored what novices discovered, the challenges they perceived, and how they approached thermograpic 173 building assessments. Findings from this study further suggested that users with minimal training can employ thermal cameras to document energy-efficiency issues in buildings and even identify previously unknown issues, though they faced challenges such as determining the severity of the issues they detect. The next thread of research focused on professional energy auditors and their perspectives on the potential for automated approaches to thermographic data collection and analysis. To that end, Chapter 5 presented two studies: a semi-structured interview study with 10 professional energy auditors that included five design probes investigating recent approaches to automated thermographic data collection and analysis as well as an observational case study of a residential energy audit. These studies provided insights into current auditing procedures, the benefits and challenges of using thermography during energy audits and elicited critical feedback on automated thermography research; the observational case study further contextualized findings and emphasized the complexities of energy auditing. Together, these studies offered reflections on current professional practice as well as guidelines for the design of future thermographic tools and approaches to thermographic automation. Building on the outcomes of the previous studies, Chapters 6 and 7 presented the third and final thread of research: the development and evaluations of a temporal thermographic sensor system including accompanying data visualization and reporting tools. Chapter 6 introduced a novel temporal thermography system and a corresponding interactive visual analytics tool for viewing and analyzing temporal thermographic data. Through a usability study and a field deployment, we found that while temporal data may make identifying transient environmental conditions easier, inexperienced users require 174 more support to meaningfully extract insights. In Chapter 7 we iterated on this system and visualization approach. Through three studies, we described: (i) the development and validation of the system, (ii) field deployments with homeowners, and (iii) interviews with professional energy auditors. We introduced computational support for thermal camera calibration and temporal data collection, showed that our resulting system is as accurate as the state-of-the art in terms of assessments of building envelope performance, and that it addresses issues with inaccurate measurements from single-image thermography. Findings from the deployments with homeowners showed that they felt an increased agency in determining whether issues existed in their homes and how severe the issues were, appreciated the holistic approach of the system (e.g., learning building codes, receiving additional information about air quality), and experienced a lasting awareness of energy efficiency issues. The interviews with professional auditors showed interest in the deployment of such a sensor system in residences, including multiple simultaneous deployments, while offering cautions about sensor placement and futures that envision urban-scale deployments. 8.1 Summary of Contributions In summary, this dissertation makes several contributions to the areas of: computer science, human-computer interaction, sustainable HCI, and building sciences. Through this work, we advance understandings of (i) current building thermography practices, including the benefits and challenges among both professional and novice thermographers, (ii) offer the first human-centered explorations into the role of automation and the potential of pervasive thermographic scanning in the built 175 environment, and (iii) advance the state-of-the-art through the development and testing of new interactive building thermography systems. 8.1.1 Contributions from Research Thread 1: Studies of Novices’ Use of Thermal Cameras and Temporal Thermography During Energy Audits A characterization of non-professional, novice end-users of thermography with a focus on their DIY energy auditing practices. Through a study investigating YouTube thermography content posted by non- professional thermographers, we identified that a community of novice end-users of thermal cameras is growing, actively performing audits, and implementing energy efficiency recommendations based on their DIY thermographic inspections. Through our second study, a field study of novice thermographers performing energy audits, we found that novice end-users may have difficulty gauging the severity of the issues they encounter and point out barriers that may impact a person’s ability to enact change. An identification of key design recommendations for future thermographic systems and applications designed to support novice use. Through our studies with novice thermography users we developed—and later implemented a subset of—design recommendations valuable for making thermographic energy auditing systems usable by novice users. 176 8.1.2 Contributions from Research Thread 2: Studies on the Practices and Perspectives of Professional Energy Auditors on Potential Automated Approaches to Thermography A characterization of professional end-users of thermography and the role of thermal cameras in professional energy auditing. Through a combination of interviews with professional energy auditors who have applied thermography to their work and a direct observation of a residential energy audit, we provided the first human-centered assessment of energy auditing and thermography’s role therein. This work revealed challenges and highlighted energy auditing as a socio- technical, dialogic process. A critical examination of recently proposed automated and semi-automated solutions to thermographic data collection and analysis in the built environment. We reviewed design probes with a total of 15 professional auditors across two studies on recent and proposed approaches to automated thermography. This work is useful for understanding the potential benefits, limitations, and challenges of these approaches while exploring how well they will integrate into professional energy auditing practices. An identification of key design recommendations for future thermographic systems and applications designed to support professional use. Through our studies with professional energy auditors and building thermographers we developed—and later applied a subset of—design recommendations that inform the design of future thermographic energy auditing sensor systems. 177 8.1.3 Contributions from Research Thread 3: Development and Deployment of a Temporal Thermographic Sensor System The design, development, and evaluation of a novel, temporal thermographic sensor system that can be used effectively by novice and professional energy auditors to collect and analyze thermography data in residential buildings. Through iterating the design of our temporal thermographic data collection system and automatically generated report of analyzed data through a series of pilot tests, validation measures, field deployments, and interviews with professional energy auditors, we offer a novel system that benefits both novice and professional energy auditors. A summary of the user benefits and challenges associated with temporal thermography sensor systems. Through a field study deployment with homeowners and interviews with professional energy auditors, we summarize potential benefits of using a temporal thermography sensor system to support residential audits (e.g., increasing homeowner agency, new auditor-client interactions) as well as the associated challenges (e.g., household coverage). An identification of key design recommendations for future temporal thermographic systems that support in-home use by novice and professional energy auditors. We identify a further eight recommendations for future temporal thermographic sensor systems that would enable them to support the data collection and analysis needs of novice and professional energy auditors. 178 8.2 Future Work In this section, we describe the limitations of the work completed in this dissertation, how future work may address those limitations, and suggest future research initiatives that build on our research. Specifically, we describe: (i) expanding data collection and report generation (i.e., further validation, multiple simultaneous sensor deployments in homes, longer deployments, and new report interactions), (ii) new practices and domains (i.e., homeowner-auditor interactions, homeowner DIY energy audits, engaging with policy makers), and (ii) technical improvements to the sensor system. 8.2.1 Expanding Data Collection and Report Generation Here we describe a number of ways that future work can build upon and expand the current longitudinal thermographic sensor system, including further validation of the system, multiple simultaneous sensor deployments in homes, longer deployments, and new report interactions. Further Sensor System Validation. The sensor system was shown to perform accurately in terms of the validation measures described in Chapter 7. However, our experiments were restricted to the greater Washington DC area, and therefore the sensor system was validated only for buildings with construction types common to that region. As different procedural recommendations exist for buildings of differing constructions [23,85], future work should investigate the validation of this sensor system on a wider range of buildings. Additionally, we validated the sensor system using the THM method [14] and future 179 Figure 8.1: Multi‐unit scenario (presented to auditors in Chapter 7, Study 3) where several of our  sensors are deployed to  increase  the simultaneous coverage area represented by  the red arcs  paired with each wireless sensor unit.  work should compare our results to the procedures in ISO 9869 [85], as it is more common in temporal thermography literature despite that it requires additional sensing hardware. More broadly, current practices and procedures for validating the accuracy of temporal thermography are ad-hoc—differing by research team in terms of data collection intervals, number of images collected, and other methodological decisions (e.g., [6,35,56,113]). While this work complements this temporal thermography research by further demonstrating the practice’s viability and potential for scalability through a novel, field tested system that is easily used by novices, future work toward a common procedure or uniform best practice would be benefit practitioners and researchers alike. 180 Multiple Sensor Deployments in Homes. Findings from Chapter 7 described how homeowners and professional auditors alike sought additional coverage from our temporal thermographic sensor system. Deploying multiple sensors simultaneously in homes could offer this coverage (Figure 8.1); however, this would likely increase end- user’s concerns over intrusiveness and exacerbate issues around data storage, management, and overload discussed throughout this dissertation. Future work should investigate these issues and explore diverse ways of presenting the data. Potential avenues of data displays in this scenario include abstractions (e.g., describing energy saving in terms of emissions reduction versus dollars saved), and, as spatial-temporal complexity increases, through hybridizing the approaches presented in Chapters 6 and 7—combining interactive visualization systems with automated data analyses. This latter approach could enable the exploration of the data (similar to [111]) while capitalizing on the advantages of summative reporting, such as the ease of consumption. Longer Deployments. As observed in previous literature [54] and by our experiments in Chapter 7, temporal thermography methods become more accurate with longer data collection periods. Future work should investigate the use of platforms such as ours for long-term structural building health monitoring systems, as they could complement current building management [29,46] and in-home sensing [63,65] systems and offer new insights. Through longer deployments of the sensor system—on the scale of weeks, months, or even years—new insights into energy efficiency and building degradation issues may be possible. Future work could investigate how seasonal changes may influence whether issues are detectable by such sensor systems and whether new issues become observable over longer data collection periods. 181 Future long-term deployments also have the potential to make new contributions to the field of material sciences. For instance, an active topic is investigating the performance and degradation of insulation materials using a variety of tools (e.g., [3,53,154]). Future work that uses a temporal thermography sensor system to monitor the degradation over time would offer empirical data about this process, which would enable better decisions surrounding construction practices, what materials to use initially in construction, and when to make efficiency upgrades. Lastly, another potential benefit of long-term temporal thermographic data collection is that deviations from previously collected results may provide early warnings to problems (e.g., leaking water). At the same time, because analysis relies on comparisons with previously collected results future work should also investigate ways to account for renovations and retrofits to a building. As temporal data analysis is based on the convergence of data through averaging [23], older data in these systems may result in artificially lower performance values after building modifications have been made, which could mislead users regarding the effectiveness of their renovations unless these changes are accounted for (e.g., by settings key points, restarting data collection). New Report Interactions. Promoting the adoption of automated thermographic reports by consumers and professional auditors is a critical avenue for future work. Participants of this work offer beginnings to this endeavor, for instance suggesting the customization of the reports, ability to curate data sets, and the creation of downloadable/shareable datasets. Future work in this area should also work with building scientists, energy auditors, and certified thermographers to develop standardized language to describe household performance and issues within the report, as no such language standard exists. 182 Additionally, participants in our study wanted more specific and actionable recommendations; understanding how to best deliver such information has received recent attention as it is critical for turning new insights from data analysis into user actions [139]. Moreover, if longer deployments are pursued as previously described, leveraging modern technologies such as smartphones to offer push notifications to users when issues are uncovered would be helpful toward improving user engagement and reducing the data fatigue on users, who would otherwise be required to regularly interact with the reporting infrastructure. 8.2.2 New Practices and Domains Here we describe the new practices and domains that future research can pursue by building upon and expanding the lessons learned in this dissertation, including regarding new homeowner-auditor interactions, homeowner DIY energy audits, and engaging with policy makers. Homeowner-Auditor Interactions. Much of the work on energy efficiency describes a problem with the public knowing what services are available to them and a reluctance to implement change due to the perceived hassle [123]. Moreover, as we saw in our studies with novices, some people prefer not to know about issues or perceive professional service providers as being “out to sell you,” which are both deterrents to uncovering and addressing issues. After using thermography, particularly the temporal sensor system, our novice participants (Chapters 4 & 7) described improved agency: they felt equipped with evidence that was trusted and which would act as a starting point for communications with professionals. Auditors were also receptive to the idea of homeowners seeking out 183 their services with data in hand (Chapter 7). Given this potential opportunity, future work should make sure that future systems and reports collect and represent the data that both parties require in such situations (e.g., easy overviews for homeowners, raw data for practitioners); it also points to the importance of ensuring data is accurate and reliable. Supporting Homeowner DIY Energy Audits. Upon completing a DIY energy audit, many homeowners, including those in our studies (Chapters 4 & 7), wanted to know what they can personally do to improve their homes. In our work, for instance, some participants were comfortable making simple improvements (e.g., caulking air leakages) while others were not sure where to begin. For future work, investigating ways to more effectively suggest low-difficulty fixes could offer users solutions to common energy efficiency issues that they would be more likely to act upon. A more substantial challenge for future research will be investigating how to motivate change for larger-scale issues, such as low insulation performance. Some participants in our studies indicated they would be willing to learn how to perform these upgrades themselves without needing a professional; however, offering direct recommendations on tools and procedures for substantial renovations may require different scaffolds. Overall, future work in this area should seek to further improve user agency by helping them act upon their insights in improving their home’s energy efficiency. Engaging with Policy Makers. Given the generation of these temporal thermographic energy auditing data in residential buildings, and the potential for many homes to adopt such technologies in the future, an area of future investigation will be how to allow homeowners to contribute this data to policy makers in such a way that it benefits 184 municipalities while preserving the privacy of individual households. Firstly, formative work investigating what data policy makers require will be necessary to determine how to meet their needs in this domain—both in terms of what data are needed and its presentation. Given that numerous programs at the local and federal levels exist that offer retrofit funding [38,141], such future work could be vastly impactful in determining their effectiveness moving beyond metrics such as number of audits performed and speculative assessments savings over time to a more quantified perspective. 8.2.3 Technical Improvements There are several opportunities to improve the temporal thermographic sensor system presented in Chapters 6 and 7. Collecting and labeling more thermographic data may improve automated analyses procedures by allowing for more advanced anomaly detection and issue analyses. For example, having more labeled data may enable us to further adapt existing material recognition models for emissivity detection or allow for the training of new models—which is particularly important given the diversity of potential deployment locations. Furthermore, incorporating additional sensing technologies could enhance the capabilities of such systems, reduce the requirements for user-input, and offer improved privacy protections. The incorporation of LIDAR, for example, would provide depth information that could be used to: (i) more accurately detect the distance to the measurement surface, (ii) map rooms and provide a robust queue for when changes in the scene occur (e.g., moving of furniture, motion), and (iii) enable other forms of context awareness (e.g., opening and closing of windows). Integration with smart home devices, such as voice-controlled assistants like Alexa™ or Google Home™, could offer new opportunities present information to end-users. 185 8.3 Final Remarks In this dissertation, we have explored thermographic energy auditing practices, approaches to automation, and temporal thermographic analyses. We believe this work serves to help make future thermographic systems and tools more congruent with the current practices of novice and professional energy auditors by highlighting benefits, challenges, and outlining paths forward. However, one additional note worth discussing is the observation that obtaining an energy audit or deploying a new sensing system does not always lead to action on the part of building owners despite the additional investment of time and money into these activities. As a result, lowering the barrier to obtaining the information produced by these activities is critical but so is getting building owners engaged in the processes and behaviors that increase their likelihood of acting. Toward this goal, another area future work should investigate is how our sensor system could be opened sourced and made more accessible to a community of interested makers and DIY enthusiast (e.g., through platforms like Instructables™). These types of end-user might be interested in leading development of the system and making it more suitable to long-term residential use. This community could potentially tackle areas of related work described earlier (e.g., determining the best suites of sensors to use in residential environments, constructing more effective interfaces that help engage building owners with their data), serve as evangelists for DIY thermographic auditing practices, and explore synergies with other home automation platforms while potentially providing the valuable data needed to design future interactions that help motivate action or power algorithms that can detect issues beyond insulation problems (e.g., moisture, air leakage). 186 Appendices 1. YouTube Study Survey 2. Novice Smartphone Thermography Study Training Guide a. Used again in later novice study 3. Novice Smartphone Thermography Study Image Analysis Codebook 4. Novice Smartphone Thermography Study Debrief Interview Codebook 5. Professional Thermography Study Design Probe Video Links 6. Novice Temporal Study Activity 1 Survey a. Iterated from prior novice study weekly surveys 7. Novice Temporal Study Activity 2 Survey 8. Novice Temporal Study Debrief Interview a. Iterated from prior professional study interview 9. Novice Temporal Study Sensor System Training Guide 10. Novice Temporal Study Codebook 11. Profession Temporal Study Debrief Interview a. Iterated from prior professional study interview 12. Professional Temporal Study Design Probes 13. Professional Temporal Study Codebook a. Iterated from prior professional study interview 187 1. YouTube Study Survey 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 2. Novice Smartphone Thermography Study Training Guide 203 204 205 206 3. Novice Smartphone Thermography Study Image Analysis Codebook 207 4. Novice Smartphone Thermography Study Debrief Interview Codebook 208 209 5. Professional Thermography Study Design Probe Video Links 1. Understanding the role of thermography in energy auditing - Design Probes Video Figure: https://www.youtube.com/watch?v=KZqKiOgRHZY 2. Understanding the role of thermography in energy auditing - Preview Video: https://www.youtube.com/watch?v=cMGwNf90z28 210 6. Novice Temporal Study Activity 1 Survey 211 212 213 214 215 216 217 218 7. Novice Temporal Study Activity 2 Survey 219 220 221 222 223 224 225 226 8. Novice Temporal Study Debrief Interview 227 228 229 230 231 232 233 234 9. Novice Temporal Study Sensor System Training Guide 235 236 237 10. Novice Temporal Study Codebook 238 239 11. Professional Temporal Study Sensor System Interview 240 241 242 243 244 245 246 247 248 249 12. Professional Temporal Study Sensor System Design Probes 250 251 13. Professional Temporal Study Sensor System Codebook 252 253 Bibliography 1. Bilal Abdulkarim, Rustam Kamberov, and Geoffrey Hay. 2014. Supporting Urban Energy Efficiency with Volunteered Roof Information and the Google Maps API. Remote Sensing 6, 10: 9691–9711. Retrieved October 21, 2015 from http://www.mdpi.com/2072-4292/6/10/9691/htm 2. Gregory D Abowd, Anind K Dey, Peter J Brown, Nigel Davies, Mark Smith, and Pete Steggles. 1999. Towards a better understanding of context and context- awareness. In International Symposium on Handheld and Ubiquitous Computing, 304–307. 3. Mohammad S Al-Homoud. 2005. Performance characteristics and practical applications of common building thermal insulation materials. Building and environment 40, 3: 353–366. 4. Alper T Alan, Enrico Costanza, Sarvapali D Ramchurn, Joel Fischer, Tom Rodden, and Nicholas R Jennings. 2016. Tariff Agent: Interacting with a Future Smart Energy System at Home. ACM Trans. Comput.-Hum. Interact. 23, 4: 25:1--25:28. https://doi.org/10.1145/2943770 5. Hamed Alavi, Denis Lalanne, Julien Nembrini, Elizabeth Churchill, David Kirk, and Wendy Moncur. 2016. Future of Human-Building Interaction. In Proceedings of the 34rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 6. Rossano Albatici, Arnaldo M Tonelli, and Michela Chiogna. 2015. A comprehensive experimental approach for the validation of quantitative infrared thermography in the evaluation of building thermal transmittance. Applied energy 141: 218–228. 7. Lisa Anthony, YooJin Kim, and Leah Findlater. 2013. Analyzing User-generated Youtube Videos to Understand Touchscreen Use by People with Motor Impairments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), 1223–1232. https://doi.org/10.1145/2470654.2466158 8. Association of Energy Engineers. 2014. Certified Energy Auditor. Retrieved from http://www.aeecenter.org/i4a/pages/index.cfm?pageid=3365 9. C.A. Balaras and A.A. Argiriou. 2002. Infrared thermography for building diagnostics. Energy and Buildings 34, 2: 171–183. Retrieved May 30, 2014 from http://www.sciencedirect.com/science/article/pii/S0378778801001050 254 10. Ashok Bardhan, Dwight Jaffee, Cynthia Kroll, and Nancy Wallace. 2014. Energy efficiency retrofits for U.S. housing: Removing the bottlenecks. Regional Science and Urban Economics 47: 45–60. Retrieved August 31, 2014 from http://www.sciencedirect.com/science/article/pii/S0166046213000677 11. Christopher J Bay, Trevor J Terrill, and others. 2017. Autonomous Robotic Building Energy Audits: Demonstrated Capabilities and Open Challenges. ASHRAE Transactions 123: 3. 12. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild with the Materials in Context Database. Computer Vision and Pattern Recognition (CVPR). 13. Linda Berry. 1993. A review of the market penetration of US residential and commercial demand-side management programmes. Energy Policy 21, 1: 53–67. 14. David Bienvenido-Huertas, Roberto Rodr\’\iguez-Álvaro, Juan José Moyano, Fernando Rico, and David Mar\’\in. 2018. Determining the U-value of façades using the thermometric method: Potentials and limitations. Energies 11, 2: 360. 15. Eli Blevis. 2007. Sustainable Interaction Design: Invention & Disposal, Renewal & Reuse. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07), 503–512. https://doi.org/10.1145/1240624.1240705 16. Mark Blythe and Paul Cairns. 2009. Critical Methods and User Generated Content: The iPhone on YouTube. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), 1467–1476. https://doi.org/10.1145/1518701.1518923 17. Dorit Borrmann, Andreas Nüchter, Marija Dakulovic, Ivan Maurovic, Ivan Petrovic, Dinko Osmankovic, and Jasmin Velagic. 2012. The Project ThermalMapper—Thermal 3D mapping of indoor environments for saving energy. In Proceedings of the 10th International IFAC Symposium on Robot Control (SYROCO), 1. 18. V Braun and V Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2: 77–101. 19. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2: 77–101. Retrieved July 11, 2014 from http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa 20. Peter Brooks. 2007. Testing Building Envelopes with Infrared Thermography: Delivering the “Big Picture.” Interface, the Technical Journal of RCI July. Retrieved from http://www.rci-online.org/interface/2007-07-brooks.pdf 255 21. Hronn Brynjarsdottir, Maria Håkansson, James Pierce, Eric Baumer, Carl DiSalvo, and Phoebe Sengers. 2012. Sustainably Unpersuaded: How Persuasion Narrows Our Vision of Sustainability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12), 947–956. https://doi.org/10.1145/2207676.2208539 22. Erin Buehler, Stacy Branham, Abdullah Ali, Jeremy J Chang, Megan Kelly Hofmann, Amy Hurst, and Shaun K Kane. 2015. Sharing is Caring: Assistive Technology Designs on Thingiverse. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15), 525–534. https://doi.org/10.1145/2702123.2702525 23. ASTM C1155-95. 2007. Standard Practice for Determining Thermal Resistance of Building Envelope Components from the In-Situ Data. 24. James Carroll. 2017. Thermal cameras capture 30 Seconds to Mars in live MTV Video Music Awards broadcast. Vision Systems Design, 1. Retrieved January 1, 2017 from http://www.vision-systems.com/articles/2017/08/thermal-cameras- capture-30-seconds-to-mars-in-live-mtv-video-music-awards-broadcast.html 25. Centers for Disease Control and Prevention (CDC). 2015. INDOOR ENVIRONMENTAL QUALITY. National Institute for Occupational Safety and Health Respiratory Health Division. Retrieved from https://www.cdc.gov/niosh/topics/indoorenv/temperature.html 26. Marshini Chetty, A J Bernheim Brush, Brian R Meyers, and Paul Johns. 2009. It’s Not Easy Being Green: Understanding Home Computer Power Management. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), 1033–1042. https://doi.org/10.1145/1518701.1518860 27. D Manning Christopher, Raghavan Prabhakar, and SCHÜTZE Hinrich. 2008. Introduction to information retrieval. An Introduction To Information Retrieval 151: 177. 28. Marisol Clark-Ibáñez. 2004. Framing the social world with photo-elicitation interviews. American behavioral scientist 47, 12: 1507–1527. 29. Adrian Clear, Sam Mitchell Finnigan, Patrick Olivier, and Rob Comber. 2017. “ I’d want to burn the data or at least nobble the numbers”: Towards data-mediated building management for comfort and energy use. 30. Rob Comber and Anja Thieme. 2013. Designing beyond habit: opening space for improved recycling and food waste behaviors through processes of persuasion, social influence and aversive affect. Personal and Ubiquitous Computing 17, 6: 1197–1210. https://doi.org/10.1007/s00779-012-0587-1 256 31. Twentieth Century Fox Film Corporation. 1987. Predator. 32. Enrico Costanza, Sarvapali D Ramchurn, and Nicholas R Jennings. 2012. Understanding Domestic Energy Consumption Through Interactive Visualisation: A Field Study. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp ’12), 216–225. https://doi.org/10.1145/2370216.2370251 33. Matthew Dahlhausen. 2017. personal communication. 34. Giuliano Dall’O, Luca Sarto, Angela Panza, and others. 2013. Infrared screening of residential buildings for energy audit purposes: results of a field test. Energies 6, 8: 3859–3878. 35. Maria Danese, Urska Demsar, Nicola Masini, and Martin Charlton. 2010. Investigating Material Decay of Historic Buildings using Visual Analysis with Multi-Temporal Infrared Thermographic Data. Archaeometry 52, 3: 482–501. 36. Maria Danese, Urška Demšar, Nicola Masini, and Martin Charlton. 2010. Investigating material decay of historic buildings using visual analytics with multi- temporal infrared thermographic data. Archaeometry 52, 3: 482–501. https://doi.org/10.1111/j.1475-4754.2009.00485.x 37. Girum G Demisse, Dorit Borrmann, and Andreas Nuchter. 2013. Interpreting thermal 3D models of indoor environments for energy efficiency. In Advanced Robotics (ICAR), 2013 16th International Conference on, 1–8. https://doi.org/10.1109/ICAR.2013.6766550 38. Department of Housing and Community Development. EmPOWER Maryland Low Income Energy Efficiency Program. Retrieved February 7, 2018 from http://dhcd.maryland.gov/Residents/Pages/lieep/default.aspx 39. John Dewey. 1910. Science as subject-matter and as method. Science 31, 787: 121–127. 40. Tawanna Dillahunt, Jennifer Mankoff, and Eric Paulos. 2010. Understanding Conflict Between Landlords and Tenants: Implications for Energy Sensing and Feedback. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (Ubicomp ’10), 149–158. https://doi.org/10.1145/1864349.1864376 41. J. R. Martinez-De Dios and Anibal Ollero. 2006. Automatic Detection of Windows Thermal Heat Losses in Buildings Using UAVs. In 2006 World Automation Congress, 1–6. Retrieved September 19, 2014 from http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=4259914 257 42. Carl DiSalvo, Phoebe Sengers, and Hrönn Brynjarsdóttir. 2010. Mapping the Landscape of Sustainable HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10), 1975–1984. https://doi.org/10.1145/1753326.1753625 43. Karen Ehrhardt-Martinez, Kat A Donnelly, Laitner Contributors, Dan York, Jacob Talbot, and Katherine Friedrich. 2010. Advanced Metering Initiatives and Residential Feedback Programs: A Meta-Review for Household Electricity-Saving Opportunities. Retrieved January 8, 2017 from http://aceee.org/sites/default/files/publications/researchreports/e105.pdf 44. Energy.gov. 2012. Blower Door Tests. U.S. Department of Energy. Retrieved from http://energy.gov/energysaver/articles/blower-door-tests 45. S Mitchell Finnigan. 2017. personal communication. 46. S Mitchell Finnigan, Adrian K Clear, Geremy Farr-Wharton, Karim Ladha, and Rob Comber. 2017. Augmenting Audits: Exploring the Role of Sensor Toolkits in Sustainable Buildings Management. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2: 10. 47. FLIR Inc. 2014. FLIR Systems Introduces First Thermal Imager Designed for Smartphones. 48. FLIR Inc. 2016. FLIR Approved Applications. Retrieved August 8, 2016 from http://www.flir.com/flirone/display/?id=69356 49. FLIR Systems. FLIR Tools. Retrieved from http://www.flir.com/instruments/display/?id=54865 50. FLIR Systems. 2017. FLIR Company History. 1. Retrieved January 1, 2017 from http://www.flir.com/about/display/?id=55679 51. B J Fogg. 2002. Persuasive Technology: Using Computers to Change What We Think and Do. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 52. Paris A Fokaides and Soteris A Kalogirou. 2011. Application of infrared thermography for the determination of the overall heat transfer coefficient (U- Value) in building envelopes. Applied Energy 88, 12: 4358–4365. 53. Matthew Fox, David Coley, Steve Goodhew, and Pieter de Wilde. 2012. Comparing transient simulation with thermography time series. In Building simulation and optimization conference. Loughborough, UK: IBPSA England, 8. 258 54. Matthew Fox, David Coley, Steve Goodhew, and Pieter de Wilde. 2014. Thermography methodologies for detecting energy related building defects. Renewable and Sustainable Energy Reviews 40: 296–310. Retrieved August 21, 2014 from http://www.sciencedirect.com/science/article/pii/S1364032114006406 55. Matthew Fox, David Coley, Steve Goodhew, and Pieter De Wilde. 2014. Thermography methodologies for detecting energy related building defects. Renewable and Sustainable Energy Reviews 40: 296–310. https://doi.org/10.1016/j.rser.2014.07.188 56. Matthew Fox, David Coley, Steve Goodhew, and Pieter De Wilde. 2015. Time- lapse thermography for building defect detection. Energy and Buildings 92: 95– 106. 57. Jon Froehlich, Tawanna Dillahunt, Predrag Klasnja, Jennifer Mankoff, Sunny Consolvo, Beverly Harrison, and James A Landay. 2009. UbiGreen: Investigating a Mobile Tool for Tracking and Supporting Green Transportation Habits. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), 1043–1052. https://doi.org/10.1145/1518701.1518861 58. Jon Froehlich, Leah Findlater, and James Landay. 2010. The Design of Eco- feedback Technology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10), 1999–2008. https://doi.org/10.1145/1753326.1753629 59. Jon Froehlich, Eric Larson, Elliot Saba, Tim Campbell, Les Atlas, James Fogarty, and Shwetak Patel. 2011. A Longitudinal Study of Pressure Sensing to Infer Real- World Water Usage Events in the Home. . Springer Berlin Heidelberg, 50–69. https://doi.org/10.1007/978-3-642-21726-5_4 60. Rikke Gade and Thomas B. Moeslund. 2013. Thermal cameras and applications: a survey. Machine Vision and Applications 25, 1: 245–262. Retrieved September 19, 2014 from http://link.springer.com/10.1007/s00138- 013-0570-5 61. Edward W Galligan, George S Bakken, and Steven L Lima. 2003. Using a thermographic imager to find nests of grassland birds. Wildlife Society Bulletin (1973-2006) 31, 3: 865–869. 62. Gerald T. Gardner and Paul C. Stern. 2008. The Short List: The Most Effective Actions U.S. Households Can Take to Curb Climate Change. Environment: Science and Policy for Sustainable Development 50, 5: 12–25. https://doi.org/10.3200/ENVT.50.5.12-25 63. Khusvinder Gill, Shuang-Hua Yang, Fang Yao, and Xin Lu. 2009. A zigbee-based home automation system. IEEE Transactions on consumer Electronics 55, 2. 259 64. Mani Golparvar-Fard and Youngjib Ham. 2014. Automated Diagnostics and Visualization of Potential Energy Performance Problems in Existing Buildings Using Energy Performance Augmented Reality Models. Journal of Computing in Civil Engineering 28, 1: 17–29. Retrieved May 30, 2014 from http://ascelibrary.org/doi/abs/10.1061/(ASCE)CP.1943-5487.0000311 65. Carles Gomez and Josep Paradells. 2010. Wireless home automation networks: A survey of architectures and technologies. IEEE Communications Magazine 48. 66. Julie Goodhew, Sabine Pahl, Tim Auburn, and Steve Goodhew. 2014. Making Heat Visible: Promoting Energy Conservation Behaviors Through Thermal Imaging. Environment and Behavior: 0013916514546218-. Retrieved September 20, 2014 from http://eab.sagepub.com/content/early/2014/09/09/0013916514546218.abstract 67. E Grinzato, P Bison, G Cadelano, and F Peron. 2010. R-value estimation by local thermographic analysis. In Thermosense XXXII, 76610H. 68. Sidhant Gupta, Matthew S Reynolds, and Shwetak N Patel. 2010. ElectriSense: Single-point Sensing Using EMI for Electrical Event Detection and Classification in the Home. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (UbiComp ’10), 139–148. https://doi.org/10.1145/1864349.1864375 69. Maria Håkansson and Phoebe Sengers. 2013. Beyond Being Green: Simple Living Families and ICT. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), 2725–2734. https://doi.org/10.1145/2470654.2481378 70. Youngjib Ham and Mani Golparvar-Fard. 2013. An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery. Advanced Engineering Informatics 27, 3: 395–409. Retrieved May 23, 2014 from http://www.sciencedirect.com/science/article/pii/S147403461300027X 71. Youngjib Ham and Mani Golparvar-Fard. 2014. Three-Dimensional Thermography-Based Method for Cost-Benefit Analysis of Energy Efficiency Building Envelope Retrofits. Journal of Computing in Civil Engineering: B4014009. Retrieved September 18, 2014 from http://ascelibrary.org/doi/abs/10.1061/(ASCE)CP.1943-5487.0000406 72. ASHRAE Handbook and others. 2001. Fundamentals. American Society of Heating, Refrigerating and Air Conditioning Engineers, Atlanta 111. 260 73. Hanna Hasselqvist, Cristian Bogdan, and Filip Kis. 2016. Linking Data to Action: Designing for Amateur Energy Management. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS ’16), 473–483. https://doi.org/10.1145/2901790.2901837 74. G.A. Helcke, F. Conti, B. Daniotti, and R.J. Peckham. 1990. A detailed comparison of energy audits carried out by four separate companies on the same set of buildings. Energy and Buildings 14, 2: 153–164. Retrieved January 9, 2015 from http://www.sciencedirect.com/science/article/pii/037877889090034G 75. T Honga, K Jeonga, and C Kooa. 2015. Framework for Evaluating the Thermal Insulation Performance of Existing Residential Buildings Using the Infrared Thermal image and Image Processing Method. In ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, 1. 76. Steven Houben, Connie Golsteijn, Sarah Gallacher, Rose Johnson, Saskia Bakker, Nicolai Marquardt, Licia Capra, and Yvonne Rogers. 2016. Physikit: Data Engagement Through Physical Ambient Visualizations in the Home. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), 1608–1619. https://doi.org/10.1145/2858036.2858059 77. Juan Pablo Hourcade, Sarah L Mascher, David Wu, and Luiza Pantoja. 2015. Look, My Baby Is Using an iPad! An Analysis of YouTube Videos of Infants and Toddlers Using Tablets. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15), 1915–1924. https://doi.org/10.1145/2702123.2702266 78. Daniel J. Hruschka, Deborah Schwartz, Daphne Cobb St.John, Erin Picone- Decaro, Richard A. Jenkins, and James W. Carey. 2004. Reliability in Coding Open-Ended Data: Lessons Learned from HIV Behavioral Research. Field Methods 16, 3: 307–331. https://doi.org/10.1177/1525822X04266540 79. Diane Hu, Liefeng Bo, and Xiaofeng Ren. 2011. Toward Robust Material Recognition for Everyday Objects. In BMVC, 6. 80. Aaron Ingle, Mithra Moezzi, Loren Lutzenhiser, and Richard Diamond. 2012. How well do home energy audits serve the homeowner? In Proceedings of the 2012 American Council for an Energy Efficient Economy Summer Study on Energy Efficiency in Buildings. Retrieved from http://homes.lbl.gov/sites/all/files/Ingle_LBNL-5712E_0.pdf 81. Aaron Ingle, Mithra Moezzi, Loren Lutzenhiser, and Richard Diamond. 2014. Better home energy audit modelling: incorporating inhabitant behaviours. Building Research & Information 42, 4: 409–421. Retrieved September 12, 2014 from http://dx.doi.org/10.1080/09613218.2014.890776 261 82. Aaron Ingle, Mithra Moezzi, Loren Lutzenhiser, Zac Hathaway, Susan Lutzenhiser, Joe Van Clock, Jane Peter, Rebecca Smith, David Heslam, and Richard C. Diamond. 2013. Behavioral Perspectives on Home Energy Audits: The Role of Auditors, Labels, Reports, and Audit Tools on Homeowner Decision Making. 1–396. Retrieved from http://www.escholarship.org/uc/item/1323m27r 83. ISO. 2012. ISO 6781:1983: Thermal Insulation -- Qualitative detection of thermal irregularities in building envelopes -- Infrared method. Retrieved from http://www.iso.org/iso/catalogue_detail.htm?csnumber=13277 84. ISO. 2014. ISO 50002:2014: Energy audits -- Requirements with guidance for use. Retrieved from https://www.iso.org/obp/ui/#iso:std:60088:en 85. B S ISO. 9869-1: 2014 Thermal insulation, Building elements, In-situ measurement of thermal resistance and thermal transmittance-Part 1: Heat flow meter method. London: BSI. 86. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, 675–678. 87. Gerald Kastberger and Reinhold Stachl. 2003. Infrared imaging technology and biological applications. Behavior Research Methods, Instruments, & Computers 35, 3: 429–439. 88. Jordan D. Kelso. 2011. Buildings energy data book. Retrieved from http://buildingsdatabook.eren.doe.gov/ 89. Bran Knowles, Lynne Blair, Mike Hazas, and Stuart Walker. 2013. Exploring Sustainability Research in Computing: Where We Are and Where We Go Next. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’13), 305–314. https://doi.org/10.1145/2493432.2493474 90. K.H Krippendorff. 2003. Content Analysis: An Introduction to Its Methodology. Sage Publications, Inc. 91. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097–1105. 92. Angeliki Kylili, Paris A. Fokaides, Petros Christou, and Soteris A. Kalogirou. 2014. Infrared thermography (IRT) applications for building diagnostics: A review. Applied Energy 134: 531–549. Retrieved January 7, 2015 from http://www.sciencedirect.com/science/article/pii/S0306261914008083 262 93. John Lafferty and Chengxiang Zhai. 2017. Document Language Models, Query Models, and Risk Minimization for Information Retrieval. SIGIR Forum 51, 2: 251–259. https://doi.org/10.1145/3130348.3130375 94. S. Lagüela, J. Armesto, P. Arias, and J. Herráez. 2012. Automation of thermographic 3D modelling through image fusion and image matching techniques. Automation in Construction 27: 24–31. Retrieved September 18, 2014 from http://www.sciencedirect.com/science/article/pii/S0926580512000842 95. S. Lagüela, J. Martínez, J. Armesto, and P. Arias. 2011. Energy efficiency studies through 3D laser scanning and thermographic technologies. Energy and Buildings 43, 6: 1216–1221. Retrieved September 4, 2014 from http://www.sciencedirect.com/science/article/pii/S0378778811000041 96. Susana Lagüela, Lucia Diaz-Vilarino, David Roca, and Julia Armesto. 2014. Aerial oblique thermographic imagery for the generation of building 3D models to complement Geographic Information Systems. In 12th International Conference on Quantitative Infrared Thermography (QIRT 2014). Retrieved from http://qirt.gel.ulaval.ca/archives/qirt2014/QIRT 2014 Papers/QIRT-2014- 041.pdf 97. Bruno Latour and Steve Woolgar. 1986. Laboratory Life: The Construction of Scientific Facts. Princeton University Press. Retrieved January 9, 2015 from http://books.google.com/books/about/Laboratory_Life.html?id=XTcjm0flPdYC &pgis=1 98. Ce Liu, Lavanya Sharan, Edward H Adelson, and Ruth Rosenholtz. 2010. Exploring features in a bayesian framework for material recognition. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, 239–246. 99. Robert Madding. 2008. Finding R-values of stud frame constructed houses with IR thermography. InfraMation 2008. 100. Sanna Malinen. 2015. Understanding User Participation in Online Communities. Comput. Hum. Behav. 46, C: 228–238. https://doi.org/10.1016/j.chb.2015.01.004 101. Jennifer Mankoff. 2012. HCI and sustainability: a tale of two motivations. interactions 19, 3: 16–19. https://doi.org/10.1145/2168931.2168937 102. Maryland Energy Administration. 2015. Your Home & The Energy Code. Retrieved from http://energy.maryland.gov/Documents/YourHomeandtheEnergyCode.pdf 103. Rob Matheson. 2015. Drive-By Heat Mapping. MIT News. Retrieved from http://newsoffice.mit.edu/2015/startup-essess-heat-mapping-cars-0105 263 104. Matthew Mauriello, Jonah Chazan, Jamie Gilkeson, and Jon E. Froehlich. 2017. A Temporal Thermography System for Supporting Longitudinal Building Energy Audits. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. https://doi.org/10.1145/3123024.3123082 105. Matthew Louis Mauriello, Cody Buntain, Brenna McNally, Sapna Bagalkotkar, Samuel Kushnir, and Jon E. Froehlich. 2018. SMIDGen: An Approach for Scalable, Mixed-Initiative Dataset Generation from Online Social Networks. In HCIL Tech Reports. Retrieved from http://hcil.umd.edu/pub-perm- link/?id=2018-01 106. Matthew Louis Mauriello, Matthew Dahlhausen, Erica Brown, Manaswi Saha, and Jon Froehlich. 2016. The Future Role of Thermography in Human-Building Interaction. In Proceedings of the 34rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 107. Matthew Louis Mauriello and Jon E Froehlich. 2014. Towards Automated Thermal Profiling of Buildings at Scale Using Unmanned Aerial Vehicles and 3D-reconstruction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’14 Adjunct), 119–122. https://doi.org/10.1145/2638728.2638731 108. Matthew Louis Mauriello, Brenna McNally, Cody Buntain, Sapna Bagalkotkar, Samuel Kushnir, and Jon E. Froehlich. 2018. A Large-Scale Analysis of YouTube Videos Depicting Everyday Thermal Camera Use. In Proceedings of MobileHCI 2018. To Appear. 109. Matthew Louis Mauriello, Leyla Norooz, and Jon E. Froehlich. 2015. Understanding the Role of Thermography in Energy Auditing: Current Practices and the Potential for Automated Solutions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 1993--2002. https://doi.org/10.1145/2702123.2702528 110. Matthew Louis Mauriello, Manaswi Saha, Erica Brown, and Jon E. Froehlich. 2017. Exploring Novice Approaches to Smartphone-based Thermographic Energy Auditing: A Field Study. In Proceedings of the 35th Annual ACM Conference on Human Factors in Computing Systems, 1768--1780. https://doi.org/10.1145/3025453.3025471 111. Matthew Louis Mauriello, Ben Shneiderman, Fan Du, Sana Malik, and Catherine Plaisant. 2016. Simplifying overviews of temporal event sequences. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2217–2224. 264 112. Peyman Moghadam and Stephen Vidas. 2014. HeatWave: the next generation of thermography devices. In SPIE Sensing Technology + Applications, 91050F. Retrieved September 18, 2014 from http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1875668 113. I Nardi, D Ambrosini, T De Rubeis, S Sfarra, S Perilli, and G Pasqualoni. 2015. A comparison between thermographic and flow-meter methods for the evaluation of thermal transmittance of different wall constructions. In Journal of Physics Conference Series. 114. Iole Nardi, Domenica Paoletti, Dario Ambrosini, Tullio De Rubeis, and Stefano Sfarra. 2016. U-value assessment by infrared thermography: A comparison of different calculation methods in a Guarded Hot Box. Energy and Buildings 122: 211–221. 115. Iole Nardi, Stefano Sfarra, and Dario Ambrosini. 2014. Quantitative thermography for the estimation of the U-value: state of the art and a case study. In Journal of Physics: Conference Series, 12016. 116. Worawan Natephra, Ali Motamedi, Nobuyoshi Yabuki, Tomohiro Fukuda, and Takashi Michikawa. Building Envelope Thermal Performance Analysis using BIM-Based 4D Thermal Information Visualization. 117. Vicki Norberg-Bohm and Chad White. 2004. Building America Program Evaluation. Retrieved from http://www1.eere.energy.gov/analysis/pdfs/ba_program_eval_09-04.pdf 118. Jarno Ojala. 2015. Content Sharing Building Social User Experiences. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI ’15), 903– 905. https://doi.org/10.1145/2786567.2786937 119. Omar Oreifej, Jason Cramer, and Avideh Zakhor. 2014. Automatic Generation of 3D Thermal Maps of Building Interiors. ASHRAE transactions. Retrieved from http://www.eecs.berkeley.edu/~oreifej/papers/Thermal_2014.pdf 120. Anna Ostańska. 2017. Thermal Imaging for Detection of Defects in Envelopes of Buildings in Use: Qualitative and Quantitative Analysis of Building Energy Performance. Periodica Polytechnica Civil Engineering. 121. Jeni Paay, Jesper Kjeldskov, Mikael B Skov, and Kenton O’Hara. 2012. Cooking Together: A Digital Ethnography. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’12), 1883–1888. https://doi.org/10.1145/2212776.2223723 265 122. Sabine Pahl, Julie Goodhew, Christine Boomsma, and Stephen R J Sheppard. 2016. The Role of Energy Visualization in Addressing Energy Use: Insights from the eViz Project. Frontiers in psychology 7. 123. Karen Palmer, Margaret Walls, Hal Gordon, and Todd Gerarden. 2012. Assessing the energy-efficiency information gap: results from a survey of home energy auditors. Energy Efficiency 6, 2: 271–292. Retrieved September 3, 2014 from http://link.springer.com/10.1007/s12053-012-9178-2 124. James Pierce and David Roedl. 2008. COVER STORY: Changing Energy Use Through Design. interactions 15, 4: 6–12. https://doi.org/10.1145/1374489.1374491 125. Mattia Previtali, Luigi Barazzetti, Raffaella Brumana, and Fabio Roncoroni. 2013. Thermographic Analysis from UAV Platforms for Energy Efficiency Retrofit Applications. J. Mob. Multimed. 9, 1–2: 66–82. Retrieved from http://dl.acm.org/citation.cfm?id=2600433.2600439 126. Catherine M Ridings and David Gefen. 2004. Virtual community attraction: Why people hang out online. Journal of Computer-mediated communication 10, 1: JCMC10110. 127. E F J Ring and K Ammer. 2012. Infrared thermal imaging in medicine. Physiological measurement 33, 3: R33. 128. Mina Seraj. 2012. We create, we connect, we respect, therefore we are: intellectual, social, and cultural value in online communities. Journal of Interactive Marketing 26, 4: 209–222. 129. Rita Shewbridge, Amy Hurst, and Shaun K Kane. 2014. Everyday Making: Identifying Future Uses for 3D Printing in the Home. In Proceedings of the 2014 Conference on Designing Interactive Systems (DIS ’14), 815–824. https://doi.org/10.1145/2598510.2598544 130. Ben Shneiderman. 1993. 1.1 direct manipulation: a step beyond programming languages. Sparks of innovation in human-computer interaction 17: 1993. 131. Ben Shneiderman. 1996. The eyes have it: a task by data type taxonomy for information visualizations. In Visual Languages, 1996. Proceedings., IEEE Symposium on, 336–343. https://doi.org/10.1109/VL.1996.545307 132. M. Six Silberman, Lisa Nathan, Bran Knowles, Roy Bendor, Adrian Clear, Maria Håkansson, Tawanna Dillahunt, and Jennifer Mankoff. 2014. Next steps for sustainable HCI. interactions 21, 5: 66–69. Retrieved September 17, 2014 from http://dl.acm.org/ft_gateway.cfm?id=2651820&type=html 266 133. SparkFun Electronics. 2014. FLiR Dev Kit. Retrieved September 1, 2016 from https://www.sparkfun.com/products/13233 134. SparkFun Electronics. 2018. Friday Product Post: It’s Clear with FLIR! Retrieved February 7, 2018 from https://www.sparkfun.com/news/2722 135. BRITISH STANDARD. 2008. BS ISO 18434-1: condition monitoring and diagnostics of machines: thermography: part 1: general procedures. 136. Yolande Strengers. 2013. Smart energy technologies in everyday life: Smart Utopia? 137. Yolande Strengers. 2014. Smart Energy in Everyday Life: Are You Designing for Resource Man? interactions 21, 4: 24–31. https://doi.org/10.1145/2621931 138. Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies 67, 8: 639–662. https://doi.org/http://dx.doi.org/10.1016/j.ijhcs.2009.03.004 139. Reuven Sussman, Maxine Chikumbo, and Robert Gifford. 2018. Message Framing for Home Energy Efficiency Upgrades. Energy and Buildings. 140. Leslie Tack. 2015. Top Ten Myths About Handheld Infrared Thermal Imaging Cameras. Pembroke Instruements, LLC, 12. Retrieved January 1, 2017 from https://pembrokeinstruments.com/_download_pdf_897/_Pembroke-PDF- Downloads/Top-Ten-Myths-IR-Thermal-Cameras-Pembroke.pdf 141. Jayne Thompson. 2017. HUD Home Repair Grants. Retrieved from http://homeguides.sfgate.com/hud-home-repair-grants-8167.html 142. Albert Thumann and Williiam J. Younger. 2008. Handbook of Energy Audits. The Fairmont Press, Inc. 143. Edward Tufte. 1990. Envisioning information. Cheshire (Conn.). 144. Rasty Turek. 2016. What YouTube Looks Like In A Day. Medium, 1. Retrieved January 1, 2017 from https://medium.com/@synopsi/what-youtube-looks-like-in-a- day-infographic-d23f8156e599 145. U.S. Green Building Council. 2016. LEED v4 for BUILDING OPERATIONS AND MAINTENANCE. Retrieved from http://www.usgbc.org/resources/leed- v4-building-operations-and-maintenance-current-version 146. US Department of Energy. 2012. Thermographic Inspections. Retrieved from http://energy.gov/energysaver/articles/thermographic-inspections 267 147. US Department of Energy. 2014. #AskEnergySaver: Home Energy Audits. Retrieved from http://energy.gov/articles/askenergysaver-home-energy-audits 148. US Department of Energy. 2016. Building America: Making Real Progress for U.S. Homes. Building Technologies Office. Retrieved from https://www.energy.gov/sites/prod/files/2016/02/f29/building_america_factsheet_2 .10.16.pdf 149. US Department of Housing and Urban Development. HOME Investment Partnerships Program. Retrieved February 7, 2018 from https://www.hudexchange.info/programs/home/ 150. V. P. Vavilov. 2011. A pessimistic view of the energy auditing of building structures with the use of infrared thermography. Russian Journal of Nondestructive Testing 46, 12: 906–910. Retrieved September 19, 2014 from http://link.springer.com/10.1134/S1061830910120065 151. Anthony J Viera, Joanne M Garrett, and others. 2005. Understanding interobserver agreement: the kappa statistic. Fam Med 37, 5: 360–363. Retrieved from http://www.stfm.org/FamilyMedicine/Vol37Issue5/Viera360 152. Paul Viola and Michael Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, I-- 511. 153. Christopher Walker and others. 1998. Expanding the Nation’s Supply of Affordable Housing: An Evaluation of the Home Investment Partnerships Program. 154. Rosanne Walker and Sara Pav\’\ia. 2015. Thermal performance of a selection of insulation materials suitable for historic buildings. Building and Environment 94: 155–165. 155. Chao Wang, Yong K. Cho, and Mengmeng Gai. 2013. As-Is 3D Thermal Modeling for Existing Building Envelopes Using a Hybrid LIDAR System. Journal of Computing in Civil Engineering 27, 6: 645–656. Retrieved May 30, 2014 from http://ascelibrary.org/doi/abs/10.1061/(ASCE)CP.1943-5487.0000273 156. David H Wolpert. 1992. Stacked generalization. Neural networks 5, 2: 241–259. 157. Allison Woodruff, Jay Hasbrouck, and Sally Augustin. 2008. A Bright Green Perspective on Sustainable Choices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08), 313–322. https://doi.org/10.1145/1357054.1357109 268 158. 2014. FLIR Systems Introduces First Thermal Imager Designed for Smartphones. FLIR Inc. Retrieved January 7, 2014 from http://investors.flir.com/releasedetail.cfm?releaseid=817445 159. 2014. Seek Thermal Announces It Will Bring Next-Generation Thermal Imaging To Mass Market. Seek Thermal Inc. Retrieved May 1, 2016 from http://www.thermal.com/press-releases/2015/2/12/seek-thermal-announces-it-will- bring-next-generation-thermal-imaging-to-mass-market 160. 2014. How do I take a good infrared thermal image? Monroe Infrared. Retrieved December 4, 2016 from http://monroeinfrared.com/knowledgebase/a-guide-to- thermal-imaging-cameras/how-do-i-take-a-good-infrared-thermal-image 161. 2014. Thermography drone for aerial recordings using the FLIR T640bx thermal imaging camera. FLIR Inc. Retrieved from http://www.flirmedia.com/MMC/THG/Brochures/T820501/T820501_EN.pdf 162. 2015. How Thermal Imaging Works: A Closer View. FLIR Inc. Retrieved May 1, 2016 from http://www.flir.com/flirone/Press/FLIR-ONE-Android-iOS-How-It- Works.pdf 163. 2015. FLIR One: At Home. FLIR Inc. Retrieved January 1, 2016 from http://www.flir.com/flirone/content/?id=62915 164. 2016. CAT S60 ANNOUNCED AS WORLD’S FIRST SMARTPHONE WITH INTEGRATED THERMAL CAMERA. Caterpillar Inc. Retrieved March 21, 2016 from http://www.catphones.com/en-us/news/press-releases/cat- s60-announced-as-worlds-first-smartphone-with-integrated-thermal-camera 165. 2016. Movidius Brings Artifical Vision Intelligence to FLIR Systems’ Latest Thermal Imaging Product. Movidius. Retrieved December 3, 2016 from http://www.marketwired.com/press-release/movidius-brings-artificial-vision- intelligence-flir-systems-latest-thermal-imaging-product-2115618.htm 269