ABSTRACT Title of Document: GENERATING AND USING EVIDENCE FOR PROGRAM DESIGN: LESSONS FROM EVALUATIONS IN PAKISTAN AND PERU Mario G. Picon, Ph.D., 2015 Directed by: Professor Carol Graham, University of Maryland School of Public Policy The design and implementation of development programs is driven by a set of assumptions on development interventions that typically overlook the role of context; particularly social norms and institutions. Moreover, evaluation is still focused in understanding if an intervention works, instead on how to make it work better. This dissertation discusses the evaluation of a marketing intervention in Pakistan, and the evaluation of a participatory development strategy in Peru. Neither intervention produced the expected results. Rather than stopping there, I discuss the reasons behind their lack of effectiveness, specifically looking at elements for program re-design. The first essay discusses the randomized evaluation of a marketing intervention in Pakistan. The hypothesis was that given the positive role models featured in brochures promoting a microfinance product, women would increase their demand for loans. The brochure, however, had a negative effect in program take up among the poorest women. The likely reason behind this: prevailing social norms regarding role of women. The second essay stresses that the randomized evaluation experiment should not be taken as indicating that marketing is ineffective to improve the impact of microfinance in rural Pakistan, and that the role of social norms in microfinance can be internalized and used in the re-design of the brochure along several dimensions. Using theory of change and realistic evaluation approaches, I propose a framework that combines formative and process evaluation to design and pilot alternative marketing intervention in Pakistan. The third essay features the evaluation of the participatory strategy of El Alto, in Peru. This was a study with very limited data and virtually no control of the research team over the intervention. A mix of quantitative and qualitative techniques is used for outcome and intervention evaluation, and the framework presented on the second essay is used to understand why the participatory strategy has not been successful in sustaining participation. Originally an evaluation of small pretensions, it was used as an opportunity to revisit the objectives of the strategy, improve intervention design, and establishing a monitoring system based on administrative data. A case is made for complementary, context-based interventions. GENERATING AND USING EVIDENCE FOR PROGRAM DESIGN: LESSONS FROM EVALUATIONS IN PAKISTAN AND PERU By Mario G. Picon Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park, in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2015 Advisory Committee: Dr. Carol Graham, University of Maryland, Public Policy, Chair Dr. David Crocker, University of Maryland, Public Policy Dr. Philip Swagel, University of Maryland, Public Policy Dr. John Wallis, University of Maryland, Economics Dr. Raj Desai, Georgetown University and Brookings Institution © Copyright by Mario G. Picon 2015 ii Acknowledgements There are numerous people without whom this work would have not been completed. First of all, I thank my advisor and mentor Carol Graham for her support and encouragement during the different stages of the work. Her friendship and guidance were crucial, especially in times of doubt and frustration due to delays in execution and restrictions in access to information around the programs the dissertation is based on. Similarly, I thank all my committee members for their insights and encouragement along the way. Special mention to David Crocker, whose class was the beginning of my interest in participation as a topic, reflected in essay three of this dissertation. A number of individuals made possible each essay. I thank my co-authors Xavier Gine and Ghazala Mansuri at The World Bank for our endless discussions and iterations to understand what was behind the results of the microfinance paper. I learned a lot from them. My colleagues at The World Bank, especially Luis Andres, George Joseph and Jennifer Solotaroff, supported and provided feedback on the links between evaluation and design underlying essay two. The selection committee of the Measuring Institutional Reform in the Americas (MIRA) competition at the Inter-American Development Bank (IDB) believed in the idea behind essay three and provided funding; in particular, Axel Radics from the IDB in Lima Peru provided invaluable guidance. I also thank Fernando Prada and the team of FORO Nacional Internacional in Peru for their logistic support and diligent work systematizing the information from neighborhood meeting minutes and citizen demands since 2010; Mayor Zarate, and the personnel of the Municipality of El Alto for their openness to the idea of going beyond what was originally requested from us iii in the evaluation, for their availability and making the research team feel at home at all times. Finally, to my entire family in Washington DC, Lima, New Jersey, Addis Ababa and Portland for their endless encouragement and believing in me. In particular, to my wife and best friend Lauren Ross: on top of her endless support, she helped with fresh eyes in reading and proofing this document. Here is to not getting the “So, are you done with your dissertation?” question ever again. iv TABLE OF CONTENTS Acknowledgements……...……………………………………………………………… ii Table of contents………………..……………………………………………………….iv List of Tables ………………….………………………………………………………..viii List of Figures and Boxes ………………………………………………………………viii INTRODUCTION…………………………………………………………………….. 1 The role of social norms and views of entrepreneurship in the success of microfinance programs in Pakistan…………………………………………………………………… 2 Institutions and social norms when evaluating development interventions: A framework and lessons from the Pakistan case study……………………………………………… 3 We built it, they came. Now what? Participatory Strategies in Municipalities of Peru: The Case of El Alto, Piura …………………………………………………………………. 5 ESSAY ONE: The role of social norms and views of entrepreneurship in the success of microfinance programs in Pakistan…...…………………………………………. 8 I. Introduction……………………………………………………………………. 8 II. Context and the Field Experiment……………………………………………... 14 II.1. Access to Credit in Pakistan ………………………………………………. 14 II.2 Description of the field experiment ……………………………………... 15 II.3 Rationale of the intervention and theory of change ………………………...19 III. Data ……………………………………………………………………………. 21 IV. Empirical Approach …………………………………………………………… 28 V. Results …………………………………………………………………………. 30 VI. Conclusions ……………………………………………………………………. 37 VI.1 On the use of Marketing tools to increase demand among women ………. 37 VI.2On interpreting the results of the ‘Money or Skills’ experiment …………. 38 VI.3 On the design of microfinance interventions …………………………….. 39 Appendix 1. Selected Results of the “Money or Skills” Experiment ………………….. 41 Appendix 2. Brochure design………………………………………………………...… 43 v Glossary………………………… ………….………………………………………….. 44 Bibliography………………………………….……………………………………….... 46 ESSAY TWO: Institutions and social norms when evaluating development interventions: A framework and lessons from the Pakistan Case Study ......……… 53 I. Motivation and Research Relevance ……………………………………………….... 53 I.1The aid effectiveness debate and the rise of randomized evaluation ……….. 54 I.2 Complexity theory and development programs ………………………….... 57 I.3 The evaluation function in a complex world ……………………………….. 60 II. Theoretical Framework…………………………………………………………….... 62 III. Analytical Framework……………………………………………………………… 66 III.1 From intervention to mechanisms: Looking at formal and informal institutions……………………………………………………………………… 66 III.2 Triggering Mechanisms: Change in a complex world……………………. 68 III.3 Evaluation: From Intervention to Outcomes………………...……………. 71 IV. Applying the Framework: Microfinance Experiment in Pakistan…………………. 74 IV.1 Understanding the Experiment’s Results…………………………………. 75 IV.1.1 Context Analysis..………………………………………………. 77 IV.1.2 Agents’ Behavior and Mechanisms……..…….………………… 78 IV.2 Using analysis for the next evaluation wave……………………………… 79 IV.2.1 Informing intervention design………………………….……….. 79 IV.2.2 Stages of Program Evaluation………..…………………………. 80 V. Epilogue: The Way Forward in Design and Evaluation….…………………………. 83 Appendix 1. Steps involved in undertaking Realistic Evaluation………………………. 86 Appendix 2. Common M&E Tools……………………………………………………... 87 Bibliography……………………………………………………………………………. 88 ESSAY THREE: We built it, they came. Now what? Participatory Strategies in Municipalities of Peru: The Case of El Alto, Piura…………………………………. 96 I. Introduction…………………………………………………………………………... 96 I.1The Participatory Development debate…………………………………….... 97 I.2 Evaluating participation…………………………………………………….100 vi I.3 The value of participation…………………………………………………. 102 I.4 Quality and potential of participatory mechanisms……………………….. 103 I.5 Participation and Local Governance………………………………………. 103 I.6 Organization of the Essay…………………………………………………. 104 II. Citizen Participation in Peru, and the Context of El Alto, Piura…..……………….. 104 II.1 Local participation and local government in Peru………………………... 104 II.2.Local participation in the context of El Alto –Piura………..…………….. 108 II.2.1 El Alto’s population is relatively homogenous…………………. 109 II.2.2 Limited social development, but low levels of poverty characterize El Alto…………………………………………………………………. 110 II.2.3 Lower investment in public works…………..………………….. 110 II.2.4 El Alto features regular political conflict and challenges to Mayor’s legitimacy…………………………………..………………………….. 111 II.2.5The new participatory strategy is detached from existing channels of engagement……………………………………………………………. 112 III. Evaluation Framework………………………………………………………… 114 III.1 Strategy as a special unit of analysis for evaluation……………………... 114 III.2 Research questions……………………………………………………….. 115 III.3 Participatory Strategy of El Alto and its Objectives……………………... 116 III.4 Data………………………………………………………………………. 118 IV. Empirical Strategy and Theory of Change………………………………………... 121 IV.1 Evaluation of Immediate Objectives……………………………………...121 IV.2 Evaluation of Strategy’s Implementation………………………………... 123 IV.3 Theory of Change and Process Mapping………………………………... 126 IV.4 Informing strategy improvements, and the evaluation of them……….…. 129 V. Results of the analysis……………………………………………………………... 129 V.1 Evaluation of Immediate Objectives……………………………………… 129 V.2 Evaluation of Strategy’ Implementation………………………………….. 135 V.2.1 Perspective vs. Patterns: From Intended to Realized Participation Strategy…………………………..……………………………………. 135 V.2.2. Alignment of different players to the Perspective……………... 137 vii V.2.3 How the Municipality increased the number of meetings…...…. 140 V.2.4 How the Mayor’s presence in meetings and the filing of meeting proceedings was decided………………………………………………. 143 V.2.5 Implementation of Neighborhood Committee election……...…. 147 V.2.6 Implementation Evaluation: General observations…………...… 148 V.3 Theory of change and process mapping………………………………...… 148 V.3.1 The strategy is incomplete………………………………….…... 151 V.3.2The strategy lacks adequate targeting……………………….….. 152 V.4 Piloting supplementary interventions and evaluation plan………………. 152 V.4.1At the wider end of the funnel: Improving participation…….…. 152 V.4.2 At the narrow end of the funnel: processes at the Municipality...154 VI. Conclusions: El Alto Participatory Strategy…………………………………….… 155 Appendix 1: List of Public Works initiated by Municipality of El Alto 2010-2013….. 158 Appendix 2: An evaluation framework for the revamped participatory strategy..……. 159 Bibliography..…………………………………………………………………………. 162 viii LIST OF TABLES ESSAY ONE: The role of social norms and views of entrepreneurship in the success of microfinance programs in Pakistan Table 1. Summary Statistics of Variables used in the Analysis……………………….. 23 Table 2. Baseline Member Characteristics of Mixed vs. Single Gender COs…………. 25 Table 3. Borrowing and Reasons for Not Borrowing, by Credit Source…………….… 26 Table 4. Verification of Randomization…………………………………………….…. 27 Table 5. Uptake of Large Loan among those exposed to the Brochure…………….…. 31 Table 6. Heterogeneous Effects of Larger Loan Uptake among those exposed to a brochure………………………………………………………………………….…….. 35 Table 7A. Loan Size, All Members…………………………………………….……… 36 Table 7B. Loan Size, Business Owners……………………………………….……….. 37 ESSAY TWO: Institutions and social norms when evaluating development interventions: A framework and application to the Pakistan Case Study Table 1. Borrowing and Reasons for Not Borrowing (Jan. 2002-Nov. 2006)………… 79 ESSAY THREE: We built it, they came. Now what? Participatory Strategies in Municipalities of Peru: The Case of El Alto, Piura Table 1. Analysis of Immediate Objectives pursued by the El Alto Municipality…… 130 Table 2. El Alto: Characteristics of People Attending Neighborhood Meetings…….. 133 Table 3. Analysis of Implementation of Mayor’s Visits and Filing of Meeting Proceedings…………………………………………………………………………... 145 ix LIST OF BOXES and FIGURES ESSAY ONE: The role of social norms and views of entrepreneurship in the success of microfinance programs in Pakistan Figure 1. Areas in Pakistan where research was undertaken (in blue)…………………. 16 Figure 2. Timeline of the Experiment………………………………………………….. 16 Figure 3. The Marketing Experiment’s Theory of Change…………………………….. 20 ESSAY TWO: Institutions and social norms when evaluating development interventions: A framework and application to the Pakistan Case Study Figure 1. A Portfolio of Results Framework…………………………………………… 59 Box 1. Basic Definitions used in this Essay……………………………………………. 64 Figure 2. The Context – Mechanism – Outcome (CMO) Hypothesis…………….……. 65 Figure 3. From Context to Mechanisms………………………………………………... 68 Figure 4. The Funnel of Attrition………………………………………………...…….. 71 Figure 5. C-M-O Hypothesis Pakistan Case Study……………………………...……... 75 ESSAY THREE: We built it, they came. Now what? Participatory Strategies in Municipalities of Peru: The Case of El Alto, Piura Box 1. Civil Society Failures: Bonding and Bridging…………………………………. 99 Figure 1. El Alto: Geographical location in the Region of Piura, Peru……..………… 108 Figure 2. El Alto: Characteristics of Population by Size of Neighborhood (over and under 100 inhabitants)……………………………………………………………………..… 110 Figure 3. El Alto: Public Investment (2010-2013) vs. Population (2012)………….… 111 Figure 4. An evaluation framework for tracking Strategy……………………….…… 115 Figure 5. The Context – Mechanism – Outcome (CMO) Hypothesis……………..…. 127 Figure 6. Mapping the Participatory Development Strategy…………………….…… 128 Figure 7. El Alto: Average Meeting Attendance by Month…………………….……. 132 Figure 8. El Alto: Total Attendance to Neighborhood Meetings per Month………… 132 Figure 9. El Alto: Average Age of Neighborhood Meeting Attendants……………... 134 x Figure 10. El Alto: Participatory Strategy’ Timeline based on Actual Implementation (January 2010- December 2013)……………………………………………………... 141 Figure 11. Correlation between Start of Public Works projects in El Alto and Number of Neighborhood Meetings……………………………………………………………… 143 Figure 12. C-M-O Hypothesis Peru Case Study……………………………………... 149 Figure 13. El Alto Participatory Strategy…………………………………………….. 151 1 Introduction In development practice, it is not uncommon to observe technically sound and evidence-backed interventions underperform in terms of development results in a given context, even with negative unintended consequences. Over the years, I have been looking for a better understanding of the factors- internal and external to programs- that improve the probability of achieving desired development outcomes. Hence my professional interest in evaluation. This dissertation is based in projects I have been working on for the past few years: the evaluation of microfinance programs in Pakistan, the assessment of participatory development strategies at the municipal level in Peru, and the evaluation framework for different projects at the World Bank South Asia Region. None of the essays that comprise this work is a traditional self-contained academic paper with a standard hypothesis, framework and discussion of results. Two of them in particular tell the story behind specific interventions designed as part of two very common strategies in development: access to finance for the poor, and participation to identify local priorities and demands. The evaluation of these interventions was pursued with very different methods and in very different settings: a randomized control trial (RCT) in rural Pakistan with significant budget and significant control of the research team over implementation; and a mixed methods evaluation in a small district in Peru with virtually no control of the evaluator over implementation. The other essay proposes a framework to understand and internalize those elements around a development program that an evaluation normally tries to control for; and use that information for the design of alternative interventions. 2 The role of social norms and views of entrepreneurship in the success of microfinance programs in Pakistan Microfinance is arguably one of the most researched and scrutinized development interventions around the world. As much has been said about its potential for bringing households out of poverty as has been discussed about its shortcomings. In fact, a recent debate on the effectiveness of microfinance in South Asia focused on the promises of microfinance proponents, with calls from academic and NGO quarters to stop over- claiming its potential benefits, and to better understand the context in which microfinance is implemented, together with the incentives of borrowers and lenders (Banerjee et al., 2009; Karlan and Mulainathan, 2010). Assessing those claims has increasingly relied on impact evaluations. In particular, RCTs have been used to evaluate variations and add-ons to microfinance programs: from access to credit itself to business training or insurance mechanisms; and their effects on a number of outcomes: from loan uptake, to income generation, to empowerment and intra-household gender relations. This paper discusses the effect of a marketing experiment in rural Pakistan on loan take up and loan size, in which two different versions of a brochure informing of a new large loan program were randomly assigned among microcredit clients organized in credit groups. One brochure design showed five businesses managed by men; the other design showed the exact same businesses but managed by women. The hypothesis of the experiment was that given the positive role models featured in the brochures, women would increase their demand for loans under the large loan 3 program. An increase in loans take up by women was of particular interest to the microfinance organization, given the prevalent low demand for loans of women in microfinance schemes. As the essay explains, the results of the randomized evaluation were puzzling at first, but a more detailed look at context surrounding participant villages provided the key for understanding the results. This had consequences not only for the marketing experiment, but for the larger experiment it was embedded in, focused in the theoretical debate between “Money or Business skills” as constraints to entrepreneurship. The findings of the marketing experiment in fact cast serious doubts on the most fundamental assumptions behind the focus on access to credit and business skills in the microfinance literature. Institutions and social norms when evaluating development interventions: A framework and lessons from the Pakistan case study While the marketing experiment generated information that is relevant and academically interesting; I contend first that the results could have been foreseen with a closer look at context in rural Pakistan that informed a better design of the brochures used, or of the way they were introduced. I emphasize that the conclusion of the marketing experiment is not that marketing does not work in rural Pakistan to increase women’s participation in microfinance. The experiment was based on a particular brochure design based on experience elsewhere, and what the brochure promoted was not microfinance in general, but a particular program offering larger than normal loans. 4 In fact, the overall microfinance program was designed along the lines of a theoretical debate and strong assumptions on the demand and supply of credit in rural Pakistan, with little attention paid to context at the time of design. The evaluation, while using a ‘gold standard’ like RCT, provides limited assistance in informing intervention adaptation, in part because the RCT was strictly used to only determine attribution, without looking much at the future of the program. This is a reflection of the emphasis in evaluation as a point of arrival instead of a point of departure, and its lack of integration with design and implementation. The proposed framework, drawing from the most recent literature on realistic evaluation, experimentation, and experiential learning, offers a systematic look at the institutional settings (formal and informal) surrounding program implementation, and how they affect the behaviors that a program looks to trigger in the target population. It then shows steps to build alternative intervention designs tested through formative and implementation evaluation even before a full-fledged impact evaluation is deployed. I argue that such approach provides for results that are more policy-oriented in a more cost-effective manner. The essay is an explicit call to subordinate and adapt the evaluation method to the development problem at hand and the complexity of the context faced; but also to better connect evaluation to implementation by testing alternative intervention designs (or ‘crawling the design space’). The model is applied to Pakistan’s marketing experiment. 5 We built it, they came. Now what? Participatory Strategies in Municipalities of Peru: The Case of El Alto, Piura The effectiveness of participatory development programs has been the focus of another of the recent debates in the development literature. Much has been discussed on the expected results of participation along the lines of improving service delivery and fostering a sense of empowerment among citizens; as well as disappointment surrounding participation in light of elite capture, little reflection of citizen views in service delivery, and limited impact on development outcomes. In fact, most impact evaluations around this topic seem to find rather limited impact of participation on human development indicators. Instead of concluding that participation does not work, I argue that underachievement in participatory programs is linked to: - A disconnect between the rationale for participation and the concrete program actions and objectives set for it. - Implementation problems consequence of an intervention design that ignores context and institutional settings. - A focus on the demand side of participation, instead of a strategy complemented with supply-side interventions. To illustrate this, I discuss an evaluation I had the opportunity to lead in El Alto, Peru. The original scope of the evaluation requested by the municipal government was very limited, with number of people attending neighborhood meetings the core indicator of success. The essay tells the story of how the research team was able to turn a retrospective evaluation with very limited data into an opportunity to assist the 6 Municipality of El Alto in redesigning the strategy and its monitoring system. Adapting the framework proposed in the previous essay, the research team showed the Mayor of El Alto and his team that the limited results achieved so far are likely unsustainable without complementing the current strategy in key areas. Thus, the recommendations include changes to the design of El Alto’s participatory strategy with interventions adapted to context, and a restructured monitoring system of its participatory program based on the systematization of information already available. The challenges faced during the evaluation are taken not as limitations to the analysis, but as valuable information about the internal processes of the implementing agency (the local Municipality) and of the political context, and how these affected implementation, vis-à-vis what was originally planned. As a matter of fact, the mixed- methods evaluation of objectives and implementation reaches different conclusions depending of the time frame used in the analysis, based on political events in El Alto.. I conclude that the participatory strategy of this small city in the north of Peru, while creating a mechanism for closer interaction between local government and citizens in times of political upheaval, lacks response mechanisms to deal with a growing flow of demands and concerns from citizens, something that compromises the overall strategy and its sustainability. The strategy evaluation identifies key elements along the participatory process that need to be reinforced in order to strengthen the short term outcomes pursued by the Municipality, but also to increase the likelihood of realizing the instrumental and intrinsic values of participation that are the focus of the academic debate. 7 Taken together, the essays point how the introduction of widely accepted development strategies such as microfinance and participation, even with the support of international organizations or local authorities, fail to fulfill the expectations they generate due to the little attention paid to local context at the time of design and implementation. Recent reviews of microfinance programs (Banerjee, Karlan and Zinman, 2015) and participatory programs (Mansuri and Rao, 2013) reveal a lack of evidence of their transformative effects. However, instead of stopping at reporting effects, evaluation can and should be used to strengthen intervention design and implementation. The framework herein presented hopes to be a contribution to the integration of evaluation to the design and implementation processes. 8 Essay One: The role of social norms and views of entrepreneurship in the success of microfinance programs in Pakistan1 “Microcredit and other ways to help tiny businesses have an important role to play in the lives of the poor, […] but we are kidding ourselves if we think these businesses can pave the way for a mass exit from poverty” Banerjee and Duflo, 2011 I. Introduction The first essay of this dissertation tells the story of an experiment designed only as a complement to a larger microfinance study and examining more closely gender- specific constraints to entrepreneurship in rural Pakistan. This evaluation shows how development interventions that are transposed to a new place without major consideration of local context, or those designed purely on an academic debate, can result in disappointing outcomes and conclusions with limited policy potential. Microfinance is arguably one of the most researched development interventions. It is strongly rooted in economic theory: in fact, the debate on the effectiveness of microfinance programs to get people out of poverty is centered in two main arguments that have their origins several decades back. Schumpeter (1942), Baumol (1968) and Solow (1975) were among the first to highlight the role of entrepreneurship in growth and economic development. The 1 The field experiment that this analysis is based on took place in the November 2006- December 2008 period with all data (including repayment outcomes) available by late 2009. The author oversaw the analysis of all the proceedings from the field and co-authored one of two peer reviewed papers from the data: Gine and Mansuri (2014) and Gine, Mansuri and Picon (2012). I thank the Development Research Group of The World Bank and my co-authors Xavier Gine and Ghazala Mansuri for allowing the use of the data related to this field experiment. All views expressed are mine and do not reflect the views of The World Bank or my colleagues; and any mistakes remain of my exclusive responsibility. 9 literature of microfinance often refers to their work when making a case for this type of programs: if entrepreneurship is the key to development, the constraints to entrepreneurship in developing countries must be identified and removed. One constraint in the literature is access to finance. Distortions in capital markets have been argued as affecting business creation and survival from Blanchflower and Oswald (1984) to more recently de Mel, McKenzie and Woodruff (2008) or Banerjee et al. (2010). Microfinance practitioners, including the pioneer of microfinance and founder of the Grameen Bank, Muhammed Yunus, justify microfinance under the premise that “giving the poor access to credit allows them to immediately put in practice the skills they already know” (Yunus, 1999). However, there is another strand of literature, building around the occupational choice models of Lucas (1978) that focuses on the role of business skills (called more generally managerial capital) as key for economic development. Research by Van Reenen (2010), Bruhn, Karlan and Schoar (2010) and Schoar (2010) specifically focuses on the role of business skills in reducing restrictions to entrepreneurship in microfinance programs. How exactly do business skills and access to credit generate economic development as part of a microfinance program? The literature around microfinance is in fact very rich in terms of intermediate development outcomes that can be achieved through such programs, from individual-level (such as empowerment, use of time, outlook of life), to household-level outcomes (income, expenditures, assets, educational attainment of children) and business-specific outcomes (sales, business creation or business practices and knowledge). Further, the claims of the potential benefits of 10 microfinance are so wide-ranging that they have triggered a debate on their accuracy (see Duvendack et al. 2011 for a systematic review). In South Asia, there have been calls from academic and NGO quarters to stop over-claiming microfinance’s potential benefits, and to better understand the context in which microfinance is implemented, together with the incentives of borrowers and lenders (see Banerjee et al., 2009; Karlan and Mulainathan, 2010). In short, the many theories surrounding microfinance are in general appealing, at times well justified with references to classical economic theory as previously discussed and in other cases, heavily relying on anecdotal evidence or practical experience. However, even when microfinance is found to have average positive and significant effects, it is worth asking not only which of these many pursued outcomes are being achieved, but also for whom does microfinance seem to be working, and why. In restricting the debate to access to credit and business skills, microfinance programs implicitly assume that individuals in credit groups2 (commonly referred as self- help groups) are entrepreneurs-in-waiting, or in the case of those already owing a business, eager to expand their operations. It may well be the case for some; however, for others, starting or scaling-up a business or operation is not an aspiration given the size of the market around the entrepreneur. In addition, fear of not being able to repay the loan prevents some households from participating in the program3. 2 Microfinance organizations typically use group-lending practices: they lend to groups of members instead of individuals to seize economies of scale, reducing the costs the costs associated with monitoring loans and enforcing repayment The loan to one participant in group-lending typically depends upon the successful repayment from another member, thus transferring repayment responsibility off of microcredit institutions to loan recipient. 3 See Chapter 9 Banerjee and Duflo (2011) for a discussion on the limits of microfinance and ‘reluctant entrepreneurs’. 11 Field et al. (2010) was among the first known publications to dig into the possibility of a ‘third constraint’ to entrepreneurship in microfinance programs, one that particularly affects women. Their research in rural India indicates that even when credit is available, women’s demand for it can be low because of social norms and traditional institutions that limit their mobility and participation in economic life. With limited business opportunities for them, it makes less sense to take up loans, and when they take one, it is a small one on average. Context of the Marketing Experiment The overall research design that generated the data for this study was focused in providing evidence on the ‘Money or Business skills’ debate in rural Pakistan, in coordination with NRSP, the second largest microfinance organization in Pakistan. The evaluation design assumed interest of participants in becoming entrepreneurs or expanding their business, since all participants were members of credit groups for some time. The initial analysis of the expansion of credit and business training provided by the program over a wide range of proposed outcomes indicated virtually no effects across the board for women, a result that admittedly puzzled the research team4. Alternative hypotheses for why women failed to realize the benefits of microfinance were widely discussed among the team.5 The key insights, however, came from a side-experiment that had been added to the original plan in 2006, inspired by the literature on behavioral change through marketing. This essay describes the result of this marketing experiment and how it relates to the larger study. 4 See Appendix 1 for a summary of results from the “Money or Business Skills” experiment. 5 Note that at the time of this initial analysis of the data, Field et al. (2010) was not yet published. Most alternative hypotheses explored at the time were focused on why female-led businesses were of lesser quality or profitability, particularly focused in the educational and use of time links, and using occupational choice models (Lucas, 1978; Emran, Morshed and Stiglitz, 2007) as a theoretical foundation. 12 To promote a loan product that offered the opportunity of borrowing up to four times the size of a regular loan, a marketing tool was designed to stimulate participation, particularly from women. In coordination with the microfinance institution, two different brochures provided information about the characteristics of the program and described the application process. The brochures were identical, except for the cover page: one featured 5 different businesses with men operating them, while the other had the exact same five businesses, but with women entrepreneurs. Groups of borrowers invited to participate in the loan program were randomized to receive either the brochure with the female or the male pictures. The impact of marketing on individual or group behavior has been well- documented (see, for example, La Ferrara et al., 2008; Jensen and Oster, 2007 and Paluck, 2009). However, the relationship between social marketing and microfinance has not been very much explored, with Bertrand et al. (2010) the immediate precedent. They analyzed a direct mail field experiment among microfinance clients in urban South Africa that randomized advertising content, loan price, and loan offer deadlines simultaneously; and found that advertising influenced behavior (particularly among men) when the treatment appealed to intuition (such as with a picture) as opposed to reason (like a comparison of interest rates across lenders). Based on this precedent and the ‘affinity’ proposition (Evans, 1963; Mobius and Rosenblat, 2006), the hypothesis for the Pakistan study was that women would be more responsive to flyers showing women managing businesses, while men would be more responsive to flyers showing men managing the same businesses. 13 The results from the marketing experiment featured in this essay, however, show a rather different kind of affinity. I argue that the results reflect social norms regarding women’s role in society and the household, which in turn affect take up of the loan product. Thus, the negative effect of female brochures among women business owners with low decision making power in the household, is likely mirroring the reaction to the female brochure of the male household head. This is consistent with men’s observed low regard for females as business owners6 in Pakistan. The results of this marketing experiment provided key information to the larger “Money or Business Skills” study. In that experiment, decision making in the household was one of the many outcome indicators explored; after the marketing experiment analysis, decision making became a key stratification variable for that analysis. The microcredit movement that has developed in past couple of decades is eminently woman-centered, because it is widely accepted that women may be the ones most in need of credit , and the ones in the household that are likely to invest more of the proceedings of the business in their household. In Pakistan, however, microfinance has focused primarily on men, precisely because the restrictions to women’s involvement in economic activity are known. One of the justifications of the overall microfinance program was increasing women take up of microfinance loans, particularly among the poorest households. This objective was not achieved. In the case of the brochures, while well rooted in theory of providing information and positive role models, they failed to consider in their design the low decision-making power of their female members, a reality that was widely known before the experiment 6 Beaman et al. (2009) discuss the issue of male attitudes toward women participation in non-traditional roles for the case of perceptions about female politicians. 14 started among NRSP field workers and practitioners in Pakistan. Among the recommendations of this essay, it is argued that other interventions, especially targeted to women (or to men) may need to be tested, such as campaigns to change stereotypes or specific incentives that stimulate not only take up, but that actually improve more structural or societal outcomes for female entrepreneurs, generating a signal or demonstration effect that challenges the prevailing stereotypes of gender roles. The rest of the essay is structured as follows. Section II describes the context in Pakistan and the field experiment. Section III discusses the data and Section IV describes the empirical strategy and the results of the experiment. Section V concludes with some recommendations and lessons from the research process that can be useful for future work in Pakistan and elsewhere. II. Context and the Field Experiment II.1. Access to Credit in Pakistan Over 60% of Pakistan’s 162 million population lives in rural areas. Even though agriculture remains the most important economic activity for people in rural areas of the country, about 45% of the rural poor also rely on non-farm activities as one of their income sources7. This rural population of Pakistan has quite limited access to financial services and credit, despite a significant growth of the country’s financial system and modernization of its regulatory framework8. According to Nenova et al (2009) the financial system only 7 Source: World Development Indicators database, April 2009 and World Development Report 2008. 8 Even considering urban and rural areas together, only 14 percent of households interviewed reported using a financial product or service (including savings, credit, insurance, payments, and remittance services) from a formal financial institution. When informal financial access is taken into account, however, this figure rises to just over 50 percent. In comparison, 32 percent of the population has access to the formal financial system in Bangladesh, and this figure amounts to 48 percent in India and 59 percent in Sri Lanka (World Bank, 2008). 15 reaches 15% of farmers in Pakistan. Financial access is the lowest among the poor, women, and small and micro enterprises in rural areas, even though market studies consistently suggest they are viable customers. Overall, rural firms and households account for about 7% of total credit disbursement (about Rs 130.7 billion in 2008) and the bulk of this is for agricultural finance (Rs 108.7 billion the same year), including both farm and nonfarm credit (see Akhtar, 2008). While microcredit volumes are skewed towards rural areas, microcredit currently accounts for only 17% of total rural credit and serves some 1.7 million clients. Comparative rates of microfinance penetration in the South Asian region are 35% in Bangladesh, 25% in India, and 29% in Sri Lanka. Another key characteristic of the microfinance sector in Pakistan is that unlike most countries, it has focused primarily on men rather than women on the grounds that there is less demand for credit from women given their low mobility levels and gender segregation in labor markets. II.2 Description of the field experiment The field experiment was conducted in the districts of Bahawalpur, Hyderabad, and Attock, spanning different agro-climatic regions of Pakistan9. Figure 1 shows the location of the study districts. A baseline survey was conducted in November 2006 in a sample of 747 credit groups (COs), selected so that their membership was between 5 and 26 members. The resulting sample consisted of a total of 4,161 members interviewed, and 2,284 members (54.9%) that were in good standing (eligible for credit)10. The timeline of the experiments is presented in Figure 2. 9 The branches considered for the study are as follows: Matiari and Tando Muhammad Khan in Hyderabad, Attock in Attock and Bahawalpur in Bahawalpur. 10 NRSP staff conducted a complete listing of the occupation of CO members in the study branches to identify those who were engaged in non-farm activities. The original sampling framework included all CO members that according to 16 Figure 1. Areas in Pakistan where research was undertaken (in blue) Source: National Rural Support Programme (NRSP) Figure 2. Timeline of the Experiment Source: Author, based on field experiment information Using data from the member’s registry, COs were randomly allocated into two groups, one of which was assigned to receive business training. Training sessions were held, from February to May 2007, with each session lasting for 6 to 8 days. After the listing exercise had a non-farm business and five other members selected at random from each CO. In practice, however, enumerators ended up interviewing everyone that attended a special CO meeting that was called to conduct the baseline survey. 17 completing the business training sessions, members from all study COs were invited to an orientation meeting that introduced the possibility of borrowing a larger loan amount11. Orientations occurred successfully in 596 COs.12 During the orientation meeting, 3,451 members who were in good standing (i.e., those who had successfully repaid at least one loan on time) received one of two versions of the marketing brochure. The brochure was identical in all respects except one. In one version, the entrepreneurs manning the businesses were male while in the other they were female. To ensure that only the gender of the businessperson differed between both versions of the brochure, the exact same business was photographed twice, the first time with a man as owner and the second with a woman. The businesses in the brochure were chosen to be representative of the type of businesses typically run by NRSP micro entrepreneurs.13 All members of a CO were given one of the two brochures, which were randomly allocated across COs. Randomized evaluation was the method of choice for the analysis as it reduces or can even eliminate selection bias in the results, increasing internal validity, and therefore, allows the results to be presented in terms of attribution (Duflo, Glennester and Kremer, 2006). The goal of the brochure was to explain how to apply for a larger loan (for a picture of the brochure, see Appendix 2). According to this, all eligible members could 11 Most orientation sessions took place in regularly scheduled CO meetings and lasted for about an hour and a half. Attendance at these sessions was high, with more than 90 percent of members attending. Message consistency during the orientation was maintained by providing training to all NRSP credit officers and other staff who were in charge of delivering the orientations. There were 12 teams of two NRSP staff each in Attock, 29 in Bahawalpur and 7 in Hyderabad. 12 In the remaining 151 COs, orientation meetings could not be held because the CO had either disbanded or was newly formed so that none of its members were eligible for the experiment. 13 The brochure thus contained two agribusinesses, two retail businesses and one tailoring business. According to baseline data, about 49% of male businesses were agribusiness, 27% were in retail and 9% were involved in handicrafts and tailoring, thus accounting for almost 85 percent of all male businesses. Among female businesses, 20% were agribusinesses, 18% were retail businesses and 57% were in handicrafts and tailoring, accounting for almost 95 percent of all female businesses. 18 make a loan request of up to Rs. 100,000. The request was subject to all the usual technical and social reviews conducted by NRSP credit officers, who could also determine the loan amount they were willing to approve for each borrower. Approved loans which were larger than the usual limit of Rs. 30,000, were to be forwarded to headquarters, for final decision. 14 Although the brochure encouraged members to borrow for productive purposes, in practice there were no restrictions on the use of the loan15. Eligible CO members had seven months, from November 2007 to June 2008, to apply for the larger loan. Of the 2,284 eligible CO members, 713 (31%) applied. NRSP approved 532 loans (75% of applications). Most applicants had their loan amounts reduced; credit officers reported that this was due to concerns that borrowers would not be able to make the required monthly installments16. A follow-up survey was conducted in December 2008. This was six months after the loan lottery concluded and about 13 months after the loan orientation meetings. In the follow up, some 45% of eligible CO members recalled attending the loan lottery orientation meeting. Among those who recalled attendance, about 70% recalled receiving a brochure and of these about half were able to correctly recall the picture they were shown. 14 The final decision on lending the money to approved applicants depended on the result of a lottery held at NRSP headquarters to assign all eligible members as large loan lottery winners or losers. Lottery winners could borrow the approved amount, while those who lost the lottery could borrow up to their regular loan size. The lottery was designed so that the chance of winning was 50 percent. 15 Qualifying members who already had an outstanding loan with NRSP were even allowed to apply for the larger loan, subject to the condition that part of the new loan would be used to pay off the outstanding debt. 16 Of the customers approved, 254 were assigned to win the lottery (48%) and 211 ended up borrowing (83%). Among the 278 loan applicants not selected through the lottery, only 161 borrowed (58%). Among the reasons cited for changing their mind are that time elapsed from request to approval (average time was 2 months), and that the new loan size was not too different from the loan they currently had access to. 19 II.3 Rationale of the intervention and theory of change The design of the microfinance experiments was well anchored in recent microfinance literature on business skills, access to finance, and the use of marketing to affect behavior through positive role models (in this case, women leading local businesses). It did, however, rely on a number of assumptions on the nature and depth of the gaps addressed by the interventions17. The marketing experiment (Figure 3) was designed to encourage female uptake of the new microcredit product that allowed eligible borrowers to borrow larger amounts of money. This focus on female uptake is why the analysis is performed in terms of the effect of a female brochure vis-à-vis a male brochure. Alternatively, the combined effect of female brochure and business training is explored. In hindsight, it may well be possible that too much was asked from introducing the brochure: the featured positive role models were expected to be enough to overcome a context of low demand for credit from women; and a microfinance market such as the Pakistani, oriented to men instead of women. Instead of customizing microfinance products to women as a target group, it was assessed if information about the program highlighting positive models sufficed to increase take up and average loan size. Women (and the males in their households and communities) were expected to be more open to female entrepreneurship after the treatment was introduced. Figure 3. The Marketing Experiment’s Theory of Change 17 This section stresses the assumptions behind the design of the intervention and the field experiment itself. Its importance will be highlighted when discussing impact evaluation designs in the other two essays of this Dissertation. 20 Source: Author’s conceptualization Two of the main outcomes of interest include applications for the larger loan program (take up) and the average size of loans requested. The analysis in this essay focuses on immediate impact in terms of demand for credit. However, the assumption behind the intervention is that larger take up of the program will unchain the benefits of microfinance in terms of individual, household and business outcomes. The “Money or Skills” experiment also relied on a few key assumptions. Strictly, the experiment did not compare the role of business knowledge to access to credit, but did so for access to larger loans used for opening a business or scaling up operations in an existing one. Some CO members had been microfinance clients since 2002, with different levels of credit demand over the 2002-2006 period (from 0 to 7 loans). Both business training and access to larger credit were expected to generate improvements in development outcomes, overcoming the same context described for the low demand for credit and prevailing social norms. III. Data NRSP Administrative data and data from baseline and follow up surveys are used for this analysis. In the case of the survey, a wide range of characteristics was collected as 21 baseline data in November 2006 with questions about the CO member, the member’s household, the business if they had one, and the CO. Household characteristics included information on expenditures, wealth (including agricultural land, livestock, housing quality and savings) and past and current debt and savings of household members. Business characteristics included age, location and type of business activity, as well as the scale of the business as measured by its assets, hired workers and monthly sales. The survey also contained information on CO cohesion, including borrowing and lending between members in a CO and the collective purchase and/or sale of products. Summary statistics from the baseline survey are presented in Table 1,18 and variable definitions are provided in Appendix 3. The average age among CO members in the sample is 38 years, with 4.1 years of average education. Households have average landholdings of 4.5 acres and monthly expenditures of Rs 5,200 (US$61). About 60% of the households in the sample run at least one business. This percentage is significantly higher than the population average in the study areas because households with businesses are more likely to be microfinance clients.19 Although most businesses have a fixed location and operate year round, the average scale is small. About 90% of businesses do not have a paid employee. Businesses managed by male and female CO members are quite different. While women are primarily engaged in small home based manufacturing (handicrafts or tailoring), men are involved primarily in the agribusiness sector which requires much 18 Indexes are calculated on the total number of observations for each variable, this is why their overall mean is close to zero. 19 According to the Demographic and Heath Survey conducted in 2006-07, 31 percent of households in rural areas reported having at least one household member engaged in non-agriculture self-employment. Among all the 6,837 microfinance clients in the study COs, roughly 40 percent have a business at the time of baseline. 22 greater contact with markets outside the village The scale at which male and female businesses operate is quite different (see also de Mel et al. 2009): average monthly sales among male-led businesses are Rs 22,820 (USD 380) while only Rs 4,827 (USD 80) among businesses run by female CO members. Women tend to operate mainly from home and are less likely to employ paid employees. In addition, 40 percent of female CO members involved in a business at baseline claim that the main business decisions, such as purchase of inputs, hiring, and marketing are made by their husband. 23 N . O bs M ea n S td . D ev . M al e Fe m al e M al e Fe m al e M al e Fe m al e (1 ) (2 ) (3 ) (7 ) (8 ) (9 ) (1 0) (1 1) (1 2) (1 3) (1 4) (1 5) Ta ke -u p S ho w n Fe m al e br oc hu re ( Y es =1 ) 3, 45 1 0. 52 0. 50 0. 49 0. 56 0. 00 0. 47 0. 57 0. 02 0. 45 0. 58 0. 02 E lig ib le fo r l oa n lo ttt er y (Y es =1 ) 3, 45 1 0. 55 0. 50 0. 62 0. 62 0. 00 0. 69 0. 67 0. 00 0. 71 0. 68 0. 01 O ffe re d B us in es s Tr ai ni ng ( Y es =1 ) 3, 45 1 0. 54 0. 50 0. 52 0. 53 0. 00 0. 55 0. 53 0. 00 0. 54 0. 56 0. 04 A pp lie d fo r l oa n (1 =Y es ) 2, 14 9 0. 31 0. 46 0. 42 0. 18 0. 00 0. 45 0. 17 0. 00 0. 44 0. 22 0. 00 A pp ro ve d, c on di tio na l o n ap pl yi ng (1 =Y es ) 66 4 0. 76 0. 43 0. 76 0. 76 0. 01 0. 78 0. 75 0. 05 0. 81 0. 74 0. 35 B or ro w ed , c on di tio na l o n be in g ap pr ov ed (1 =Y es ) 50 3 0. 69 0. 46 0. 75 0. 52 0. 62 0. 75 0. 55 0. 85 0. 78 0. 63 0. 87 A m ou nt b or ro w ed (' 00 0 R s) 44 5 33 .0 0 16 .9 4 33 .8 3 30 .0 0 0. 06 34 .5 9 29 .5 5 0. 02 36 .0 4 32 .5 0 0. 43 B as el in e C ha ra ct er is tic s I nd iv id ua l C ha ra ct er is tic s Fe m al e (Y es =1 ) 3, 45 1 0. 46 0. 50 - - - - - - - - - A ge 3, 45 1 37 .8 8 12 .1 7 38 .1 8 37 .5 1 0. 01 37 .5 36 .9 7 0. 29 38 .4 9 39 .0 2 0. 98 Y ea rs o f E du ca tio n (0 -1 6) 3, 45 1 4. 09 4. 51 5. 31 2. 63 0. 00 5. 53 2. 73 0. 00 5. 29 2. 65 0. 00 M ar rie d (Y es =1 ) 3, 45 1 0. 83 0. 37 0. 82 0. 84 0. 12 0. 82 0. 86 0. 13 0. 84 0. 87 0. 09 D ig it S pa n R ec al l ( 0- 8) 3, 45 1 3. 31 2. 24 3. 84 2. 68 0. 00 4. 03 2. 74 0. 00 3. 87 2. 48 0. 00 M em be r o f a M ix ed G ro up (Y es =1 ) 3, 45 1 0. 06 0. 24 0. 04 0. 09 0. 00 0. 05 0. 09 0. 00 0. 03 0. 13 0. 00 In de x of F em al e M ob ili ty 1, 57 1 0. 06 1. 35 - 0. 06 - - 0. 12 - - 0. 15 - In de x of N o P ur da h 1, 57 1 0. 15 1. 62 - 0. 15 - - 0. 13 - - 0. 37 - B us in es s O w ne r ( Y es =1 ) 3, 45 1 0. 60 0. 49 0. 61 0. 58 0. 01 - - - - - - R is k To le ra nc e (0 =R is k A ve rs e; 1 0= R is k Lo ve r) 3, 45 1 3. 61 3. 03 3. 81 3. 37 0. 00 3. 84 3. 39 0. 00 4. 08 3. 59 0. 00 In te re st in T ra in in g (Y es =1 ) 3, 45 1 0. 57 0. 49 0. 64 0. 49 0. 00 0. 68 0. 52 0. 14 0. 70 0. 55 0. 01 M on th s as C O M em be r 3, 45 1 26 .5 5 23 .9 5 24 .5 7 24 .5 6 0. 00 26 .9 2 22 .1 1 0. 00 28 .3 8 25 .3 5 0. 01 H ou se ho ld C ha ra ct er is tic s Y ea rs o f E du ca tio n, S po us e (0 -1 6) 3, 45 1 3. 31 4. 34 2. 12 4. 75 0. 00 2. 24 4. 66 0. 00 2. 06 4. 53 0. 00 To ta l H ou se ho ld In co m e (lo g) 3, 45 1 11 .5 3 0. 81 11 .6 2 11 .4 4 0. 00 11 .7 1 11 .4 5 0. 00 11 .5 3 11 .6 1 0. 99 E xp en di tu re s (lo g) 3, 45 1 8. 28 0. 63 8. 29 8. 27 0. 00 8. 32 8. 29 0. 00 8. 19 8. 39 0. 40 N um be r o f C hi ld re n un de r 9 3, 45 1 2. 65 2. 17 2. 89 2. 37 0. 00 2. 89 2. 43 0. 00 2. 77 2. 31 0. 01 La nd (a re a) 3, 45 1 4. 48 18 .6 9 5. 94 2. 74 0. 00 5. 84 2. 57 0. 01 4. 99 3. 48 0. 01 C re di t c on st ra in ts ( Y es =1 ) 3, 45 1 0. 14 0. 35 0. 12 0. 16 0. 00 0. 12 0. 12 0. 00 0. 12 0. 13 0. 01 Fa m ily e ve r i n bu si ne ss ( Y es =1 ) 3, 45 1 0. 61 0. 49 0. 61 0. 61 0. 31 0. 75 0. 7 0. 06 0. 74 0. 69 0. 14 D ec is io n M ak in g (0 -8 ) 3, 45 1 2. 65 3. 11 3. 34 1. 84 0. 00 3. 35 1. 59 0. 00 3. 20 1. 79 0. 00 S ou rc es o f C re di t % b or ro w in g Fo rm al S ec to r 2 00 6- 08 2, 93 1 0. 05 0. 22 0. 07 0. 03 0. 01 0. 08 0. 03 0. 02 0. 09 0. 01 0. 01 A m ou nt b or ro w ed 2 00 6 ('0 00 s) 2, 93 1 5. 94 50 .9 2 9. 67 1. 37 0. 08 11 .5 7 0. 85 0. 19 13 .3 5 0. 15 0. 31 % b or ro w in g M ic ro fin an ce In st itu tio ns / M ic ro fin an ce B an ks 2 00 6- 08 2, 93 1 0. 82 0. 38 0. 79 0. 86 0. 00 0. 86 0. 93 0. 00 0. 87 0. 91 0. 00 A m ou nt b or ro w ed 2 00 6 ('0 00 s) 2, 93 1 31 .1 7 26 .3 5 34 .2 3 27 .4 2 0. 00 37 .6 7 28 .7 9 0. 00 39 .6 2 31 .5 4 0. 13 % b or ro w in g Fr ie nd s an d Fa m ily (o th er th an C O m em be rs ) 2 00 6- 08 2, 93 1 0. 22 0. 41 0. 21 0. 23 0. 11 0. 2 0. 22 0. 30 0. 19 0. 22 0. 47 A m ou nt b or ro w ed 2 00 6 ('0 00 s) 2, 93 1 5. 57 18 .7 3 6. 19 4. 81 0. 00 6. 19 4. 78 0. 01 6. 04 4. 61 0. 13 % b or ro w in g In fo rm al L en de rs 2 00 6- 08 2, 93 1 0. 48 0. 50 0. 50 0. 47 0. 16 0. 47 0. 42 0. 27 0. 49 0. 45 0. 20 D eb t t o in fo rm al le nd er s 20 06 ( '0 00 s) 2, 93 1 15 .0 7 68 .8 8 23 .2 1 5. 06 0. 00 26 .4 5 4. 32 0. 00 18 .3 8 4. 73 0. 02 B us in es se s A gr ib us in es s, D ai ry , L ive st oc k (Y es =1 ) 1, 96 2 0. 40 0. 49 0. 54 0. 22 0. 00 0. 54 0. 22 0. 00 0. 56 0. 56 - R et ai l a nd F oo d S er vic es (s ho pk ee pi ng ) (Y es =1 ) 1, 96 2 0. 24 0. 42 0. 28 0. 19 0. 00 0. 28 0. 19 0. 00 0. 29 0. 29 - H an di cr af t, Ta ilo rin g, V oc at io na l T ra de ( Y es =1 ) 1, 96 2 0. 29 0. 46 0. 09 0. 55 0. 00 0. 08 0. 55 0. 00 0. 05 0. 05 - O th er ( Y es =1 ) 1, 96 2 0. 07 0. 44 0. 09 0. 04 0. 00 0. 09 0. 03 0. 00 0. 09 0. 09 - A ge o f B us in es s 2, 06 5 14 .1 0 14 .6 9 16 .3 6 11 .2 2 0. 03 16 .3 5 11 .2 2 0. 03 16 .8 1 14 .6 9 0. 11 A ll sa le s in v ill ag e (Y es =1 ) 2, 06 5 0. 82 0. 38 0. 78 0. 88 0. 00 0. 78 0. 88 0. 00 0. 78 0. 84 0. 01 O pe ra te s al l m on th s (Y es =1 ) 2, 06 5 0. 79 0. 41 0. 80 0. 77 0. 02 0. 8 0. 77 0. 03 0. 77 0. 72 0. 27 B us in es s Li te ra cy 2, 06 5 0. 47 1. 42 0. 68 0. 22 0. 00 0. 79 0. 29 0. 00 0. 75 0. 06 0. 00 S al es (' 00 0 R s) 2, 06 5 14 .5 1 10 3. 18 22 .3 4. 74 0. 01 22 .3 4. 74 0. 00 7. 06 7. 01 0. 48 Ta bl e 1. S um m ar y S ta tis tic s of V ar ia bl es u se d in th e A na ly si s P -v al ue of t- te st (1 3) -(1 4) N ot e: A ll va ria bl es a re fr om th e ba se lin e su rv ey (N ov em be r 2 00 6) , e xc ep t f or S ou rc es o f C re di t i nf or m at io n, w hi ch re ca lls c re di t f or th e 20 06 -2 00 8 pe rio d. P le as e se e A pp en di x 3 fo r v ar ia bl e de fin iti on s M at ch ed B us in es s O w ne rs P -v al ue of t- te st (1 0) -(1 1) B us in es s O w ne rs P -v al ue of t- te st (7 )-( 8) A ll m em be rs A ll m em be rs 24 Entrepreneurs are also in fact different from other CO members. They are more likely to be older, more educated and to report a family history of business ownership. In addition, business owners have better digit span recall20, have a better outlook on life and not surprisingly, also score higher on a business knowledge test. They are also more likely to be risk tolerant. Among women, those that have a business are less likely to report mobility constraints, but are somewhat more likely to observe purdah21, perhaps because they are relatively wealthier. Women are also more likely than men to belong to a mixed gender CO in the sample (about 63 percent of members in mixed COs are women). While this is a small sample overall, given that only 7 percent of COs have both men and women, there are some indicative differences between women in mixed and single gender COs. In particular, the former seem to have less ‘voice’ in the COs they belong to. While the odds of holding office in the single gender COs are about the same for men and women, at 20 and 18 percent respectively, 48 percent of men and only 7 percent of women in mixed COs report holding any office. Women in mixed COs are also significantly more likely to observe purdah ,even though mixed COs are more than twice as likely to be composed of 20 Individuals are asked to recall eight digits after 30 seconds. Evidence from enterprising psychometrics indicates that better digit span recall is linked to selection into entrepreneurship (Djankov et al. 2005). 21 Purdah is the practice among women in certain Muslim and Hindu societies of living in a separate room or behind a curtain, or of dressing in all-enveloping clothes, in order to stay out of the sight of men or strangers. The practice varies in intensity across communities. What is common to the areas of the study is covering head and body, and restrictions to mobility. 25 members of the same zaat (caste)22. Mixed Gender CO/CG Single gender CO/CG Mixed Gender CO/CG Single gender CO/CG (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Individual Characteristics Age 82 1798 36.26 38.27 0.16 137 1434 34.06 37.85 0.00 Own Education 82 1798 7.22 5.22 0.06 137 1434 2.54 2.65 0.61 Business Owner (Yes=1) 82 1798 0.63 0.61 0.66 137 1434 0.59 0.58 0.84 Digit Span Recall 82 1798 3.87 3.85 0.94 137 1434 2.39 2.71 0.12 Risk Tolerance (0=Risk Averse; 10= Risk Lover) 82 1798 4.09 3.80 0.38 137 1434 2.80 3.43 0.02 Married (1=Yes) 82 1798 0.77 0.82 0.01 137 1434 0.83 0.85 0.48 Index of Optimism 82 1798 -0.23 -0.02 0.21 137 1434 -0.87 -0.29 0.00 Business Literacy 82 1798 0.97 0.66 0.07 137 1434 0.14 0.23 0.67 Index Female Mobility 82 1798 - - - 137 1434 0.08 0.00 0.36 Index No Purdah 82 1798 - - - 137 1434 -0.30 0.19 0.00 Trust in Formal System 82 1798 -0.01 -0.03 0.31 137 1434 -0.28 -0.14 0.00 Months as CO member 82 1798 16.48 28.74 0.00 137 1434 20.28 24.98 0.45 Holds office in CO (1=Yes) 82 1798 0.48 0.20 0.00 137 1434 0.06 0.18 0.01 Eligibility 82 1798 0.49 0.63 0.00 137 1434 0.53 0.63 0.00 Household Characteristics Household size 82 1798 7.88 7.93 0.70 137 1434 7.45 7.00 0.00 Years of Education of Spouse 82 1798 3.87 2.67 0.02 137 1434 4.93 4.88 0.91 Number of children under 9 82 1798 1.78 1.97 0.40 137 1434 1.69 1.40 0.05 Land 82 1798 4.38 6.01 0.39 137 1434 4.48 2.58 0.30 Fraction of CO Members of same Zaat (caste) 82 1798 0.41 0.43 0.12 137 1434 0.45 0.21 0.00 Ever in Business (1=Yes) 82 1798 0.60 0.62 0.11 137 1434 0.54 0.61 0.76 Log of Value of Livestock 82 1798 5.77 9.36 0.00 137 1434 6.99 4.78 0.00 Distance to meeting place 82 1798 6.24 7.50 0.07 137 1434 5.91 9.17 0.00 Credit Constraints (1=Yes) 82 1798 0.09 0.12 0.28 137 1434 0.19 0.16 0.31 Expenditures (1,000 Rupees) 82 1798 5.42 4.98 0.91 137 1434 4.86 4.58 0.36 Decision making 82 1798 3.10 3.35 0.71 137 1434 1.95 1.83 0.39 Source: Baseline survey, 2006 N. Obs Single gender CO/CG Means P-val of t-test (3)-(4) N. Obs. Mixed gender CO/CG N. Obs Single gender CO/CG Means Table 2. Baseline Member Characteristics of Mixed vs single gender COs P-val of t-test (8)-(9) Male Female N. Obs. Mixed gender CO/CG Table 3 shows the main sources of credit for sample households at baseline. Only 5 percent of members have any loans from formal financial sources, mainly commercial banks. In contrast, 34 percent report borrowing from informal lenders, mainly shopkeepers, and 21 percent report borrowing from relatives23. Average loans taken from an MFI is only Rs. 12,000 (around USD 140). There is little variation in the relative share of lenders by gender, however, though female CO members tend to borrow less overall. 22 Seclusion practices are much less stringent within zaat/biradari (caste) groups. See Jacoby and Mansuri (2011). 23 Informal lenders also include traders and wholesalers and, to a smaller extent, professional moneylenders and landlords. 26 Table 3. Borrowing and Reasons for Not Borrowing, by Credit Source Commercial Bank MFI Informal Lenders Relatives and Friends Percent borrowing from [source] in 2006 5.07 83.43 33.77 21.12 Do not like to borrow 52.55 81.39 70.81 74.69 Inadequate collateral 18.50 1.98 11.94 6.77 Lender's procedures are too cumbersome 14.49 9.31 6.70 5.82 Lender's loan terms are unfavorable 5.48 0.59 3.35 0.91 Lender is too far away 3.18 0.40 0.78 1.35 Need to pay bribes 2.02 0.79 2.18 0.08 Past default with lender 1.96 0.79 0.89 1.15 CO members not willing to lend to me 1.25 4.55 2.51 6.53 Bad credit history 0.56 0.20 0.84 2.69 The main [reason] for not borrowing from [source]? Credit Source Source: Baseline Survey, 2006 Table 3 also reports reasons for not borrowing for those who reported not using a credit source. Interestingly, most respondents without loans report not a lack of access, but an unwillingness to borrow – either because they have no need for loans or because they dislike borrowing – as the main reason for not taking a loan and this varies little across lenders. Lack of collateral and cumbersome loan application procedures come in next, and are particularly important when dealing with a formal lender. 27 M al e Br oc hu re Fe m al e Br oc hu re M al e Br oc hu re Fe m al e Br oc hu re M al e Br oc hu re Fe m al e Br oc hu re (1 ) (2 ) (3 ) (4 ) (5 ) (6 ) (7 ) (8 ) (9 ) (1 0) (1 1) (1 2) Ch ar ac te ris tic s at B as el in e Ag e 3, 45 1 37 .9 5 37 .8 1 0. 74 1, 88 0 37 .9 8 38 .4 0 0. 89 1, 57 1 37 .9 1 37 .2 0 0. 85 Nu m be r o f c hi ld re n un de r 9 3, 45 1 1. 73 1. 83 0. 77 1, 88 0 1. 92 1. 99 0. 53 1, 57 1 1. 45 1. 40 0. 17 Bu si ne ss O wn er (Y es =1 ) 3, 45 1 0. 60 0. 59 0. 57 1, 88 0 0. 63 0. 59 0. 75 1, 57 1 0. 57 0. 60 0. 79 Di gi t S pa n Re ca ll 3, 45 1 3. 36 3. 28 0. 85 1, 88 0 3. 82 3. 88 0. 83 1, 57 1 2. 72 2. 65 0. 82 Ri sk T ol er an ce (0 =R is k Av er se ; 1 0= R is k Lo ve r) 3, 45 1 3. 71 3. 52 0. 25 1, 88 0 3. 81 3. 81 0. 53 1, 57 1 3. 57 3. 22 0. 49 La nd 3, 45 1 4. 38 4. 58 0. 82 1, 88 0 5. 83 6. 05 0. 77 1, 57 1 2. 36 3. 05 0. 55 Ye ar s of E du ca tio n of S po us e 3, 45 1 3. 69 3. 71 0. 69 1, 88 0 2. 72 2. 73 0. 86 1, 57 1 5. 05 4. 75 0. 55 M em be r o f a M ix ed G ro up (Y es =1 ) 3, 45 1 0. 04 0. 08 0. 10 1, 88 0 0. 02 0. 06 0. 06 1, 57 1 0. 07 0. 10 0. 89 O wn E du ca tio n 3, 45 1 4. 26 3. 95 0. 14 1, 88 0 5. 27 5. 36 0. 78 1, 57 1 2. 84 2. 47 0. 07 De ci si on M ak in g 3, 45 1 2. 74 2. 57 0. 35 1, 88 0 3. 41 3. 26 0. 77 1, 57 1 1. 82 1. 85 0. 61 Pc t. Bo rr ow in g fro m … a t t he ti m e of B as el in e* Co m m er ci al B an k 2, 93 1 2. 38 1. 7 0. 64 1, 61 6 3. 69 2. 71 0. 60 1, 31 5 0. 51 0. 69 0. 74 M ic ro fin an ce In st itu tio n 2, 93 1 71 .3 4 67 .6 2 0. 30 1, 61 6 69 .4 4 63 .7 4 0. 98 1, 31 5 74 .0 6 71 .7 4 0. 46 Fr ie nd s an d Re la tiv es 2, 93 1 7. 71 6. 58 0. 48 1, 61 6 7. 85 5. 16 0. 05 1, 31 5 7. 5 8. 09 0. 14 In fo rm al le nd er s 2, 93 1 0. 7 0. 2 0. 07 1, 61 6 0. 83 0. 26 0. 19 1, 31 5 0. 51 0. 13 0. 23 Pc t. O ffe re d Bu si ne ss T ra in in g 3, 45 1 52 .4 7 51 .8 2 0. 83 1, 88 0 51 .2 9 52 .0 3 0. 93 1, 57 1 54 .1 51 .6 0. 48 M em be r i s el ig ib le fo r l oa n lo tte ry (Y es =1 ) 3, 45 1 63 .7 2 69 .0 0 0. 54 1, 88 0 64 .6 3 59 .6 9 0. 82 1, 57 1 62 .4 5 62 .2 1 0. 66 0. 19 0. 24 0. 14 Ta bl e 4. V er ifi ca tio n of R an do m iz at io n M al e Fe m al e P- va l o f t-t es t (1 1) -(1 2) P- va l o f t-t es t (6 )-( 7) N. O bs . Al l m em be rs M ea ns No te s: * de no te s va ria bl e m ea su re d at fo llo w- up , c on du ct ed in D ec em be r 2 00 8. P ct . O ffe re d Bu si ne ss T ra in in g an d M em be r e lig ib ilit y co m e fro m a dm in is tra tiv e da ta fr om NR SPP -va l o f F -te st th at a ll ba se lin e ch ar ac te ris tic s ar e jo in tly in si gn ific an t M ea ns M ea ns N. O bs . P- va l o f t-t es t (2 )-( 3) N. O bs . 28 Table 4 presents the means of baseline variables for the sample as a randomization check. The difference in means for members receiving the male and female brochure is significant for only two of the 16 baseline variables considered. Study participants who received the male brochure borrow more from informal sources and are less likely to be members of a mixed group (both significant at the 10% level) than members receiving the female brochure. For the sample of males, the difference in means in the two groups is again significant for only two of the 16 variables considered. Male participants who received the male brochure are somewhat less likely to belong to a mixed CO (significant at the 10% level)24 and are somewhat less likely to borrow from friends and relatives (significant at the 5% level). For the sample of females the difference is only significant for own education. Women who received the male brochure appear to have higher formal education by about one third of a year. Table 4 also reports the p-value of an F-test that all baseline characteristics are jointly insignificant. The hypothesis cannot be rejected in any of the three samples (lowest p-value is 0.58), which strengthens the argument of successful random assignment. IV. Empirical Approach Because the type of brochure is assigned randomly at the CO level, its impact can be estimated via the following regression equation: Yij = α + βFBj + ɣXij + εij (1) where Yij is the outcome of interest for individual i in CO j (e.g., and indicator variable that takes the value 1 if the respondent had applied for a larger loan, and 0 otherwise), FBj is an indicator variable that take the value 1 if the respondent was shown the female 24 Since brochure-type was assigned at the CO level, the lower representation of men in mixed COs likely explains this. 29 brochure, Xij is a vector of individual characteristics collected at baseline and εij is a mean-zero error term. Standard errors are clustered at the CO level throughout since the CO level treatment assignment creates spatial correlation among members of the same CO (Moulton, 1986). The vector Xij includes the following baseline characteristics reported in Table 1: eligibility, being offered business training, a dummy if decision-making power is above the median in the same gender sample, own education, digit span recall, spouse education, landholdings, membership in a mixed group, age, number of children and risk tolerance. It also includes field unit (branch) dummies, which is the stratification variable in the model. This set of variables includes characteristics showing imbalances in the randomization check as well as variables that are likely to affect the decision to borrow 25. Lastly, the coefficient β captures the impact of being shown a brochure with pictures of female entrepreneurs on the cover and is the coefficient of interest. Interactions are also examined between the type of brochure shown and baseline characteristics which could proxy for attitudes towards women Yij = α + π(FBj * Zij)+βFBj + ɣXij + εij (2) Zij is a subset of individual characteristics included in the vector Xij that could represent an individual’s attitudes towards women. The coefficient π on the interaction term FBj * Zij reveals the extent to which the impact of the female picture (FB) on loan uptake varies according to the members attitudes towards women. 25 The regression results to come are not substantially affected by the exclusion of this set of control variables. 30 V. Results Two main outcomes were explored for the marketing experiment: the decision to borrow and the loan amount requested. Table 5 presents regression results for the decision to borrow. Columns (1)-(3) present results for the full sample, combined and disaggregated by gender. On average, only 14.6 percent of members, roughly 31 percent of the eligible, applied for a larger loan. This is somewhat higher for men, at 20 percent, and correspondingly lower for women, among whom only 8 percent of the eligible applied. Given that the sample consists of existing microfinance clients, this number appears low; when asked anecdotally, many borrowers report either that the monthly installment amount for the larger loan was too high or that the maturity period was too short. This indicates that NRSP clients are not, by and large, constrained by the current loan size, but that some could benefit from larger loans. Among men, older members and business owners are more likely to apply, while among women, being offered business training increases loan demand. In general, however, and after controlling for different characteristics, exposure to the female brochure does not seem to have any effect on loan take up. Columns (4)-(6) report on all CO members who had a business at baseline. Perhaps surprisingly, the female brochure impacts negatively the uptake by women entrepreneurs. The effect is not small: from a base of 8.3 percent, the point estimate indicates a reduction of 39 percent the probability of applying for the larger loan26. 26 In contrast, businessmen appear indifferent to the female picture in the brochure and, as before, being offered business training increases loan demand while membership in a mixed CO depresses it, but there are no differences by gender in these or any other characteristics. 31 S am pl e: A ll M al es Fe m al es A ll M al es Fe m al es A ll M al es Fe m al es (1 ) (2 ) (3 ) (4 ) (5 ) (6 ) (7 ) (8 ) (9 ) Fe m al e B ro ch ur e -0 .0 26 -0 .0 33 -0 .0 15 -0 .0 29 -0 .0 17 -0 .0 51 ** -0 .0 29 -0 .0 07 -0 .0 43 (0 .0 21 ) (0 .0 34 ) (0 .0 19 ) (0 .0 27 ) (0 .0 44 ) (0 .0 23 ) (0 .0 35 ) (0 .0 58 ) (0 .0 39 ) O ffe re d B us in es s Tr ai ni ng 0. 04 8* * 0. 04 6 0. 04 6* * 0. 05 8* * 0. 06 7 0. 03 7 0. 07 1* * 0. 05 6 0. 09 5* * (0 .0 21 ) (0 .0 35 ) (0 .0 19 ) (0 .0 27 ) (0 .0 44 ) (0 .0 24 ) (0 .0 35 ) (0 .0 55 ) (0 .0 47 ) Fe m al e -0 .0 16 -0 .0 34 -0 .0 29 (0 .0 21 ) (0 .0 31 ) (0 .0 46 ) A ge 0. 00 6* ** 0. 00 8* * 0. 00 0 0. 00 6 0. 00 9 -0 .0 03 0. 00 9 0. 01 1 -0 .0 03 (0 .0 02 ) (0 .0 04 ) (0 .0 04 ) (0 .0 04 ) (0 .0 06 ) (0 .0 06 ) (0 .0 06 ) (0 .0 08 ) (0 .0 11 ) A ge 2^ -0 .0 00 ** -0 .0 00 ** 0. 00 0 0. 00 0 0. 00 0 0. 00 0 -0 .0 00 * 0. 00 0 0. 00 0 (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) H ig h D ec is io n- m ak in g po w er (1 =y es ) 0. 00 1 -0 .0 03 -0 .0 22 0. 00 2 0. 00 9 -0 .0 16 -0 .0 02 -0 .0 37 -0 .0 39 (0 .0 02 ) (0 .0 20 ) (0 .0 17 ) (0 .0 03 ) (0 .0 25 ) (0 .0 22 ) (0 .0 05 ) (0 .0 51 ) (0 .0 36 ) O w n E du ca tio n 0. 00 2 0. 00 5 0. 00 0 0. 00 3 0. 00 4 0. 00 1 0. 00 4 0. 00 4 0. 00 7 (0 .0 02 ) (0 .0 03 ) (0 .0 03 ) (0 .0 03 ) (0 .0 04 ) (0 .0 04 ) (0 .0 05 ) (0 .0 07 ) (0 .0 06 ) D ig it S pa n R ec al l 0. 00 3 0. 00 2 0. 00 1 -0 .0 01 -0 .0 02 -0 .0 02 0. 00 2 -0 .0 01 -0 .0 01 (0 .0 03 ) (0 .0 05 ) (0 .0 04 ) (0 .0 05 ) (0 .0 08 ) (0 .0 05 ) (0 .0 08 ) (0 .0 13 ) (0 .0 11 ) R is k To le ra nc e 0. 00 1 -0 .0 01 0. 00 2 0. 00 1 -0 .0 01 0. 00 1 0. 00 4 -0 .0 02 0. 00 6 (0 .0 02 ) (0 .0 03 ) (0 .0 03 ) (0 .0 03 ) (0 .0 04 ) (0 .0 03 ) (0 .0 05 ) (0 .0 08 ) (0 .0 06 ) Y rs . O f E du ca tio n of S po us e 0. 00 0 -0 .0 05 0. 00 2 -0 .0 01 -0 .0 03 0. 00 0 -0 .0 04 -0 .0 11 * -0 .0 01 (0 .0 02 ) (0 .0 03 ) (0 .0 02 ) (0 .0 02 ) (0 .0 04 ) (0 .0 03 ) (0 .0 03 ) (0 .0 06 ) (0 .0 04 ) N um be r o f c hi ld re n un de r 9 0. 00 3 0. 00 3 0 0. 00 7 0. 00 7 0. 00 6 0. 01 0. 01 7 0. 00 1 (0 .0 03 ) (0 .0 04 ) (0 .0 04 ) (0 .0 04 ) (0 .0 06 ) (0 .0 07 ) (0 .0 08 ) (0 .0 11 ) (0 .0 11 ) B us in es s O w ne r ( Y es =1 ) 0. 02 8* * 0. 04 0* 0. 01 (0 .0 14 ) (0 .0 22 ) (0 .0 18 ) La nd 0. 00 0 0. 00 1 0. 00 0 0. 00 0 0. 00 1 0. 00 0 -0 .0 01 0. 00 0 -0 .0 02 (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 01 ) (0 .0 01 ) (0 .0 02 ) M em be r o f a M ix ed C O /C G -0 .0 92 ** * -0 .0 86 * -0 .1 05 ** -0 .1 50 ** * -0 .1 21 * -0 .1 96 ** * -0 .1 66 ** 0. 02 1 -0 .2 72 ** * (0 .0 32 ) (0 .0 47 ) (0 .0 48 ) (0 .0 43 ) (0 .0 66 ) (0 .0 70 ) (0 .0 73 ) (0 .2 36 ) (0 .1 00 ) E lig ib ili ty 0. 18 8* ** 0. 24 6* ** 0. 13 0* ** 0. 20 3* ** 0. 29 9* ** 0. 11 1* ** 0. 21 9* ** 0. 30 4* ** 0. 14 8* ** (0 .0 15 ) (0 .0 26 ) (0 .0 16 ) (0 .0 19 ) (0 .0 33 ) (0 .0 18 ) (0 .0 24 ) (0 .0 40 ) (0 .0 32 ) M ea n D ep en de nt V ar ia bl e 0. 15 0. 20 0. 08 0. 15 0. 20 0. 08 0. 18 0. 25 0. 11 N . O bs er va tio ns 3 ,4 51 1 ,8 80 1 ,5 71 2, 06 5 1, 14 9 91 6 72 6 36 3 36 3 R -s qu ar ed 0. 17 0. 18 0. 11 0. 18 0. 17 0. 13 0. 17 0. 17 0. 16 Ta bl e 5. U pt ak e of L ar ge r Lo an , a m on g th os e ex po se d to a B ro ch ur e O LS A ll m em be rs M at ch ed B us in es s O w ne rs A ll B us in es s O w ne rs N ot e: T he d ep en de nt v ar ia bl e ta ke s va lu e 1 if th e m em be r a pp lie d fo r a la rg er lo an . A ll re gr es si on s in cl ud e br an ch fi xe d ef fe ct sa nd a re e st im at ed u si ng O LS m et ho ds . S ta nd ar d er ro rs a re c lu st er ed a t t he b or ro w er g ro up le ve l.T he fo llo w in g sy m bo ls * , * * an d ** * d en ot e si gn ifi ca nc e at th e 10 , 5 , an d 1 pe rc en t l ev el , re sp ec tiv el y. S ee A pp en di x B fo r d ef in iti on o f v ar ia bl es . 32 The result among women appears counterintuitive: if anything, one might expect that a woman who owns a business would be attracted to a loan product that makes salient her identity as a business woman. But since women operate businesses that are much smaller than those of men, it is possible that the negative impact is an artifact of the business scale. Specifically, women may find the scale of the business pictured in the brochure too large and they may be discouraged from applying27. To explore this hypothesis further, the sample is restricted to female business owners who operate at a scale comparable with men by sector28. Given differences in business scale by gender, this matched sample subsequently consists of businesses in the upper tail of the distribution of female businesses (and lower tail of distribution of male businesses). In particular, female businesses in the matched sample have on average Rs. 4,016 higher sales compared to the sample of all female businesses, while male businesses in the matched sample have Rs. 8,129 lower sales than the average for all male businesses. Sample size is also considerably reduced. Only 363 female businesses can be matched with a corresponding male business. Columns (7)–(9) report these results. The coefficient for female brochure remains negative but loses precision: the coefficient falls by about 16 percent while the standard error increases by about 70 percent. In sum, a 27 There is some anecdotal evidence that businesswomen felt that some of the businesses featured in the brochure were run on a larger scale than the typical female business and that this fact discouraged them from borrowing. As mentioned, many women operated each of the three business types featured in the brochure, which together account for 95 percent of all businesses run by women. In addition, the brochure clearly stated that the loan could be used for any purpose and for any business, but nonetheless, the picture may have triggered a more deliberative response (see Kahneman, 2003 for a discussion on perception in decision-making). 28 For each female business, the absolute difference in sales with each male business in the same sector is computed and then the business with the smallest difference is picked. A male business is matched only once with a female business. In the final sample only those female businesses where the absolute difference in sales is less than Rs 1000 are kept. 33 negative effect of exposure to the female brochure cannot conclusively be ruled out even among women business owners who run larger businesses. Note, however, that in this matched group, being offered business training increases the probability of a woman applying for a larger loan by 86 percent from a base of 11 percent and appears to be driving the overall increase in loan applications due to the offer of business training. On the other hand, decision making power29 does not appear to encourage loan applications from women in any of these samples. Table 6 reports the regression results from specification (2), for males (columns 1-3) and females (columns 4-6) on all three samples. All regressions include the baseline controls used in Table 5. The first striking fact is that a subset of men respond to the psychological content of the brochure. Specifically, business owners with low scores on the digit span recall question, a proxy for ability, and those whose wives are poorly educated (p-value of FB x Years Education of Spouse is 0.11), respond negatively to the female brochure. In this group, loan demand falls by about 13 percentage points (more than a 50% decline over the base demand). While the experimental design does not allow a distinction between lack of affinity towards women and an outright distaste for female-run businesses as possible explanations for the negative response, it suggests one channel through which exposure to the female brochure among female business owners could depress loan demand. Specifically, women who have low decision making power may turn to their husbands for 29 The decision-making variable is built by taking the number of household decisions out of a list of eight that the member usually takes on his or her own. The decisions are: children’s schooling, consumption expenditures, major investments in business or land, the respondent’s participation in community or political activities, the respondent’s spouse participation in community or political activities, whether or not the respondent should work for an income, whether or not the spouse should work for an income and how much the household saves. In the analysis, a dummy is used that takes value 1 if the variable is above the median for each gender subsample. 34 permission to borrow and may face a negative response from them when shown the female brochure. The experiment provides evidence of this. Among female business owners, only those with low decision making power in their household react negatively to the female brochure. Interestingly, female business owners with high decision making autonomy shown the male brochure also react negatively by roughly the same magnitude, while there is no effect on female business owners with autonomy shown the female brochure (p-value is 0.28).30 This suggests that women with decision-making autonomy react negatively to the brochure of opposite sex by about as much as men without business, with low digit span and poorly educated wives. Given the disadvantaged position of women in rural Pakistan, it can be deducted that men may have less respect for female businesses whereas women may feel more alienated when shown the male brochure. In the matched sample (column 6), the results for female business owners with high decision making autonomy become stronger (the point estimate increases in absolute value). 30 The test is whether the coefficient on the female brochure plus the coefficient on decision-making power above median plus the coefficient on the interaction between the female brochure and decision- making power above median is different from zero, that is β+γ+ρ=0 following the notation of Equation (2) . 35 Al l M em be rs Al l B us in es s O wn er s M at ch ed B us in es s O wn er s Al l M em be rs Al l B us in es s O wn er s M at ch ed B us in es s O wn er s (1 ) (2 ) (3 ) (1 ) (2 ) (3 ) Fe m al e Br oc hu re (F B) -0 .1 32 ** -0 .1 34 * -0 .1 56 -0 .0 1 -0 .0 71 * -0 .0 77 (0 .0 51 ) (0 .0 71 ) (0 .1 13 ) (0 .0 34 ) (0 .0 38 ) (0 .0 54 ) Bu si ne ss T ra in in g 0. 06 1 0. 09 7* 0. 07 5 0. 02 7 0. 01 3 0. 03 3 (0 .0 45 ) (0 .0 54 ) (0 .0 70 ) (0 .0 26 ) (0 .0 32 ) (0 .0 52 ) FB x B us in es s Tr ai ni ng -0 .0 42 -0 .0 64 0. 00 2 0. 03 0. 02 2 0. 05 4 (0 .0 65 ) (0 .0 80 ) (0 .1 08 ) (0 .0 35 ) (0 .0 44 ) (0 .0 71 ) Hi gh D ec is io n- m ak in g po we r ( 1= ye s) -0 .0 17 -0 .0 10 -0 .0 39 -0 .0 22 -0 .0 53 * -0 .1 01 * (0 .0 24 ) (0 .0 31 ) (0 .0 68 ) (0 .0 24 ) (0 .0 30 ) (0 .0 54 ) FB x Hi gh D ec is io n m ak in g po we r 0. 00 8 0. 00 8 0. 00 6 0. 02 3 0. 07 7* * 0. 13 8* * (0 .0 05 ) (0 .0 07 ) (0 .0 14 ) (0 .0 31 ) (0 .0 39 ) (0 .0 68 ) Di gi t S pa n Re ca ll -0 .0 05 -0 .0 14 -0 .0 16 0. 00 8 0. 00 8 0. 01 5 (0 .0 06 ) (0 .0 10 ) (0 .0 17 ) (0 .0 05 ) (0 .0 07 ) (0 .0 13 ) FB x D ig it Sp an R ec al l 0. 01 7* * 0. 02 4* 0. 02 3 -0 .0 09 -0 .0 13 -0 .0 19 (0 .0 08 ) (0 .0 12 ) (0 .0 21 ) (0 .0 06 ) (0 .0 09 ) (0 .0 16 ) Yr s. E du ca tio n of S po us e -0 .0 38 -0 .0 63 * -0 .1 11 -0 .0 11 -0 .0 1 0. 03 4 (0 .0 28 ) (0 .0 38 ) (0 .0 69 ) (0 .0 25 ) (0 .0 33 ) (0 .0 64 ) FB x Y rs . E du ca tio n of S po us e 0. 03 9 0. 09 0. 12 3 0. 04 3 0. 04 1 -0 .0 12 (0 .0 40 ) (0 .0 56 ) (0 .1 01 ) (0 .0 32 ) (0 .0 39 ) (0 .0 71 ) Bu si ne ss o wn er (Y es =1 ) 0. 03 1 0. 04 5* * (0 .0 30 ) (0 .0 22 ) FB x B us in es s ow ne r 0. 03 3 -0 .0 57 * (0 .0 42 ) (0 .0 31 ) M ea n De pe nd en t V ar ia bl e 0. 20 0. 24 0. 25 0. 08 0. 08 0. 11 N. O bs . 18 80 11 49 36 3 1, 57 1 91 6 36 3 R- sq ua re d 0. 19 0. 19 0. 18 0. 12 0. 13 0. 18 No te : T he d ep en de nt va ria bl e ta ke s va lu e 1 if th e m em be r a pp lie d fo r a la rg er lo an . A ll re gr es si on s co nt ro l f or e lig ib ilit y, e xp os ur e to B T, d ec is io n m ak in g po we r r el at ive to m em be rs o f t he s am e ge nd er , ow n ed uc at io n, d ig it sp an re ca ll, s po us e ed uc at io n, b us in es s ow ne rs hi p, l an dh ol di ng s, b el on gi ng to a m ix ed g ro up , a ge , n um be r o f c hi ld re n, ri sk to le ra nc e, a nd in cl ud e br an ch fi xe d ef fe ct s. L og o f a ve ra ge s al es a t t he ti m e of b as el in e is in tro du ce d fo r b us in es s ow ne rs a nd m at ch s am pl es . S ta nd ar d er ro rs a re c lu st er ed a t th e bo rro we r g ro up le ve l.T he fo llo wi ng s ym bo ls *, * * an d ** * de no te s ig ni fic an ce a t t he 1 0 , 5 , a nd 1 p er ce nt le ve l, re sp ec tiv el y. S ee A pp en di x B fo r d ef in iti on o f v ar ia bl es . O LS Ta bl e 6: H et er og en eo us E ffe ct s of L ar ge r L oa n Up ta ke a m on g th os e e xp os ed to B ro ch ur e M al e Fe m al e 36 Males Females Males Females (1) (2) (3) (4) (5) (6) (7) (8) Female Brochure 0.088 0.085* 0.161*** -0.035 0.0036 -0.006 0.008 -0.098 (0.054) (0.049) (0.052) (0.104) (0.029) (0.030) (0.030) (0.069) Offered Business Training -0.041 -0.032 -0.092 0.004 -0.01 0.007 (0.042) (0.046) (0.104) (0.029) (0.031) (0.068) Female 0.078 -0.054 (0.079) (0.055) Age 0.010 0.015* -0.007 0.010** 0.013** 0.002 (0.007) (0.009) (0.018) (0.005) (0.005) (0.017) Age 2^ 0.000 -0.000* 0.000 -0.000* -0.000** 0.000 (0.000) (0.000) (0.000) (0.000) (0.000) (0.000) High Decision-making power (1=yes) 0.015*** 0.087*** 0.024 0.008* 0.021 0.1 (0.005) (0.028) (0.071) (0.005) (0.026) (0.064) Own Education 0.018*** 0.016*** 0.016 0.016*** 0.014*** 0.016^ (0.005) (0.004) (0.010) (0.003) (0.003) (0.010) Digit Span Recall -0.012 -0.018** 0.024 -0.013** -0.012* -0.001 (0.009) (0.008) (0.024) (0.006) (0.006) (0.016) Risk Tolerance 0.003 0.001 0.005 -0.002 -0.004 -0.001 (0.004) (0.004) (0.013) (0.004) (0.005) (0.008) Yrs. Of Education of Spouse 0.002 0.003 0.002 0.004 -0.004 0.018** (0.004) (0.005) (0.010) (0.005) (0.005) (0.008) Number of children under 9 0.004 0.004 0.007 0.004 0.003 -0.017 (0.009) (0.009) (0.029) (0.006) (0.006) (0.022) Business Owner (Yes=1) 0.023 -0.006 0.064 0.050 0.058 0.027 (0.032) (0.036) (0.069) (0.035) (0.042) (0.061) Land 0.006*** 0.005*** 0.007 0.001 0.000 0.005** (0.001) (0.001) (0.005) (0.001) (0.001) (0.002) Member of a Mixed CO/CG 0.096 0.377** -0.233 0.025 -0.06 0.076 (0.153) (0.170) (0.167) (0.176) (0.176) (0.197) Mean Dependent Variable 10.93 10.93 10.92 10.97 10.67 10.67 10.72 10.52 N. Observations 664 664 491 173 503 503 372 131 R-squared 0.21 0.27 0.32 0.33 0.27 0.34 0.26 0.44 Table 7A. Loan Size, All Members OLS Log of Amount Approved All Members Note: The dependent variable is log of loan amount requested (columns 1-4) and log of loan amount approved (columns 5-8). The sample includes loan applicants. All regressions include branch fixed effects and are estimated using OLS methods. Standard errors are clustered at the borrower group level.The following symbols *, * * and ** * denote significance at the 10 , 5 , and 1 percent level, respectively. See Appendix B for definition of variables. Log of Amount Requested All Members Table 7A reports the impact of the female brochure and baseline characteristics on the loan amount requested (columns 1-4) and approved (columns 5-8) among loan applicants. Table 7B reports the same regressions among loan applicants with a business at baseline. In both tables a positive and significant effect of the picture is found on the loan amount requested among males but not females. The result in both tables is driven primarily by selection because men without a business and low digit span recall tend to borrow less and decide not to borrow when shown a female brochure. 37 Males Females Males Females (1) (2) (3) (4) (5) (6) (7) (8) Female Brochure 0.093 0.08 0.135** -0.061 0.027 0.011 0.002 -0.068 (0.057) (0.050) (0.052) (0.129) (0.032) (0.033) (0.030) (0.081) Offered Business Training -0.032 -0.058 -0.021 0.054** 0.026 0.032 (0.046) (0.047) (0.145) (0.027) (0.026) (0.098) Female 0.089 -0.094 (0.082) (0.064) Age 0.013 0.017 -0.024 0.007 0.015** -0.006 (0.009) (0.011) (0.021) (0.005) (0.006) (0.023) Age^2 -0.000* -0.000* 0.000 0.000 -0.000** 0.000 (0.000) (0.000) (0.000) (0.000) (0.000) (0.000) High Decision-making power (1=yes) 0.025 0.095** -0.022 0.029 -0.007 0.094 (0.038) (0.038) (0.091) (0.030) (0.024) (0.070) Own Education 0.022*** 0.020*** 0.027 0.012*** 0.013*** 0.012 (0.005) (0.005) (0.017) (0.004) (0.004) (0.010) Digit Span Recall -0.014 -0.019* 0.003 -0.009 -0.009 0.001 (0.011) (0.010) (0.034) (0.006) (0.007) (0.020) Risk Tolerance 0.002 0.001 0.014 -0.001 0.000 -0.003 (0.005) (0.005) (0.017) (0.005) (0.006) (0.011) Yrs. Of Education of Spouse 0.001 0.007 -0.007 0.002 -0.005 0.013 (0.005) (0.005) (0.013) (0.006) (0.006) (0.010) Number of children under 9 0.004 0.008 -0.032 0.001 -0.004 -0.011 (0.011) (0.011) (0.037) (0.007) (0.007) (0.019) Business Owner (Yes=1) Land 0.005*** 0.005*** 0.005 0.002 0.000 0.004 (0.001) (0.002) (0.005) (0.001) (0.001) (0.003) Member of a Mixed CO/CG 0.138 0.314 -0.293 -0.007 -0.256*** 0.030 (0.167) (0.213) (0.293) (0.239) (0.086) (0.331) Mean Dependent Variable 10.92 10.92 10.92 10.92 10.69 10.69 10.69 10.69 N. Observations 459 459 356 103 355 355 278 77 R-squared 0.23 0.30 0.34 0.42 0.28 0.35 0.29 0.58 Business Owners Business Owners Note: The dependent variable is log of loan amount requested (columns 1-4) and log of loan amount approved (columns 5-8). The sample includes loan applicants. All regressions include branch fixed effects and are estimated using OLS methods. Standard errors are clustered at the borrower group level.The following symbols *, * * and ** * denote significance at the 10 , 5 , and 1 percent level, respectively. See Appendix B for definition of variables. Table 7B. Loan Size, Business Owners OLS Log of Amount Requested Log of Amount Approved VI. Conclusions The results of the marketing experiment have several consequences for future work of NRSP and other microfinance organizations in rural Pakistan. VI.1 On the use of Marketing tools to increase demand among women The rationale behind using brochures to promote a new microfinance product in rural Pakistan was that information targeted to women would increase female take up 38 through affinity. It did not happen. In a context of already low take up (especially of women, but also of men), the female brochure actually reduced take up among female business owners. This result is concentrated among women with low levels of numeracy (proxy for ability), relatively low levels of education, and limited decision-making in the household. In fact, results and the analysis that followed them suggest that the decision to not borrow money reflects the reaction of males in the household instead of that of women. This is deduced by a similar effect among men with low education levels and limited numeracy. In this way, the gender segregation in the labor market of Pakistan finds itself reflected in microfinance programs: women can be business owners, but men in the household are the ones making most of the decisions, including borrowing money. If the goal of NRSP is to increase female participation in microfinance, marketing tools need to be better designed and customized so they sensitize both women and men. The results in this sense confirm what Field et al. (2010) find for India on the importance of understanding traditional views and social norms about gender roles when assessing and designing microfinance programs. VI.2 On interpreting the results of the ‘Money or Skills’ experiment As previously mentioned, the field work was designed to inform the ‘Money or Skills’ debate. As part of the data collection process, information on decision making within the household was included in the baseline and follow up questionnaires. The idea 39 was to treat decision making as an outcome indicator, one that would serve as a proxy for female empowerment31. The analysis of the marketing experiment changed the role of this proxy in the analysis. The “Money or Skills” experiment indicated negligible effects of both interventions on development outcomes for women, despite a strong effect of business training on business knowledge among them. This suggests that barriers remain that constrain women from transforming business knowledge into profitable businesses and improved socio-economic indicators. The marketing experiment generated information on the role of social norms that guided the rest of the analysis. Using information from use of time surveys and proxies for decision-making in the household helped reach a better understanding of the effects of both business skills and increased line of credit. As a consequence, it has also become clear that NRSP microcredit products need to be customized in such a way that they respond to social constraints faced by a significant number of CO members. VI.3 On the design of microfinance interventions It would be easy to conclude by arguing that the marketing experiment found that social norms governing women in society are in fact a key constraint to female entrepreneurship in Pakistan, in line with the observations of Field et al (2010) work in India. However, there were clear indications of this being the case before the field experiment was designed given the patterns of participation in previous NRSP programs, national trends in women’s access and demand for credit, and information from the 31 The complete analysis of the ‘Money or Skills’ experiment can be found in an upcoming paper by Gine and Mansuri (2014). The experiment did not have significant effects on decision-making in the household either. 40 baseline, such as use of time surveys and reasons for not borrowing. A more careful analysis of context previous to the experiment could have significantly altered the design of the marketing tools used, including the comparison between tools or the interaction between them. Instead, the only piece of actionable information for NRSP the experiment produced is that female brochures, as designed, fail to increase women’s take up of their microcredit products. This however, should not be taken as an indication that marketing does not work for microfinance in Pakistan. The conclusion reached is only valid for this specific tool, and a consequence of applying a standard intervention without adapting it to the local context. I will return to this point in the next essay of this dissertation. 41 Appendix 1. Selected Results of the “Money or Skills” Experiment Business Knowledge New Business CO member involved (1=Yes) New Business CO member not involved (1=Yes) Main Business Failed (1=Yes) Aggregate Business Practices Aggregate Business Operations Aggregate Sales and Profits (1) (2) (3) (4) (5) (6) (7) Panel A: Intent to Treat Effect Business Training (1=Yes) 0.058* -0.006 -0.001 -0.034 0.131** 0.043 -0.021 (0.031) (0.008) (0.012) (0.028) (0.062) (0.027) (0.054) Lottery Winner (1=Yes) -0.014 -0.012 -0.007 -0.002 0.099 0.081** 0.013 (0.037) (0.013) (0.019) (0.036) (0.082) (0.035) (0.071) BT and LW 0.075* -0.004 0.010 -0.014 0.166** 0.047 -0.080 (0.038) (0.013) (0.018) (0.037) (0.079) (0.035) (0.066) R-Squared 0.09 0.01 0.01 0.03 0.06 0.09 0.33 P-value of t - test of ... BT = BT and LW 0.63 0.88 0.51 0.55 0.61 0.89 0.32 LW = BT and LW 0.03 0.53 0.42 0.77 0.45 0.40 0.26 BT = LW 0.05 0.59 0.77 0.37 0.69 0.29 0.66 Panel B: Intent to Treat Effects with Gender Interactions Business Training (1=Yes) 0.058 -0.011 -0.012 -0.061* 0.122 0.067* 0.023 (0.043) (0.013) (0.016) (0.037) (0.085) (0.036) (0.070) BT x Female 0.000 0.013 0.023 0.060 0.018 -0.061 -0.111 (0.062) (0.017) (0.025) (0.055) (0.118) (0.052) (0.102) Lottery Winner (1=Yes) 0.014 -0.019 -0.015 -0.004 0.061 0.095** -0.008 (0.045) (0.018) (0.024) (0.045) (0.105) (0.046) (0.092) LW x Female -0.066 0.016 0.017 0.000 0.097 -0.034 0.065 (0.073) (0.023) (0.035) (0.068) (0.150) (0.067) (0.138) BT and LW 0.066 -0.016 -0.017 -0.047 0.246** 0.084* -0.07 (0.051) (0.016) (0.020) (0.046) (0.102) (0.044) (0.085) BT and LW x Female 0.021 0.028 0.062* 0.077 -0.225 -0.098 -0.022 (0.073) (0.024) (0.036) (0.069) (0.138) (0.069) (0.128) R-Squared 0.09 0.01 0.01 0.03 0.06 0.09 0.34 BT = BT and LW 0.87 0.74 0.77 0.75 0.17 0.64 0.20 LW = BT and LW 0.46 0.50 0.92 0.00 1.00 0.59 0.32 BT = LW 0.35 0.63 0.90 0.20 0.55 0.54 0.76 BT + BT x Female = 0 0.19 0.88 0.56 0.98 0.10 0.88 0.26 LW + LW x Female = 0 0.38 0.85 0.94 0.34 0.18 0.22 0.58 BT and LW = BT and LW x Female = 0 0.12 0.53 0.14 0.58 0.84 0.79 0.35 Mean of dependent variable among controls 0.01 0.05 0.10 0.40 0.02 -0.09 0.47 N. Observations 3494 3494 3494 2137 1333 1311 1265 A. Business Outcomes Note: The reported mean of the dependent variable is computed using CO members not offered business training nor chosen as winners of the lottery.The dependent variables are aggregates of standardized z-scores. See Appendix B for a definition of the aggregates. All regressions are estimated using OLS methods and include as covariates the stratification variables (eligibility for loan lottery, business ownership at baseline, gender and branch dummies). Standard errors reported in parentheses are clustered at the CO level. The following symbols *, * * and ** * denote significance at the 10 , 5, and 1 percent level, respectively. OLS 42 Expenditures and Assets CO Cohesion Outlook on Life Decision- Making (1) (2) (3) (4) Panel A: Intent to Treat Effect Business Training (1=Yes) 0.070*** 0.089*** 0.082*** 0.082 (0.021) (0.027) (0.024) (0.080) Lottery Winner (1=Yes) 0.036 0.037 0.074** -0.021 (0.027) (0.032) (0.029) (0.113) BT and LW 0.069** 0.084** 0.115*** -0.027 (0.029) (0.039) (0.032) (0.116) R-Squared 0.44 0.02 0.08 0.03 P-value of t - test of ... BT = BT and LW 0.99 0.88 0.25 0.32 LW = BT and LW 0.30 0.28 0.22 0.97 BT = LW 0.25 0.14 0.79 0.38 Panel B: Intent to Treat Effects with Gender Interactions Business Training (1=Yes) 0.094*** 0.102*** 0.050 0.123 (0.028) (0.039) (0.033) (0.123) BT x Female -0.056 -0.031 0.067 -0.083 (0.041) (0.053) (0.047) (0.162) Lottery Winner (1=Yes) 0.046 0.032 0.058 0.045 (0.036) (0.037) (0.039) (0.163) LW x Female -0.02 0.012 0.032 -0.146 (0.050) (0.061) (0.054) (0.205) BT and LW 0.144*** 0.145** 0.115*** -0.067 (0.038) (0.060) (0.040) (0.161) BT and LW x Female -0.167*** -0.139** 0.000 0.096 (0.051) (0.068) (0.059) (0.207) Decision-making power 0.46 0.50 0.08 0.00 P-value of t - test of ... BT = BT and LW 0.15 0.13 0.08 0.20 LW = BT and LW 0.02 0.05 0.20 0.55 BT = LW 0.18 0.39 0.84 0.64 BT + BT x Female = 0 0.21 0.05 0.00 0.70 LW + LW x Female = 0 0.50 0.39 0.03 0.46 BT and LW = BT and LW x Female = 0 0.53 0.87 0.01 0.84 Mean of dependent variable among controls 0.01 -0.11 -0.12 0.12 N. Observations 3494 3494 3494 3494 Note: The reported mean of the dependent variable is computed using CO members not offered business training nor chosen as winners of the lottery.The dependent variables are aggregates of standardized z-scores. See Appendix B for a definition of the aggregates. All regressions are estimated using OLS methods and include as covariates the stratification variables (eligibility for loan lottery, business ownership at baseline and gender). Standard errors reported in parentheses are clustered at the CO level. The following symbols *, * * and ** * denote significance at the 10 , 5, and 1 percent level, respectively. B. Individual and Household Outcomes OLS 43 Appendix 2. Brochure design 44 Glossary Data used in this paper come from two surveys: a baseline conducted in November 2006, a follow-up survey in December 2008. Administrative data about loan take-up, obtained from NRSP’s internal records, is also used. Treatments and Take Up (from administrative records) • Female picture brochure takes the value of 1 if the member was shown a brochure with female business owners on the cover, 0 if the brochure showed the same businesses with men. • Eligibility, takes the value of 1 if member is eligible for the loan lottery, 0 otherwise. • Business Training, a dummy taking the value of 1 if the individual was offered business training, 0 otherwise. • Applied for larger loan, 1 if the member actually requested a loan, 0 otherwise. • Approved, conditional on applying for a loan, takes a value of 1 if the member was approved by NRSP, 0 otherwise. • Borrowed, takes a value of 1 if the individual actually borrowed money from NRSP; while conditional on approval, not all those that applied and were approved took actually a loan. • Amount borrowed, is measured in thousands of Rupees. Baseline characteristics Individual 45 • Female equals 1 for women and 0 for men. • Age is respondent’s age in years. • Years of education is years of completed schooling, and is top-coded at 16. • Married, a dummy taking the value of 1 if member is married, 0 if single, divorced or widowed. • Digital span recall reports the number of digits correctly recalled after being shown an eight digit number for 30 seconds. • Member of a mixed group, dummy takes the value of 1 if the member belongs to a borrowing group with mixed gender, 0 if the group is of the same gender. • Index of female mobility and No purdah index are principal components of several variables with negative values indicating less mobility (or observing more types of purdah). • Business owner equals 1 if the member had a business at baseline, 0 otherwise. • Aversion to risk general is measured on a 0-10 scale where 0 indicates the most risk averse and 10 the most risk-tolerant/lover. • Months as member, number of months as member of NRSP group. • Holds Office in Group, takes value 1 if member has or has had in the past a leadership position in group. • Fraction of Members of same Zaat (caste), is a percentage of members in the group that share the same cast of the member. Household • Education of spouse is years of completed schooling of the respondent’s partner, if any. Top coded at 16. 46 • Total HH income and Expenditures, transformed to logs for analysis. • Number of children under 9 • Land is the total owned land inside and outside the village. • Credit constraints, dummy taking a value of 1 if the member faced any type of credit constraint, formal or informal. • Ever in business, captures business experience within the household. Equals 1 when this is the case, 0 otherwise. • Decision Making, is the number of household decisions out of a total of eight that the member usually takes on his or her own. The decisions are: children’s schooling, consumption expenditures, major investments in business or land, the respondent’s participation in community or political activities, the respondent’s spouse participation in community or political activities, whether or not the respondent should work for an income, whether or not the spouse should work for an income and how much the household saves. In the analysis, a dummy is used that takes value 1 if the variable is above the median for each gender subsample. Business characteristics • Type of business, dummy variables for businesses shown on brochure • Business Literacy, scores of component 1 of a PCA for a set of questions about knowledge about how to run a business, and of competition. • Sales in ‘000 rupees, sales of business in an average month at the time of baseline. 47 Bibliography Akhtar, Shamshad, 2008. “Financial Sector–Ten Year Vision and Strategy”. Presentation at the inauguration of the Development Finance Conference on “Expanding Frontiers of Financial Access in Pakistan” organized by State Bank on the eve of 60th Anniversary of the State Bank of Pakistan, Karachi Banerjee, A. and E. Duflo, 2011. Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty. Oxford: PublicAffairs Banerjee, A., E. Duflo, R. Glennerster, C. Kinnan, 2009. “The miracle of microfinance? Evidence from a randomized evaluation”, unpublished paper. Banerjee, Abhijit and Esther Duflo, 2008. “Do Firms Want to Borrow More? Testing Credit Constraints Using a Directed Lending Program”, MIT Department of Economics Working Paper No. 02-25. Beaman, L. Raghabendra Chattopadhyay, Esther Duflo, Rohini Pande and Petia Topalova, 2009. “Powerful Women: Does Exposure Reduce Bias?”, Quarterly Journal of Economics 124(4): 1497-1540. Berge, L.I.O., K. Bjorvatn, and B. Tungodden, 2010. On The Role of Human and Financial Capital for Microenterprise Development: Evidence from a Field Experiment in Tanzania. Unpublished NEUDC conference paper. Bertrand et al., 2010. “What is advertising content worth? Evidence from a consumer marketing field experiment”, Quarterly Journal of Economics, v. 125, issue 1, pp. 263- 306. 48 Croson, R. and U. Gneezy, 2004. Gender differences in preferences. Journal of Economic Literature 2009, 47:2, 1–27 De Mel, Suresh, David McKenzie and Christopher Woodruff, 2009.“Are Women More Credit Constrained? Experimental Evidence on Gender and Microentreprise Returns”, American Economic Journal: Applied Economics, 1(3):1-32. De Mel, Suresh, David McKenzie and Christopher Woodruff , 2010. “Getting Credit to High Return Microenterprises: The Results of an Information Intervention” mimeo, World Bank. De Mel, Suresh & McKenzie, David & Woodruff, Christopher, 2008. "Who Are the Microenterprise Owners? Evidence from Sri Lanka on Tokman v. de Soto," IZA Discussion Papers 3511, Institute for the Study of Labor (IZA) De Soto, H., 1989. The Other Path: The Invisible Revolution in the Third World. Harpercollins. Djankov, S., Y. Qian, G. Roland and E. Zhuravskaya, 2005. “Entrepreneurship in China and Russia Compared”, Center for Economic and Financial Research (CEFIR) Working Paper 49. Drexler, A. , G. Fischer, A. Schoar, 2011. Keeping it Simple: Financial Literacy and Rules of Thumb. CEPR Discussion Paper No. DP7994 Duflo, E., R. Glennmester, M. Kremer, 2006. Using Randomization in Development Economics: A Toolkit. NBER Technical Working Paper No. 333. 49 Evans, Franklin B., 1963. "Selling as a Dyadic Relationship", American Behavioral Scientist 6: 76-79. Field, E., S. Jayachandran, and R. Pande, 2010. “Do Traditional Institutions Constrain Female Entrepreneurship? A Field Experiment on Business Training in India,” American Economic Review: Papers and Proceedings, 100, 125–129. Giné, X. and G. Mansuri, 2011. “Together We Will: Evidence from a Field Experiment on Female Voter Turnout in Pakistan”, mimeo Giné, X. and G. Mansuri, 2014. “Money or Ideas? A Field Experiment on Constraints to entrepreneurship in rural Pakistan? Upcoming, World Bank Policy Research Working Paper. Gine, X., G. Mansuri, and M. Picon, 2012. “Does a Picture Paint a Thousand Words? Evidence from a Microcredit Marketing Experiment”, World Bank Economic Review 25(3):508-542. Gine, X. and D. Karlan , 2006. “Group versus Individual Liability: A field experiment in the Philippines”, World Bank Policy Research Working Paper 4008. Jacoby, Hanan and Ghazala Mansuri, 2011. “Crossing Boundaries: Gender, Caste and Schooling in Rural Pakistan”, mimeo Jensen, Robert and Emily Oster, 2007. “The Power of TV: Cable Television and Women's Status in India”,NBER Working Paper No. 13305 Kahneman, Daniel, 2003. “Maps of Bounded Rationality: Psychology for Behavioral Economics”, American Economic Review, 93(5): 1449-1475. 50 Karlan, D. and S. Mulainathan, 2010. “Rigidity in Microfinance: Can One Size Fit All?” Kirzner, 1985. Discovery and the Capitalist Process. Chicago: University of Chicago Press. Kling, J.R., J. Liebman and L. Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, 01 La Ferrara, Eliana, Alberto Chong and Suzanne Duryea, 2008. “Soap Operas and Fertility: Evidence from Brazil”, BREAD Working Paper No. 172 Mansuri, Ghazala, 2011. “Bottom up or Top Down? Participation and the Provision of Local Public Goods”, mimeo. McClelland, D.C. 1961. The Achieving Society. Princeton, NJ: Van Nostrand McKernan, S.-M. (2002). "The Impact of Microcredit Programs on Self-Employment Profits: Do Noncredit Program Aspects Matter?" Review of Economics and Statistics 84(1): 93-115. Microfinance Connect, 2008. MicroWatch, Quarterly Update on Microfinance in Pakistan. Mobius, Markus M. and Tanya S. Rosenblat, 2006. “Why Beauty Matters”, American Economic Review 96(1): 222-235. Moulton, Brent, 1986. “Random Group Effects and the Precision of Regression Estimates”, Journal of Econometrics, 32, 3:p. 385-397. 51 Nenova, T., and C. Nenang, with A. Ahmad, 2009. “Bringing Finance to Pakistan’s Poor: A Study on Access to Finance for the Underserved and Small Enterprises”, World Bank report. Washington, DC: World Bank. Paluck, Elizabeth L., 2009. “Reducing intergroup prejudice and conflict using the media: A field experiment in Rwanda,” Journal of Personality and Social Psychology 96(3): 574- 587. Paulson, A. L., and R.M. Townsend. 2004. “Entrepreneurship and Financial Constraints in Thailand.” J. Corporate Finance 10 (March): 229–62. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701. Schumpeter, J.A., 1934. The theory of economic development: an inquiry into profits, capital, credit, interest, and the business cycle translated from the German by Redvers Opie (1961) New York: OUP. Sen, A., 1999. Development and Freedom, New York: Anchor Books. Tokman, V., 2007, “Modernizing the Informal Sector,” UN/DESA Working Paper No. 42. Smith, S. C., 2002, Village banking and maternal and child health: Evidence from Ecuador and Honduras, World Development 30, 707-723. UNDP, 1998. Human Development Report 1998: Consumption for Human Development. New York: Oxford University Press. 52 Yunus, M., 1999. Banker to the Poor, London: Aurum Press Ltd. World Bank, 2008. “Finance for All? Policies and Pitfalls in Expanding Access”, World Bank: Washington, DC. 53 Essay Two: Institutions and social norms when designing and evaluating development interventions: An application to the Pakistan Case Study “A major challenge in understanding real-world feedback processes is precisely this tendency to look at problems not as they actually are but in line with existing intellectual and conceptual frameworks- at least until is too late” Ben Ramalingam, Aid in the Edge of Chaos: Rethinking International Cooperation in a Complex World (2014) I. Motivation and Research Relevance The first essay of this dissertation featured a randomized evaluation in a very well controlled setting to assess whether an intervention of interest (a brochure designed on the basis of marketing practices and the concept of affinity) increased loan uptake in a microfinance program, particularly among women. This essay illustrates the lessons from the Pakistan case study32 and builds a framework to improve the design of development interventions. The research uses theory-based evaluations, and in particular realistic evaluation for: a) better understanding the role of context on intervention results; b) improving the design of monitoring systems through the selection of indicators that pick up information on the internal processes and mechanisms triggered (or not) by a development intervention; c) phasing in formative, implementation, and impact evaluation to facilitate corrective measures over time. In this essay, context is defined as prevailing institutional settings (formal and informal) that can affect the magnitude or 32 The Pakistan case study provides a good link to the debate in the literature on the appropriateness of randomized evaluation vis-à-vis other evaluation methods 54 direction of development outcomes and impacts observed after the intervention is introduced. The framework intends to be a practitioner’s response to the challenge posed to the design of development interventions by complexity theory; and to generate knowledge that is usable for policy throughout implementation. This is occurring at a time in which donors and international development organizations look for alternatives to introduce or scale up in a more systematic manner promising interventions33. In doing this, the essay starts with an overview of the development aid debate and a discussion on complex adaptive systems. Then, experiential learning (Pritchett et al, 2013) and realistic evaluation (Pawson and Tilley, 1997) are introduced as key elements to connect design and evaluation approaches. I.1 The aid effectiveness debate and the call for rigorous evaluation Around a decade ago, the debate on development aid was very much represented by those calling for its expansion (with Sachs (2005) one of its most recognizable faces) and those calling for a significant reduction in aid (with Easterly (2007) among the most vocal in this line). Poor performance of many aid recipient countries (Morrisey, 2001), characterized by poor governance structures, lack of institutions, or lack of incentives (Collier, 2007; Easterly, 2008); plus accounts of potentially perverse effects of aid (Moyo, 2009) strengthened the arguments of aid skeptics, while the global economic 33 This quest for better ways to generate and use knowledge for policy has been translated in the recent proliferation of ‘Labs’ across donor organizations and also civil society; integrating the identification of promising solutions, adapting them to context and introducing experimentation and its evaluation in development projects. 55 crisis of 2008 increased pressure on elected officials in donor countries to justify aid budgets to tax payers34. The aid debate and the increased calls for accountability led to a stronger demand for rigorous evaluation of aid programs’ effectiveness. This generated an inherent tension between implementing organizations (typically with narrow missions and limited budgets) and donor or executive organizations that provide resources to them (Pritchett et al., 2013). For implementing organizations, there is pressure to demonstrate as rigorously as possible that their programs worked, and donors focused more than ever in allocating resources to ‘what works’ in development. In this context, the importance of the evaluation function in development rapidly grew. In particular, the profile of randomized evaluations as a method was heightened, promising to determine if development outcomes could actually be attributed to development programs. Randomized Control Trials (RCTs) have been used in the pharmaceutical industry for decades, and their use in social sciences has increased dramatically since the mid-90s (Heckman and Smith, 1995), with a few paradigmatic papers regularly cited to highlight the method’s power and advantages (for example, Kremer and Miguel (2004) and Gertler (2004)). In Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty, Banerjee and Duflo (2011) make a strong case for rigorous evaluation that focuses on well-defined interventions to identify ‘what works’. Part of the problem, they argue, is that the aid debate has been too focused on ‘big questions’ for which an appropriate 34 However, it must be said that perceptions on the size of aid budgets were seriously overblown, particularly in the United States (World Public Opinion, 2010) 56 counterfactual does not exist. Instead, they call for pursuing concrete questions that can be answered through experimentation and that can affect life outcomes for the most disadvantaged. Their book is in fact a compilation of cases in which mainly experimental methods were used to generate evidence on the effectiveness of development programs in given circumstances and countries, from microfinance schemes, to health programs, to institutional reforms. This view of development and of what constitutes evidence that Banerjee and Duflo propose has received its own share of criticism. Critics have warned of a temptation to subordinate the subject of development research to those policies for which randomization is suitable (Ravallion, 2011; Rodrik, 2010). Some, like Easterly and Cohen (2010) speak of how “embracing RCTs has led development researchers to lower their ambition”, by arguing that randomized evaluation is not useful to answer some of the most important questions in development. Others point out that even when the method shows practitioners what to do, the conclusions may not be useful neither for policy nor the understanding of development problems (Das, Devarajan and Hammer, 2011). Deaton (2010) formalizes the criticism as ‘the misunderstanding of exogeneity and the handling of heterogeneity’. On top of this, the method produces results with limited external validity. If the objective of randomized evaluation35 is that the policy or intervention can be then scaled up or replicated with the confidence of having established attribution, the problem is that the circumstances surrounding the experiment are quite different to those surrounding the 35 Or alternatively, instrumental variables or natural experiments as second-best alternatives recommended by most randomized evaluation proponents. 57 scaling up or replication of a policy, with the comparison of one situation to the other based on observables, which is the basic criticism that randomization advocates make of other estimation methods. Deaton ends his paper calling for experimentation aimed at the evaluation of theoretical mechanisms over the evaluation of projects, because heterogeneity in particular “is not a technical problem, but a symptom of something deeper, which is the failure to specify causal models of the processes we are examining”36. In short, Deaton concludes that the evaluation of development programs has been focusing too much on asking ‘what works’, instead of understanding how and why a policy does (or does not) work. Philosopher of science Nancy Cartwright (2007) stresses this point, highlighting how the fundamental issue in development is not the purity of the methods, but the inherent complexity of the problem being studied. I.2 Complexity theory and the design and evaluation of development programs Banerjee and Duflo’s book goes well beyond evaluation and evidence. It also highlights how so many predictions of standard economic theory fail when translated into development interventions. They explain those failures in development policies in design flaws and the presence of the “ubiquitous three I’s: Ignorance, Ideology and Inertia”37. Part of the problem with the latter characterization is that human behavior (individual or collective) may follow a different logic than the rational choice expected from program stakeholders when a development intervention is designed. Beinhocker (2006) criticizes neoclassical economics for its assumption of perfect rationality, 36 Deaton (2008, p. 47) 37 Banerjee and Duflo (2011, p.270) 58 “modeling smart people in unbelievably simple situations, […] when the real world involves simple people coping with unbelievably complex situations”. Instead, he favors the view of bounded rationality developed by Simon (1957) which argues that people have access to limited or biased information and make the best decisions they can with it. Beinhocker further argues that the economy is a complex adaptive system38 , a system never in equilibrium, in which diverse and autonomous components are linked through many different connections and where all components adapt to changes in the environment and to changes in the other components. The concept of complexity has been progressively embraced in development economics. Ostrom (2007) was adamant in her criticisms of panaceas in development, where simplistic analyses of complex systems lose sight of what really happens on the ground. Complexity in her view needs to be analyzed, understanding the different elements in a system being studied. A common practice in development is to design a development intervention around a logical framework approach, which maps a rather linear theory of change. The logical framework approach is an attempt to simplify reality in a causal chain that typically overlooks local context, and is based on a number of assumptions about behavior of stakeholders during implementation and along the causal chain. Alternative theory-driven approaches39 do accept and embrace complexity better than the logical framework, but the development community is still far from mainstreaming such a way 38 Following Beinhocker (2006) and Ramalingam (2014), five characteristics of complex adaptive systems can be identified: a) difficult to predict in detail; b) broad predictions about the system as a whole can be made, identifying patterns; c) emerging properties, patterns that arise from the system overall, not from the individual components; d) a trend toward greater complexity; and e) systems don’t tend to an equilibrium, meaning that each component is evolving and adapting. 39 See, for example, Eyben’s (2008) work on meta-theories, power, and mutual accountability. 59 of thinking; even though several actors are contributing to this either piloting interventions or developing frameworks and toolkits that can be applied to specific programs40 . Figure 1. A Portfolio of Results Framework Source: Ramalingam (2014) Once a development intervention is designed and mapped considering alternative theories of change, the evaluator is expected to select the most appropriate tools for assessing each element of the chain, from the intervention itself to the impact it has on different drivers of change, and from those to different levels of results (output, outcome, impact). Ramalingam (2014) proposes a very basic bi-dimensional framework for mapping and assessing different development interventions based on their nature and the context in which they are introduced. Figure 1 summarizes this view. Notice in it how 40 An example of the former is recent experimentation in social accountability and capacity development done by Oxfam in Africa, in which a range of possible theories are identified and applied to a particular problem, and then their usefulness is assessed. An example of the latter is recent work done by the Methods Lab, a joint initiative of the Overseas Development Institute (ODI), the Australian Department of Foreign Affair and Trade, and the BetterEvaluation think tank. 60 RCT’s have a very specific application, when the context of the program is stable and homogeneous and the nature of the intervention is relatively simple. I.3 The evaluation function in a complex world Randomization was already growing in importance in the evaluation discipline before the latest iteration of the aid debate. It did so as part of the move towards measuring impact. Pritchett et al (2013) describe this evolution: they point out how the term project evaluation was not long ago still loosely and alternatively used to mean up to three different things: project valuation, implementation evaluation, and impact evaluation. The first use of the term evaluation was to describe an ex-ante analysis of development projects based on cost-benefit analysis (Dreze and Stern, 1987). It was assumed that the inputs in an intervention would produce the desired outcomes; the focus was in determining if the value of aggregated benefits was larger than the cost, compared to alternatives. Evaluation later became a retrospective look or certification process to see if the project was implemented as designed. Evaluation understood in this way focuses on disbursements, inputs, activities and outputs. Evaluation, however, is increasingly identified with impact evaluation, the ex-post analysis on how outcomes for intended beneficiaries are affected. Impact evaluation requires a counterfactual: a way to know what would have happened without the project. Randomized evaluations are one way to perform an impact evaluation, one that bases its appeal in the claim that when carefully done can virtually eliminate selection bias. 61 The rise of randomized evaluation was then a response to the claims of development effectiveness regularly made through project valuation and implementation evaluation, and the call for a focus in outcomes that would require a good counterfactual. In the quest for unambiguous attribution, practitioners have for a few years leaned towards identifying randomized evaluation as the only method for rigorous impact evaluation. However, a complex world and the limited capacity of randomized evaluation to generate understanding on the interaction between intervention and context requires a different way to design and implement evaluation. There are in fact many academics and practitioners developing and testing such evaluation protocols. Blattman (2008) proposes what he calls ‘Evaluation 2.0’ which takes into account context specificity and a focus on performance management and process learning, while Szekely (2011) develops the idea of integrating processes for institutionalized learning, in what he calls Results-based Social Policy Design and Implementation Systems (RSPDI). Khagram et al. (2011) suggest diagnostic, contextual approaches to experimentation and innovation, or Impact Planning Assessment Reporting and Learning Systems (IPARLS). Andrews et al. (2012) propose what they call Problem-Driven Iterative Adaptation (PDIA). And lastly, Pritchett et al. (2013) focus on the monitoring function and how it informs Evaluation through experiential (little ‘e’) learning, or MeE. All these different methods (and acronyms) are attempts to respond to that failure in identifying causal models of the processes studied that Cartwright (2007) and Deaton (2008) refer to. Deaton in particular called to move beyond the techniques and focus on how we generate and use knowledge, something that 62 can actually be traced all the way back to Greenberg’s (1968) Evaluation of Social Programs seminal paper, and also is a common feature in the action research literature. This essay does not intend to propose yet another method for design and evaluation. It identifies, from a practitioner’s perspective, key elements of the emerging models, and applies them to the microfinance program case in rural Pakistan. In particular, steps to consider social norms and institutional arrangements in analysis and design are emphasized. Section II introduces and then discusses the links between theory of change and realist evaluation. Section III presents the analytical framework that will help us reinterpret the results and context surrounding the cases study of Pakistan, and that is heavily influenced by complexity theory, realistic evaluation and structured experiential learning. Section IV discusses the evaluation results from Pakistan in light of the proposed framework, with recommendations for how to continuing the learning process in both cases. Section V concludes and discusses how the proposed framework could be useful for the international development community. II. Theoretical Framework The key to the discussion in this chapter is the concept of Theory of Change (ToC), which emerged from both the fields of program theory and program evaluation in the mid-1990s as a method to analyze the theories and assumptions behind programs, particularly in the social and political arenas (see Weiss, 1995). Thus, at its core, theory of change connects program design and program evaluation, a link that I emphasize here, and which is the same connection that Deaton (2010) refers to when 63 calling for experimentation to be used in the evaluation of theoretical mechanisms instead of the evaluation of projects. Theory of Change makes explicit and details the expected process of change of an intervention, explaining the causal linkages of an intervention: this allows to map outcomes over causal chain and the relationship between them over time. This forces stakeholders to distinguish desired and actual outcomes and to model desired outcomes before deciding on the intervention. From the evaluation point of view, ‘theory-oriented’ evaluation was developed as a response to what was seen as ‘the original sins’ of the evaluation profession: playing a low key role in program design and management, avoiding the discussion of theoretical implications, and focusing in developing evaluation methods suitable to projects designed under ‘absolute rationality’ (Stame, 2004). Chen and Rossi’s (1989) paper is considered one of the earliest formulations of theory-oriented41 evaluation, and in it they describe some basic steps to follow for it: studying treatment; discussing the views on outcomes of stakeholders and evaluators; and examining why and how a program performs. This paper focuses one of the main strands of theory-oriented evaluation: Realistic (or Realist) Evaluation developed by Pawson and Tilley (1997). Realistic evaluation is interested in ‘program theory’, thinking about the responses of people to program activities. Appendix 1 lists the steps in undertaking realistic evaluation. 41 Chen and Rossi’s formulation was called theory-driven evaluation, while the latest approaches are identified as theory-based evaluation. 64 In practical terms, realistic evaluation parallels theory of change since it involves hypothesizing in advance the mechanisms that are likely to operate, the contexts in which they may operate, and the outcomes that will be observed if they operate as expected, in what is known as Context-Mechanism-Outcome (CMO) hypotheses. Whether mechanisms work as expected at the time of implementation depends on context. As Box 1. Basic Definitions used in this Essay After an extensive literature review to support the essays in this dissertation, it is clear that a major problem in the way many existing approaches are presented is in the semantics. There are a number of concepts used freely in the academic and practitioner literature that can easily lead to confusion. The following definitions are drawn from Wong et al. (2013) and Westhorp (2014) and are the ones considered for this essay in particular. Impacts are positive or negative changes produced by a development intervention – directly or indirectly, intended or unintended- in the context of its environment, as it interacts with multiple factors affecting development change. Impact implies changes for people in their lives. Outcomes are short, medium and long term changes, intended and unintended, resulting from an intervention. Outcome implies change for people and their lives, but can also include other kind of changes (for organizations, workers, governments, etc.) Impact evaluation is a systematic and empirical investigation of the impacts produced by an intervention – it seeks to establish whether an intervention has made a difference in the lives of people. It aims to answer questions about what works, how, for whom and why. An impact evaluation addresses causation (how programs cause change) and attribution (whether observed changes can be attributed to the program or were caused by other things). Mechanisms are the underlying causal processes that happen before the outcomes we observe. Development interventions in a given context trigger mechanisms, and these condition the outcomes we observe. Program mechanism is the interaction between what the program provides and the reasoning of its intended target population that causes the outcomes. Source: Wong et al. (2013) and Westhorp (2014) 65 complexity theory literature argues, the most relevant networks of causation for the problem at stake should be selected to articulate solutions that are most likely to have an effect on desired outcomes (Ramalingam, 2014). In Westhorp’s (2014) framework, “what matters about context is what influences, whether mechanisms operate, and which mechanisms operate”42. Figure 2 helps in visualizing this. What follows is a framework to identify elements of context and the mechanisms at play when designing an evaluation. Figure 2. The Context – Mechanism – Outcome (CMO) Hypothesis Source: Author’s conceptualization 42 Westhorp (2014, p. 6). Wong et al. (2013) also highlight the role of realism and context: “In summary, realism holds that mechanisms matter because they generate outcomes, and that context matters because it changes…the processes by which an intervention produces an outcome. Both context and mechanism must therefore be systematically researched along with intervention and outcome. By implication, research or evaluation designs that strip away or ‘control’ for context with a view to exposing the ‘pure’ effect of the intervention limit our ability to understand how, when and for whom the intervention will be effective” (p.13). 66 III. Analytical Framework III.1 From intervention to mechanisms: Looking at formal and informal institutions The focus of this paper is on context as prevailing formal and informal institutions, using North (1990) and Little (1991) framing. This look at institutions when analyzing context is based on the little attention that interventions based on best practice approaches pay to the role of institutional practices and organization on program impact43. North defines institutions as “the humanly devised constraints that structure political, economic and social interaction […] consisting of both informal constraints (sanctions, taboos, customs, traditions, and codes of conduct), and formal rules (constitutions, laws, property rights)”. Little, on the other hand, discusses the constituents of institutions: norms (formal and informal, deliberately inculcated and reinforced and those influencing the behavior of agents); procedures; routinized behaviors; and rules. Context, in the form of formal and informal institutions, determines how good a given intervention is at triggering a targeted mechanism. The four constituents of institutions act through development actors at three levels44: the participating agents (workers, mid-level officers and leaders), the organization itself (particularly looking at its structures and how it preserves its 43 Andrews (2010) calls this ‘isomorphic mimicry’, a view of institutional reform that is based on the form of the desired change instead of the functions that the related institutions need to have for an intervention to work. 44 I borrow Andrews et al. (2012) elements of organizations that bring about change under their Problem- Driven Implementation Approach (PDIA). The selection of the PDIA approach for this framework is based on the characteristics it shares with realistic evaluation, particularly the heavy weight it assigns to context, and the focus on problems over pre-defined solutions, parallel to the emphasis on not defining the tool to be used in the evaluation before establishing what are the questions that the tools need to answer. 67 legitimacy), and the ecosystem of the organization (an open or closed system, and the incentives provided from above to innovate or not). The magnitude and direction of the intervention’s effect will then depend on the presence or absence of: i. An ecosystem with ‘authorizing environments’ for decision-making (particularly if they encourage experimentation and ‘positive deviance’) ii. Organizations with active learning mechanisms and iterative feedback loops, and iii. Broad engagement of agents for assuring viability, legitimacy and relevance. Thus, when evaluating a program, this framework warrants initially questioning how the originally planned intervention is modified by the different elements of context when implemented. The evaluation tools for this part of the analysis can include a mix of qualitative and quantitative tools, aiming at answering questions such as: i. Is the change introduced by the intervention supported by all the relevant players ii. Are the internal structure and functions of the implementing organization conducive to a successful implementation? iii. Are all elements of society, particularly the target population, aware and supportive of the intervention? iv. Is the intervention, due to context, triggering other mechanisms instead or aside of the targeted one, with potential consequences (positive or negative) in terms of outcomes? Organizational analysis, behavioral analysis, political economy analysis, among others can be used as part of implementation evaluation, formative evaluation, or process evaluation to assess whether the intervention was successfully implemented (and why it was or not the case), to evaluate if the intervention actually triggered the targeted 68 mechanism, and if other behaviors were triggered during implementation. Appendix 2 features a tentative list of tools used as part of the monitoring and evaluation process that could generate information on the path from intervention to mechanism. Figure 3 on the other hand, shows how the intervention may be affected by context and end up triggering other mechanisms (such as resistance from a key stakeholder to the reform introduced), or affecting the strength with which the intervention triggers the targeted mechanism. Figure 3. From Context to Mechanisms Source: Author’s conceptualization III.2 Triggering Mechanisms: Change in a complex world A development intervention is designed and implemented under the assumption that it will trigger a mechanism, causing changes in behavior from key actors that lead to expected outcomes. Once the intervention is introduced, context may either: 69 i. alter the magnitude of the effect of the intervention over the mechanism altering expected outcomes; or ii. trigger alternative mechanisms that may influence the final outcome. Mechanisms are subject to context, and this interaction is what determines how change happens in a given place and time. Here I follow Eyben et al.’s (2008) discussion of Meta-theories and Archetypes, to understand the drivers of change. They provided an analysis focused on political change, but that can be adapted to other types of programs (including microfinance). In a previous work, Eyben (2008) argued how, even if unconsciously, people use theory for explaining the world to themselves and to others, and what is perceived as usual way of thinking about the world shapes their practice. New ways of thinking offer the potential to make choices about practice. Hence, when we interpret a development problem, we use different lenses (systemic meta-theories) to understand its dynamics. The common way to interpret reality used in development projects is the framework of rational choice, where change is the unintended consequence of individual choice. However, it is important to recognize the existence of other ways to see the world, and understand that in certain scenarios, rational choice might not be the best way to understand the outcome of a development program, because some of the development actors engaged (or whose behavior the program aims to change) might manage a different view of the same reality, and this needs to be considered in the policy design. Alternative meta-theories include, for example, evolution, where variation, selection and amplification are the dominant drivers; structuralism, for which changes in relations of 70 production and economic power structures are the driving forces; long term shifts in deep underlying norms, values and beliefs; among others45. An archetype, on the other hand, is the identification of how under a dominant meta-theory, change can actually happen in a given place and time. Think for example of participatory development: if change in a given context can happen through active citizenship, then a participatory program merely providing spaces for participation (demand side of governance) may be enough to trigger change; if experience shows that change does not happen unless local elites are on board, then elite networking or coalitions may be the needed focus to consider when designing the program; if experience shows that it is through contagion that change occurs, demonstration effects and localized pilots of new approaches should likely be the focus of the intervention. Thus, when Banerjee and Duflo (2011) say that many of the failures in development are due to the “ubiquitous three Is: ignorance, ideology and inertia”, it may well be that stakeholders’ reaction to the intervention is due to a non-linear, bounded rationality scheme as meta-theory, or the assumption of a certain archetype of change, while the intervention was designed with a rational choice approach. In realistic evaluation lingo, the intervention, as designed, did not trigger the expected mechanisms, generating unexpected results46. In this environment, realistic evaluation looks at context and the mechanisms that were supposed to be triggered by the intervention, and helps 45 Beinhocker (2006) does an abstraction of ‘drivers’ of complexity economics and that are behind many of these meta-theories. These are dynamics, agents, networks, emergence and evolution. 46 Results can also be unexpected and negative ones when there are alternative mechanisms triggered (such as the negative reaction of men to the female brochure in the Pakistan case study) 71 better understand why an intervention may not have had the expected result for a given group of expected beneficiaries. III.3 Evaluation: From Intervention to Outcomes Realistic evaluation can assess the intermediate steps from intervention to outcomes and how the assumptions behind the original hypothesis are strengthened or weakened along the process. This exercise feeds information that can be used in the re- design of development interventions. A useful representation of how the intervention loses steam through context or acting mechanisms and at different levels of outputs is the one provided by White (2014) as the “Funnel of Attrition”. As Figure 4 shows, achieving development outcomes depends on a number of factors that should be considered at the design stage. Figure 4. The Funnel of Attrition Source: White (2014) 72 Along the funnel, only the people that reach a given step can experience the benefits of the step that follows. In other words, only those people for which the outputs have been realized can actually achieve the desired outcomes in the last step. The literature review that supports this essay informs the identification of different elements of an evaluation that is built upon the C-M-O hypothesis that underlies a development program: i. Before Implementation: For the type of intervention explored, the C-M-O hypothesis is defined and informs the discussion of strengths and potential pitfalls of the alternative intervention designs with local experts and practitioners, with intensive evidence review also conducted. At this design stage, limitations in the intervention can be corrected. The information may even lead to the conclusion that the proposed intervention is not adequate given the prevailing institutions and prevailing agents’ behavior. ii. Formative and Implementation Evaluation: Most impact evaluation specialists prefer analyzing large scale programs to achieve enough statistical power, but this is of no relevance if the program has technical problems during implementation, or if the targeted population is not interested in the program. This stage encourages the experimentation of promising approaches identified in the pre- evaluation stage at a small scale, with strong emphasis in monitoring key indicators developed on the basis of the C-M-O hypothesis, for an iterative learning process, similar to what Pritchett et al. (2013) propose as ‘experiential learning’. The idea is to try a design of the intervention in one community, learn, adapt and try it in a few more. Each intervention tested is monitored doing causal 73 chain analysis along the funnel of attrition. This part of the evaluation can very well be based on factual analysis. Advanced planning is highly desirable so appropriate monitoring indicators are tracked, and the information generated is policy oriented. iii. Impact Evaluation. The previous elements help understand the process at play during program implementation. There is however, still an important role to play for experimental and quasi-experimental methods that are able to answer if a given intervention does work or not. This evaluation, however, has a stronger factual basis, because the design of the intervention has considered context and potential alternative mechanisms triggered, and the previous analysis generates information to better understand the outcomes.47 West et al. (2008) classifies alternatives to RCT designs for strengthening causal inference in three major categories: - Randomized Encouragement, with random assignment to different intervention designs. - Non-Random Quantitative Assignment, with treatment only applied to those people falling below a threshold for one of the outcome variables. An example of a method in this category is regression discontinuity. - Observational Studies, where participants in preexisting or constructed groups receive various treatment conditions, and selection bias is controlled using matching and stratifying techniques. 47 Alternatively, the use of mainstream impact evaluation methods can be embedded in program implementation. For example, this can include randomizing the assignment of small variations (nudges) of the intervention as part of formative evaluation. On this particular point, see Pritchett et al. (2013). 74 I now revisit the Pakistan case study, highlighting the context, the mechanisms at play and the outcomes of the program introduced, and proposing an evaluation design that is able to assess not only if the intervention works, but how and why it does so, and provides recommendations for adapting the interventions to better respond to context. IV. Applying the Framework: Microfinance Experiment in Pakistan The marketing experiment testing the effect of a brochure with women managing small businesses (compared to a brochure with the same businesses but managed by men) concluded that exposure to the brochure decreased the demand for the new microfinance product among the poorer and least educated women (and men) members of NRSP credit groups. Further, essay one ended explaining how, in hindsight, the results of the experiment as it was designed could have been reached with a comprehensive data analysis of available information on women’s demand for credit in Pakistan, demand for credit among female members of the participating microfinance organization, and even using the data from baseline, collected one full year before the brochures were distributed (November 2006, compared to orientation and brochure distribution in November 2007). This factual analysis could have been used to inform an alternative design of the brochures that considered a context of limited participation of women in economic activity. Still, the experiment provided evaluators and microfinance organization with very rich information for similar programs. What follows is a systematic review of how (unexpected) change happened as a consequence of the intervention, and a proposed framework to design evaluation of similar initiatives in rural Pakistan moving forward. 75 IV.1 Understanding the Experiment’s Results Figure 5 synthetizes the C-M-O hypothesis evaluated; the relevant characteristics of context; how the context translates into agents’ behavior; the expected and additional mechanisms triggered and a comparison between expected and actual outcomes48. The development problem that the microfinance organization tries to solve is a low loan take up among female members of its credit groups in rural Pakistan. The literature on microfinance has a very rich spectrum of prospective solutions for the problem, including preferential interest rates, insurance schemes, and business skills training, among others. Marketing is only one alternative solution among many, a type of intervention particularly appealing given its relatively low cost, and with the backing of well documented literature on its impact in behavioral change. The conclusion of the experiment was not that marketing is ineffective as a tool to increase take up rates among women in rural Pakistan; but that one particular design, the brochure featuring women leading five small businesses considered typical of average microcredit clients, does not achieve the desired outcomes, compared to the brochure showing men leading the same businesses. 48 For illustration purposes, only the C-M-O hypothesis or theory of change for the marketing experiment is featured here. However, consider that the marketing intervention represents one element of the larger C-M-O hypothesis for the microfinance experiment of expanding the loan amount that credit group members can have access to. 76 77 IV.1.1 Context Analysis Evaluating the C-M-O hypothesis behind the experiment increases the evaluator’s understanding of the ‘how’ and ‘why’ the intervention works or not. Data from the experiment and the baseline provided evidence on elements of context at play, particularly social norms about women’s role in the participating communities. Three particular patterns arise from the analysis: i. Limited decision making of women in the household. Even among female business owners, 40 percent declare that their husband makes the main business decisions, such as purchase of inputs, hiring, and marketing. ii. Limited mobility of women within and beyond the community. Earlier studies of microfinance in Pakistan such as the one from Nenova et al. (2009) indicate that microfinance has traditionally been targeted to men rather than women precisely because low female mobility and gender segregation in labor markets. The baseline confirms this, as female business owners operate the business mainly from home, as they are the ones more likely to observe purdah. iii. Low regard for the household business as economic activity. Use of time modules included as part of the baseline and follow up surveys indicate that female business owners dedicate very little time to the business, and the same applies to their husbands, who on average dedicate more time to agricultural activities. In this sense, the data challenges the assumption of ‘entrepreneurs-in-waiting’ only lacking access to finance or knowledge that justifies the introduction of microfinance. 78 IV.1.2 Agents’ Behavior and Mechanisms Factual analysis, confirmed by the results of the experiment, provides a picture of an environment restrictive of women engagement in economic activities and general mobility. As a result, microfinance organizations’ rules and programs typically did not target women in any special way before the marketing experiment was introduced and as a consequence, lent more to men than women in terms of number and amount of loans. On top of this, credit group members (men and women) show a relatively low demand for loans. Average size of loan before the experiment was Rs. 12,000 (US$140), even though the regular program allowed borrowing up to Rs. 30,000 (US$350). And as Table 1 shows, four out of five credit group members that never took a loan between 2002 and 2006 indicated having not done so because they ‘do not like to borrow’. This is quite an important piece of information, because the outcome being measured by the experiment is take up of the new loan program (which offered access to loans for up to Rs. 100,000), not to the regular loan. Again, in hindsight, with most business of credit group members being small in scale and sales (which is particularly the case for female business owners), and an already low demand for credit in terms of the loan amount, the content of the brochure probably needed a more careful design in order to have a stronger likelihood of affecting take up in a significant way. 79 Table 1. Borrowing and Reasons for Not Borrowing (Jan. 2002-Nov. 2006) Commercial Bank MFI Informal Lenders Relatives and Friends Percent borrowing from [source] in 2006 5.07 83.43 33.77 21.12 Do not like to borrow 52.55 81.39 70.81 74.69 Inadequate collateral 18.50 1.98 11.94 6.77 Lender's procedures are too cumbersome 14.49 9.31 6.70 5.82 Lender's loan terms are unfavorable 5.48 0.59 3.35 0.91 Lender is too far away 3.18 0.40 0.78 1.35 Need to pay bribes 2.02 0.79 2.18 0.08 Past default with lender 1.96 0.79 0.89 1.15 CO members not willing to lend to me 1.25 4.55 2.51 6.53 Bad credit history 0.56 0.20 0.84 2.69 The main [reason] for not borrowing from [source]? Credit Source Source: Baseline survey 2006 As a result of this context, the female brochure triggered a negative reaction in terms of take up from the households that from a development point of view should be the main target of the program, relative to the male brochure. This lower take up restricts the access to the expected development outcomes down the results chain behind microfinance programs. IV.2 Using analysis for intervention design and evaluation IV.2.1 Informing intervention design As previously stated, the results of the marketing experiment do not provide conclusive evidence on the lack of effectiveness of marketing in microfinance, only of one particular design, for one particular objective (take up of large loan program), and for segments of the target population (women, and particularly the poorest among them). With a stronger understanding of the context and the mechanisms at play, it is now possible to propose alternative marketing interventions by ‘crawling the design 80 space’ (Pritchett et al., 2013). This requires the identification of the different elements of the design of marketing tools that can be used in the Pakistani rural setting. Assuming that within the category of marketing interventions the focus is on a low cost solution such as a brochure49, alternative brochure designs could consider: i. Brochure recipients (e.g. every member, only women, only women identified as belonging to the poorest households, only women that have not borrowing history with the microfinance organization). ii. Brochure Product (e.g. access to the increased loan or the regular loan program) iii. Brochure Content (e.g. pictures - maybe showing only household-based businesses- cartoons, success stories of people similar to the selected brochure recipient) iv. Brochure Main characters (e.g. only women, only men, or men and women working together in the business) v. Brochure Support (e.g. local leader publicly supporting the possibility of women leading a business, successful borrowers sharing their stories). vi. Brochure Timing (e.g. brochure distribution at the time of business skills training, at orientation, or repeated exposure during the loan application period). IV.2.2 Stages of Program Evaluation The new evaluation is then designed following the phases previously described: i. Before Implementation: a shortlist of alternative designs of the marketing intervention is discussed with practitioners, local leaders and actual clients to fine 49 Distributing a brochure with program information is only one particular way of marketing the program. Radio, advertising in public areas, door to door information provision are examples of alternative marketing tools that could be introduced. 81 tune and define which ones will be field-tested. As an example of what a brochure informed by different dimensions of the design space could include two candidate brochures that would be distributed among women members of credit groups with relatively worst socio-economic indicators: - Brochure 1: distributed only to female credit group members from the poorest households in the community, promoting the regular and the increased loan program, using pictures of women leading household-based businesses, distributed at orientation, with no additional backing. - Brochure 2: exact same design, but with pictures of the same businesses in the original brochure, showing both men and women leading them. ii. Formative/ Implementation Evaluation: The next step is to test the selected designs in limited areas and for a short period of time. Each brochure is tested in one community where NRSP works, but that was not included in the original experiment. Any changes in the actual demand for loans but also in the attitudes towards borrowing and towards owning a business are registered. For this to work, a monitoring system is set up that includes not only administrative data, but also qualitative information on household attitudes towards the brochure and towards borrowing in general. This data is collected through focus groups with members, analysis of existing female-led businesses and potential market for new businesses. For a sample of credit group members (men, women, business owners and not business owners) this can be complemented with behavioral analysis inquiring about people reactions to different scenarios to assess prevalence of risk aversion and perceived social restrictions to economic activities. 82 At the end of the formative evaluation, each brochure design would have been assessed against the different stages of the Funnel of Attrition, and the brochure design(s) that feature the least attrition along the causal chain is (are) selected for scale-up and impact evaluation. Information collected during the monitoring of the pilot is used for improving design if needed. iii. Impact Evaluation. The most promising brochure designs identified during the formative evaluation stage and for which monitoring indicators have been collected during implementation evaluation are scaled-up and evaluated. A control group is set up consisting of credit group members not exposed to any of the brochure designs, past or present. The monitoring system of the pilot phase informs the system set up for the scale-up phase, which will introduce questions in the baseline and follow up surveys of attitudes and household and business dynamics identified as relevant during the pilot phase. This includes the inclusion of qualitative indicators that keep track of how context still affects program performance and how. Depending on resources, timeframe and the group of people exposed to the brochures, a combination of methods can be used. Randomized encouragement designs with alternative intervention arms can be combined with regression discontinuity if the brochure is applied to members of different socio-economic levels in the community, and a household income threshold can be defined. 83 V. Epilogue: The Way Forward in Design and Evaluation Twelve years ago, I started my international development career working in program evaluation at a multilateral organization. Back then, the evaluation function was still very much about pure cost-benefit analysis, despite an emergent trend towards results-based management. A lot has happened since then. The turn towards rigorous impact evaluation that emphasizes the elimination of selection bias could be seen as a pendulum movement, from the days of low ambition in the evaluation function, to the meticulous quest for unequivocal attribution. The price of the transition has been a trend towards disregarding context and focusing too much in significance levels of variables of interest. At the height of the movement towards randomized evaluation, I recall sitting in numerous presentations where questions about how institutions, behaviors or human interaction influence and condition the results posed to the presenter were answered in a similar way: random assignment controls for that. The evaluation profession, however, seems to be in the path to a more balanced approach to the problem. There is an increasing understanding that the causal chain from intervention to outcomes needs to be understood, looking at how the intervention was implemented, how and why it worked, and for whom. Evaluation set up in this way is less prone to speculation on how to interpret results, and more useful for policy. As part of this process, there is emerging interest at promoting a better connection between evaluation and design. The consolidation of theory of change approaches and the recognition of complexity theory open an opportunity for a better dialogue between 84 program officers in charge of design and evaluators. This essay shows how a focus on context and its effect on mechanisms can contribute to improve the design of development programs, and lead to evaluation that answers the questions needed for policy prioritization and scale up. The result is an approach that emphasizes ‘best fit’ in intervention design by leveraging what has been learned in previous experiences and adapting the intervention type to respond better to context. This has significant consequences in terms of cost of implementation and of evaluation, by prioritizing the piloting and later scale-up of promising interventions that internalize context by design, disregarding intervention designs that while popular or appealing based on experience elsewhere, are not likely to be adopted by key stakeholders, and making a more rational and systematic use of the wide range of quantitative and qualitative methods in the evaluation toolbox. In the Pakistan case study, the proposed approach frames the problem in a different way than the original experiment. In the latter, the design of the overall field experiment was set up to answer if marketing would increase take up in microfinance programs in rural Pakistan. Moreover, that evaluation was embedded in a larger endeavor of assessing if either lack of access to credit or lack of business skills constrained entrepreneurship. The experiment failed to recognize that only one of many possible interventions tackling each of those questions was being evaluated. Thus, the failure of the female brochure in stimulating take up among poorest households is not enough to conclude that marketing tools are useless in rural Pakistan, in the same way that the apparent success of business training among men does not mean that it is skills and not money the constraint to entrepreneurship in rural Pakistan. In fact, access to credit was 85 not a problem for participants: many of them had been members of credit groups for years, with average loan size less than half of what was available to them. The proposed framework changes the research question from the onset. The focus is not in knowing if marketing works; instead, the question is how marketing works better in overcoming restrictive social norms and risk aversion. And that is a significantly more interesting question from the policy point of view. 86 Appendix 1. Steps involved in undertaking Realistic Evaluations The Realistic Evaluation has no set steps, but Blamey and Mackenzie (2007) map it in the following way for comparison purposes: Step 1: Through dialogue with program implementers, evaluator looks to understand the nature of the program, of the target population at whom it is aimed; in what kind of context will it operate; and what are the prevailing theories around the type of program under analysis. Step 2: Evaluator maps out a series of potential mini-theories that relate the various contexts of a program to the multiple mechanisms by which it might operate to produce different outcomes. For example, practitioner knowledge and existing evidence base might suggest that a component of a program should focus on one particular topic. Step 3: The evaluator undertakes an ‘outcome inquiry’ in relation to these mini theories. This involves building up a quantitative and qualitative picture of the program in action. This includes an assessment of the extent to which underlying mechanisms (motivation, incentives, social norms, or institutional culture) are triggered by specific interventions in target groups. Step 4: Through exploration of how context, mechanisms and outcome (CMO) configurations play out, the evaluator refines and develops tentative theories of what works for whom in what circumstances. These theories are the ones to be tested. 87 Appendix 2. Common M&E Tools Quantitative Qualitative Efficiency Effectiveness Outcomes Audits or Counts Budget Tracking Dartboard Deemed Savings Focus Group Footprint Calculators Lessons Learnt Workshop Literature Review LogFrame Matrix Meter Reading Metering Observation Outcome Hierarchy Problem Tree / Solution Tree Analysis Program Logic Project Diary Questionnaire Semi-structured Interview Stakeholder Analysis Storytelling Time Tracking Helps you measureType of Data CollectedTools 88 Bibliography Acemoglu, D. and J. Robinson, 2008. ‘The role of institutions on growth and development’ Commission on Growth and Development Working Paper n. 10. Washington DC: The World Bank. Andrews, M. 2009. ‘Isomorphism and the Limits to African Public Financial Management Reform’, HKS Faculty Research Working Paper Series RWP09-012, May 2009. Andrews, M., L. Pritchett, and M. Woolcock (2012a). ‘Escaping Capability Traps through Problem-Driven Iterative Adaptation (PDIA)’. UNU-WIDER Working Paper 2012/64. Helsinki: UNU-WIDER. Banerjee and Duflo, 2011. Poor Economics: A radical rethinking of the way to fight poverty. Washington: PublicAffairs. Banerjee, A., E. Duflo, R. Glennerster, C. Kinnan, 2009. “The miracle of microfinance? Evidence from a randomized evaluation”, unpublished paper. Barder, O., 2012. “Complexity, Adaptation and Results”. Center for Global Development: views from the Center Blog. Beinhocker, E. 2006. The Origin of Wealth: Evolution, Complexity, and the Radical Remaking of Economics. Cambridge: Harvard Business Press. Blamey, A. and M. Mackenzie, 2007. Theories of change and realistic evaluation: Peas in a pod or apples and oranges? Evaluation, vol. 13, n. 4, pp. 439-455. 89 Blattman, C., 2008. Impact Evaluation 2.0. Presentation at the Department for International Development, London, UK. Bovens, M., T Hart, P. & Kuipers, S., 2006. The Politics of Policy Evaluation (pp. 317- 333) in M. Moran, M. Rein, & R.E. Goodin (eds.), The Oxford Handbook of Public Policy,. Oxford: Oxford University Press. Card, D. S. Dellavigna and U. Malmendier, 2011. The Role of Theory in Field Experiments. Journal of Economic Perspectives, vol. 25, n. 3, pp. 39-62. Cartwright, N., 2007. Are RCTs the Gold Standard? BioSocieties n. 2, pp.11-20. Centre for the Future of the State, 2010. An Upside-down View of Governance. Brighton: IDS. Chen, H.T. and P. Rossi (1989). Issues of the Theory- Driven Perspective. Evaluation and Program Planning n. 12, pp. 299-306. Collier, P. 2007. The Bottom Billion: Why the poorest countries are failing and what can be done about it. Oxford: Oxford University Press. Das, J., S. Devarajan and J. Hammer, 2011. Lost in Translation. Article part of the Forum Small Changes, Big Results. Boston Review, March 2011. Deaton, A. 2010. Instruments, Randomization, and Learning about Development. Journal of Economic Literature 48 (June 2010) pp. 424–455. Dreze, J. and N. Stern, 1987. The theory of cost-benefit analysis, in A. Auerbach and M. Feldstein (eds.) Handbook of Public Economics, vol. 2, pp. 909-989. Cambridge: Elsevier. 90 Easterly, W. and J. Cohen, eds. 2010. What works in Development? Thinking big and thinking small. Washington: Brookings Institution Press. Easterly, W. 2008. Reinventing Foreign Aid. Cambridge: MIT Press. Easterly, W. 2007. The White Man's Burden: Why the West's Efforts to Aid the Rest Have Done So Much Ill and So Little Good. London: Penguin Books. Eyben, R., 2010. 'Hiding relations. The irony of 'effective aid'', European Journal of Development Research vol. 22, n.3, pp. 382-397. Eyben, R., 2008. 'Power, Mutual Accountability and Responsibility in the Practice of International Aid: A Relational Approach', IDS Working Paper 305, Brighton: IDS Eyben, R., 2005. 'Donors Learning Difficulties', IDS Bulletin 36.Brighton: IDS Eyben, R., Kidder, T., Rowlands, J., and Bronstein, A., 2008. 'Thinking about change for development practice: a case study from Oxfam GB', Development in Practice 18.2, Oxford: Development in Practice Gertler, P., 2004. Do conditional cash transfers improve child health? Evidence from PROGRESA control randomized experiment. American Economic Review, vol. 94, n. 2, pp. 336-341. Greenberg, B.G., 1968. Evaluation of Social Programs. Review of the International Statistical Institute, vol. 36, n. 3, pp. 260-278. Guijt, I. 2008. Critical Readings on Assessing and Learning for Social Change: A Review. IDS Development Bibliography 21. Brighton: IDS. 91 Howes, S. and P. Wyrwoll, 2012. Asia’s Wicked Environmental Problems. ADB Working Paper Series n. 348. Kahneman, D., 2011. Thinking Fast and Slow. New York: Farrar, Strauss and Giroux. Khagram, S., P. Rogers, D. Bonbright, S. Earl, F. Carden, Z. Ofir, and N. Macpherson, 2011. Principles for Action. The Impact Evaluation for Development (IE4D) Group. Kremer, M. and E. Miguel, 2004. Worms: identifying impacts on education and health in the presence of treatment externalities. Econometrica vol. 72, n. 1, pp. 159-217. Leonard, D., 2008. Where Are ‘Pockets’ of Effective Agencies Likely in Weak Governance States and Why? A Propositional Inventory. IDS Working Paper 306. Brighton: IDS. Levy, B. and F. Fukuyama, 2010. Development Strategies: Integrating Governance and Growth. World Bank Policy Research Working Paper Series.2010. Little, D., 1991. Varieties of social explanation. Boulder: Westview Press. Morrissey, O., 2001. Does Aid Increase Growth?, Progress in Development Studies, 1:1, 37-50. Moyo, D., 2009. Dead Aid: Why aid is not working and how there is a better way for Africa. Vancouver: Douglas & McIntyre. Nenova, T., and C. Nenang, with A. Ahmad, 2009. “Bringing Finance to Pakistan’s Poor: A Study on Access to Finance for the Underserved and Small Enterprises”, World Bank report. Washington, DC: World Bank 92 North, D. J. Wallis and B. Weingast, 2007. Limited Access Orders in the Developing World: A new approach to the problem of development. World Bank Policy Research Paper, n.4359. North, D., 1990. Institutions, Institutional Change and Economic Performance. Cambridge: Cambridge University Press. Ostrom, E., 2007. “A diagnostic approach for going beyond panaceas” in PNAS 4(39): 15181-7. Patton, M. Q., 2011. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. New York, NY: The Guilford Press. Pawson, R. and Tilley, N., 1997. Realistic Evaluation. London: Sage Publications. Pritchett, L., S. Samji, J. Hammer, 2013. It’s all about MeE, Center for Global Development Working Paper n. 322, April 2013.Pritchett, L., M. Woolcock, and M. Andrews, 2010. Capability Traps? The Mechanisms of Persistent Implementation Failure, Center for Global Development Working Paper No. 234. Ramalingam, B., 2014. Aid in the Edge of Chaos: Rethinking International Cooperation in a Complex World. Oxford: Oxford University Press. Ravallion, M. 2011. Are we really assessing development impact? Development Impact World Bank blog, May 2011. http://blogs.worldbank.org/impactevaluations/guest-post- by-martin-ravallion-are-we-really-assessing-development-impact. Ravallion, M., 2009. Evaluation in the Practice of Development . The World Bank Research Observer, vol. 24, n. 1, pp. 29-53. 93 Ridell, R., 2008. Does Foreign Aid Really Work? Oxford: OUP. Rodrik, D., 2010. The New Development Economics: We shall experiment, but how shall we learn? In W. Easterly and J. Cohen (eds.) What works in Development? Thinking big and thinking small. Washington: Brookings Institution Press. Rodrik, D., 2010. ‘Diagnostics before Prescription’, Journal of Economic Perspectives, vol. 24, n. 3, pp. 33-44. Rodrik, D., 2008. ‘Second-Best Institutions’, American Economic Review, Papers and Proceedings, May 2008. Roll, M., 2014.. The state that works: “Pockets of Effectiveness” as a perspective on stateness in developing countries‟, in T. Bierschenk and J.P. Olivier de Sardan (eds): States at work, Leiden: Brill. Roll, M., 2011: Pockets of Effectiveness: A theoretical framework, draft paper. Sachs, J., 2005. The End of Poverty: Economic Possibilities for Our Time. London: Penguin Books. Senge, P., 1992. Systems thinking: A Language for Learning and Acting. Farmington, MA: Innovation Associates. Serrat, O., 2009. “Understanding complexity”, in Knowledge Solutions 66, pp. 1-9. Shepherd, G., 2003, Civil Service Reform in Developing Countries: Why is it Going Badly? Paper presented at the 11th International Anti-Corruption Conference, 25-28 May 2003, Seoul, Republic of Korea. 94 Sida, 2005. The Use and Abuse of the of the logical framework approach. Resultatredo visnings projekt. Simon, H.A., 1957. Models of Man. New York: Wiley. Stame, N., 2004. Theory-based evaluation and varieties of complexity. Evaluation, vol. 10, n. 1, pp.58-76. Stern, E., 2008. Evaluation: Critical for Whom and Connected to What? Evaluation, vol. 14, n. 2, pp. 249-257. Stern, E., 2005. ‘Contextual Challenges for Evaluation Practice’ in Shaw, I., J.C. Greene and M. Mark (eds.) The Sage Handbook of Evaluation. London - Thousand Oaks: Sage Publications. Sutton, R., 1999. The Policy Process: An Overview. Working Paper 118. London: ODI. Szekely, M. (2011). Toward Results-Based Social Policy Design and Implementation. Center for Global Development, Working Paper 249. Weiss, C., 1998. Have we learned anything new about the use of evaluation? American Journal of Evaluation vol.19, n. 1, pp. 21–33. Weiss, C., 1995. Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families in Connell, J, Kubisch, A, Schorr, L, and Weiss, C. (Eds.) ‘New Approaches to Evaluating Community Initiatives’. Washington, DC: Aspen Institute. 95 West, S., N. Duan, W. Pequgnat, 2008. Alternatives to the Randomized Control Trial. American Journal of Public Health, vol. 98, n. 8, pp. 1359-1366. Westhorp, G., 2014. Realist Impact Evaluation: An introduction. Methods Lab publications, ODI. White, H., 2014. Current Challenges in Impact Evaluation. European Journal of Development Research n. 26, pp 18-30. Wong, G., G. Westhorp, R. Pawson, and T. Greenhalgh, 2012. Realist Synthesis. RAMESES. Woolcock, M., 2009. Interim Institutions and the Development Process: Opening Spaces for Reform in Cambodia and Indonesia‖, BWPI Working Paper 86. World Bank Institute, 2011. The Coalition Building Toolkit (CBT), Meeting the Multiple and Interconnected Needs of Change Agents and Reform Coalitions., Unpublished Draft paper, The World Bank Institute, Washington D.C. World Public Opinion, 2010. Misinformation and the 2010 US Election: A study of the US Electorate. 96 Essay Three: We built it, they came. Now what? Participatory Strategies in Municipalities of Peru: The Case of El Alto, Piura50 “Unfortunately, policy decisions on local participatory development have historically been driven by fads […] Passionate advocates spark a wave of interest, followed, after a few years, by disillusionment, which gives ammunition to centralizers to engineer a sharp reversal…” (Mansuri and Rao, 2011b) I. Introduction The final essay deals with the challenges of introducing an institutional reform: local government-led participatory processes arising in districts across Peru during the last decade, with a case study for El Alto, a district in the northern region of Piura. In late 2012, the Mayor’s office asked the Peruvian think tank FORO Nacional- Internacional to evaluate their participatory strategy51, launched in January 2011, and to provide recommendations for its future. I joined FORO in this endeavor, under the Measuring Institutional Impact in the Region of the Americas (MIRA) initiative of the Inter- 50 I acknowledge funding from the Inter-American Development Bank, through its Measuring Impact of Institutional Reform in the Americas (MIRA initiative) competition for the design and implementation of this evaluation; and total collaboration from local authorities of the El Alto district in Piura, Peru, particularly of Mayor Zarate. The original design of the evaluation was selected for funding through a regional competition, and had to be adapted to the realities of the field, the political economy of the area, and the political events during the fieldwork. In collaboration with the local think tank FORO Nacional- Internacional, field workers were trained and sent to El Alto to monitor and record proceedings of neighborhood meetings and collect information from official records and through structured interviews. All analysis and any mistake or omission found in this paper is of my exclusive responsibility, and does not represent the views of the Inter-American Development Bank, FORO Nacional Internacional, or the Municipality of El Alto. 51 A “participatory strategy” is defined as a set of interventions to gather preferences and concerns of citizens and use them in the design, monitoring or evaluation of policy. As this study shows, the participatory strategy of El Alto featured a number of interventions phased in a three year period, and with constant changes in implementation due primarily to poor planning and political events at the local level. 97 American Development Bank. Together with the research team at FORO, we turned this retrospective evaluation with very limited data into an opportunity to collaborate in the redesign of the participatory strategy in El Alto. Accordingly, this study goes beyond the assessment of immediate program objectives, and evaluates the implementation of the strategy, identifying the mechanisms at play, and highlighting existing gaps in the strategy. This information leads then to concrete policy recommendations to improve the strategy by “crawling the design space” (Pritchett, Samji and Hammer, 2013). . Another particular feature of this evaluation is the unit of analysis. It focuses on a strategy, understood as a set of interventions and the institutional setting surrounding them. Thus, the research is built on recent literature related to strategy evaluation (Patrizi and Quinn Patton, 2010), participation and its alleged shortcomings (Mansuri and Rao, 2011a; Banerjee, Duflo, Glernnerster and Khemani, 2010); the call for participatory approaches that integrate demand and supply-side of governance (Fox, 2014); and experiential learning (Pritchett, Samji and Hammer, 2013) I.1 The Participatory Development debate Recent impact evaluations of participatory development programs tend to highlight their shortcomings. For example, Banerjee et al. (2010) conclude their influential randomized evaluation of participatory mechanisms in the education sector in India with a categorical statement: “Whatever the explanation, it seems clear that the current faith in participation as a panacea for the problems of service delivery is unwarranted. It is possible that it can be made to work on a more systematic basis, but it would take a lot of patience and experimentation to get there” (p.25, italics are mine). 98 The study of participatory development programs has in fact fluctuated from overly optimistic, generating high expectations, to grim and leading to categorical and negative assessments52. Mansuri and Rao (2011a) refer to the manner in which participation has been introduced in community-based development programs and part of decentralization processes as ‘a tarnished silver bullet’. They call for a contextually sensitive diagnosis of the relationship between civil society, government and markets, so the strategy of local development can be adapted accordingly. They have a very valid point: participatory programs are typically conceptualized as the solution to market and government failures; so citizen involvement is assumed to be a guarantee of service delivery targeted to a community’s needs, and of downward accountability. This view however, ignores that ‘civil society failures’ are also possible and may distort the expected outcomes of participation (see Box 1). Gaventa and Barrett (2010) show a mixed but more encouraging picture of participation. Their systematic review and outcome mapping of 100 case studies in twenty countries shows that negative examples are many times related to citizen engagement hitting ‘bureaucratic brick walls’, government failure to implement decisions, and even reprisals (sometimes, violent). They also recommend to carefully study the context of civil society, but with a different aim: to identify the ways citizens already participate (e.g., social organizations, associations, self-help groups), and then build programs around these, instead of imposing imported participatory mechanisms 52 The World Bank Participation Sourcebook (1996) can be considered an example of the former, and Cooke and Khotari (2001) of the latter, and in between, there is a rich literature. See, among many others, Mansuri and Rao (2004), Chambers (2009), Uphoff (1998), Gould (1996), Alkire (2002). 99 ms. In a similar line, Crocker (2006) classifies and responds to the major objections to the ideal of deliberative participation: asymmetry of power (also called indeterminacy), autonomy and realism, all of which can be linked to civil society bonding failures53. These objections are used as arguments to conclude that participation ‘does not work’, or at least, that its use is not warranted. Crocker, however, stresses that they can be overcome through better institutional designs. At least two of those objections, namely asymmetries of power and realism, are in fact reasons for which participation may not work as expected, not flaws of participation itself: the participatory method chosen could be perfected or adapted to improve its performance. The autonomy objection, on the other hand requires a differentiated response, since the imposition of participatory development in contexts where hierarchical decision-making is the rule might backfire and even compromise the integrity of minorities or activists. Crocker provides again criteria to respond to this, consistent with the value of agency (Sen, 1999), inquiring if everyone is really in agreement with the 53 Realism objection can be linked to both binding and bridging failures of civil society. Box 1. Civil Society Failures: Bonding and Bridging Bonding failures are internal to the group, e.g. elite in a village capture public resources for private gain, intergroup violence, or when a community cannot devise equitable and efficient rules of engagement or to manage common property. Bridging failures arise when citizens cannot organize to correct for government or market failures, e.g. when a community cannot hold service providers or local officials accountable for resource allocation or quality of public services. Source: Mansuri and Rao (2011b) 100 prevailing system; if everyone in the community has information about alternatives; and if those in disagreement have the right and means to exit the group (Reich, 2002). I.2 Evaluating participation A literature review of participatory development programs observes that their effectiveness is typically assessed in terms of ‘it works’ or ‘it does not work’. This is not exclusive of the evaluation of participation, but it is notorious because participatory programs are typically implemented as a method to plan, monitor, implement or assess a development process54, but in the literature they are judged with the same criteria used for more concrete interventions that have mechanisms more amenable to measurement, such as cash transfers, infrastructure provision, or the increase in the number of nurses or teachers in a locality. Understanding success in participatory programs, requires a look beyond individual interventions, that allows exploring how and why participation works (or does not work) in a given context. Fox (2014), for example, offers quite an interesting perspective on how to assess social accountability, a concept that in general refers to development interventions attempting to improve public sector performance by bolstering both citizen engagement and government responsiveness in one or more of three dimensions: transparency, participation and accountability (O’Meally, 2013; World Bank, 2004). Fox argues that the existing evidence from impact evaluations leads to different conclusions on social accountability when it is seen either as a tactical or a strategic approach. 54 In this regard, Bell (1994) talks of ‘the tyranny of methodology’ in participatory approaches and the way they are exported to developing countries. 101 A tactical approach involves the use and analysis of a bounded intervention. When applied to social accountability, a tactical approach typically refers only to the demand-side of the problem (civil society based efforts) and assumes that information provision alone will inspire a strong enough collective action to influence public sector performance. A strategic approach, on the other hand, involves the use or analysis of multiple tactics or tools that are (at least in theory) mutually-reinforcing. This type of approach focuses on understanding enabling environments for collective action, and attempts to coordinate demand-side initiatives with government reforms (supply-side)55. Thus, Fox’s review concludes that while the evidence of results of tactical approaches is mixed (and tends to be interpreted as negative), the evidence on strategic approaches is much more promising. Assessing participation as a strategy, however, requires looking at it as the sum of different elements that taken individually may have a partial impact, only under certain conditions, and only for certain dimensions of the problem at stake. The tactical versus strategic debate poses indeed a challenge for evaluation. The literature has been relatively late at providing methodological alternatives that suit particularly the evaluation of strategies, even though evaluation of the work of organizations and of large policy frameworks has been common for a long time. Patrizi and Quinn Patton (2010) try to fill this void with case studies and a proposed methodology, which will be used throughout this essay. I.3 The value of participation 55 The tactical vs. strategic debate is not original from the social development arena: it originated in the business strategy and planning literature for the private sector. Mintzberg et al. (2005) and Mintzberg (2007) are common references in the discussion of strategy and strategy evaluation in the private sector. 102 Even after defining the best way to evaluate a participatory strategy, the question remains of what outcomes of participation to assess. Participation is typically designed as a component of a larger program, with its goal a combination of improved service delivery and downward accountability of service providers or local authorities56. This instrumental value is important, but it is not the only value of participation. Alkire (2002), based on Sen’s assessment of the value of democracy, argues that participation has also intrinsic and constructive value57. Indeed, Mansuri and Rao (2011b) admit that “there appears to be some intrinsic value of participation: people are generally happier when consulted…” However, measuring changes in collective capacity or the achievement of agency is quite challenging, as the limited use of such indicators across the literature indicates. And even when proxies linked to intrinsic and constructive value are considered, there is pressure (in the operational, political and academic realms) to report results for a very limited span of time, even though the kind of transformational effects related to intrinsic and constructive values typically require a long time to be observed. I.4 Quality and potential of participatory mechanisms The other key dilemma is how to tackle and measure the quality of participation. Crocker’s (2006) classification from thinner to thicker modes of participation in group 56 The latest World Bank report on Demand for Good Governance for Human Development (June 2011) still carries this view, institutionalized at the Bank since the World Development Report Making Services Work for the Poor (2004). 57 Development practitioners typically translate the intrinsic value of participation as ‘empowerment’, but this term is used in the literature more to signal ‘opportunity’ or ‘choice’, while Alkire and Sen actually mean agency-achievement. Typical indicators used to evaluate empowerment are decision-making power, freedom of movement, among others. Constructive value refers to the possibility of value formation through participation, as an opportunity for citizens to learn from each other and form the value and priorities of a society. 103 decision-making is a good way to identify the starting point of the participatory program under analysis, and assess if the introduction of a thicker mode is possible in each context. In a more recent work, Crocker (2008) calls to ‘improve the democratic body along one or more of the dimensions of breadth, range, depth or control’. I.5 Participation and Local Governance Finally, even if participation can be adapted to respond and overcome context, the question remains if participation can or cannot change local governance processes and dynamics. Institutional and political context mediates (in a positive or negative manner) the impact of participation. Acemoglu and Robinson (2012) are rather skeptic of development programs in general being able to break poor governance practices, while Banerjee and Duflo (2011) argue that while good policy is affected by local politics, it can also create the incentives for good politics. Considering this, the study explores if providing El Alto citizens with more opportunities to voice their demands and concerns has altered in any way the organization and decision-making processes within the Municipality, and if the strategy has engaged other stakeholders that may see the process as a threat to their own position of advantage in the community58. Instrumental for this analysis is recent literature on public sector reform, particularly Andrews’ (2009) framework for driving change in public administration and Malinga’s (2008) research on the role of ‘middle managers’ in public sector to understand incentives of municipal office managers. I.6 Organization of the Essay 58 On this point, see Selee (2012) for case studies in Mexico, where the parties pushing for more and better participatory mechanisms were the ones undermining them once their own party reached power. 104 Section I briefly introduced the research project and laid out the relevant literature. Section II features the national and local context in which the strategy was introduced. Section III features the evaluation framework and describes the data collected for the analysis. Section IV describes the empirical strategy and the introduction of theories of change and process mapping as elements of the evaluation to inform a redesign of the strategy and its evaluation; and Section V presents the results of the analysis. Lastly, Section VI presents the conclusions and recommendations to improve the performance over time of El Alto’s participatory strategy. II. Citizen Participation in Peru, and the Context of El Alto, Piura II.1 Local participation and local government in Peru Native forms of participation in community affairs have existed in Peru for centuries, and some elements of them can still be observed in the life of many rural communities of the Andes and the Amazonia. However, the rapid urbanization process of the twentieth century, the internal conflict of the 1980s and the political centralism and authoritarianism of the Fujimori years in the 1990s shaped citizen relationships between and within communities, creating new spaces of heterogeneous actors for which no strong and inclusive social organization existed (Panfichi, ed. 2007; Remy, 2005). This context changed in the first years of the new millennia. As part of an effort from the State and civil society to promote reconciliation, decentralization and a closer relationship State-citizens, the Mesas de Concertación (Consensus Roundtables) first, and Participatory Budgeting later were introduced to foster decision-making processes that respond to local needs. 105 Participatory budgeting at the district and regional level in Peru is mandated by law59. A recent evaluation by The World Bank (2011) found encouraging facts around Peru’s participatory budgeting experience: notably, 36% of investment budget was allocated through the process by 2007. The same report, however, recognizes serious challenges to its sustainability: for example, by the same year, over 20% of Municipalities had not executed any project prioritized through participatory budgeting. Moreover, its impact has been limited because: a) only capital investment can be discussed (therefore excluding social expenditure by design) and b) it targets organizations, not individuals, in this way likely excluding most disadvantaged groups. In this particular point, McNulty (2012) questions the representativeness of participatory budgeting in Peru at the district level based on evidence that the most influential civil society organizations and government officials control the process60. Recent literature on participation in Peru has primarily focused on participatory budgeting and its potential to improve service delivery. However, one of the consequences of both participatory budgeting and the establishment of consensus roundtables has been repeatedly overlooked: the Municipality-led organization of neighborhood-based groups in participatory processes at the district level, particularly in urban areas of Peru61. 59 Peru’s Participatory Budget Law (Law 28056) was passed by Congress in 2003, and dictates that the capital investment costs of each regional, provincial, and local budget must be developed with civil society input. Following a series of steps—established by the Ministry of Economy and Finance (MEF)— subnational governments must demonstrate that they have complied with this process in order to receive their annual budgets. 60 The cited World Bank evaluation also describes how the costs of participation remain prohibitive in Peru, particularly in rural areas. 61 In most rural districts, the consensus roundtables (Mesas de Concertación) have become one of the main channels to connect civil society and local government beyond the participatory budgeting process. In other rural areas, native participatory arrangements are still strong and have their own channels of communication with their local government. 106 Neighborhood Committees (Juntas Vecinales) and their meetings (Reuniones Vecinales) have existed in urban Peru at least since the 1960s (Schonwalder, 2002). Originally, they were spaces to coordinate citizen-based social action in the absence of a strong presence of local and national government. During the height of internal conflict in the 1980s and early 1990s, their role shifted to coordinating citizen security at the local level. In parallel, neighborhood-level leadership developed over the years along party/political movement lines. Only after the 2003 Law of Municipalities (Ley Orgánica de Municipalidades), neighborhood-based participation was formally linked to the concept of local government. As the law states: “Local governments are fundamental entities of the territorial organization of the State and immediate channels of neighborhood participation in public affairs, which institutionalize and manage with autonomy the particular interests of their corresponding community, with essential elements of local government the territory, population and organization” Ley Orgánica de Municipalidades N. 27972, Article I This formal link between local government and neighborhood-based participation has been followed by the creation or formalization of Neighborhood Participation Offices as part of the organigram of many Municipalities in Peru, and the design of new forms of 107 engagement with local residents, through participatory strategies. Such strategies share a number of characteristics62: i. Electoral origin, with candidates before municipal elections promising to establish or strengthen a participatory system, or incumbent Mayors thinking of how to connect with people before the following election. ii. Multiple interventions to increase the channels of communication between the Municipality and citizens, year round. iii. Rhetoric around participation at the individual-level, but in practice a strong reliance either on civil society organizations or established leadership as intermediaries, particularly in larger districts. II.2 Local participation in the context of El Alto -Piura The case study focuses on El Alto district in Piura, with around 7,000 inhabitants organized in 36 neighborhoods (only 30 of them recognized by the Municipality)63. What follows is a description of the context in which the participatory strategy was implemented. II.2.1.El Alto’s birth and development is closely related to its Oil and Fishing industries El Alto district is in the Talara province, Piura Region, in the north of Peru. Its history is closely linked to that of oil exploration in the country, with origins that go back 62 As already stated, there is very limited literature on local participatory processes in Peru beyond participatory budgeting. The characteristics laid out here are the result of interviews with officials in charge of the Municipality of Lima’s Neighbor Participation Office, the office of the Mayor in El Alto, Piura, interviews with local leaders, NGO representatives and discussions with academics. 63 A similar retrospective analysis also funded by the Inter-American Development Bank was made for el Cercado de Lima (downtown Lima). That study can be found in a forthcoming book of the Inter-American Development Bank that presents eight case studies across Latin America. El Alto was selected for this essay because it allowed a more in-depth analysis thanks to the availability of socio-economic information from citizens attending neighborhood meetings. 108 to the 1920s, after a government concession to the British company Petrolera Lobitos Limited. The area received a constant influx of settlers to work in exploration and later extraction activities. The first neighborhoods (called barrios, some of them still existing nowadays) were formed along occupational lines. For example, Sanchez Cerro was residence to people selling liquor and food, and El Volante was home to the drivers of the oil company. Jobs were abundant and attracted people from different parts of the Peruvian north and beyond. During its times of affluence in the 1950s, the coastline of El Alto (Cabo Blanco, originally a very small fishermen village) became a vacation magnet for celebrities and millionaires from Europe and the United States. The confluence of the cold Humboldt Current and the warm waters near the Equator translates into an exceptional biological wealth. Figure 1. El Alto: Geographical location in the Region of Piura, Peru Source: Plan Concertado de Desarrollo Distrito de El Alto 2012-2021 Presently, the economic activity continues to gravitate around oil extraction and fishing. Petrobras (of Brazilian capital) and Savia (of Colombian-South Korean capital), lead oil extraction in El Alto and are key sources of employment and budget support (through oil 109 contributions, or canon). This economic structure, however, is expected to radically change. Savia and Petrobras have considerably reduced operations in the area, and a firm of Chinese capital will take over operations in 2015. There is also a lot of interest in fostering investment in the tourism sector, particularly around Cabo Blanco. While this transition takes place, the local government has become an important employer through investment in public works and maintenance, albeit jobs that are temporary and subject to availability of public resources. II.2.2 El Alto’s population is relatively homogenous As stated, 7,000 people live in El Alto divided in 30 recognized neighborhoods64. Size of the neighborhoods varies from 16 people in Los Jazmines to more than 600 people in El Volante and Ciudad del Pescador. Extended family members tend to live in the same neighborhood or nearby, many with family names of original inhabitants of the area dedicated to fisheries or migrants of the 1920-1960 period. Immigration into El Alto is limited nowadays, but there is a transient population flow for seasonal work with the oil companies. Broadly speaking, neighborhoods share very similar demographic characteristics regardless of size (Figure 2). Smaller neighborhoods (under 100 inhabitants) have on average a younger population, more non-remunerated workers, and less people with college education. Figure 2. El Alto: Characteristics of Population by Size of Neighborhood (over and under 100 inhabitants) 64 In practical terms, recognition implies that the neighborhood meetings organized by a Neighborhood Committee are considered representative of a neighborhood’s population and their proceedings formally received by the local government. 110 Source: Author’s calculations based on Population Census 2007. II.2.3 Limited social development, but low levels of poverty characterize El Alto The district of El Alto is comprised of 1,844 households. While 92% households have access to electricity, 35% of the population lives in households with no access to sanitation. Access to water, on the other hand, is virtually universal, with only 1% of households not having piped water; supply of water, however, is scarce. Human development index for the area is 0.63, compared to 0.73 for the country, with a 4.2% poverty rate and low levels of malnutrition partially explained by a diet rich in nutrients from fish and seafood. Literacy reaches 93%, with 13% having technical school education and 7% a completed college education. In short, El Alto is not particularly poor in terms of socio-economic indicators, but its relatively highly educated inhabitants struggle to find stable and well-paid jobs. II.2.4 Lower investment in public works due to smaller oil contributions Resources from the oil canon have supported public investment in the area. Figure 3 shows that, except the Cabo Blanco and Castro Pozo neighborhoods, public investment has been uniform vis-à-vis population distribution, mainly to build roads, sidewalks and water connection. Figure 3. El Alto: Public Investment (2010-2013) vs. Population (2012) 111 Source: Census Data and Municipality of El Alto records. In the last year of the previous administration (2010), S/. 17 million (roughly US$6 million) was invested in public works in the district. In three years of Mayor Zarate’s administration under study (2011-2013) a total of S/.33 million (or about US$12 million) has been invested, for an average of S/. 11 million a year, reflecting a decline in contributions from oil companies. The composition of investment in public works also changed, with a predilection for road/sidewalk construction. Between January 2011 and December 2013, Mayor Zarate’s administration invested 92% of public works investment in roads/sidewalks, compared to 8% invested in services related to education, water or security65. II.2.5 El Alto features regular political conflict and challenges to Mayor’s legitimacy Because its natural resources and the access to oil contributions, El Alto is a coveted district with regular political conflict. Peru’s electoral law contemplates the possibility of Revocatoria processes against elected mayors, regional presidents and 65 To put this in perspective, the last year of the previous administration (2010), saw 26% public works expenditure in education, security, tourism and water infrastructure (against 74% expenditure in roads/sidewalks). 112 elected magistrates66. The previous elected Mayor was voted out from office through a Revocatoria in 2009. A year later the current Mayor, Juan Zarate, was elected under an electoral platform that promised a renewed relationship between the local government and the people of El Alto. Then, in 2012, the administration of Mayor Zarate also faced a Revocatoria process, but survived the challenge67. II.2.6 The new participatory strategy is detached from existing channels of engagement The participatory strategy unfolded after Mayor Zarate took office in early 2011. However, before the strategy was introduced, there were already in place two channels for gathering citizen feedback and identify citizen demands: la Mesa de Partes and the Participatory Budgeting process. It is quite indicative that neither channel was considered as part of the participatory strategy of the new Administration68, and that both are managed by municipal offices different to the one in charge of the strategy. 66 Revocatoria, a legal process similar to the impeachment, is one of the direct democracy instruments introduced in Peru’s 1993 Constitution together with referendum, popular initiatives and demands for accountability. It is a procedure that has been increasingly used in the last few years. In 2012, 260 districts of Peru (out of a total 1,789 districts, or 14.5%) went through this process, with 69 Mayors (26% of districts undergoing revocation) losing their jobs after the new election. Critics point out that while a legitimate instrument, revocatorias in Peru do not feature ample deliberation to discuss and justify their application, and they only require enough signatures that can be collected and presented even by an individual citizen. The incumbent mayor or regional president cannot advertise his/her work during the process. There is a perception that these processes are tools for political revenge at the local level (Kresalja, 2012), a debate that intensified only after the Mayor of Lima, the capital and largest city in the country was subjected to a revocation process in 2012. 67 The story behind this is a good example of the misuse of the revocation process. After the vote, local press reported that the person promoting the revocation of Mayor Zarate’s mandate was his predecessor, former Mayor Rosa Machuca, who lost the 2009 Revocatoria vote. At that time, she was accused of a number of irregularities, but never sentenced. After leaving her post as Mayor she moved to the provincial capital, Talara to become Manager of the Human Development Office of the Municipality of Talara. She was removed from this office only after her participation in the process against Mayor Zarate was confirmed (Diario Virtual de Talara, November and December 2012). 68 In particular, there is no coordination between participatory budgeting and neighborhood meetings during the year, in spite of the participatory budgeting conditioning the agenda of neighborhood meetings. 113 i. Mesa de Partes. Residents can make requests to the municipal government through a letter to the Municipal Manager Office delivered in person at City Hall. There were 420 such requests between January 2011 and December 2013: 30% are related to employment opportunities, 18% requests for meeting municipal managers, 15% information requests, and among the rest, only a few are related to neighborhood problems such as safety, neighborhood upkeep, or land (either delimitation or use of it)69. ii. Participatory Budgeting. As is common practice in Peru, the process is led by the local Budget and Planning Office (GDP). In most districts of Peru there is deliberation among neighborhood participants proposing projects for the year to come, and then local authorities prepare investment profiles to be cleared by the Ministry of Economics and Finance regional office; however, in El Alto, under the argument of accelerating the process, the Budget and Planning Office prepares public works files considered priorities by the Municipality and then subjects those files to vote among neighbors for prioritization70. The participatory budgeting process of El Alto is therefore not deliberative, and it can be argued that not quite representative either, as citizen priorities are not discussed beforehand. Further, although neighborhood attendance during the participatory budgeting process increased from 30 people in 2011 to 75 in 2013, this still 69 In a context of increasing demand for job opportunities, neighbors of El Alto regularly push the Municipality to negotiate with the oil companies in the area for higher quotas in local labor. The local government also hires temporary labor for public works. Over time, the SPV has become an intermediary between neighbors and the Municipality, channeling job offers and collecting information on workers in the district. 70 Apparently, this process for Participatory Budgeting was established after serious delays in the first couple of years of its implementation attributed to the original deliberative practice (Source: interview with GDP Manager). 114 represents barely 1% of El Alto’s population71, compared to an average 3% nationwide. During the 2011-2013 period, neighborhood representatives attending participatory budgeting meetings averaged 14 years of education and were 55 years old on average. III. Evaluation Framework III.1 Strategy as a special unit of analysis for evaluation “However beautiful the strategy, you should occasionally look at the results.” Sir Winston Churchill (1874-1965) There is a reason why there is still so little in the development economics literature on how to evaluate strategies. Evaluators are used to have as unit of analysis either interventions or theories of change. Patrizi and Quinn Patton (2010) postulate the evaluation of a strategy as a different dimension of the evaluation problem. In general, a strategy is defined as “about where an organization is headed and how it intends to get there” (Patrizi, 2010). Mintzberg (2007) underscores that in evaluating strategies, the researcher should not focus on what was planned but in what actually happened during execution. In fact, strategy evaluation typically expects that some of what is planned will go unrealized, some will be implemented roughly as expected, and some new elements will emerge during implementation. The standard approach to evaluation, in contrast, is based on accountability, where program results are judged based on what was originally planned in terms of actions and intended outcomes. Figure 4 presents a representation of the subject of strategy evaluation. 71 In fact, recent reports indicate a reduction of 15% in participation during the budgeting process for 2014 compared to a year earlier. 115 Figure 4. An evaluation framework for tracking Strategy Source: Quinn Patton and Patrizi (2010), adaptation of Mintzberg (2007) In fact, Mintzberg claims that any strategy is one or more of four Ps: a plan, a perspective, a position, or a pattern72. I argue that the story of how the participatory strategy of El Alto came to be combines elements of a perspective and a pattern. While a perspective reflects internal drivers and preferences on how things should be done, a pattern arises when an organization learns from what worked (or not) in the past to guide future action. III.2 Research questions The nature of the strategy matters to determine the questions that the evaluation needs to answer and the tools to be used. Thus, to evaluate a strategy that is defined by perspective and patterns the analysis should assess the consistency of stakeholder behavior over time, understanding how the perspective changes or is adapted during 72 Previous work by Mintzberg (2005) referred to Strategy as 5P’s, with the fifth one being Ploy, and the use of tools such as game theory instrumental to build this type of strategy. Ploy has been absorbed as a Plan in more recent versions. 116 implementation (Mintzberg (2007), based on previous work by Simon (1957)). Tracking behavior can lead to important baseline observations—about actions, values, practices, processes, structures, competencies, networks, and beliefs—that can guide a strategy constructively. In order to do this, data for patterning should be as tangible as possible (Patrizi, 2010). Adapting Patrizi (2010) and Wehipeihana and Davidson’s (2010) guidelines for the evaluation of strategies as perspectives and patterns, the following are the evaluation questions posed in the analysis: A. On the Overall Strategy i. What is the organization’s strategic perspective? ii. What aspects of the intended strategy were implemented as planned? iii. What planned elements of the strategy were dropped? Why? iv. What unplanned/emergent strategies were implemented? Why? How? B. On the interventions used to implement the Strategy i. What is the value of a particular policy initiative as a contributor to strategic policy outcomes? ii. How well does each initiative fit with and complement the other initiatives that make up the strategic policy mix? iii. What is the collective value of the suite of initiatives to achieve a particular strategic outcome? III.3 Participatory Strategy of El Alto and its Objectives One of the first striking elements observed by the research team was the gap between the objectives of the participatory strategy as promised by Mayor Zarate during 117 the electoral campaign (and later the 2012-2021 Development Plan for El Alto); and the outcomes the Municipality requested to be measured through the evaluation. The Municipal administration had participation at the center of its Government Plan as an instrument for citizen empowerment and improving service delivery; but the evaluation request was limited to assess if the strategy had or not increased the number of citizens attending neighborhood meetings, with particular interest in attendance of youth living in El Alto. The reason for this limited ambition was the lack of information available on higher level outcomes, and as it will be argued in a moment, a view of participation limited to the demand side, without an introspection of the Municipalities’ own systems and response mechanisms. Implicitly, the strategy assumed that offering more spaces for participation and having as many citizens as possible using those spaces would be guarantee achieving the intrinsic and the instrumental values of participation. Another concern for the evaluation team after a first look at the strategy was how its components had been designed. Interviews with the Mayor and office managers at the Municipality revealed how when Mayor Zarate took office there was no clear idea of what the participatory strategy would entail. After a few months the participatory strategy of El Alto took shape as a set of four interventions (called pillars by El Alto officials): i. Increase in the frequency of neighborhood meetings. This entails more than just having more meetings: it demands deploying personnel from the Neighborhood Participation Sub-Office more often, and local leadership and citizens allocating more of their time to discussions. 118 ii. Systematization of proceedings from neighborhood meetings (Tramite Documentario, or TRADOC). Traditionally, content of meetings was only registered and kept by the Neighborhood Committee. El Alto has progressively moved towards generating for each meeting a Summary of issues and demands raised by citizens that could be referenced and sent to the Municipality. iii. Presence of the Mayor in neighborhood meetings. Introduced to increase the opportunities of direct contact of the Mayor with the population, and improve coordination with neighborhood residents. The plan was to schedule his presence so all neighborhoods would be visited by him at some point every year. iv. Third-Party monitored election of Neighborhood Committees’ members. In the past, election of Neighborhood Committees was not mandatory and when it happened, it was managed by the incumbent members in each neighborhood. The election of representatives through regular third-party monitored elections was conceived during the first few months of 2011, but only took place in September 201373. Three representatives for each of the 30 neighborhoods were then elected and among them, three were elected by their peers to represent all Neighborhood Committees at the El Alto Municipality and Council. III.4 Data A research team of four travelled to El Alto three times for extended stays collecting and systematizing data during 2013. Coordination with the Sub-Office of Neighbor Participation (SPV) allowed for continued data collection in between visits. The focus of data collection was in the following areas: 73 Information collected during interviews and unofficial discussions speaks of the support given by some of the exiting leaders to the Revocatoria process against Mayor Zarate. Neighborhood committees were typically dominated by a handful of families and had not changed in many years. 119 i. Neighborhood-level information: using 2007 Census data, aggregate socioeconomic data was generated to have a profile of each of the 30 recognized neighborhoods, including population and unsatisfied basic needs indicators. ii. Neighborhood meeting records: Only a limited number of meeting records had been filed with the Municipality and reports from facilitators were typically very generic. We collected and photocopied Minutes and facilitator reports from all 294 neighborhood meetings since January 2010 to December 2013, classifying them based on the topics discussed and gathering information on attendance The file generated has been shared with the Municipality and this tracking will continue for their internal monitoring. iii. Observation and qualitative analysis of meetings. A total of 12 neighborhood meetings in 8 different neighborhoods large and small were attended by the research team during 2013. We attended and observed meeting dynamics, the extent to which the agenda was followed, other topics raised by attendants, number of people present, registration process, and informally talked to neighborhood participants about how they heard about the meeting and what topics they wished had been discussed. iv. Socio-economic information on attendants. Even before the current administration, the attendance of citizens to the meetings was recorded by getting their national identification number (DNI)74. We requested the Municipality to provide the latest census data of the El Alto inhabitants by 74 When inquired about this, the SPV Manager said they simply continued a practice already in place during the previous administration, but that it made sense to her to prevent outsiders influencing the discussion. 120 DNI but without names, and that list was matched to the attendance records75. Matching records were generated for 1,086 (60%) out of 1,819 different people attending neighborhood meetings between January 2010 and December 2013. The remainder 40% could not be matched either because the DNI provided had missing/wrong numbers, or because the people involved did not have DNI at the time of the Census. v. Information on public works investment in el Alto 2010-2013. Appendix 1 presents the list of public works that took place in El Alto between 2010-2013 including the budget assigned to each, begin and end date, and the neighborhoods benefitted. The information on public works was coded and matched to a database with information on meetings and on attendants, so the analysis could identify what public works were being executed when a meeting took place in a given neighborhood. Notice, however, that the public works records do not include other more routine actions by the Municipality such as city beautification. Records requested to the Community Services Office on the latter were incomplete and unreliable, so they are not considered in this analysis. vi. Semi-structured interviews76 with municipal authorities and focus group with neighborhood facilitators. 75 To protect the privacy of meeting attendants, DNI numbers were replaced by unique identifiers before analysis was produced. 76 People interviewed at the Municipality were: Mayor Juan Zarate, Municipal Manager Ernesto Coronado, Social Development Manager Teresa Risco, Neighbor Participation Sub-Office Manager Cinthia Yamunaque, Planning and Budget Manager Darwin Quinde, Urban Development Manager Denis Tantajulca, Communal Services Manager Eduardo Tenque. 121 IV. Empirical Strategy and Theory of Change The evaluation undertaken in El Alto is retrospective: it attempts to assess the effectiveness of the overall strategy three years into its implementation. The Municipality of El Alto received no technical assistance for intervention design, or in deciding the objectives and indicators attached to the strategy. This research turns a retrospective evaluation into an opportunity to apply realistic evaluation and provide recommendations to improve the strategy’s design and future evaluation. Thus, the research is structured in four parts: the evaluation of the attendance objectives set by the Municipality; the evaluation of the strategy’s implementation; the comparison of the assumed theory of change behind the strategy to what is actually happening in El Alto; and a proposed evaluation framework that introduces experimentation to test complementary interventions informed by context. Every element of the research employs qualitative methods, complemented by quantitative methods whenever data availability allows for it. IV.1 Evaluation of Immediate Objectives As indicated, the Municipality of El Alto was interested in assessing if the participatory strategy launched in 2011 had or not increased attendance to neighborhood meetings, particularly of young people. Interviews with Municipal officers indicate a common assumption that attracting more people to the meetings would automatically lead to the empowerment of citizens. The analysis looks at the trends, the drivers of attendance, and explores how sustainable attendance is in the near future given the 122 current strategy setup, all within the parameters allowed by the very limited data available. For simplicity in interpreting results, a better understanding of magnitudes of effects and comparability across sets of results, Ordinary Least Square (OLS) regressions are employed77. In each set, regressions are run for the overall period of implementation, and for before, during and after the Revocatoria period (January-September 2012). The period before the Revocatoria has a relatively low number of observations, so the results for this period must be taken just as indicative78. The effect of Mayor’s presence in the meeting, any ongoing infrastructure project in the neighborhood, the timing of the meeting (before or after the election of Neighborhood Committees)79, and the topics discussed during the meeting are included as dummy variables to explain attendance to each meeting and average age in the January 2011-December 2013 period. In the case of attendance, neighborhood population size is also included as a control variable80. (1) (2) 77 Alternative methods were considered, particularly Instrumental Variables that are frequently used in the literature to deal with potential measurement error and simultaneity. However, it was judged that even using this method, the limited information available on the meetings and relatively small samples did not guarantee that the IV method would produce better results. If there is a better IV that should be used then the estimates are not efficient; and IV estimates are biased in small samples, in the same direction as OLS. 78 Low number of observations in an OLS regression affects the degrees of freedom of the estimation process, with resulting estimates biased. 79 The dummy takes the value of 1 if the meeting took place after the JUVECO election, 0 otherwise. 80 Population is not included as control in the Average Age regression given the homogeneous distribution of ages across neighborhoods, regardless of size. 123 Notice that only two components of the strategy are considered in the models. The regressions are at the meeting level so it makes no sense to include number of meetings; while there were not enough observations of TRADOC (the filing of meeting proceedings) across the sample81. RoadWorks and Topic are used to explore potential drivers of attendance independent of the strategy components82. IV.2 Evaluation of Strategy’s Implementation Looking at the data for the analysis of attendance to meetings, it was clear that implementation of the four components of the strategy was not evenly accomplished, or even purposely sequenced, over time. Only towards the very end of the period of analysis the four components were fully in place. Implementation evaluation focuses on how the introduction of interventions that constitute the strategy was influenced by context, examining changes to the intended strategy over time and across neighborhoods. The semi-structured interviews explore the circumstances that motivated the selection of participation as a key strategy for municipal government and the changes in scope, priority and behavior patterns of key actors. Following the research questions proposed, interviews aimed at: i. Understanding of the rationale and scope of the participatory strategy ii. Documenting changes in the menu of interventions during strategy implementation 81 There is a serious imbalance in the distribution of TRADOC observations over the period analyzed, particularly before and during the Revocatoria process. This will be explained in the implementation evaluation section. 82 The topic ‘Infrastructure’ is excluded from the Topic matrix due to its high correlation with having RoadWorks active in the neighborhood. It must be noted that RoadWorks and Topic may drive Mayor’s presence in a meeting, something that is explored in the implementation evaluation section too. 124 iii. Identifying context-related challenges in implementation, particularly if they were motivated by budget constraints, resistance of organized neighbors or internal coordination programs. iv. Determining the level of alignment of internal and external actors to the overall strategy, given citizens’ incentives and alternative channels of participation. v. Identifying patterns in the behavior of local government employees in terms of implementing the different interventions, or in scheduling public works and other local government activities over the study period. Next, regressions are run for two of the strategy components, using OLS for the same reasons as exposed above, while the implementation of the other two strategy components is analyzed qualitatively given their nature. i. Organization of Neighborhood Meetings Finding what determined the pattern of neighborhood meeting organization between 2011 and 2013 is quite challenging, as municipal officers differ in their assessment of it. The only pieces of information available, besides the record of a meeting occurring, are the list of concurrent public works of the Municipality, the timing of the meeting (before, during or after Revocatoria), and of course the neighborhood where the meeting took place. However, the limited number of months (36) in which to group meetings and the variation across neighborhoods means likely biased estimates in a regression analysis setting. Thus, a basic correlation analysis between meetings and public works is presented just to identify change in patterns over time. 125 ii. Mayor’s presence at meetings The reasoning behind having the Mayor attend meetings is to increase the credibility of the strategy, but also the visibility of the Mayor as the person promoting more opportunities for participation. Since the Mayor has limited time to attend meetings given competing priorities, the analysis tries to determine if the selection of the meetings he did attend is based on the size of the neighborhood, the topics on agenda, or ongoing public works. Thus: (3) where the attendance of the Mayor at a meeting is a function of population in the neighborhood based on the 2007 Census data, having ongoing infrastructure projects in the neighborhood at the time of the meeting (RoadWorks83) and the topics discussed in the meeting: Jobs, Representation, Others (mainly security and beautification)84. Analysis is clustered at the neighborhood level. iii. Filing Meeting Proceedings (TRADOC) This component was first introduced as a suggestion made to neighborhood committees and facilitators, and only in November 2013 it was turned into a mandatory requirement. Patterns in the implementation of this component are explored before it became mandatory. The analysis explores if filing was more likely or not based on Mayor’s presence at the meeting, size and gender composition of the audience, socio-economic 83 There 22 infrastructure projects in El Alto that started between January 2011 and December 2013. Only 3 of them are not related to roads, traffic or sidewalks: 2 projects are of educational infrastructure and 1 of water access infrastructure. Two of these three projects benefitted the entire district, so cannot be attached to a single neighborhood. Hence the characterization of infrastructure projects as road works in the model. 84 Notice that these dummies are not exclusive: in the same meeting two or more topics can be discussed. The dummy Others excludes infrastructure as a topic, since the latter is highly correlated with RoadWorks. 126 status of attendants (based access to sanitation, housing status, and roof materials), ongoing road works in the neighborhood and the agenda of the meeting. (4) where TRADOC is a dummy that takes the value of 1 if Minutes were filed in the Municipality, 0 otherwise; Mayor is a dummy with value 1 if the mayor attended that meeting, 0 otherwise; Attendance is expressed in numbers, PercWomen as proportion of women present in meeting and RoadWorks has already been explained. SocioEc85 is a matrix of variables linked to average socioeconomic conditions of attendants, and Topic another matrix for the different themes than can be discussed. iv. Neighborhood Committees Election The fourth component of the strategy, the Neighborhood Committee Election, is a one-time event so a regression to explain it is not warranted. The qualitative analysis, however, will provide some insights on its timing; the component will also be considered as a control to explore any structural change for variables explored in the regression models that are linked to it. IV.3 Theory of Change and Process Mapping As discussed, a strategy evaluation framework emphasizes not the intended strategy, but the realized strategy. The analysis of attendance and of implementation facilitates the identification of key elements of context that shape the strategy and its 85 One precision that must be made about SocioEc is that while it could be expected to have a meeting attendance composition that follows the one of the neighborhood, the idea is precisely to allow for a different composition, because certain members of the community could well be over-represented in a given meeting. 127 results over time. Based on this information, I construct the Context-Mechanism- Outcome (C-M-O) hypothesis (as featured in Westhorp, 2014) behind the participatory strategy of El Alto, describing both the assumptions of Mayor Zarate and his team when they designed the strategy and what actually happened once the strategy was implemented. Figure 5 features the basic C-M-O hypothesis with its focus on institutional settings and the mechanisms triggered by the intervention (the strategy) on the ecosystem (El Alto’s social interactions), organization (Municipality), and participating agents (citizens, community leaders, private sector, the Mayor, among others). Figure 5. The Context – Mechanism – Outcome (CMO) Hypothesis Source: Author’s conceptualization The nature of a strategy – a set of interventions pursuing the same objectives- demands a more detailed look at how its components complement each other. By mapping the participatory process of El Alto, it is possible to identify elements of the strategy that are currently missing or incomplete, so the recommendations can go beyond answering if the strategy is or not working, but how it could work better. The process map featured here is the same one used by the research team in discussions with the Municipality of El Alto, and it proved to be a useful way to frame 128 the discussion. It represents any participatory development process as a ‘funnel’ with holes on it86: the role of a participatory strategy is to expand the wide end of the funnel (increased and more representative participation), repair the holes (minimizing the loss of information), and improve the flow in the narrow end of the funnel (improving internal systems and executing with consideration of information provided). The specific components of El Alto’s strategy will be mapped along the funnel and missing elements identified. Figure 6. Mapping the Participatory Development Strategy Source: Author’s conceptualization of Participatory Strategy This framework responds to calls by authors such as Imae et al. (2011) and Deaton (2010) to avoid a “black box” approach to causality, focusing in the internal 86 This is an adaptation of the ‘funnel of causality’ model for decision making commonly used in political science, with an institutional constraints approach. 129 dynamics of the program being analyzed to explain “how” causal relationships within the system arise. IV.4 Informing strategy improvements, and the evaluation of them Based on the evaluation of objectives, the evaluation of implementation, the framing of the C-M-O Hypothesis, and the process mapping, the research proposes supplementary interventions to strengthen the participatory strategy of El Alto to be piloted, and an evaluation plan for the new components, including alternative outcome indicators that can be tracked through a revamped M&E system that is fed through administrative data87. V. Results of the analysis V.1 Evaluation of Immediate Objectives Table 1 presents the results for the short term objective set up by the Municipality: attracting more people to neighborhood meetings, particularly youth. After controlling for the population size of each neighborhood, and considering the overall period, attendance seems to have been driven by having employment on the agenda of the meeting, while attendance at meetings held after the neighborhood committee election of November 2013 seems to be consistently lower, which may be a temporary effect due to saturation of neighborhood activities during the period leading to the election. 87 This section follows the steps proposed in Essay 2 for informing design and sequencing evaluation. 130 The table also looks at drivers of average age among attendants. The analysis indicates that Mayor’s presence in meetings actually drives average age up, while no significant effect is registered after the committee elections. In terms of topics on the agenda, Jobs is again the one that attracts younger people on average, but only during the Revocatoria period. M ay or Za ra te Ti m e i n Of fic e Be fo re Re vo ca to ria Re vo ca to ria Pe rio d Af te r Re vo ca to ria M ay or Za ra te Ti m e i n Of fic e Be fo re Re vo ca to ria Re vo ca to ria Pe rio d Af te r Re vo ca to ria (1 ) (2 ) (3 ) (4 ) (5 ) (6 ) (7 ) (8 ) M ay or A tte nd an ce 1. 33 2 -8 .1 59 1. 66 3 7. 42 6* 2. 89 8* * -2 .2 07 4. 88 6* ** 4. 31 5* * (2 .4 48 ) (6 .4 53 ) (6 .9 83 ) (3 .9 59 ) (1 .1 61 ) (1 .5 46 ) (1 .4 63 ) (1 .9 87 ) Ne ig hb or ho od Po pu la tio n 0. 04 6* ** 0. 05 0* ** 0. 06 6* ** 0. 02 6* (0 .0 11 ) (0 .0 11 ) (0 .0 18 ) (0 .0 13 ) On go in g R oa d W or ks 2. 12 8 11 .1 32 ** -2 .2 54 4. 80 9 -2 .1 1 3. 73 -4 .0 35 -1 .6 58 (3 .5 19 ) (3 .3 45 ) (3 .0 49 ) (7 .1 70 ) (2 .4 27 ) (3 .4 45 ) (2 .4 56 ) (3 .6 14 ) Jo bs d is cu ss ed 4. 04 7* * -5 .2 92 4. 94 6* ** 5. 24 1 0. 46 7 -2 .3 37 2. 95 6* ** -0 .6 94 (1 .7 09 ) (8 .2 48 ) (1 .4 42 ) (3 .3 20 ) (0 .4 60 ) (2 .3 75 ) (0 .8 01 ) (1 .1 91 ) Re pr es en ta tio n Di sc us se d 0. 02 9 10 .8 99 ** -6 .2 07 1. 83 6 1. 07 3 1. 64 1 1. 14 8 0. 61 8 (1 .9 63 ) (4 .5 69 ) (5 .6 06 ) (3 .1 51 ) (1 .0 39 ) (1 .4 82 ) (1 .6 22 ) (1 .2 13 ) So ci al Is su es Di sc us se d 2. 92 6 8. 61 3* 8. 95 2 -0 .0 32 -0 .9 65 -2 .6 -1 .4 41 0. 60 6 (2 .3 17 ) (3 .9 67 ) (5 .2 99 ) (4 .2 13 ) (1 .0 81 ) (2 .4 02 ) (2 .3 64 ) (0 .9 03 ) Ot he r i ss ue s Di sc us se d -0 .8 24 -0 .1 21 -5 .5 52 2. 93 7 2. 11 9 -0 .5 38 1. 66 1 2. 32 2 (1 .9 62 ) (6 .0 41 ) (5 .0 65 ) (4 .7 45 ) (1 .2 69 ) (2 .1 52 ) (1 .8 67 ) (1 .8 34 ) M ee tin g a fte r co m m itt ee el ec tio n -8 .5 48 * -9 .2 37 * 0. 6 -0 .2 23 (4 .7 63 ) (5 .1 26 ) (2 .1 39 ) (2 .1 32 ) _c on s 15 .1 06 ** * 19 .3 9. 55 5* * 17 .1 46 ** * 40 .3 05 ** * 41 .3 86 ** * 39 .1 84 ** * 41 .0 88 88 8 (2 .9 45 ) (1 2. 87 3) (3 .8 73 ) (2 .3 38 ) (1 .2 53 ) (1 .0 24 ) (1 .8 29 ) (1 .2 17 ) M ea n de p va ria bl e 27 27 28 26 40 .7 40 .6 39 .1 41 .6 N. ob s 27 4 46 89 13 9 27 5 46 89 14 0 R- Sq 0. 35 0. 46 0. 56 0. 26 0. 07 0. 14 0. 16 0. 09 p 0 0. 46 0 0 0. 04 0 0 0. 12 So ur ce : A ut ho r's ca lcu la tio ns , u sin g sy ste m at ize d in fo rm at io n fro m m ee tin gs ' r ec or ds # P eo pl e at te nd in g n ei gh bo rh oo d m ee tin g Av er ag e Ag e Pe op le at te nd in g n ei gh bo rh oo d m ee tin g Ta bl e 1. A na lys is of Im m ed iat e Ob je ct ive s p ur su ed b y E l A lto M un ici pa lit y 131 The determinants of attendance, however, look very different during the time of political noise due to the Revocatoria (Jan-Sept. 2012): social and representation issues in the agenda drive attendance, as does the ongoing implementation of public works in the neighborhood. Mayor’s presence seems to influence attendance only after the Revocatoria process. The numbers give an idea of how our assessment of the strategy varies based on the period of time considered. Figure 7 shows that compared to the last year of the previous administration (2010), the average attendance to neighborhood meetings has, in fact, decreased during Mayor Zarate’s tenure. However, this is not unexpected: meetings are no longer rare events held only in the larger neighborhoods. As part of the strategy, meetings are now more common, and cover most neighborhoods of El Alto, large and small. After the Revocatoria (September 2012), attendance seems stabilized at between 20-30 people attending each meeting on average. The larger number of meetings, however, meant a larger number of people of El Alto attending (Figure 8). One claim that the administration can make is that 1,758 different people (38% of all adults in El Alto) have attended at least one neighborhood meeting between January 2011 and December 2013. This also indicates interest among El Alto’s inhabitants in using this opportunity for engaging the Municipality. 132 Figure 7. El Alto: Average Meeting Attendance by Month Source: Neighborhood Committees’ records, author’s calculations Figure 8. El Alto: Total Attendance to Neighborhood Meetings per Month Source: Neighborhood committees’ records, author’s calculations 133 Table 2. El Alto: Characteristics of People Attending Neighborhood Meetings Demographics Value Attended a Meeting that discussed: % Female (%) 58 social issues 13 Age 42 infrastructure 35 Years of Education 10 Jobs 33 Household Size 4.6 representation 35 Average Attendant… Characteristic Attended Meeting: % Occupation Not in Labor Force with Mayor present 27 Housing arrangement Own dwelling, no title while road works ongoing 48 Roof material Plank, metal in 2013 57 Sanitation arrangement Some access more than once 48 Source: Neighborhood committees’ records Some characteristics of the average person attending these meetings between January 2011 and December 2013 are described in Table 3: female, over 40, educated, with access to services, outside the labor force. In terms of participation, it is indicative that 52% have only been to one meeting, 43% attended their last meeting in 2012, and 73% of people participating have not been present in a neighborhood meeting with the Mayor present. In the case of average age, no element of the strategy was specifically tailored to attract young people: the assumption was that increasing the mere spaces of participation would attract more people in this age bracket. Figure 9 shows that the strategy of increasing the number of meetings to increase younger turnout has worked, but only to certain extent. While the average age of neighborhood participants in meetings has steadily declined since 2010, the largest effect was registered in the first year of the strategy, when more frequent neighborhood meetings were a novelty in the community. Since then, average age has slightly increased. 134 Figure 9. El Alto: Average Age of Neighborhood Meeting Attendants Source: Neighborhood committees’ records, author’s calculations In conclusion, the increase in number of meetings (first component of the strategy) and the effort to hold meetings across neighborhoods in the period 2011-2013 has paid off in terms of attendance and average age. This was expected, given the very limited access citizens of El Alto had before to direct engagement and discussion with representatives of their Municipal Government. From the perspective of the Municipality, this element of the reform meant a significant mass of voters that could be engaged year-round, which was particularly helpful during the Revocatoria process. However, the impact of increasing number of meetings has slowed down as they are less of a novelty and there is no systematic response to citizen feedback provided during the meetings. 135 The analysis also makes clear a lack of connection between other interventions in the strategy and the objectives set by the Municipality. First, the Mayor’s presence (second component of the strategy) has not been a driver of attendance in the overall period and in fact, it seems to attract older instead of younger people. While it can be argued that it is too soon to see any effect from the filing of meeting proceedings (third component of the strategy), this element is unlikely to have an effect on attendance if it is not complemented with response mechanisms to the information that reaches the Municipality. In other words, the promise that filing meeting Minutes carries from the authority receiving a concern is irrelevant if over time there is no visible action from the Municipality’s side. Finally, the election of neighborhood committees (fourth component of the strategy) assumed that increased legitimacy of representatives would attract more people to neighborhood meetings; however, the election of the same leaders in most neighborhoods indicates that either legitimacy was not a problem, or that there is a lack of viable alternatives to existing leadership. V.2 Evaluation of Strategy’ Implementation V.2.1 Perspective vs. Patterns: From Intended to Realized Participation Strategy88 During the 2010 election campaign, Mayor Zarate and his team included in their Government Plan for the district the promise of ‘increasing local participation in local decision making for a more inclusive El Alto’ (Plan de Gobierno Distrito de El Alto - APRA, Elecciones Municipales 2010). Interviews with Mayor Zarate and collaborators 88 Section based on interviews with the Mayor of El Alto, the Head Municipal Manager and municipal office Managers. 136 during his campaign, who are now part of the Administration, reveal that the idea was motivated by: i. Discussions with people of El Alto revealing frustration with the way the municipality handles their demands and requests. People living in smaller neighborhoods, in particular, did not have regular meetings, and when there was one, follow up of issues was quite limited; ii. A perceived lack of trust from the population in local authorities, particularly after the Revocatoria process lost by the previous Mayor, accused of improper use of public funds; iii. A sense of political instability, and the need to improve the relationship of Mayor with citizens throughout the term in office. Two of the previous three Mayors had been through processes to vacate their offices supported by Mayor Zarate’s political party (APRA, so the Mayor’s team feared retribution. iv. The concentration of Neighborhood leadership among families with larger power or networks in their area, which do not necessarily represent the wider population in each neighborhood. Thus, while formally justified in a rhetoric of citizen empowerment during the campaign, the design of the participatory strategy was strongly guided by political considerations: government stability, building the Mayor’s relationship with citizens beyond the election, and weakening potential opposition. This is reflected in the selection of indicators of success tracked by the Municipality: attendance and average age of attendants. While the original justification of participation was heavy on empowerment 137 arguments, the behavior pattern of municipal officers and staff during implementation, particularly during the Revocatoria process (January-September 2012) indicates a dominance of the political angle of the strategy. V.2.2. Alignment of different players to the Perspective89 A. Internal Actors (Municipality) 90 Municipal managers interviewed confirmed their full support of the Mayor’s participatory strategy. However, they face competing demands and restrictions that complicate a stronger commitment to participation in day to day operations. Below is an analysis of how the most important players in the implementation of the strategy interact, and how engaged they are in the strategy. i. Office of Social Development (OSD). The Sub-Office of Neighborhood Participation (SPV) of this department organizes neighborhood meetings. It receives requests from citizens at neighborhood meetings, redirecting them either to other municipal offices or to private sector companies when they are employment-related. However, its capacity of response to citizen demands is very limited. It does not manage any resources linked to infrastructure, and since El Alto receives income through mining 89 Section based on information from interviews with municipal office and sub-office managers and representatives of Neighborhood Committees. 90 It must be noted that participation was not the only priority for the new Municipal government in 2011, and attention to implementation of the strategy varied over time. The key to the administration’s success was the access to resources and jobs provided by the oil companies in the area: in this sense lower oil prices have translated in less contributions to local authorities in the past few years; on top of that, Petrobras and Savia had already announced in late 2011 that they were likely to sell their operation in El Alto in 2015. This is the context that the different players of the participatory process navigate. 138 contributions, the district is not eligible for State allocations for social programs. Therefore, projects that OSD would like to undertake in areas such as health and education are often stalled. ii. Office of Planning and Budget (OPB). Manages the participatory budgeting (PB) process. It has a larger budget so it can announce PB meetings through media and provide incentives to residents for their participation. Its work is not coordinated with SPV. In fact, the OPB manager reported that response to citizen requests channeled through SPV is not in its mandate. OPB focuses in capital investment priorities under the Ministry of Finance guidelines and those agreed through participatory budgeting. iii. Office of Urban Development (OUD). In charge of responding to citizen requests for infrastructure and maintenance. Only an executing role, so no direct outreach to the population. OUD reports overwhelming requests: besides projects approved through participatory budgeting, it executes other infrastructure projects discussed and approved through the Mayor’s office and the municipal council. iv. Office of Community Services (OCS). Receives the most demands involving cleaning, safety, trading, environment and traffic. Response to demands depends on budget, order in which the demand was received, or priorities established by the Mayor’s office or the Council. B. External Actors 139 In general, El Alto shows very limited alignment of neighborhood representatives and private sector with the participatory strategy. i. Neighborhood Committees and the citizens of El Alto. The Committees can call the population for other meetings than those sponsored by the Municipality if deemed convenient, and may resolve some basic disputes and demands within their neighborhood. Some of the larger Committees, like the one in Cabo Blanco, have their own resources from business activities in the area allowing them to have an annual budget for immediate actions in the neighborhood. There is a sense among office managers at the Municipality that Committees are not very active in spreading the word about meetings with the Municipality’s facilitators, something that was confirmed during the field visits. Besides, even after an increase in the number of meetings with the Municipality, a large number of requests from individual citizens or Committees was still channeled through Mesa de Partes in the 2011-2013 period. ii. Oil and other private sector companies of El Alto. A reported perverse consequence of the restrictions for social spending at the Municipality of El Alto is that many Neighborhood Committees maintain the practice of directly contacting the oil companies for assistance. The oil companies have reinforced this practice by having their own meetings and activities with the Committees, in a very ad hoc manner. The analysis of alignment of key players in the participatory strategy indicates that within the Municipality, there is a lack of identification with the process of most 140 municipal offices beyond the Sub-Office of Neighborhood Participation. Most offices lack a mandate to collaborate and coordinate with SPV the strategy’s implementation and even less to establish response mechanisms for citizen requests. Moreover, the Mayor’s Office and the Council have not modified the internal structure of the Municipality to be more responsive to participatory processes, and failed at providing adequate budget for implementation during the first two years. The strategy’s impact has also been restricted by the lack of engagement of players outside the Municipality, in particular the Neighborhood Committees and private sector companies. V.2.3 How the Municipality increased the number of meetings The organization of a meeting starts with a request of members of a neighborhood or by initiative of the Municipality to discuss topics of interest such as participatory budgeting projects, jobs, or safety. In practice, it is the Municipality that typically reaches out to the Neighborhood Committees to organize meetings and listen to the population. Meetings attended by the research team had always the same format: a presentation by a Municipality facilitator on latest developments in the district, and then open floor for residents to express their opinion or concerns. Figure 10 shows implementation of the different components of the strategy over time and major landmarks during the period of analysis. 141 Figure 10. El Alto: Participatory Strategy’ Timeline based on Actual Implementation (January 2010- December 2013) Source: Author, based on diverse Municipality of El Alto records and interviews Organizing more meetings has associated costs, from mobilizing facilitators and meeting preparation to the provision of snacks and refreshments to people attending. The lack of budget for the incremental cost associated was reflected in a slow start in implementation. In 2010, the last year of the previous administration, the Municipality organized only 19 meetings in 2 neighborhoods: Divino Nino (3) and Cabo Blanco (16). During the first year of the new Administration (2011), 46 meetings were organized in 8 neighborhoods, even though there were no meetings during the first three months. According to the Sub-Office of Neighborhood Participation, the initial selection of new 142 neighborhoods was opportunistic: larger neighborhood were prioritized, but also neighborhoods where leadership was willing to share organization expenses. In 2012 and 2013, with a larger budget assigned, the Sub-Office of Neighborhood Participation, organized 115 meetings each year. The increase however, has not been evenly distributed over time or across neighborhoods. Thus, over a third of 2012 meetings (40) took place within a month before the Revocatoria election, and while in 2012 those 115 meetings were distributed across 11 neighborhoods (37% coverage), in 2013 the 115 meetings were distributed across 24 (80% of El Alto’s neighborhoods). Looking for patterns, Figure 11 shows another interesting fact about the decision to hold a meeting in a given place at a given time. Taking the overall period of strategy implementation, meetings do not seem motivated by public works in a given neighborhood: correlation between number of meetings and start of public works in a given neighborhood is 0.15. However, when only the Revocatoria period (Jan-Sept. 2012) is considered, the correlation is up to 0.56. While this evidence is not conclusive, it indicates how the way neighborhood meetings were scaled up changed during the times of most political noise: the build up toward the Revocatoria vote of 2012, and at a lower extent before the Neighborhood Committees’ election. 143 Figure 11. Correlation between Start of Public Works projects in El Alto and Number of Neighborhood Meetings Source: Author, based on Municipality of El Alto records. V.2.4 How the Mayor’s presence in meetings and the filing of meeting proceedings was decided Table 3 helps the discussion of implementation patterns for the two components for which an econometric analysis was possible. After controlling for neighborhood size, Mayor’s presence in meetings does not seem explained by either topics discussed or ongoing public works; in any case, results indicate that the mayor tends to avoid meetings 144 discussing representation issues. During the Revocatoria period, his presence is less likely if jobs, security or beautification are on the agenda91. The results bear a more detailed look at the implementation process. The first visit of the Mayor to a neighborhood meeting was only on July 2011, to Cabo Blanco. Even during the Revocatoria process (Jan.-Sept. 2012), the Mayor only attended 9 neighborhood meetings; although 7 of those meetings occurred within two months of the vote. In fact, Mayor Zarate has attended 8% of meetings during his tenure, but 16% within two months of the Revocatoria vote. When inquired on this, the Manager of the Sub-Office of Neighborhood Participation stated that many Neighborhood Committees across the district requested during this time the presence of the Mayor to respond to allegations made by promoters of the revocation process. Comparing intended and realized strategies, the original plan of the Mayor of attending meetings at least once in each neighborhood per year proved to be unrealistic. This intervention, however, was instrumental from the political point of view for the Mayor to reach out to citizens when his own job was at risk. 91 It may well be that neighborhood representatives were not interested in discussing upkeep when the Mayor was present, but it is highly unlikely that the discussion of jobs was not of interest when he visited. It could also be an issue of acknowledging the discussion in the meeting Minutes. 145 Ju ly 2 01 1- De ce m be r 20 13 Be fo re Re vo ca to ria Re vo ca to ria Pe rio d Af te r Re vo ca to ria Pr e + P os t Re vo ca to ria M ay 2 01 1- O ct 20 13 Re vo ca to ria Pe rio d Pr e+ Po st Re vo ca to ria (1 ) (2 ) (3 ) (4 ) (5 ) (6 ) (7 ) (8 ) b/ se b/ se b/ se b/ se b/ se b/ se b/ se b/ se On go in g R oa d W or ks 0. 00 7 -0 .1 04 0. 11 -0 .0 54 -0 .0 67 -0 .0 14 -0 .0 55 0. 01 6 (0 .0 40 ) (0 .0 58 ) (0 .0 73 ) (0 .0 52 ) (0 .0 41 ) (0 .0 29 ) (0 .0 33 ) (0 .0 19 ) Jo bs d is cu ss ed -0 .0 84 -0 .0 91 -0 .2 15 ** 0. 01 -0 .0 22 -0 .0 07 -0 .0 06 -0 .0 05 (0 .0 50 ) (0 .1 12 ) (0 .0 76 ) (0 .0 73 ) (0 .0 75 ) (0 .0 34 ) (0 .0 20 ) (0 .0 40 ) Re pr es en ta tio n D is cu ss ed -0 .1 17 ** -0 .1 82 -0 .1 35 -0 .0 67 -0 .0 9 0. 11 0* * 0. 11 7* 0. 10 2* (0 .0 55 ) (0 .1 08 ) (0 .1 04 ) (0 .0 56 ) (0 .0 59 ) (0 .0 48 ) (0 .0 59 ) (0 .0 54 ) So ci al Is su es D is cu ss ed 0. 08 7 -0 .0 64 -0 .0 12 0. 15 7 0. 11 3 -0 .0 32 -0 .1 36 * -0 .0 53 (0 .0 68 ) (0 .0 60 ) (0 .1 03 ) (0 .1 13 ) (0 .1 04 ) (0 .0 32 ) (0 .0 69 ) (0 .0 49 ) Ot he r i ss ue s D is cu ss ed -0 .0 77 -0 .0 42 -0 .1 96 ** -0 .0 02 -0 .0 15 -0 .0 07 -0 .0 03 -0 .0 12 (0 .0 56 ) (0 .1 58 ) (0 .0 77 ) (0 .0 79 ) (0 .0 82 ) (0 .0 36 ) (0 .0 44 ) (0 .0 42 ) Ne ig hb or ho od P op ul at io n 0. 00 0* 0 0. 00 0* 0 0 (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) (0 .0 00 ) M ay or P re se nt 0. 06 4 0. 01 8 0. 15 4 (0 .0 66 ) (0 .0 69 ) (0 .1 11 ) # At te nd an ts -0 .0 01 -0 .0 01 0. 00 1 (0 .0 01 ) (0 .0 01 ) (0 .0 01 ) % W om en in a tte nd an ce -0 .2 1 -0 .0 71 -0 .3 72 (0 .1 56 ) (0 .1 33 ) (0 .2 29 ) W &S s er vi ce s (H ig he r= Be tte r) 0. 00 9 0. 00 1 0. 02 4* * (0 .0 10 ) (0 .0 29 ) (0 .0 11 ) Ho us in g (H ig he r= Ow ns ) -0 .1 06 -0 .3 06 ** -0 .0 16 (0 .0 80 ) (0 .1 27 ) (0 .0 61 ) Ro of M at er ia l ( Hi gh er =B et te r) 0. 39 0. 05 4 0. 32 8 (0 .3 84 ) (0 .3 30 ) (0 .4 26 ) _c on s 0. 08 5* * 0. 13 4 0. 13 0. 01 1 0. 04 -0 .2 81 0. 93 3 -0 .3 56 (0 .0 35 ) (0 .0 98 ) (0 .0 94 ) (0 .0 53 ) (0 .0 57 ) (0 .7 88 ) (0 .7 50 ) (0 .8 81 ) m ea n de pe nd en t v ar ia bl e 0. 08 0. 09 0. 1 0. 06 0. 06 0. 06 0. 06 0. 07 N. ob s 26 2 34 89 13 9 17 3 25 7 89 10 4 R- Sq 0. 07 0. 07 0. 13 0. 12 0. 09 0. 13 0. 35 0. 31 p- va lu e 0. 02 0. 19 0 0. 06 0 0. 42 0. 35 0. 7 M ay or P re se nc e in N ei gh bo rh oo d M ee tin gs Fil in g o f N ei gh bo rh oo d M ee tin g P ro ce ed in gs Ta bl e 3. A na ly sis o f I m pl em en ta tio n o f M ay or 's Vi sit s a nd Fi lin g o f M ee tin g P ro ce ed in gs So ur ce : A ut ho r's ca lcu la tio ns , u sin g sy ste m at ize d in fo rm at io n fro m m ee tin gs ' r ec or ds 146 Table 3 also takes a look at factors that increased the likelihood of filing meeting proceedings with the Municipality between May 2011 and October 2013. Before the reform, it was common practice among Neighborhood Committees to keep Minutes of the topics discussed in each meeting, but not share these summaries with the Municipality. Facilitators, in addition, did not provided detail report on the meeting back to the Municipality. The filing of citizen requests posed during meetings was introduced in 2011 as part of the reform, but it was not systematically adopted or enforced: in 2011, only 2 out of 46 meetings filed their Minutes; in 2012, only 5 out of 115 meetings did so, and between January and October 2013, 9 out of 98 meetings did. Thus, this component of the strategy is an example of an intervention that suffered serious implementation shortcomings due to a focus in the form of the innovation instead of a clearer understanding of its function. The Sub-Office of Neighborhood Participation in coordination with the Municipal Manager, decided in November 2013 to declare mandatory for all facilitators and recently elected Neighborhood Committees to file the Minutes of each meeting with the Municipality for appropriate recording and follow up. In addition to this, a standard format for reporting was introduced so key information from the meeting could be highlighted. It is still early to see if this change will have any impact on outcomes. However, it is possible to have an idea of the incentives to file before it became mandatory, as this provides information on alignment of the different stakeholders with the strategy. Table 147 3 results are just indicative: the available indicators are in fact not jointly significant in explaining filing, given the low variability in the value of the objective variable. The only pattern observed is that the likelihood of filing increases when issues of representation are discussed in the meeting; which means that neighborhood leaders are more likely to file when their own role in the community is discussed. If appropriately enforced, making the filing of meeting proceedings, however, has the potential for significantly change the participatory dynamic in El Alto, as the Municipality now has a regular flow of information on demands and concerns of the citizens of El Alto, effectively turning an element of the strategy into a monitoring tool. V.2.5 Implementation of Neighborhood Committee election Before the participatory strategy was introduced, Neighborhood Committee representatives were elected every two years by show of hands at a special neighborhood. Mayor Zarate was eager to have an election monitored by the ONPE (National Office of Electoral Processes). The election was originally planned for mid-2012, but postponed due to the Revocatoria, so it only took place in September 2013. This intervention is different than the others in the strategy in the sense that it is a one-time event expected to take place every two years. The intended outcome of this intervention was to improve the representativeness of leadership around the district, generating a sense of empowerment in citizens. There is, additionally, a political motivation behind it: weakening the family-based leadership system predominant at the neighborhood level. 148 It is not clear that the de-concentration of power was realized. Even though leadership did change in the largest neighborhood (El Volante), small neighborhoods are largely inhabited by members of the same extended family. V.2.6 Implementation Evaluation: General observations The review of the implementation of the different elements of the strategy reveals uneven but steady progress over time. It took the Sub-Office of Neighborhood Participation two years to have adequate resources and figure out the logistics to spread the organization of meetings across neighborhoods. Mayor’s presence in meetings was downsized compared to what was originally planned because it became clear that the original plan of visiting all neighborhoods every year was not realistic. Filing of neighborhood Minutes cannot even be considered as implemented until it became mandatory in November 2013. Finally, the Municipality officers interviewed note that the Neighborhood Committees’ election was only possible in large part because of the Mayor’s strengthened position after the Revocatoria process92, and it is unclear if it achieved its objectives. V.3 Theory of change and process mapping Figure 12 provides a synthesis of relevant information on the intended and realized participatory strategy of El Alto using the Context-Mechanism-Outcome hypothesis framework. It highlights the differences between the assumptions behind the original design and the mechanisms actually triggered. Any changes to the strategy to be suggested are to consider the informal and formal institutions surrounding the 92 It is worth pointing out that a Mayor can only be subject to a Revocatoria once during his/her term. 149 150 participatory process in El Alto, as well as the behavior patterns of key players. The C- M-O hypothesis also questions the immediate objectives of the strategy since they are not necessarily leading to fulfill the instrumental, intrinsic or constructive values of participation. Finally, the process map is reviewed to identify the challenges tackled by the current strategy components, and the missing elements of the strategy. Visualizing the strategy, it is clear that all interventions in El Alto have so far focused on expanding the demand side of governance, or in the analogy used, the wider end of the funnel, from the environment and actual collection of citizen concerns to the point those concerns and requests reach the Municipality, preventing information loss as much as possible. It is argued that additional measures need to be taken to guarantee the representation of vulnerable groups in terms of attendance to the meeting and actual participation during the meeting. Additionally, the process mapping highlights that measures are needed to adapt the response mechanisms of the Municipality so citizen feedback can be translated into policy. 151 Figure 13. El Alto Participatory Strategy Source: Author’s conceptualization. V.3.1 The strategy is incomplete The strategy has neglected the narrow end of the funnel, i.e. the handling of citizen requests and concerns once they reach the Municipality. As the literature review highlighted, success in participatory programs is typically linked to synergies between the demand side (citizens) and the supply sides of governance (local government). By improving the channeling of citizen demands, a larger flow of requests reaches the Municipality, and citizens are aware that their requests have been received by the authority. The Municipality, however, has not adapted its internal structure and organization to this increased flow of demands. The handling of citizen requests from neighborhood meetings is exactly the same as during the previous administration. 152 The Sub-Office of Neighbor Participation (SPV) has very little leverage within the organizational structure of the Municipality: SPV passes along requests received from neighbors to the relevant municipal offices, but there remains no follow up or accountability system in place. If experience elsewhere is any indication, this lack of attention to the supply side will likely lead to a loss in credibility of the neighborhood meetings system (Fox, 2014). V.3.2 The strategy lacks adequate targeting However, the strategy is incomplete even in its demand-side focus. Increasing the number of meetings across neighborhoods improves the voice of residents living in smaller and normally less represented neighborhoods, but it is not clear that among those attending meetings or expressing their views are the poorest segments of the population. The strategy lacks instruments to specifically collect the voice of the poor, either because the announcement of the meeting does not reach them, or because even if they attend the meeting, they do not necessarily participate in the discussion. V.4 Supplementary interventions Based on the gaps found in the strategy and the performance of the existing components, the following are some concrete changes proposed to the Municipality of Alto for improving the strategy. V.4.1 At the wider end of the funnel: Improving participation The participatory planning stage aims at solving one of the biggest challenges of participatory development: representation of the views of the neighborhood as a whole 153 (i.e. correcting for bonding failures, responding to the asymmetry of power and realism challenges). There are two suggested interventions linked to this stage: a. Changes in format and dynamic of neighborhood meetings A pilot in which the thirty neighborhoods will be divided in two groups comparable groups in terms of size and social indicators. One group will organize business as usual neighborhood meetings while meetings for the second group will follow a different dynamic: a combination of timed discussions and opportunity during the meeting for explicitly requesting feedback from normally underrepresented or less vocal groups within the neighborhood. For both groups, every attendee fills a registry upon arrival to the meeting, requesting information on how each attendee learned about the meeting, and on issues they would like to discuss with the Municipality. Additionally, a box to collect suggestions and complaints will be kept at each meeting, and opinions systematized. Minutes of the meetings will be publicly displayed in municipal dependencies. Associated costs with this intervention are: special training on participatory planning for facilitators, additional personnel for the organization of meetings and the logistics they imply, and snacks and refreshments for participants93. Appendix 2 describes how this change in the strategy implementation could be rigorously evaluated. 93 One of the explored options for additional personnel is to recruit volunteers from the local Technical School, with a recognition of their work through certificates and a small monetary compensation. 154 b. Changes in how neighborhood meetings are announced One of the big challenges of the strategy is to be able to generate decisions that are representative of the demands of each zone’s whole population. Since municipal budget does not allow for monetary compensation for neighbors’ time, meetings tend to be attended (and discussion monopolized) by a reduced number of social organizations and individuals. One alternative is changing the way the meetings are announced and promoted. Currently, it is ‘word of mouth’ or announced the same day of the meeting in the local street markets. An alternative is to pilot in a few neighborhoods an aggressive campaign to announce neighborhood meetings using flyers, door-to-door invitations, and timely announcements in community areas (for example, local market, convenience stores, and offices offering government services) V.4.2 At the narrow end of the funnel: processes within the Municipality The Municipality of El Alto has vowed to change its processes towards improved coordination and information exchange between its offices. Changes within the structure of the Municipality are not easy to achieve or sustain, since the incentives tilt towards the focus on infrastructure programs. The recommendation is to track what happens after the Sub-Office of Neighborhood Participation reports the results of the participatory planning process (including the priorities established by citizens under the new meeting dynamic proposed) to the Municipal Council. Resolutions from the Municipality and City Orders (Ordenanzas) to the lists produced by participatory meetings could be compared for future evaluation. 155 VI. Conclusions: El Alto Participatory Strategy Following the recommendation of Mansuri and Rao (2011a), this evaluation has been heavy in contextually sensitive diagnosis, understanding the relationship between citizens, local government and other players relevant to the participatory process. The case of El Alto, in fact, reflects many of the characteristics of programs studied by them that were not successful in improving service delivery and empowerment. For starters, by limiting the objective of the strategy to citizen attendance to meetings (not even actual participation during the meeting) El Alto’s strategy assumes that attendance is enough guarantee of effective service delivery (one that responds to citizen needs and promotes downward accountability. This view crucially overlooks: a) that bonding failures of civil society such as elite capture can still happen during the meetings, with interest groups dominating the agenda and the decision-making process; and b) that the quality of deliberation (across the dimensions of breadth, range, depth and control) determines the effectiveness of the meetings as tools. Thus, the current objective indicators do not say much about citizen empowerment (the goal used to justify the strategy), and do not guarantee the underlying goal of political stability either. There is however a perception among Municipal officials that the interventions have been instrumental in generating political stability during the Revocatoria process. Thus, meetings have been prioritized in neighborhoods with ongoing public works in certain periods of time; and Mayor Zarate increased his presence in meetings during the campaign to remove him from office. Even if it is the case that the strategy helped the Mayor to keep his job, such an effect rests on the credibility that the neighborhood 156 meeting system has among citizens. At this point, no element of the strategy is trying to preserve or increase the credibility of the participatory system. The theoretical framework suggests that credibility is sustained in effectiveness: the existence of response mechanisms as part of a strategic (Fox, 2014) or ‘sandwich’ approach (Mansuri and Rao, 2011b) to participation, so citizens know that the local government does takes their feedback into consideration. Moreover, the strategy as it stands today lacks any attempt to improve the Municipality’s management of citizen requests. Even if achieving the instrumental value of participation (improving service delivery) was not set as a short term goal for El Alto’s strategy, it may well be the element that guarantees its sustainability by strengthening citizen credibility in the system. Presently, however, all planning done by the Municipality is circumscribed to public works (decided or not through a participatory budgeting process of limited-deliberation), and very little attention is given to the feedback provided by citizens in neighborhood meetings. The existence of other channels for local leadership to channel assistance to their neighborhood, such as meetings between neighborhood leaders and oil companies, weakens even more the local government’s position. The evaluation team left in place a managerial and monitoring tool for the collection of information from meetings based on the mandatory filing of meeting proceedings and the systematization of past information. It also left a strategy design and evaluation plan based on the process mapping and identification of gaps in the strategy. 157 The application of this plan will allow the regular collection of information related to the quality of participation, and the piloting of mechanisms in neighborhood meetings to elicit the opinions and ideas of people that attend but do not participate. Suggestions to improve the announcement of the meetings in the community and to gather citizen requests and concerns beyond the meetings and Mesa de Partes, are being explored. The case of El Alto reflects a common view that urban Municipalities of Peru have of participation: the more people present at meetings, the more legitimacy the participatory process has. However, there is no accountability in terms of answering to citizens, and no monitoring of how representative the views expressed during the meetings are of the community. The selection of attendance as main outcome indicator mirrors the limited ambition of the strategy. Short term gains in attendance and youth attendance registered in El Alto mask the underlying limitations of the strategy. The analysis recommends complementing the evaluation of objectives with the evaluation of implementation and benchmarking under a strategy evaluation. Such an approach better identifies limitations and weaknesses in the strategy in an environment of information shortage and changing political environment and creates a strong argument for the piloting of alternative interventions. 158 YE A R D ES C R IP TI O N ST A R T EN D VA LU E (S /. M IL LI O N S) SO C IA L IN FR A ST R U C TU R E R O A D / SI D EW A LK N EI G H B O R H O O D Ro ad im pr ov em en t S an ta E lis a A ve nu e 20 -J an -1 0 15 -S ep -1 0 1. 33 N o Y es Ba rri o El V ola nt e St ro m d ra in co ns tru ct ion 27 -M ay -1 0 23 -S ep -1 0 1. 06 N o Y es A A .H H . J ua n Pa blo II y Ba rri o Lo s C om er cia nt es Co ns tru ct ion a nd e qu ipm en t P oli ce S ta tio n 10 -A ug -1 0 29 -N ov -1 0 0. 89 Y es N o El A lto In fra str uc tu re im pr ov em en t C rn l. Ze ga rra H igh S ch oo l 17 -J un -1 0 16 -D ec -1 0 1. 10 Y es N o El A lto Pa ve m en t c on str uc tio n 3- Ju l-1 0 15 -O ct -1 0 0. 52 N o Y es A A .H H . L as M er ce de s Pa ve m en t c on str uc tio n 14 -J ul- 10 16 -N ov -1 0 0. 97 N o Y es El A lto Pa ve m en t c on str uc tio n 14 -J ul- 10 15 -O ct -1 0 1. 30 N o Y es El A lto In sta lla tio n of m et er s 9- N ov -1 0 29 -M ar -1 1 0. 51 Y es N o El A lto y C ab o Bl an co Ro ad im pr ov em en t B olo gn es i A ve nu e 21 -J ul- 10 2- N ov -1 0 1. 34 N o Y es El A lto Ex te ns ion o f E rn es t H em ing w ay se a fro nt 21 -J ul- 10 22 -S ep -1 0 0. 72 Y es N o Ca bo B lan co Pa ve m en t c on str uc tio n 20 -N ov -1 0 31 -M ar -1 1 1. 24 N o Y es A A .H H . S an M ar tín d e Po rre s Pa ve m en t c on str uc tio n 21 -O ct -1 0 19 -J an -1 1 0. 85 N o Y es A A .H H . L a Cu m br e Pa ve m en t c on str uc tio n 5- N ov -1 0 22 -F eb -1 1 0. 60 N o Y es A A .H H . S an P ed ro Pa ve m en t c on str uc tio n 30 -S ep -1 1 28 -D ec -1 1 1. 16 N o Y es A A .H H . P rim av er a Pa ve m en t c on str uc tio n 28 -S ep -1 1 10 -J an -1 2 1. 33 N o Y es Ba rri o Pl om o St re et im pr ov em en t 16 -D ec -1 1 28 -F eb -1 2 0. 79 N o Y es A A .H H . J ua n Pa blo II Pa ve m en t c on str uc tio n 6- O ct -1 1 14 -J an -1 2 0. 86 N o Y es A A .H H . L uc ian o Ca sti llo Ex pa ns ion a cc es s r oa d an d co ns tru ct ion o f r oa ds a nd si de w alk s 5- Ju l-1 1 21 -J an -1 2 3. 46 N o Y es Ca bo B lan co Pa ve m en t c on str uc tio n 7- Se p- 11 16 -J an -1 2 1. 29 N o Y es A A .H H . R od olf o Sá nc he z To rre s Co ns tru ct ion o f r oa ds a nd si de w alk s 16 -D ec -1 1 10 -M ar -1 2 0. 85 N o Y es A A .H H . E ud eli o M uñ oz Pa ve m en t c on str uc tio n 5- Fe b- 12 O ng oin g by N ov '1 3 1. 62 N o Y es Ba rri o Bl an co Co ns tru ct ion o f r oa ds a nd si de w alk s 6- M ar -1 2 O ng oin g by N ov '1 3 3. 69 N o Y es Ba rri o Ca str o Po zo Pa ve m en t i m pr ov em en t 4- M ay -1 2 O ng oin g by N ov '1 3 3. 67 N o Y es El A lto Tr an sit im pr ov em en t A v Sa nt os V ilc he z 10 -A ug -1 2 O ng oin g by N ov '1 3 1. 73 N o Y es El A lto Pa ve m en t c on str uc tio n 1- A ug -1 2 O ng oin g by N ov '1 3 0. 41 N o Y es A A .H H . V illa Ja rd ín St re et im pr ov em en t 10 -A ug -1 2 O ng oin g by N ov '1 3 1. 83 N o Y es Ba rri o Pi ur a Ex pa ns ion w at er a nd sa nit at ion se rv ice s 22 -J un -1 2 6- A ug -1 2 0. 40 Y es N o A A .H H . J iró n Tu m be s Pa ve m en t a nd si de w alk im pr ov em en t 28 -J an -1 3 28 -A pr -1 3 0. 98 N o Y es Ba rri o Co m er cia nt es Pa ve m en t a nd si de w alk im pr ov em en t 1- Fe b- 13 O ng oin g by N ov '1 3 1. 35 N o Y es A A .H H . V illa M ilit ar y V illa H er m oz a Pa ve m en t a nd si de w alk im pr ov em en t 9- A pr -1 3 6- A ug -1 3 1. 54 N o Y es A A .H H . C iud ad d el Pe sc ad or Pa ve m en t a nd si de w alk im pr ov em en t 13 -M ar -1 3 11 -J ul- 13 1. 44 N o Y es Jir ón A re qu ipa Pa ve m en t a nd si de w alk im pr ov em en t 18 -A pr -1 3 O ng oin g by N ov '1 3 1. 59 N o Y es Jir ón T um be s Ex pa ns ion e du ca tio na l s er vic es se co nd ar y sc ho ol 26 -A pr -1 3 23 -A ug -1 3 1. 69 Y es N o El A lto Ex pa ns ion e du ca tio na l s er vic es p rim ar y sc ho ol 18 -A pr -1 3 O ng oin g by N ov '1 3 0. 66 Y es N o El A lto Pa ve m en t a nd si de w alk im pr ov em en t 29 -M ay -1 3 27 -J ul- 13 0. 64 N o Y es A A .H H . L os Ja zm ine s y Ba rri o Ca str o Po zo So ur ce : M un ici pa lit y o f E l A lto re co rd s 20 10 20 11 20 12 20 13 AN NE X 1. Li st o f P ub lic W or ks in iti at ed b y t he M ni cip al ity o f E l A lto Ja nu ar y 2 01 0- Se pt em be r 2 01 3 159 Appendix 2 An evaluation framework for the revamped participatory strategy The new evaluation would measure how much neighborhood preferences are matched by the topics discussed in the participatory meetings, the decisions taken by those meetings, and the policies adopted by the Municipal Council and the different municipal managers. To this end, two variables at the individual level (each expressed demand collected before the meetings) are constructed, following Ban and Rao (2009): a matched-decision dummy (DD), and a decision intensity (DI). The matched-decision indicates if the issue of interest of each individual is reflected in the meetings minutes, while the decision intensity tracks how important was the issue during the discussion. Given a set of issues (I) treated during meetings in zone z, and each issue ikz taking a fraction fkz of the discussion, the matched-decision dummy is defined as: 1 if id e Iz DD= 0 otherwise While decision intensity fdz if id e Iz DI= 0 otherwise 160 Given the more deliberative meetings (Meeting), and the reorganization of Municipal administration (Reform) proposed, together the effect of the participatory strategy on the match of demands to decisions taken by the Municipality is represented by two equations to be estimated: DDdz= α + β1 * Meeting + β2 * Reform + β3 * Zone + β4 * Source + e DIdz= α + β1 * Meeting + β2 * Reform + β3 * Zone + β4 * Source + e Where Meeting is a dummy indicating if the matched demand was collected in a zone exposed to the intervention; Reform is 1 if the Municipal Office that receives the demand from the Sub-Office of Neighborhood Participation does participate of the realignment and coordination process (i.e. , if it produces a formal response to the requested received)94; Zone is a vector of median characteristics of the zone where the demand was collected, and Source indicates if the demand was expressed from an individual neighbor, or by a representative from a social organization.. Using the assumption of sequential ignorability (Imai et al, 2011), it is possible to estimate the total average treatment of a change in meeting dynamics, and the average causal mediation effect (ACME) of mechanisms aimed at reflecting views of people non- attending meetings, and of prevalent processes within the Municipality. This specification is for the overall participatory strategy, but it can be broken down to first evaluate the match of demands with issues discussed at the meeting: DDMeetingdz= α + β1 Meeting + β2 * Zone + β3 * Source + e 94 This will control for the type of demand expressed (infrastructure, citizen safety, sanitation, etc.). Alternatively, it can be complemented with a variable indicating the relative size of the issue demanded in the municipal budget for El Alto. 161 DIMeetingdz= α + α + β1 Meeting + β2 * Zone + β3 * Source + e And also, to compare the decisions taken at the meeting, to the issues discussed at the Municipal Council, or directly executed by the different Municipal Offices (Stage 2): DDPolicydz= α + β1 * Meeting + β2 * Reform + β5 * Budget + β6 * Source + e DIPolicydz= α + β1 * Meeting + β2 * Reform + β5 * Budget + β6 * Source + e Where Budget indicates the relative size of the issue demanded in the municipal budget for the Municipal Office in charge of executing it. 162 Bibliography Acemoglu, D. and J. Robinson, 2012. Why Nations Fail: The origin of power, prosperity and poverty. New York: Crown. Alkire, S., 2002. Valuing Freedoms: Sen’s Capability Approach and Poverty Reduction, Oxford University Press, Oxford. Andrews, M., 2009. Isomorphism and the Limits to African Public Financial Management Reform. HKS Faculty Research Working Paper Series RWP09-012. Ban, R., and V. Rao. 2009. “Is Deliberation Equitable? Evidence from Transcripts of Village Meetings in South India.” Policy Research Working Paper 4928, World Bank, Washington, DC. Banerjee, A. and E. Duflo, 2011. Poor Economics: A radical rethinking of the way to fight poverty. Washington: PublicAffairs. Banerjee, A., R. Banerji, E. Duflo, R. Glernnerster, S. Khemani, 2010. Pitfalls of Participatory Programs: Evidence form a Randomized Evaluation in Education in India, American Economic Journal: Economic Policy 2 (1): 1-30. Beath, A. et al., 2010. Randomized impact evaluation of Phase II of Afghanistan’s National Solidarity Programme (NSP). First report. Bjorkman, M., J. Svensson, 2007. Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda . Policy Research Working Paper Series 4268, The World Bank. 163 Centre for the Future of the State, 2010. An Upside-down View of Governance. Brighton: IDS. Chambers, R., 2009. So that the poor count more: using participatory methods for impact evaluation. Journal of Development Effectiveness, Vol. 1 2009, n.3, p. 243. Chambers, R., 2007. ‘Participation and Poverty’, Development, Palgrave Macmillan Journals, vol. 50(2), pages 20-25, June. Congreso de la República del Perú, 2006. Participación Política y Ciudadanía, Curso V. Lima: Oficialía Mayor y Dirección de Participación Ciudadana. Cooke, B. and U. Kothari, 2001. Participation: The New Tyranny? London: Zed Books. Crocker, D., 2008. Ethics of Global Development: Agency, Capability, and Deliberative Democracy. Cambridge: Cambridge University Press. Crocker, D., 2006. “Deliberative Participation in Local Development,” Journal of Human Development, 8, 3 (2007): 431-55. Deaton, A., 2010. Instruments, Randomization, and Learning about Development. Journal of Economic Literature 48 (June 2010) pp. 424–455. Easterly, W. and J. Cohen, eds. 2010. What works in Development? Thinking big and thinking small. Washington: Brookings Institution Press. Fox, J., 2014. Social Accountability: What does the evidence really say? GPSA Working Paper n.1, September 2014. 164 Fung, A. and Wright, E.O., 2003. Deepening Democracy: Institutional Innovations in Empowered Participatory Governance. London: Verso. Gastil, J. 2008. Cultivating Deliberative Development: public deliberation as a means of improving local, state, and federal governance, in Odugbemi, S. and T. Jacobson, eds. Governance Reform under Real-World Conditions. Washington DC: The World Bank. Gastil, J. and P. Levine, 2005. The Deliberative Democracy Handbook, Jossey-Bass, San Francisco. Gaventa, J. and G. Barrett, 2010. 'So What Difference Does it Make? Mapping the Outcomes of Citizen Engagement', IDS Working Paper 347, Brighton: IDS. Gould, C., 1996. Diversity and Democracy: representing differences, in S. Benhabib ed. Democracy and Differences: contesting the boundaries of the political. Princeton: Princeton University Press. Imae, K., L. Keele, D. Tingley, T., 2011. Unpacking the Black Box: Learning about Causal Mechanisms from Experimental and Observational Studies. Working paper Princeton Department of Politics. Inter-American Development Bank, forthcoming. Measuring impact of institutional reforms in the Americas. Washington DC: IDB. Klitgaard, R., 1997. Cleaning up and invigorating the civil service. Public Administration and Development, 17: 487–509. Kresalja, B., 2012. 15 Puntos sobre la Revocatoria. Revista Ideele N. 225. 165 Ludwig J., J. R. Kling, and S. Mullainathan, 2011. Mechanism Experiments and Policy Evaluations. NBER Working Paper No. 17062 May 2011 Lupia, A. 2004. "Can Political Institutions Increase Citizens’ Competence? Findings from a Formal Model and Two Experiments." In Irwin Morris, Joe A. Oppenheimer, and Karol Soltan (eds.) Politics from Anarchy to Democracy: Rational Choice in Political Science. Stanford, CA: Stanford University Press, 132-156. Malinga, P. 2008. Public sector middle managers: The critical link to driving public sector reforms, in Odugbemi, S. and T. Jacobson, eds. Governance Reform under Real- World Conditions. Washington DC: The World Bank. Mansuri, G. and V. Rao, 2011a. Localizing development. Washington: World Bank. Mansuri, G. and V. Rao, 2011b. Participatory Development Reconsidered, Development Outreach, May 2011. Mansuri, G. and V. Rao, 2004. Community-Based and -Driven Development: A Critical Review. World Bank Research Observer 19 (1): 1-39. Mintzberg, H. et al., 2005. Strategy safari: The complete guide through the wilds of strategic management. New York: Free Press. Mintzberg, H., 2007. Tracking strategies. New York: Oxford University Press. McNulty, S., 2012. "An Unlikely Success: Peru’s Top-Down Participatory Budgeting Experience," Journal of Public Deliberation: Vol. 8: Iss. 2, Article 4. Olken, B., 2007. "Monitoring Corruption: Evidence from a Field Experiment in Indonesia." Journal of Political Economy 115, 2 (April 2007): 200-249. 166 O’Meally, S., 2013. Mapping context for Social Accountability: A resource paper. Social Development Department, World Bank, Washington DC. Panfichi, A. 2007. Participación ciudadana en el Perú: Disputas, confluencias y tensiones. Lima: Fondo Editorial Pontificia Universidad Católica del Perú. Patrizi, P. A., 2010. Strategy evaluation: Emerging processes and methods. New Directions for Evaluation, 2010: 87–102. Patrizi, P. A. and M. Quinn Patton, 2010. Strategy as the focus for evaluation. New Directions for Evaluation, 2010: 5–28. Plan de Desarrollo Concertado El Alto Piura, 2012. Municipality of El Alto, Piura. Plan de Gobierno Candidato Juan Zarate, 2010 Pretty, J.N., 1994. Alternative Systems of Enquiry for Sustainable Agriculture, IDS Bulletin, 2 (25), pp. 37-48. Pritchett, L., S. Samji, J. Hammer, 2013. It’s all about MeE, Center for Global Development Working Paper n. 322, April 2013. Ravallion, M. 2011. Are we really assessing development impact? Development Impact World Bank blog, May 2011. http://blogs.worldbank.org/impactevaluations/guest-post- by-martin-ravallion-are-we-really-assessing-development-impact. Reich, R., 2002. Bridging Liberalism and Multiculturalism in American Education, University of Chicago Press, Chicago. 167 Remy, M.I., 2005. Los múltiples campos de la participación ciudadana en el Perú. Lima:IEP. Rodrik, D. 2010. The New Development Economics: We shall experiment, but how shall we learn? In W. Easterly and J. Cohen, eds. What works in Development? Thinking big and thinking small. Washington: Brookings Institution Press. Schonwalder, G., 2002. Linking Civil Society and the State: Urban Popular Movements, the Left, and Local Government in Peru, 1980-1992. State College: Pennsylvania State University Press. Selee, A., 2011. Decentralization, Democratization, and Informal Power in Mexico. State College: Penn State University Press. Sen, A., 1999. Development a Freedom, Random House, New York. Simon, H., 1957. Models of man. New York: Wiley Uphoff, N. 1998. Community-based natural resource management: Connecting micro and macro processes, and people with their environments. Keynote for international workshop on Community-Based Natural Resource Management, Economic Development Institute, World Bank, Washington, 1998. Wehipeihana, N., and Davidson, E. J., 2010. Strategic policy evaluation: Answering macro-level cross-project questions. Presented at Anzea Regional Symposium, Wellington, Aotearoa, New Zealand. Westhorp, G., 2014. Realist Impact Evaluation: An introduction. Methods Lab publications, ODI. 168 World Bank, 2011. Evaluación del presupuesto participativo y su relación con el presupuesto por resultados. Lima: World Bank. World Bank, 2004. World Development Report, Making Services Work for the Poor. Washington: The World Bank. World Bank, 1996. The World Bank Participation Sourcebook. Washington: The World Bank.