ABSTRACT Title of dissertation: SUSTAINABILITY OF HEALTHY TOMORROWS PARTNERSHIP FOR CHILDREN PROGRAM Latricia C. Robertson, Doctor of Philosophy, 2005 Dissertation directed by: Associate Professor Sharon M. Desmond Department of Public and Community Health The purpose of this study was to determine sustainability, the main reason(s) for sustainability, and the relationship between the amount of annual matching funds (as well as the extent of overmatch) and sustainability of the Healthy Tomorrows Partnership for Children Program (HTPCP) projects. In addition, the development of a predictive sustainability model was proposed. Ninety-four HTPCP projects received federal funding from 1989 through 1997. Five of these projects participated in the pilot-test for this study. The remaining 89 project directors (PDs) were mailed the Level of Institutionalization (LoIn) Instrument, developed by Goodman, et al. (1993), to measure institutionalization/sustainability in preventive health programs. Eighty-one PDs responded to the HTPCP survey questionnaire (91% response rate). Project directors were predominately female (80%) and had up to eight years of formal education after high school (59%). Thirty-five percent of the PDs were pediatricians, and their ages ranged from 32 ? 80 years of age. Sustainability of the HTPCP projects was demonstrated in this study. Another major finding was that a HTPCP project with a pediatrician PD (as opposed to all other disciplines) was less likely to have written and operationalized program objectives. SUSTAINABILITY OF HEALTHY TOMORROWS PARTNERSHIP FOR CHILDREN PROGRAM by Latricia C. Robertson Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2005 Advisory Committee: Associate Professor Sharon M. Desmond, Chair Associate Professor Bonnie Braun Professor Robert H. Feldman Professor Jerrold S. Greenberg Associate Professor Lori Simon-Rusinowitz ? Copyright by Latricia C. Robertson 2005 ii DEDICATION This dissertation is dedicated to my mother, Ada Belle Weaver Robertson; my sister, Brenda Ann Robertson Dallas; my brother-in-law, Charles Raymond Dallas; my nephew, Charles Ross Dallas; my niece, Latricia Ann Dallas Milton; and my cousin, Judye Diane Butler Roberts. I also dedicate this work in loving memory of my father, R.T. Robertson; my brother, Lewis Grey Robertson; and my friend, Fran Gover. iii ACKNOWLEDGEMENTS The assistance and support of many people made this dissertation a reality for me. I would like to thank the following individuals. Dr. Desmond, chair of my committee---special thanks for making this dissertation possible for me. Your ongoing guidance and support was very much appreciated. The members of my committee, Dr. Braun, Dr. Feldman, Dr. Greenberg, and Dr. Simon-Rusinowitz---you were constructive, helpful, and supportive. Dr. Carol Werlinich---special thanks for being a committee member for my proposal meeting. Special thanks to Carol Jackson and Beverly Monis, who provided information and guidance throughout my years at school. Thank you for each and every gesture performed on my behalf as well as my fellow students. To my classmate and friend, Dr. Jo Anne Gambrell Jensen---special thanks for being there for me. Dr. Margaret Ann Drum---special thanks for your assistance on behalf of the Maternal and Child Health Bureau. Dr. Thomas Tonniges and Jane Bassewitz---special thanks for your ongoing assistance and support on behalf of the American Academy of Pediatrics. iv All 94 project directors of the Healthy Tomorrows Partnership for Children Program (HTPCP) projects studied here?you were responsive, enthusiastic, supportive, and consistently willing to walk that extra mile. I could not have done this work without you and I thank you from the bottom of my heart. Most of all, thanks be to God, from whom all blessings flow. v TABLE OF CONTENTS List of Tables .............................................................................................................vii List of Figures ............................................................................................................ix Chapter I: The Problem.............................................................................................1 Introduction .......................................................................................................................1 HTPCP Description .......................................................................................4 Theoretical Framework..................................................................................6 Purpose of the Study ......................................................................................7 Significance of the Study. ..............................................................................8 Hypotheses.....................................................................................................8 Chapter II: Review of the Literature Introduction .......................................................................................................................11 Sustainability Defined....................................................................................11 Value of Sustainability...................................................................................13 Theoretical Framework..................................................................................14 LoIn Instrument..............................................................................................19 Sustainability and Institutionalization Research ............................................28 HTPCP Study Variables.................................................................................40 Summary........................................................................................................47 Chapter III: Methodology.........................................................................................48 Introduction....................................................................................................48 Study Population and Recruitment of Participants ........................................48 Study Procedures............................................................................................50 Scoring of HTPCP Survey Instrument...........................................................51 Data Collection ..............................................................................................59 Operationalization of Variables in Hypotheses..............................................61 Data Analysis .................................................................................................65 Summary........................................................................................................66 Chapter IV: Results ..................................................................................................67 Introduction....................................................................................................67 Study Population Characteristics ...................................................................67 Data Analyses Findings..................................................................................69 Summary........................................................................................................94 vi Chapter V: Discussion, Recommendations, and Conclusions..................................96 Introduction...................................................................................................96 Discussion.....................................................................................................97 Limitations ....................................................................................................104 Recommendations for Further Research.......................................................105 Recommendations for Health Education Practice ........................................107 Conclusions...................................................................................................108 Appendix A HTPCP Requirements/Criteria............................................................110 Appendix B LoIn Instrument ...................................................................................115 Appendix C Location of HTPCP Projects ..............................................................127 Appendix D Institutional Review Board .................................................................128 Appendix E Cover Letter for HTPCP Survey..........................................................129 Appendix F Consent Form.......................................................................................130 Appendix G HTPCP Record Data Collection Form................................................132 Appendix H Pilot HTPCP Survey Instrument Administration Form .....................135 Appendix I HTPCP Survey Instrument...................................................................136 Appendix J Scoring the HTPCP Survey Instrument Scale .....................................145 Appendix K HTPCP Score Sheet Grid ....................................................................147 Appendix L Table13. Multiple Regression Analyses of Hypotheses Nine..............148 Glossary .....................................................................................................................152 References..................................................................................................................155 vii LIST OF TABLES 1. LoIn Instrument: Institutionalization Factors....................................................22 2. HTPCP Questionnaire Items for Institutionalization Factors ............................53 3. HTPCP Survey Instrument Item Scoring ..........................................................57 4. Scoring for Routine and Niche Saturation Items ...............................................58 5. Item Mean Scoring Categories Indicating Level of Institutionalization ............59 6. Classification of Variables ................................................................................66 7. Demographic Variables of HTPCP Respondents ..............................................68 8. Project Director Age, Length of Time in Position, and Years with the HTPCP .........................................................................................................69 9. HTPCP Mean Scores .........................................................................................70 10. Routine and Niche Saturation Items Mean Score Range ..................................74 11. Sustainability Organizational Sub-system Variables and Institutionalization Factor Means................................................................76 12. Chi-square Analysis of Project Advisory Committee by Organizational Subsystems ........................................................................................................77 13. Chi-square Analysis of Pediatric Involvement by Organizational Subsystems ........................................................................................................79 14. Chi-square Analysis of Level of Matching Funds by Organizational Subsystems ........................................................................................................81 15. Chi-square Analysis of Hard Money Versus In-kind Funding by Organizational Subsystems...............................................................................83 16. Chi-square Analysis of Project Director Discipline by Organizational Organizational Subsystems................................................................................85 viii 17. Chi-square Analysis of Project Director Education by Organizational Subsystems ........................................................................................................87 18. Chi-square Analysis of Project Director Turnover by Organizational Subsystems ........................................................................................................89 19. Chi-square Analysis of Evaluation by Organizational Subsystems ........................................................................................................91 ix LIST OF FIGURES 1. Level of Institutionalization Matrix .....................................................................16 2. Eight Factor Model ..............................................................................................21 1 CHAPTER I: THE PROBLEM Introduction ?Benjamin Franklin once wrote: In this world, nothing can be said to be certain except death and taxes. Today, there should be a modification to this statement: In this world, nothing can be said to be certain except death, taxes, and being evaluated.? From Sarvela, P.D. and McDermott, R.J. (1993). Health Education Evaluation and Measurement. Madison, WI: WBC Brown & Benchmark, p. 2. In the United States (US), considerable federal funds appropriated to the federal Maternal and Child Health Bureau (MCHB) are spent each year to implement community-based maternal and child health (MCH) promotion programs nationwide. Many of these programs do not continue after the initial federal funding ends. The focus of these MCH programs has been on program efficacy and not the long-term viability of successful programs (Goodman and Steckler, 1987/88). Recently, sustainability has become an issue of growing concern to legislators, policy makers, funders, administrators, and health care providers. Attention must be given to the long-term viability of MCH intervention programs if the allocation of scarce resources is to be performed effectively and efficiently (Shediac and Bone, 1998), particularly as the MCHB funding is the largest federal funding source for MCH programmatic infrastructure 2 and systems development nationwide. Federal infrastructure building for health services for women and children in the US goes back over 90 years to the creation of the US Children?s Bureau on April 9, l912 (62 Congress, 1911-1912). The Bureau linked health services and social welfare concerns (Grason and Morreale, 1997). President William Howard Taft approved an Act of Congress that created the Children?s Bureau and directed it to investigate and report on all matters pertaining to the welfare of children and child life among all classes of people in the US (62 Congress, 1911-1912). This legislation was the federal government?s first mandate that it had a responsibility to promote the welfare of our nation?s children (Eliott, 1960). In 1921, the Sheppard-Towner Act was passed to provide resources for state health agencies to establish and improve health services for women and children. When this legislation expired in 1929, a service infrastructure had been developed in most states, providing a foundation for the national Maternal and Child Health (MCH) program, established in 1935 through Title V of the Social Security Act (Grason and Morreale, 1997). The enactment of Title V was the result of forces that represented the culmination of federal, state, and local efforts to protect and promote the well being of children (Lesser, 1985). As early as 1939, some of the funds were reserved for special projects of regional and national significance (SPRANS) to enable states to develop new kinds of programs and to include children with 3 diagnostic problems not previously included (Lesser, 1985). Examples included programs for the hospitalization of premature infants; for women with pregnancy complications, which included hospital delivery care; for children with rheumatic fever, epilepsy, hearing impairment, mental retardation, and congenital heart disease, as well as grants to institutions of higher learning for the training of nurse-midwives, social workers, physicians, and others in specialty areas including public health (Lesser, 1985). The MCHB, which administers Title V, is one of four bureaus in the Health Resources and Services Administration (HRSA) within the US Department of Health and Human Services (DHHS). In addition to administering the MCH Block Grant, the MCHB is also responsible for two programs authorized under the Public Health Service Act: Healthy Start, a targeted infant mortality reduction initiative begun in 199l and the Emergency Medical Services for Children program, enacted in 1984. A new Section 510 of Title V, Separate Program for Abstinence Education, was added in 1996 (USDHHS, HRSA, MCHB, 2001). Since 1982 through the present (2004), the 1981 Omnibus Budget Reconciliation Act (OBRA) Amendment provides for 10-15% of the annual Title V funding authorized by Congress to be set aside from the formula block grant funds for states and used for SPRANS. Since 1982 when OBRA 1981 was implemented through 2001, the annual total SPRANS funding ranged from $57,550,000 to $109,147,750 (USDHHS, HRSA, MCHB, 2001). The SPRANS 4 funding categories include research, training, hemophilia, genetics, and maternal and child health improvement projects (MCHIP). The maternal and child health MCHIP grants that are approved and awarded SPRANS funding are service demonstration projects. The federal funding is considered ?seed money? and the services provided by the project are expected to be sustained after the federal funding period is completed. Sustainability is the continuation of programmatic services in the community after the federal funding is completed. To date, sustainability of SPRANS projects in the MCHIP category has not been evaluated by the MCHB- HRSA. Therefore, the number of these projects being sustained and the reason(s) for their sustainability is unknown. In order to measure sustainability of SPRANS projects in the MCHIP category, one of the sub-groups of MCHIP-SPRANS programs, the Healthy Tomorrows Partnership for Children Program (HTPCP), was evaluated for sustainability in this study. HTPCP Description In 1989, the HTPCP began as a health promotion program and was initiated to engage communities in problem solving and development of preventive strategies to promote access to health care for mothers and children through community-based infrastructure building. The HTPCP is funded and administered by the MCHB-HRSA, in partnership with the American Academy of 5 Pediatrics (AAP). Due to the MCHB-HRSA/AAP Partnership, the HTPCP projects are linked to the AAP?s health professional network, consisting of 59 state chapters and 53,000 child health experts across the US. Also, a technical assistance site visit team led by a pediatrician, who is an AAP Fellow, makes a site visit to each HTPCP project in its first and fourth year and provides a written report with recommendations and follow-up as appropriate. Through 2004, the HTPCP has approved and funded 178 community-based projects, located in 45 states, the District of Columbia, Guam and Puerto Rico. The HTPCP has 50-70 active projects at any given time. In addition, there is an annual grant review cycle with an objective review committee (ORC) process that reviews 45-90 new competing applications, and funds approximately 10 new projects from the list of approved projects generated by the ORC process. The federal funding level for the HTPCP projects is capped at $50,000 per project year for each of the five funding years of the project. There is also a HTPCP requirement for each project to secure 66.67 percent (a minimum of $100,000) matching funds per year in funding years two through five. The HTPCP matching funds requirement is intended to support community collaboration and partnerships to promote sustainability of the programs. To date, all of the HTPCP projects have met the matching funds requirement annually in funding years two through five and many overmatch the minimum amount required. However, the relationship of the amount of annual matching funds as well as the extent of the overmatch and sustainability of the 6 HTPCP projects is unknown. Theoretical Framework In this study, sustainability of the HTPCP projects was examined using an instrument developed for measuring the level of institutionalization (LoIn) of health promotion programs (Goodman, McLeroy, Steckler, and Hoyle, 1993). Institutionalization is thought to occur when a program becomes an integral part of an organization; the LoIn instrument measures the extent of program integration into organizations (Goodman, et al., 1993). The LoIn instrument is based on organization innovation and change theory (Goodman, 1987). Innovation is defined as a policy, program, or technology which is new to its potential users (Mohr, 1969; Rogers and Shoemaker, 1971; Greer, 1977; Downs and Mohr, 1979; and Basch, 1984). ?There is a well developed organizational theory that accounts for the growth and developmental processes of program innovations, i.e., Stage Theory of Innovation? (Goodman, 1987, p. 12). The origin of Stage Theory of Innovation has been attributed to a combination of Lewin?s conceptualization of change as unfreezing, movement, and refreezing (Beyer and Trice, 1978; Yin, 1979; Ledford, 1984) and to Diffusion of Innovation Theory (Kaluzny, 1974; Greer, 1977; Hall and Loucks, 1977; Scheirer and Rezmovic, 1983). According to the organization innovation and change theories upon which the LoIn instrument is based, organizations consist of production, maintenance, supportive, and managerial subsystems. Institutionalization occurs 7 when a program becomes imbedded into these subsystems. Purpose of the Study The purpose of this study was to determine sustainability, the main reason(s) for sustainability, and the relationship between the amount of annual matching funds, as well as the extent of overmatch, and sustainability in the HTPCP projects. The HTPCP project requirements/criteria (Appendix A) were consistent with other SPRANS grants in areas such as goals and objectives, methodology, tracking, evaluation, cultural relevance/competence, community- based, family-centered, etc. (US Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Bureau, 1999). However, selected additional requirements specific to the HTPCP included the project advisory committee (PAC), pediatric involvement, and the matching funds requirement. Other demographic and background variables included in this study as potential reasons for sustainability were types of matching funds, discipline of the project director (PD), educational level of the PD, and the number of PDs for each project over the five-year project period. In addition, whether or not an evaluation of the HTPCP projects occurred was also examined as all approved and funded grantees do not necessarily implement an outcome evaluation as required (Eisen, Evans, Kavanagh, Athey and Schwab, 1999). 8 Significance of the Study This study predicted some project variables considered important when determining sustainability of a federally initiated MCH health promotion program. It also provided an indication of the extent to which the HTPCP projects have developed infrastructure geared toward sustainability. The HTPCP projects in this study demonstrated sustainability and this finding should influence future SPRANS grant policy and funding decisions in MCHB-HRSA. The LoIn of health promotion programs instrument (Goodman, et al., 1993) was further tested in this study. In addition, this was the first study conducted with the LoIn Instrument with the intent to measure sustainability and at the same time develop a predictor model for sustainability. Therefore, this study has contributed to the measurement of sustainability literature. Hypotheses Nine hypotheses guided this study. They were: Hypothesis #1: There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high project advisory committee (PAC) activity levels. 9 Hypothesis #2: There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high levels of pediatric involvement. Hypothesis #3: There will be no significant difference in sustainability among HTPCP projects with minimal, medium, and high levels of matching funds. Hypothesis #4: There will be no significant difference in sustainability among HTPCP projects with a majority of hard money matching funds versus in-kind matching funds. Hypothesis #5: There will be no significant difference in sustainability among HTPCP projects that have PDs with disciplines in medicine (predominantly pediatricians) as opposed to all other disciplines. Hypothesis #6: There will be no significant difference in sustainability among HTPCP projects with PDs with minimal, moderate, and high levels of education. 10 Hypothesis #7 There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high PD turnover rates over the five-year project period. Hypothesis #8: There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high levels of evaluation. Hypothesis #9: Matching funds will be the strongest predictor of sustainability when entered with all other predictor variables (PACs, pediatric involvement, matching funds, types of matching funds, PD discipline, PD level of education, PD turnover, and evaluation). 11 CHAPTER II: REVIEW OF LITERATURE Introduction This chapter presents information on the definition of sustainability, the value of sustainability, the theoretical framework used in the study, and research related to sustainability and institutionalization. In addition, a review of literature was conducted on the variables of the HTPCP projects which included PACs, pediatric involvement, matching funds, types of matching funds, discipline of PD, educational level of PD, number of PDs, and program evaluation. Sustainability Defined Little to no consensus exists in the literature on the conceptual and operational definitions of sustainability. The phenomenon of program continuation is referred to by various terms such as program ?maintenance,? ?sustainability,? ?institutionalization,? ?incorporation,? ?integration,? and ?routinization? (Shediac and Bone, 1998). The literature has clarified these concepts as follows. Claquin (1989) described sustainability as the capacity to maintain service coverage at a level that will provide continuing control of a health problem. The World Commission on Environment and Development (Bamberger and Cheema, 1990) referred to sustainability as the ability to maintain the desirable elements of the status quo 12 into the future. The World Bank definition of project sustainability found in Bamberger and Cheema (1990) was the capacity of a project to continue to deliver its intended benefits over a long period of time. According to the US Agency for International Development (1988), a development program is sustainable when it is able to deliver an appropriate level of benefits for an extended period of time after major financial, managerial, and technical assistance from an external donor is terminated. Steckler and Goodman (1989, p. 34) stated ?institutionalization refers to the long-term viability and integration of a new program within an organization.? Organizational change involves the process by which new practices become standard business in a local agency. The organizational change process is central to all organizations regardless of whether it is called routinization, institutionalization, incorporation, or some other term (Shediac and Bone, 1998). Shediac and Bone (1998, p. 93) summed up the concept by stating ?sustainability appears to better capture the dynamic process involved in program continuation and the broad range of its potential forms than the notion of institutionalization.? In this study, institutionalization was considered a form of sustainability; the terms sustainability and institutionalization were used synonymously. 13 Value of Sustainability There is not total agreement in regard to the assumption about the value of sustainability. Green (1989) stated that institutionalization of programs requires more conceptualization before assuming it is best measured by the long term viability of organizations. Goodman and Steckler (1987) list criteria for determining whether a program should survive. These criteria include that a health promotion program: 1) is based on established theory; 2) is well implemented; 3) is cost effective; 4) is desired by both a client constituency and a host organization; and 5) is producing desired outcomes. Glaser (1981) observed that not all innovations should endure. Changes in circumstances, people, situations, and problems occur. ?When a validated, more efficacious, more suitable or more cost-effective means for meeting a given problem comes to light, the former modus operandi very appropriately may be supplanted? (Glazer, 1981, p. 174). Researchers also argue that while not all programs should continue, there are reasons why the failure to sustain programs over a long term may present serious problems. One problem is that termination of a program is counterproductive when the disease that a program was established to address has not been alleviated (Shediac-Rizkallah and Bone, 1998). Additonally, some organizations/communities incur significant start- up costs in human, fiscal and technical resources and their funds are withdrawn before activities have reached 14 full fruition (Shediac-Rizkallah and Bone, (1998). Further, failure to institutionalize can be more costly than failure to adopt or implement because an innovation is likely to have incurred full-scale costs and shown some merit, unlike innovations that may have failed at the implementation stage (Yin, 1979). Another problem is that failure to institutionalize may be disruptive of careers, since workers tend to make significant investments in programs to the exclusion of other professional options (Yin, 1979). Finally, Goodman and Steckler (1987- 88) propose that it is considerably more difficult for health organizations to reestablish community trust after successful programs are terminated abruptly or inappropriately. Theoretical Framework A theoretical framework allows examination of behavior using a systematic arrangement of variables drawn from a body of evidence. A theory represents an abstract generalization that explains systematically the relationships among phenomena (Polit and Hungler, 1978). It permits an investigator to impose order on a set of variables chosen from the universe of all possible variables. Theories consist of a set of statements, each of which expresses a relationship, arranged in a logically interrelated deductive system, permitting new statements to be derived from them (Polit and Hungler, 1978). Conceptual frameworks represent a less well developed attempt at organizing phenomena than theories and deal with abstractions (concepts) which are assembled together by virtue of 15 their relevance to a common theme (Polit and Hungler, 1978). With a conceptual framework there is no deductive system of propositions which assert a relationship between concepts (Polit and Hungler, 1978). Goodman and Steckler (1989) developed a framework based on extensive study of ten health promotion programs that characterized institutionalization of health promotion programs. The framework (Figure 1) is based on Yin?s (1979) use of passages and cycles and Katz and Kahn?s (1978) typology of organizational subsystems, described below. Yin (1979) summed ?passages? and ?cycles? to calculate institutionalization. ?Passages are formal transitions in organizations, such as program funding moving from temporary to permanent; cycles are repeated organizational events, such as the yearly renewal of funding for a program? (Goodman, et al., 1993, p. 164). Figure 1 shows institutionalization conceptualized on two planes as demonstrated by the rows and columns. The rows, based on Katz and Kahn?s (1978) typology, indicate how extensively a program has been institutionalized within its host organization. The columns represent Yin?s (1979) work and reflect the program institutionalization intensiveness for each of the subsystems or rows of Figure 1. ?The level of institutionalization (LoIn) for any program is a composite of the number of dimensions (extensiveness), and the degrees of depth (intensiveness)? (Goodman, et al., 1993, p. 164). Katz and Kahn (1978) found that organizations consist of generic subsystems: production, maintenance, support, and managerial. The production 16 subsystem contains the major functions of the system and is concerned with ?throughput? or the activities related to the end product. Examples of these functions are the implementation of product plans, schedules, and quality control mechanisms such as evaluation. Dimensions (Extensiveness) Organizational Subsystems Degrees (Intensiveness) Passages Routines Niche Saturation Production Maintenance Supportive Managerial Figure 1. Level of institutionalization matrix. Modified from: Goodman, R.M., McLeroy, K.R., Steckler, A.B., and Hoyle, R.H. (1993). ?Development of Level of Institutionalization Scales for Health Promotion Programs.? Health Education Quarterly, 20(2), Summer, p. 164. In contrast, the maintenance subsystem is personnel-directed and assures continuity throughout the system?s operations. The maintenance subsystem accomplishes this by reinforcing the roles of its organizational members through recruitment, retention, socialization, and rewarding or sanctioning. While the production and maintenance subsystems of an organization have an internal focus, the supportive subsystem is directed outward toward the 17 environment. It helps the organization establish itself within the larger social environment through legitimacy and positive organizational relationships and linkages with other organizations. Examples of organizational supports are stable funding and facility space for the program. The managerial subsystem, the fourth subsystem, ??. is the lubricant which controls, coordinates, and directs all of the other subsystems? operations? (Katz and Kahn, 1978, p. 118). The adjustments of the whole system to the external environment are the responsibility of the managerial subsystem. Examples of positive components within the managerial subsystem are formally assigned managers, written job descriptions for program staff, and routinized progress reports. Goodman and Steckler (1989) found that institutionalization is characterized by the ?built-in-ness? of a program within its organization. In order for this to happen, a health promotion program must become ?rooted? within the host organization?s production, maintenance, support, and managerial subsystems. The degree to which a program becomes rooted within these subsystems is reflected through passages, cycles, and niche saturation. Passages represent the first degree of program institutionalization, and are signified by one-time sentinel events (Goodman, et al., 1993). Examples of these events are formal documentation and implementation of program plans (production), transferring the program?s funding from soft to hard money (support), and the program being 18 acknowledged with status within the organization by being added to the organizational chart as a bona fide unit (managerial). ?Routines, the second degree of program institutionalization, are signified by routinizing the program passages? (Goodman, et al., 1993, p. 165). During routinization, the program plan continues to be implemented and the activities become routine. For example, the program?s hard funding is renewed annually and therefore is maintained at a stable funding level or the program remains on the organizational chart when reorganization occurs within the host organization (Yin, 1979). ?Niche saturation, the third degree of program institutionalization, occurs when the health promotion program expands to its optimum limits within the host organization?s subsystems? (Goodman, et al., 1993, p. 166). Examples are that the program implementation has gone beyond routine, and the program has optimum staffing and reaches the maximum number of clients that it can maintain; stable funding is renewed annually but at an optimum level; and the health promotion program is not only maintained as a unit during reorganizations, but also becomes a core unit within the organization (Goodman and Steckler, 1989). Goodman, et al. (1993) maintain that program passages, routines, and niche saturation exist for each organizational subsystem. Thus, the implication of the matrix (Figure 1) is that the more cells that are occupied by a particular health promotion program, the more institutionalized the program has become. 19 LoIn Instrument Goodman, et al. (1993) conducted a study to test an instrument for measuring the level of institutionalization (LoIn) of health promotion programs. In order to test the validity of the matrix (Figure 1) as an operational construct, Goodman, et al. (1993) developed questionnaire items corresponding to each cell of the matrix. The initial questionnaire was composed of 32 three-part items representing the four subsystems of the matrix. The three parts of each item corresponded to its respective passage, routine, and niche saturation within its designated subsystem. The 32-item instrument was reviewed by an expert panel consisting of five representatives from the fields of organizational theory, community health development, health care administration, health education research, and bio- statistics. Based on the feedback from the panel, the instrument was revised and pilot-tested in a local health department that operated health promotion programs; several items were dropped and others modified. The resulting LoIn questionnaire contained 15 three-part items (Appendix B). In addition to the 15 items, there was a question asking the number of years that the program being studied had operated within the organization. The researchers assumed that number of years was a surrogate measure for program permanence. Respondents were also asked to rate on a 4-point scale their perception of how permanent the program was within the organization (?not at all permanent? to ?extremely permanent?). These two items were included to assess 20 how closely they correlated with the 15 items in the LoIn questionnaire. Thus, the researchers used the two items to measure construct validity by assessing (using a correlation coefficient) how closely they converged with the 15-item instrument. Goodman and colleagues (1993) mailed the revised questionnaire to organizations operating health promotion programs. Four hundred and fifty-three administrators in 141 organizations in three southeastern and three western states (North Carolina, South Carolina, Virginia, Arizona, Montana, and Utah) were sent study questionnaires. The organizations included county health departments, public schools, and nonprofit health agencies. The health promotion programs included community-based chronic disease prevention, school health education, and work site health promotion programs. Three-hundred-seventy-three administrators responded representing 103 organizations; 322 of the questionnaires were sufficiently complete to include in the study (82% response rate). For data analysis purposes, the researchers reconfigured the matrix in Figure 1 as an eight-factor model (Figure 2). Prior research (Yin, 1979; Miles, 1983) has shown that the first phase of program routinization was demonstrated by passages. For example, funding must be included as a budget item (a passage) before program is routinized in annual budgets. In the reconfigured eight-factor model, passages were subsumed under routines, in one of four factors for each organizational subsystem (production, maintenance, support, and management). The remaining four factors indicated a program?s niche saturation within a host 21 Dimensions (Extensiveness) Katz and Kahn, (1978) Organizational Subsystems Degrees (Intensiveness) Yin, (1979) Routines Niche Saturation Production Maintenance Supportive Managerial Figure 2. Eight Factor Model Modified from: Goodman, R.M., McLeroy, K.R., Steckler, A.B., and Hoyle, R.H. (1993). ?Development of Level of Institutionalization Scales for Health Promotion Programs.? Health Education Quarterly, 20(2), Summer, p. 164. organization?s subsystems (production, maintenance, support, and management). Therefore, the model specified eight correlated factors with four representing routines and four representing niche saturation; each factor corresponded to an organizational subsystem. The researchers analyzed the data derived from the LoIn Instrument to test the eight-factor model. Table 1 demonstrated an abbreviated form of the questionnaire items for the eight factors. Factor I, routinization of program production, indicated the repeated, or routine, carrying out of program activities as 22 Table 1. LoIn Instrument: Institutionalization Factors Item __________________________________________________________________ Factor I: Production Routine 1a. Number of years written goals and objectives followed 2a. Number of years written plans and procedures followed. 3a. Number of years written schedules followed. 4a. Number of years locally adapted strategies followed. 5a. Number of times program formally evaluated. Factor II: Production Niche Saturation 1b. Aspects having written goals and objectives. 2b. Aspects having written plans and procedures. 3b. Aspects having written schedeules. 4b. Aspects having locally adapted strategies. 5b. Aspects that have been formally evaluated. Factor III: Maintenance Routine 6a. Number of years permanent staff assigned to program. 7a. Number of years with administrative-level advocate. 8a. Number of years other staff contributed to operation. 23 Table 1. LoIn Instrument: Institutionalization Factors (Continued) Item __________________________________________________________________ Factor IV: Maintenance Niche Saturation 6b. Number of staff members in permanent positions. 7b. Level of activity of admin- istrative?level advocate. 8b. Proportion of staff in organ- ization that contribute. Factor V: Supportive Routine 9a. Number of years having permanent status. 10a. Number of years having permanent space. 11a. Number of years having established funding source. 12a. Number of years key staff hired from stable funds. Factor VI: Supportive Niche Saturation 9b. Estimate of permanency of program. 10b. Proportion of needed permanent space occupied. 11b. How permanent is program?s funding source? 12b. Permanency of funding for key staff. 24 Table 1. LoIn Instrument: Institutionalization Factors (Continued) Item __________________________________________________________________ Factor VII: Managerial Routine 13a. Number of years super- visor formally assigned. 14a. Number of years formal job descriptions followed. 15a. Number of years scheduled evaluation reports. Factor VIII: Managerial Niche Saturation 13b. Aspects that receive formal supervision. 14b. Proportion of staff having written job description. 15b. Extent to which evaluation reports are produced. Modified from: Goodman, R.M., McElroy, K.R., Steckler, A.B., and Hoyle, R.H. (1993). ?Development of Level Of Institutionalization Scales for Health Promotion programs.? Health Education Quarterly, 20(2), Summer, pp.169-170. indicated in written program goals, plan, schedules, implementation and monitoring strategies, and evaluations (items 1a-5a). Factor II, niche saturation of program production, represented the extent to which all possible program activities were formally documented and implemented (items 1b-5b). Factor III, routinization of program maintenance, showed the host organization staff?s day- to-day (routine) involvement in and priority given to the program?s operations 25 (items 6a-8a). Factor IV, niche saturation of program maintenance, indicated the extent to which staff are involved in and committed to the day-to-day program activities (items 6b-8b). Factor V, routinization of program support, represented the ongoing, or routine, priority placed on the program by the host organization?s administrative staff through the manifestation of permanent funding, staffing, and priority placed on the program (items 9a-12a). Factor VI, niche saturation of program support, demonstrated the extent of commitment by the host organization?s administration as indicated through the provision of permanent funding, staffing, and status placed on the program (items 9b-12b). Factor VII, routinization of program management, indicated the formal and routine implementation of program supervision consisting of program supervisors, written job descriptions for all staff, and an ongoing accountability process reflected through program evaluation (items 13a-15a). Factor VIII, niche saturation of program management, represented the extent to which the program had supervisors assigned on a regular basis, written job descriptions for staff were in place, and program evaluation was ongoing (items 13b-15b). Both routine and niche saturation items on the LoIn Instrument were scored using a 4-point scale. For the routine items (Factors I, III, V, and VII), a score of one (1) was equal to no passage; two (2) indicated passage plus one iteration of a routine; three (3) meant two or three routine iterations; and four (4) represented 26 four or more routine iterations. Niche saturation items (Factors II, IV, VI, and VIII) were scored as one (1) to show no niche saturation; two (2) meant minimum niche saturation; three (3) demonstrated moderate niche saturation; and four (4) was equivalent to maximum niche saturation (Goodman, et al., 1993). The hypothesized eight-factor model by Goodman, et. al., (1993) produced a chi-square of 765.27 (df = 362). When the researchers applied the Tucker and Lewis (1973) incremental fit index, the value of rho was .90. This was equal to their critical value and ?evidence that the hypothesized model?s variances and co- variances were consistent with the sample of variances and co-variances? (Goodman, et al., 1993, p. 171). Several alternative models were tested and compared and with the except ion of one, ?all alternative models that were tested produced highly significant chi-square differences and values of rho that ranged from .62 to .84 ? (Goodman, et al., 1993, p. 171). Thus, Goodman and colleagues (1993) concluded that none of the alternative models improved the fit over the hypothesized model using the sample data. The researchers also performed standardized parameter estimates, equivalent to factor loadings (standardized maximum likelihood estimates), on the 15 item-pairs. They found that all estimates were statistically significant and exceeded the usually accepted minimum of .40 (Nunnally, 1978). Factor analytical methods, particularly confirmatory factor analysis, may be useful in establishing construct validity (Goodman, et al., 1993). Confirmatory factor analysis (referred to as ?factorial? validity, one type of construct validity) 27 allows scale developers to statistically test whether the items designed to measure the underlying construct cluster as expected (Comrey, 1988). This analysis was conducted. Inter-factor correlations and reliability estimates for the eight-factors were done. ?The interfactor correlations (corrected for attenuation) were largest among the four routines? subscales (average = .85), and the four niche saturation subscales (average = .76). The correlation between the niche saturation factors for maintenance and managerial subsystems was 1.00 (corrected for attenuation), suggesting the possibility that these factors were not distinct in the sample. Routines and niche saturation factors within the same subsystem produced the next largest correlation coefficients (average = .53). Routines and niche saturation factors across subsystems produced the smallest correlations (average = .43)? (Goodman, et al., 1993, p. 172). Cronbach alphas were conducted and demonstrated moderate to high reliability for the subscales, despite the small number of items forming the scales (Goodman, 1993). The researchers found that ?even the value of .44 for the niche saturation-maintenance subscale falls within acceptable limits. The average corrected item-total correlation of .28 for that subscale is equivalent to an alpha of .80 for a 15-item scale? (Goodman, et al., 1993, p. 173). Based on Goodman?s study, the LoIn instrument was found to be valid and reliable. The LoIn instrument is grounded in theory (Katz, 1978; Yin, 1979) and in empirical study (Goodman, 1987) and has potential for measuring the extent to 28 which an innovative program is integrated within its host organization (Goodman, 1993). Therefore, it was used in this study. Sustainability and Institutionalization Research Federal, state, and philanthropic funding organizations increasingly require prospective grantees to include strategies in their grant applications for sustaining health promotion programs after funding ends. More recently, the importance of ?institutionalization,? or developing community and organizational support for health promotion programs so that they remain viable in the long term has been recognized (Goodman, et al., 1993). Although there is a growing appreciation of the concept of institutionalization, there is no clear understanding of how to bring it about or track and measure it (Green, Wilson, and Bauer, 1983). ?Without clear definitions and measures, it is difficult, if not impossible, to develop adequate studies of the phenomenon and the factors that produce and sustain it? (Goodman, et al., 1993, p. 162). Shediac and Bone (1998) reviewed the literature on sustainability of community-based programs both in the US and abroad. These researchers noted that little consensus existed regarding the conceptual and operational definitions of sustainability and more importantly, an empirical knowledge base about the determinants of sustainability was still in the early stages. 29 Researchers often use the concepts of institutionalization and sustainability in describing the results of their work without an operational definition and actually without having studied these concepts per se according to their reports. Some of these studies (Bruckner, Mangan, Godin, and Pogach, 1999; Rubardt et al., 1999; Dasgupta and Priya, 2002) were identified in a literature search with these concepts as key words and others (Chalmers, et al, 2002; Crist and Escandon-Dominguez, 2003; Lienhardt and Ogden, 2004) used the words in the text only but again, the concepts themselves were not studied and reported. In both situations, the concepts were simply not addressed in the studies and thus were not reported in the literature articles. During the past fifteen years, there have been several approaches implemented by researchers to address the measurement and operational definitions of health promotion program institutionalization and sustainability. Goodman?s (1987) dissertation study Factors Affecting the Long-Term Viability of Health Promotion Programs: An Institutionalization Perspective, examined factors which inhibit and facilitate the long-term viability, or institutionalization, of health promotion programs. The research was based on case studies of ten health promotion programs in nonprofit organizations, i.e., county health departments, universities, public schools, and community health agencies. The investigation tested a theoretical model that explained how innovative health promotion programs became institutionalized and produced a matrix for defining the dimensions of institutionalization. 30 Since 1987, Goodman has used this research as a basis for other studies looking at institutionalization. For example, Goodman and Steckler (1987-88) conducted a study on Life and Death of A Health Promotion Program: An Institutionalization Case Study and looked at a program directed at preventing use of alcohol and tobacco among young adolescents. It was determined that this program was not institutionalized. Goodman and Steckler (1989) presented a framework for assessing the institutionalization of programmatic innovations in nonprofit community agencies and in schools. This research was aimed at specifying the dimensions that comprise the institutionalization of an organizational innovation. The authors suggested that institutionalization of innovative health promotion programs in organizations can be characterized as the axes of a two dimensional matrix (Figure 2). This framework (Steckler and Goodman, 1989) was applied in a study of ten health promotion programs funded by the Virginia State Health Department and operated by local schools and community health agencies. The study addressed six implications for practice that suggested how to optimize the institutionalization of health promotion programs. These practice implications were: (1) Cultivate A Program Champion; (2) Favor Strong Subsystems; (3) Fit with Organization Mission; (4) Avoid Brokers; (5) Establish Appropriate Funding Periods; and (6) Fund Existing Programs. As a result of studying these ten health promotion programs, Goodman and Steckler (1989) proposed a model for the 31 institutionalization of health promotion programs. The model was described as exploratory in nature, although grounded in prior research and theory. Lefebvre (1990) looked at program institutionalization in relation to the planning and implementation of three community cardiovascular disease prevention projects funded by the National Heart, Lung and Blood Institute: the Minnesota Heart Health Program, the Pawtucket Heart Health Program, and the Stanford Five City Project. All three of these projects struggled with the problem of designing community interventions that would outlive the federal funding. Lefebvre viewed institutionalization of these community health programs as an outcome in search of a process and thus presented a marketing strategy for maintaining such programs, specifically for the Pawtucket Heart Health Program. The strategy involved four steps (Lefebvre, 1990). The first was to specify individual, organizational, network, and community objectives for institutionalization. The second was to develop a marketing plan that specified the (a) practices, (b) audiences, (c) channels, (d) messages, (e) products, (f) services, and (g) resources that were available or necessary to reach the objectives. The third step involved developing a marketing strategy that translated the marketing plan into a series of steps which allowed the objectives to be met. The last step consisted of a portfolio analysis which assessed the viability and marketing needs of existing products and services, as well as the program in its entirety, with respect to institutionalization objectives, the marketing plan, and the marketing strategy. Strategic planning was an ongoing process in which each step 32 had an impact on the others. It was imperative that the community itself was a partner in planning to help define, implement, monitor, and refine the institutionalization strategy. Miller, Bedney, and Guenter-Grey (2003) described their attempt to develop and implement a method for assessing whether community organizations had the organizational capacity to collaborate in a national study to prevent HIV infection among young men who have sex with men and what, if any, needs these institutions had for organizational capacity development assistance. The Feasibility, Evaluation Ability, and Sustainability Assessment (FEASA) used by the researchers combined qualitative methods for collecting data (interviews, organizational records, observations) from multiple sources to document an organization?s capacity to provide HIV prevention services and its capacity- development needs. The researchers piloted FEASA in 13 communities and examined the benefits of using a systematic approach to partnership development. Through FEASA, the researchers assessed the individual organization?s competence in board development and management, fiscal development, grant writing, leadership development, human resource management, and volunteer management. Three sustainability principles were proposed. (1) Effectively developing and managing a board of directors is essential to long-term organizational health. (2) Grant writing and financial management are related but separate areas of competence. (3) Developing and managing the organization?s human resources, including volunteers, and cultivating leadership promote the 33 organization?s ability to function well (feasibility) and its ability to survive and evolve over time (sustainability). Goodson, Smith, Evans, Meyer, and Gottlieb (2001) evaluated the institutionalization of Put Prevention Into Practice (PPIP), which consists of a kit of office-based tools intended to support the provision of preventive services by primary care providers. The study examined the institutionalization of PPIP within five primary care clinics funded by the Texas Department of Health to implement PPIP, and to examine the organizational determinants of program institutionalization. The researchers utilized an adaptation of the Goodman et al., (1993) LoIn scales for qualitative data collection and for development of an institutionalization score for each site. To assess the organizational determinants of institutionalization, a review of the health promotion and organizational behavior literatures was carried out and a list of factors was identified. These factors were used as categories for the analysis of the qualitative data collected between 1994 and 1996. Chart audit data for three documentation behaviors were also collected to examine the level of program implementation within each site between December 1996 and May 1997: (1) use of flowsheets (2) use of health risk assessments, and (3) documentation of assessment of risk coupled with appropriate counseling. The three documentation behaviors reflected the major systems changes involved in the use of PPIP and were considered a proxy measure for integration of the program within a site. 34 The study results showed that PPIP had been maintained at varying degrees of integration in four of the five sites studied for six years after adoption. Organizational factors that facilitated the institutionalization process were the site?s institutional strength, the integration of PPIP within extant programs and services, visibility of the program within and outside the site, planning for the termination of grant funding, and presence of a program champion with middle to upper-level managerial authority. In other words, if a program is successfully implemented, this does not necessarily mean the program will be institutionalized. Recently, there have been six evaluation studies that included institutionalization and sustainability issues in MCH programs receiving ongoing HRSA-MCHB funding. In May 1996, Solloway, Gotschall, Barta, and Avery conducted a study, Emergency Medical Services For Children: An Evaluation Of Sustainability In Seven States, on the national Emergency Medical Services for Children (EMSC) Program. The intent of EMSC, established in 1984, was to assist states in developing sustainable, statewide EMSC programs. A major goal of the EMSC grant program was to develop or increase the states? capacity to provide emergency care to ill and injured children. This particular evaluation used a multifaceted case study approach to assess EMSC in seven states. One of the main findings was that in all seven states, some aspects of the EMSC grant program were successfully institutionalized (Solloway, et al., 1996, p. 23). The Implementation of Healthy Start (Howell et al., 1997) focused on the Healthy Start program launched in 1991 by the HRSA- MCHB. The purpose of 35 the Healthy Start program was to demonstrate innovative ways to reduce infant mortality in some of the areas with the highest infant mortality rates in the country. HRSA-MCHB chose 13 urban areas and two rural areas in which to implement a five-year Healthy Start demonstration of community-based approaches to reduce infant mortality. The goal of the 15 projects was to reduce infant mortality by 50 percent during this period. The HRSA-MCHB required the following Healthy Start program components: community involvement through a consortium; outreach and case management; support services such as transportation and nutrition education; enhanced clinic services; and community-wide information campaigns. Sustaining Healthy Start activities became a major issue for all projects as the date for significant reductions in federal funding approached in the fall of 1997 (Howell et al., 1997). Plans to sustain program activities varied widely by project. Projects implemented a mix of strategies to sustain some or all program components, including forming a nonprofit organization, integrating Healthy Start activities with health department activities, negotiating with managed care organizations or Medicaid programs to provide services, submitting grant applications to new funders, and giving technical assistance to their contractors to help them secure alternative funding. All Healthy Start projects expected a drop in federal funding and adopted one or more strategies for sustaining many of their Healthy Start activities. It seemed apparent that much of the work of Healthy Start would continue through 36 (1) federal funding for some components, (2) integration with existing health department activities, and (3) new sources of funding such as revenue from managed care plans and grants from foundations. It was too soon to tell whether this near-term continuation of much of the Healthy Start program would endure to provide a long-term legacy of the demonstration (Howell et al., 1997). In June 1999, eight years into the extended demonstration program, a study of nine city Healthy Start projects was initiated to look at effects of community involvement (Thompson, et al., 2000). Sites studied included Pittsburgh, Philadelphia, Boston, Chicago, Cleveland, New Orleans, Kansas City, New York City, and the rural site of Pee Dee, South Carolina. These sites represented a wide diversity of geographic regions, consortia structures, and variations in grantee organizations from local and state health departments to non-profit agencies. The study found that sustained community involvement significantly enriched programs. The study showed that community involvement created community acknowledgement of the infant mortality crisis; effective outreach to families at risk for infant mortality; positive changes in individual behaviors; identification of key community issues that impacted maternal and infant health; innovative programs of health and social service delivery systems to address needs of participants and the community; new abilities to address issues of race and significant programmatic partnerships that were likely to be sustained beyond 37 Healthy Start?s funding cycle; and institutionalized programs, policies and practices that linked health interventions with the achievement of health outcomes. The results of the study concluded that community involvement practices must remain integral to programs for the success of Healthy Start to be preserved and replicated. If maintained as a separate program, this is a relatively easy recommendation to follow. However, if Healthy Start program funds were moved into the Title V block grant, new specific language and oversight mechanisms would be required to ensure that community involvement practices were effectively implemented. This study also pointed to the potential value of including community involvement principles in broader public health policy and in other health programs. The Evaluability Assessment and Evaluation of the Community Integrated Service System (CISS) Program (Teitelbaum, Irwin, Mason, Foster, and Thomas, 1998) of the CISS Phase I projects funded by the HRSA-MCHB reported on a cohort of 32 projects funded in 1992 and a cohort of eight projects funded in 1993. The purpose of the CISS Program was to enhance development of comprehensive, coordinated maternal and child health service systems. Under the MCHB program, utilizing Title V funding, each site received grant funding, ranging from $103,000 to $296,000 (averaging $187,000) per year for four years. The grantees were located in 22 states and served a diverse array of needy populations. Given the absence of a common dataset for the CISS program, the 38 evaluation team primarily used modified techniques of network analysis to: 1) characterize the extent to which the CISS service systems achieved service systems integration at several levels; 2) investigate factors that facilitated or hindered success of the projects, and 3) derive meaningful indicators of service systems integration. The major finding on the issue of sustainability was that sustainability was tempered by the availability of resources at local and state levels and by historical antecedents at these levels. Therefore, if local and state funds had historically been available for maternal and child health service systems, these systems were more than likely sustained. The fifth evaluation study funded by the MCHB was The Healthy Tomorrows Partnership for Children Program in Review: Analysis and Findings of a Descriptive Survey (Eisen, et al. 1999). The HTPCP evaluation was a descriptive study that utilized a self-administered survey questionnaire mailed to project directors of all 86 HTPCP projects initially funded between 1989 and 1996. The response rate was 87 percent. There were seven conclusions (Eisen, et al., 1999, pp. 6-7): 1) The HTPCP appears to be an effective strategy for promoting children?s access to health services at the community level; 2) Modest funding provided to community organizations, with a matching fund requirement, can leverage significant amounts of money for children?s health care; 3) The HTPCP includes elements (matching funds requirement and the five-year federal project period) that successfully foster the long-term sustainability of services; 4) Small, community-based projects do not have the expertise or resources to 39 conduct outcome evaluation; 5) Meaningful multi-agency partnerships and collaborations can greatly improve the delivery of services for children, but they are challenging to develop and attempts to do so frequently fail; 6) Pediatricians and other pediatric health professionals, when provided with support through a mechanism such as a grant, can serve as leaders and advocates in improving children?s access to services; and 7) The activities of staff at the federal level and at the AAP provide important guidance and leadership to HTPCP projects and contribute to the program?s success. Unfortunately, this study did not provide substantive information about how the conclusions were drawn. For example, the third conclusion was related to sustainability, however, the elements that foster sustainability were not clearly identified in the study. The number/percent of HTPCP projects sustained out of the total number of projects was not provided in the report. Reducing Infant Mortality: Lessons Learned from Healthy Start (Devaney, Howell, McCormich, and Moreno, 2000), the national Healthy Start evaluation, covered the five-year demonstration period, fiscal years 1992 through 1996. The national evaluation of Healthy Start included a detailed process and outcome analysis. The process analysis described and documented the Healthy Start program?the community context, interventions, and implementation. The outcome analysis assessed whether Healthy Start achieved its goals of reducing infant mortality and improving maternal and infant health. There was minimal focus on sustainability in this evaluation study, although the following lesson 40 learned was stated: Health departments could help to sustain important program components when federal funding declined (Devaney et al., 2000, p. xii). The six evaluation studies described above shared some commonalities. None of the six studies used a formal research design, and thus, the findings on sustainability were not generalizable to other programs and situations. There were no operational definitions of sustainability in the six evaluation studies; therefore, the conclusions drawn in each of the studies were not linked to operational definitions. In addition, there were no substantive findings related to sustainability in any of the six study reports. HTPCP Study Variables The HTPCP was conceptualized to stimulate community members to identify a local MCH problem, problem-solve as a group and come up with programmatic objectives and strategies to address the problem, plan the development and implementation of the resulting programmatic intervention, and continue to provide ongoing input and feedback into the implementation and evaluation of the community-based program. The underlying intent of PACs, pediatric involvement, and matching funds as HTPCP requirements/criteria was that with buy-in from the community, resources would be forthcoming and thus sustainability would occur. An evaluation component was also a requirement for all SPRANS grants; the HTPCP required an outcome evaluation. 41 Although most agree about the value of sustainability as a general goal, there is some question about ?what? is to be sustained. For example, Green (1989) questioned whether long-term program continuation or ?institutionalization? is the proper goal of grant-funded programs, arguing that long-term program effects may best be seen ?as investments in people rather than investments in programs.? Green saw the developmental functions of grants to be the development of problem-solving abilities, experience, and leadership and confidence in the community. Further, ?A community will be better equipped to adapt and respond to new challenges and programs if the legacy of a program or a grant is the increased ability and will of people to tackle the next problem?.? (Green, 1989, p. 44). Community organization and ownership can also be enhanced as a result of institutionalization of a program. Community organization is a planned process to activate a community to mobilize its own social structures and resources to accomplish community goals, decided primarily by community representatives and consistent with local values (Bracht, et al., 1994). The community organizing process is a critical aspect of health action and is often the ?glue? that strengthens citizen interest, nourishes participation in programs, and encourages support for long-term maintenance of successful intervention efforts (Bracht and Kingsbury, 1990). An important outcome of the process of community and citizen participation is community ownership (Bracht, et al., 1994). Also, local 42 community board involvement in the planning and implementation of projects leads to ownership and provides the foundation for local incorporation or maintenance success (Bracht, et al., 1994). Pediatric involvement as a HTPCP project requirement (criterion) is intended to include all community providers of pediatric health care, including pediatricians, other physicians, pediatric nurse practitioners, family nurse practitioners, physicians? assistants, etc. In order to explore pediatric involvement in the HTPCP, it is necessary to look at the AAP?s definition of community pediatrics to remind all pediatricians, generalists and specialists alike, of the profound importance of the community dimension in pediatric practice. Community pediatrics is all the following (AAP, 1999, p. 1304): A perspective that enlarges the pediatrician?s focus from one child to all children in the community; A recognition that family, educational, social, cultural, spiritual, economic, environmental, and political forces act favorably or unfavorably, but always significantly, on the health and functioning of children; A synthesis of clinical practice and public health principles directed toward providing health care to a given child and promoting the health of all children within the context of the family, school, and community; A commitment to use a community?s resources in collaboration with other professionals, agencies, and parents to achieve optimal accessibility, appropriateness, and quality of services for all children, and to advocate especially for those who lack access to care because of social or economic conditions or their special health care needs; and An integral part of the professional role and duty of the pediatrician. Eisen, et al. (1999) reported that 77, 50, and 40 percent of the HTPCP projects (funded between 1989 and 1997) had pediatricians, nurses or nurse 43 practitioners, and pediatric nurse practitioners, respectively, on staff. They also reported that approximately one-half of the HTPCP project directors were individuals with medical degrees, the majority of whom were pediatricians. Weitzman (1997) reported that during 1987, pediatricians provided approximately 50 percent of all office-based visits for children and youth, ages 0-19 years in the US. Weitzman also found that with the marked geographic variations in the availability of pediatricians, with significant shortages in many rural and indigent urban areas, that family practitioners, internists, and pediatric nurse practitioners provide the majority of the primary care services to children who have no access to pediatricians. These providers may be new to the geographic areas in which they practice and must be mindful and conscientious in regard to all day-to-day community aspects related to their pediatric medical practice and involvement in the specific community. Failure to pay attention to political issues represented by individuals within communities can prevent the best of programs from getting off the ground or from being sustained at a later date. Coalitions with broad community representation are recommended to address the political issues that inevitably arise in communities (Green and Kreuter, 1991). Effective involvement of key organizational and community leaders influences the quality and thoroughness of the planning process that precedes implementation and thus, the leveraging approach to sustainability. Leveraging is defined as ?the use of initial investments and commitments to draw larger investments and commitments? 44 (Green and Kreuter, 1991, p. 208). The leveraging approach to sustainability puts even more weight on the quality and thoroughness of the planning process and precedes implementation and the effective involvement of key organizational and community leaders in that process (Green and Kreuter, 1991). The principle of participation begins to pay off in sustained commitment to the program from multiple sources. Green and Kreuter (1991, p. 164) reference the matrix in Figure 1 (Goodman, et al., 1993) and state, ? the more cells a particular health promotion program occupies, the more institutionalized it is.? Another implication of the leveraging strategy for institutionalization is that evaluation data are even more eagerly awaited. Early data from the program on how the program is taking hold, how it is being received, and what impact it is having on short-term objectives can become political and marketing data. Strengths and the weaknesses of the program can be presented so that supporters can find some points to take pride in and others where they can offer help. ?The prospects for sustaining programs depend in part on evaluation results? (Green and Kreuter, 1991, p. 208). Evaluation is the comparison of an object of interest against a standard of acceptability (Donabedian, 1980; Green and Kreuter, 1991; Sarvella and McDermott, 1993). Donabedian (1980) offers structure, process, and outcome evaluation as indicators of quality of care. Structure addresses a sufficiency of resources and proper system design (Donabedian, 1980). Process evaluation 45 examines the procedures and tasks involved in implementing a program (US Department of Health and Human Services, Public Health Service, National Institutes of Health, Office of Cancer Communications, National Cancer Institute, 1992). Outcome evaluations are designed to examine the long-term effects of the program (Green and Kreuter, 1991). At the very least, evaluation is an assessment of the worth of a program, a method, or some other object of interest. It may provide an estimate of the degree to which resources result in specified activity and the degree to which performed activities attain objectives and goals, and thus, demonstrate program effectiveness. The question always arises of how to judge the effectiveness and or success of the program itself (McQueen, 2003). Effectiveness is difficult to define and even more difficult to measure, but is often associated with cost (Graziani, 1996; Skillen, et al., 2002). Rogers (1993) suggested that effectiveness be assessed using Donabedian?s framework of structure, process, and outcome. The evidence-based movement has been promoted vigorously in medicine and medical clinical practice where the environment, practice and relative straightforwardness of treatments mean that researchers are accustomed to the use of a randomized controlled trial (RCT) to answer the question of effectiveness (Waters and Doyle, 2002). Effectiveness has also recently been measured with RTCs in programs such as breastfeeding peer counseling (Chapman et al., 2004), home visiting by paraprofessionals and by nurses (Olds, et al., 2002), and rural 46 youth health and safety initiative (Lee, et al., 2004). Effectiveness researchers evaluate the benefit of an intervention under usual conditions (Schwartz, Ball, and Moser, 1982) and examine the benefit of an intervention in actual practice (Whittemore and Grey, 2002). Effectiveness research is a critical step to determine the robustness of the intervention under practice conditions (Newman and Tejada, 1996). All MCHB grantees, including the HTPCP projects, are required to submit an evaluation component in the initial grant application. When the objective review committee (ORC) approves a grant, the ORC approves the evaluation component as part of the whole program and stipulations regarding the evaluation component can be placed on the potential grantee at that time. Eisen, et al. (1999) reported that the majority of the HTPCP survey respondents undertook a process evaluation. Although the requirement is for each potential HTPCP grantee to submit an outcome evaluation component as a part of the application, only six out of 74 HTPCP respondents actually attempted outcome evaluations (Eisen, et al., 1999). However, it does appear that the HTPCP projects are evaluating themselves for program effectiveness in their individual evaluations. In addition to the above variables (PACs, pediatric involvement, evaluation), this researcher believed other variables should be considered, including level of matching funds, type of matching funds, discipline of PDs, educational level of PDs, and the number of PDs for each project over the five- year project period. There was no literature found on these five variables and 47 their relationship to sustainability; therefore, this study explored these variables. Summary This review of literature defined institutionalization and sustainability and established that in this study institutionalization was considered a form of sustainability and the terms ?institutionalization? and ?sustainability? were used interchangeably. The literature showed that Goodman (1987) studied factors affecting long-term viability of health promotion programs in his dissertation study. In 1989, Goodman and Steckler developed a framework that characterized institutionalization of health promotion programs. Goodman, et al. (1993) used the framework and conducted a study to test an instrument for measuring the LoIn of health promotion programs; the LoIn instrument was used in this study. This review also demonstrated that six MCH institutionalization/sustainability evaluation studies did not employ operational definitions of the construct, therefore, conclusions drawn in each of these studies were not substantive nor generalizable. 48 CHAPTER III: METHODOLOGY Introduction This chapter describes the process used to conduct this study. The process included: study population selection and recruitment of participants; study procedures; scoring of HTPCP Survey Instrument; data collection; operationalization of variables; and data analyses. A pilot-test was conducted with five, out of 94, randomly selected HTPCP project directors (PDs). The purpose of the pilot-test was to identify any problems with the data collection instruments and data collection procedures. After the pilot-study (there were no suggested changes to the instrument or process), the survey questionnaire was mailed to the remaining 89 PDs who were funded from 1989 through 1997 and had completed their federal five-year project period and funding by 2002. This study was conducted within a six month timeframe, August 2003 ? January 2004. Study Population and Recruitment of Participants The participants in this study were all of the remaining (after subtracting the five pilot projects from the total of 94) 89 project directors (and/or select project staff if the PDs chose) who participated in projects funded between October 1, 1989 and September 30, 1997. A total of 94 HTPCP projects were funded in 34 states plus Washington, DC and Puerto Rico during this timeframe (Appendix C). 49 The greatest number of projects were funded in California (12) followed by Massachusetts (7), Texas (7), and New York (5). Fifteen of the remaining 31 states, including Puerto Rico, had only one funded project. The DHHS/HRSA Region V (Midwest US with Regional Office located in Chicago, Illinois) had the highest number of funded projects (20) and was thus over-represented, while Region VIII (West Central US with Regional Office located in Denver, Colorado) had the fewest funded projects (2) and was under-represented. An incentive (the opportunity to win $100) was offered to participants for helping with the study. Each HTPCP project director was mailed a postcard to return separately from the completed questionnaire. The participant was instructed to provide his/her name, phone number, and mailing address on the postcard. Postcards were destroyed after the random drawing. The postcards were placed in the same container upon receipt by the researcher. The drawing was conducted after time was allowed for the follow-up of incomplete and unreturned questionnaires from the participants. This was planned to occur approximately eight weeks after the initial mailing of the study survey instrument but did not occur until the end of 16 weeks, due to the time required to reach some of the PDs and to receive their responses. The winner was mailed a cashier?s check for $100. The researcher mailed and e-mailed a letter to all participants announcing the winner. 50 Study Procedures Permission to conduct the study was sought from the Institutional Review Board at the University of Maryland, College Park, Maryland. Once obtained (Appendix D), the HTPCP Survey Instrument was mailed to five randomly selected PDs from the 94 HTPCP projects for the pilot-test. A cover letter (Appendix E) from the researcher was included in the instrument packet. Both the mailing envelope and the cover letter to the PDs bore a colored 1.5 x 2.0? yellow, black, and white colored HTPCP logo to promote recognition of the HTPCP by the recipients. This letter explained the nature of the study and requested that the PDs self-administer (including other project staff as appropriate) the instrument. In addition, respondents were asked to complete the enclosed Consent Form (Appendix F), and reminded their responses were anonymous. The cover letter (Appendix E) was addressed to the PD and included an individual code number assigned by the researcher. The code number was needed so the data on the instrument could be matched with other data (PACs, pediatric involvement, levels of matching funds, type of matching funds, discipline of PD, educational level of PDs, number of PDs over the five-year funding period, and evaluation) obtained from project records by the researcher (Appendix G). Pilot-test participants were asked how long it took them to complete the questionnaire and about any items that were unclear or ambiguous (Appendix H). A return preaddressed stamped envelope was provided to all. 51 Telephone contact was made by the researcher to assure responses from the five randomly selected pilot-test participants. All five pilot-test participants responded so it was not necessary to randomly select additional participants to bring the pilot-test respondent number to five. After the pilot was complete, survey packets were distributed to the 89 remaining respondents. If a response was not received by the end of six weeks, new packets were mailed and a telephone and/or e-mail contact was attempted to encourage participation as the researcher had a master control sheet. An additional request for a response was made three weeks after the second mailing. Scoring of HTPCP Survey Instrument The 46-item HTPCP Survey Instrument contained items representing the eight factors of the LoIn Instrument. Passages represent the first phase of program routinization. In the Goodman, et al. (1993) eight-factor model, passages were subsumed under routines, one of four factors for each organizational subsystem (production, maintenance, support, and management). The other four factors of the model make up the program?s niche saturation within an organization?s subsystems (production, maintenance, support, and management). In Table 2, the eight factors and corresponding survey items of the HTPCP Survey Instrument (Appendix I) are identified. Factor I, routinization of program production, demonstrated the repeated, or routine, carrying out of program activities as indicated in written program goals 52 and objectives, plans, schedules, implementation and monitoring strategies, and evaluations (HTPCP items 1, 2, 4, 5, 7, 8, 10, 11, 13, and 14). Factor II, niche saturation of program production, represented the extent to which all possible program activities were written and implemented (HTPCP items 3, 6, 9, 12, and 15). Factor III, routinization of program maintenance, showed the organization staff?s day-to-day (routine) involvement in and priority given to the program?s operations (HTPCP items 25, 26, 28, 29, 31, and 32). Factor IV, niche saturation of program maintenance, indicated the extent to which staff were involved in and committed to the day-to-day program activities (HTPCP items 27, 30, and 33). Factor V, routinization of program support, represented the ongoing, or routine, priority placed on the program by the organization?s administrative staff through the manifestation of permanent funding, staffing, and priority placed on the program (HTPCP items 34, 35, 37, 38, 40, 41, 43, and 44). Factor VI, niche saturation of program support, demonstrated the extent of commitment by the host organization?s administration as reflected through the permanent funding, staffing,and status placed on the program (HTPCP items 36, 39, 42, and 45). Factor VII, routinization of program management, indicated the formal and routine implementation of program supervision consisting of program supervisors, written job descriptions for all staff, and an ongoing accountability process represented through program evaluation (HTPCP items 16, 17, 19, 20, 22, and 23). Factor VIII, niche saturation of program management, demonstrated the extent to which the program had formal supervision, written job descriptions 53 Table 2. HTPCP Questionnaire Items for Institutionalization Factors Item __________________________________________________________________ Factor I: Production Routine 1. Written goals and objectives 2. Number of years written goals and objectives followed 4. Written plans and procedures 5. Number of years written plans and procedures followed. 7. Written schedule for program implementation of activities 8. Number of years written schedules followed 10. Strategies adapted to local circumstances 11. Number of years locally adapted strategies followed 13. Formal evaluation conducted 14. Number of times program formally evaluated Factor II: Production Niche Saturation 3. Proportion having written goals and objectives. 6. Proportion having written plans and procedures. 9. Proportion having written schedeules. 12. Proportion having locally adapted strategies. 15. Proportion having been formally evaluated. 54 Table 2. HTPCP Questionnaire Items for Institutionalization Factors (Continued) Item __________________________________________________________________ Factor III: Maintenance Routine 25. Permanent staff assigned 26. Number of years permanent staff assigned 28. Administrative-level individual actively involved in advocacy 29. Number of years of advocacy by administrative-level staff 31. Staff other than program staff actively contribute to operations 32. Number of years such staff have contributed Factor IV: Maintenance Niche Saturation 27. Number of program staff in permanent positions 30. How active administrative- level individual has been in advocacy 33. Proportion of organization staff other than program staff who actively contribute to operations 55 Table 2. HTPCP Questionnaire Items for Institutionalization Factors (Continued) Item __________________________________________________________________ Factor V: Supportive Routine 34. Transition from pilot status to permanent status 35. Number of years having permanent status 37. Permanent physical space within the organization 38. Number of years maintained permanent space 40. Source of funding similar to those for other programs 41. Number of years having similar funding source 43. Staff hired from stable funding source 44. Number of years staff hired from stable funding source Factor VI: Supportive Niche Saturation 36. Permanency of program 39. Proportion of needed permanent space occupied 42. Permanent source of funding 45. Permanency of funding for key staff 56 Table 2. HTPCP Questionnaire Items for Institutionalization Factors (Continued) Item __________________________________________________________________ Factor VII: Managerial Routine 16. Supervisor formally assigned 17. Number of years supervisor formally assigned 19. Written job descriptions 20. Number of years formalized job descriptions followed 22. Evaluation report schedule similar to other evaluation reports in organization 23. Number of years evaluation reports have been produced on a similar schedule as others Factor VIII: Managerial Niche Saturation 18. Proportion of program receiving supervision 21. Number of program staff with written job descriptions 24. Extent to which evaluation reports produced on a similar schedule as other programs __________________________________________________________________ were in place for staff, and program evaluation was ongoing (HTPCP items 18, 21, and 24). 57 Initially, both routine and niche saturation items on the HTPCP survey instrument were scored (Appendix J) using a 4-point scale. Scoring is specified in Table 3. Scoring for routine and niche saturation items is shown in Table 4. Table 3. HTPCP Survey Instrument Item Scoring A If ?No? or ?Not sure/not applicable? was checked for routine items 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, and 43, then the score for the item was 0. B If ?Yes? was checked for these routine items (identified in A) and ?0? or ?1? was written for items 2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, and 44 (these items had a raw score range of 0 ? 14), then the score for the item was 1. C If ?Yes? was checked for the routine items (identified in A) and ?2? or ?3? was written for the items identified in B, then the score for the item was 2. D If ?Yes? was checked for the routine items (identified in A) and ?4? or ?5? was written for the items identified in B, then the score for the item was 3. E If ?Yes? was checked for the routine items (identified in A) and ?6? or more was written for the items identified in B, then the score for the item was 4. F Niche saturation items 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, and 45) were scored according to the number circled for that item. The number circled ranged from 1-4. 58 Table 4. Scoring for Routine and Niche Saturation Items Score Routine Item Score Niche Saturation Item 1 2 3 4 Equal to no passage Passage plus one iteration of a routine Two or three routine iterations Four or more iterations 1 2 3 4 No niche saturation Minimum niche saturation Moderate niche saturation Maximum niche saturation Following the assignment of scores of 1-4 to the routine and niche saturation items, mean scores were determined. A HTPCP Score Sheet Grid (Appendix K) was used to summarize the scores on the survey items and determine these mean scores. Routine items 1 and 2; 4 and 5; 7 and 8; 10 and 11; 13 and 14; 16 and 17: 19 and 20; 22 and 23; 25 and 26; 28 and 29; 31 and 32; 34 and 35; 37 and 38; 40 and 41; and 43 and 44 were added together resulting in one score for two sub- items. The scores for the sub-items were added together and divided by the number of sub-items in the subsystem (production, managerial, maintenance, and support) to produce the mean for the individual sub-system. Niche saturation items 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36 39, 42, and 45 were scored separately. Table 5 provides routine and niche saturation item mean scores indicating level of institutionalization (Goodman, et al., 1993). Therefore, the 59 Table 5. Item Mean Scoring Categories Indicating Level of Institutionalization Routine Mean Score Level of Institutionalization Niche Saturation Mean score Level of Institutionalization 1 or 2 3 4 Low to moderate Moderate to high High  2 > 2 but  3 > 3 Low Moderate High mean scores demonstrated low, low to moderate, moderate, moderate to high, and high levels of institutionalization. Demographic data from survey participants were requested in the last item (#46) on the HTPCP Survey Instrument (Appendix I). Participant data were collected on age, sex, years of formal education after high school, discipline/occupation, job title, length of time in position, and years with the HTPCP. Data Collection In addition to the data gathered with the HTPCP survey instrument, data were collected from a review of the unofficial (copy) individual records of the 89 HTPCP projects made available to the researcher by the MCHB. This researcher was the Director of the HTPCP from June 1, 1991 through May 31, 2000. During that time the researcher assembled and maintained the majority of the HTPCP records to be used in this study. Ordinarily, the MCHB keeps unofficial records 60 for the length of the project period (five years in the case of the HTPCP) and then records are destroyed. The official (original) HTPCP records are maintained by the MCHB Office of Grants Management, located in Rockville, Maryland. There are two sets of unofficial records maintained for the HTPCP; one set is housed within the MCHB in the Division of Research, Training, and Education (RTE) and an additional set is maintained in the AAP Division of Community Pediatrics, located in Elk Grove Village, Illinois. Since the researcher left RTE, the Director of RTE made the HTPCP records available to her. Additionally, the AAP offered the researcher access to its HTPCP records. The researcher developed a record data collection form (Appendix G). This form was used to collect information from reviews of the original application for funding year one and continuation applications (progress reports) for the subsequent four years of federal funding for each HTPCP project, bringing the total number of records reviewed for each grantee to five. In addition, the Objective Review Committee Summary (which contains all strengths and weaknesses of the initial application along with the programmatic recommendations and conditions developed by the Objective Review Committee (ORC) with their decision to approve the grant) and all annual correspondence were reviewed by the researcher for each HTPCP project. The annual correspondence usually consisted of project requests for technical assistance, approval to change either budgets or program implementation and evaluation, as well as responses to any request for additional information from MCHB/AAP in 61 areas such as budget, program implementation and evaluation, and all program requirements. Raw data for each of the first eight independent variables: PACs, pediatric involvement, level of matching funds, type of matching funds, PD discipline, PD educational level, number of PDs over the five-year funding period, and the type of evaluation conducted were collected from the HTPCP five-year project records. This raw data were then converted to categories (Appendix G) for data analysis purposes. Operationalization of Variables Hypothesis #1: There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high PAC activity levels. To answer this hypothesis and the following seven hypotheses, the independent variables were categorized. The PAC was categorized into three levels of activity. Category one (1) represented minimal activity level which meant the PAC was in place and functioning by the end of the fifth funding year. Category two (2) indicated moderate activity level which meant the PAC was in place and functioning by the end of the third funding year. Category three (3) demonstrated a high activity level which meant the PAC was in place and functioning by the end of the first funding year. The intent of the PAC was that it would be made up of a group of representatives from the community who participated in a system of governance 62 for the HTPCP project for planning, implementing, monitoring, and evaluating it. The HTPCP requirement was that there was a PAC in place or a plan for one with some of the membership identified at the time the application was approved and funded. The expectation was that the HTPCP project had a PAC membership roster available (at the time the application was submitted for funding or prior to completion of the first year of funding) that demonstrated representation of community pediatric providers, parent/consumers, local businesses, education, social services and other organization representation as appropriate to the individual project. Hypothesis #2: There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high levels of pediatric involvement. There were three categories of pediatric involvement. Category one (1) reflected a minimal level of pediatric involvement which meant it was occurring by the end of the fifth funding year. Category two (2) demonstrated a moderate level of pediatric involvement which meant it was occurring at the end of the third funding year. Category three (3) indicated high pediatric involvement, meaning it was occurring by the end of the first funding year. The operational definition of pediatric involvement was any and all participation of community pediatricians/physicians and other pediatric health professionals (primarily nurses) in the community-based health service program activities in the role(s) of PD, administrator, provider, manager, advocate, educator, PAC member, or evaluator. 63 Hypothesis #3: There will be no significant difference in sustainability among HTPCP projects with minimal, medium, and high levels of matching funds. There were three categories of matching funds. Category one (1) represented a minimal matching funds level and meant the individual project met the minimal matching funds requirement of $100,000. Category two (2) demonstrated a moderate matching funds level of between $100,000 and $150,000. Category three (3) reflected a high matching funds level of over $150,000. The matching funds contributions came from both hard money and in- kind funds. Therefore, the following hypothesis was proposed. Hypothesis #4: There will be no significant difference in sustainability among HTPCP projects with a majority of hard money matching funds versus in- kind matching funds. (Majority was determined by greater than 50%.) Respondents were divided into two categories. Category one (1) indicated the majority of matching funds represented hard money (cash or potential to be liquidated to cash) while category two (2) meant the majority of matching funds were in-kind (cannot be liquidated, e.g. office space or dedicated time of a staff member). The researcher reviewed and collected information on the matching funds amounts from the five years of HTPCP record (budgets and justifications) narratives. Hypothesis #5: There will be no significant difference in sustainability between programs directed by PDs with disciplines in medicine (predominantly pediatricians) compared to all other disciplines. There were two categories of 64 disciplines. Category one (1) was assigned to those in medicine while category two (2) meant the PD?s background was in any other discipline. If there was more than one PD for the five-year project period of the HTPCP project, the latest or most current PD discipline was used. Hypothesis #6: There will be no significant difference in the measure of sustainability among HTPCP PDs with minimal, moderate, and high levels of education. Three categories of education level were used. Category one (1) meant the PD had completed up to four (4) years of formal education after high school. Category two (2) indicated the PD had completed eight (8) years of formal education after high school. Category three (3) demonstrated the PD had nine (9) or more years of formal education completed after high school. Hypothesis #7: There will be no significant difference in sustainability among HTPCP projects with no, moderate, and high turnover in PDs over the five-year project period. There were three categories of turnover in PDs over the five-year project period. Category one (1) represented no turnover, which meant the same PD oversaw the program throughout the five-year project period. Category two (2) indicated moderate turnover which meant there were two PDs over the five-year project period. Category three (3) demonstrated high turnover, meaning, there were three or more PDs over the five-year project period. Hypothesis #8: There will be no significant difference in sustainability among HTPCP projects with minimal, moderate, and high levels of evaluation. Three categories of evaluation level were used. Category one (1) represented 65 minimal evaluation which meant the evaluation component was identified as weak initially and through funding year one. Category two (2) indicated moderate evaluation and meant the evaluation component continued to be weak through funding year three. (This project would have a stronger evaluation component than category one as the project director would have been requested to strengthen the evaluation process initially, and at the end of funding years one, two, and/or three.) Category three (3) demonstrated the highest level of evaluation which meant an outcome evaluation looking at long-term effects of the program was conducted by the end of funding year five. Hypothesis 9: Matching funds will be the strongest predictor of sustainability when entered with all other predictor variables (PACs, pediatric involvement, type of matching funds, PD discipline, PD level of education, PD turnover, and evaluation). Sustainability was measured via the LoIn instrument while the predictor variables were computed as indicated above. Data Analysis Data were analyzed using the Statistical Package for the Social Sciences (SPSS, 2002). The collected data were coded and entered into a SPSS database. The socio-demographic variables were analyzed using frequency distributions, percents, means and standard deviation (SD). Nonparametric statistical tests were used given that all variables were nominal (Table 6). Therefore, chi-squares analyses were conducted on hypotheses 66 one through eight. Sustainability was the dependent variable in the eight hypotheses. The independent variables in hypotheses #s 1, 2, 3, 4, 5, 6, 7, and 8 were PACs, pediatric involvement, level of matching funds, type of matching funds, PD discipline, PD educational level, PD turnover, and evaluation, respectively. Multiple regression was performed to analyze Hypothesis 9 using the forward regression procedure. Table 6. Classification of Variables Variable Name Type of Variable Dependent Independent Sustainability Nominal Project Advisory Committee Nominal Pediatric Involvement Nominal Matching Fund Levels Nominal Types of Matching Funds Nominal PD Discipline Nominal PD Education Nominal PD Turnover Nominal Evaluation Nominal Summary A pilot-test was conducted with five randomly selected HTPCP project directors to identify any problems (none were identified) with the data collection instruments and procedures. Following the pilot, the remaining 89 HTPCP project directors whose projects were funded between October 1, 1989 and September 30, 1997 were mailed survey packets. Data analyses were conducted using chi-squares for hypotheses one through eight and multiple regression for hypothesis nine. 67 CHAPTER IV: RESULTS Introduction This chapter presents the results of the study conducted to evaluate the HTPCP and determine the main reason(s) for sustainability. A description of the study population demographics and background variables are presented here, as well as results from the analyses to support acceptance or rejection of the nine hypotheses. Study Population Characteristics A total of 81 (out of 89) HTPCP project directors responded to the questionnaire (91 % response rate). Only one PD reported that he/she had assistance from other project staff in completing the questionnaire. Table 7 presents frequencies and percents found for demographic and background variables. The mean, standard deviation (SD), range of subject age, length of time in position, and years with the HTPCP are shown in Table 8. The majority of respondents were female (80%) and had up to eight years (46%) or nine or more years (41%) of formal education after high school (Table 7). When examining the discipline of respondents 35% were physicians, 25% came from the business or administration field, and 15% were nurses (Table 7). The mean age of 68 Table 7. Demographic Variables of HTPCP Respondents __________________________________________________________________ Variable N (%) __________________________________________________________________ Gender Female 65 (80.2) Male 16 (19.8) Formal Education After High School Minimal - Up to 4 years 11 (13.6) Moderate - Up to 8 years 37 (45.7) High - 9 or more years 33 (40.7) Discipline Business/Administration 20 (24.7) Education 2 (2.5) Evaluation 2 (2.5) Public Health 5 (6.2) Medical Doctor 28 (34.6) Nurse (RN, PNP, FNP) 12 (14.8) Psychology (Psychologist) 3 (3.7) Social Work 5 (6.2) Other 4 (4.9) Job Title Assistant/Associate Professor 3 (3.7) Executive Director 13 (16.1) Pediatrician 28 (34.6) Program Director/Manager/Coordinator 35 (43.2) Health Care Provider 2 (2.5) RN = registered nurse; PNP = pediatric nurse practitioner; FNP = family nurse practitioner; Other = dental hygiene, occupational therapy, and dietetics 69 Table 8. Project Director Age, Length of Time in Position, and Years with HTPCP __________________________________________________________________ Variable X (SD) Range Age (years) 50.38 (8.84) 32-80 Length of Time in Position (years) 11.95 (7.13) 01-33 Years with HTPCP 7.00 (3.60) 01-14 __________________________________________________________________ N = 81 respondents was 50 years (SD = 8.8) and the mean length of time they had been with the HTPCP was 7 years (SD = 3.6) (Table 8). Data Analyses Findings Table 9 provides the frequencies, mean (calculated on raw data before being scored), and standard deviation of each item on the HTPCP Survey Instrument (Appendix I). All 45 items are represented within one of eight factors: Factor I - Production Routine; Factor II - Production Niche Saturation; Factor III - Maintenance Routine; Factor IV - Maintenance Niche Saturation; Factor V - Supportive Routine; Factor VI - Supportive Niche Saturation; Factor VII - Managerial Routine; Factor VIII - Managerial Niche Saturation. 70 Table 9. HTPCP Item Mean Scores Item N X (SD) __________________________________________________________________ Factor I: Production Routine 1. Written goals and objectives 81 1.00 (.00) 2. Number of years written goals and objectives followed 81 9.31 (2.77) 4. Written plans and procedures 81 1.01 (.11) 5. Number of years written plans and procedures followed 80 9.09 (2.84) 7. Written schedule for program implementation of activities 81 1.02 (.16) 8. Number of years written schedules followed 79 8.51 (2.97) 10. Strategies adapted to local circumstances 81 1.00 (.00) 11. Number of years locally adapted strategies followed 81 9.48 (2.72) 13. Formal evaluation conducted 81 3.84 (15.24) 14. Number of times program formally evaluated 45 5.69 (3.69) Factor II: Production Niche Saturation 3. Proportion having written goals and objectives 81 3.12 (.48) 6. Proportion having written plans and procedures 81 2.86 (.61) 9. Proportion having written schedeules 81 2.77 (.66) 12. Proportion having locally adapted strategies 81 3.19 (.53) 15. Proportion having been formally evaluated 79 2.49 (.80) 71 Table 9. HTPCP Item Mean Scores (Continued) Item N X (SD) __________________________________________________________________ Factor III: Maintenance Routine 25. Permanent staff assigned 81 1.14 (.38) 26. Number of years permanent staff assigned 72 9.14 (2.92) 28. Administrative-level individual actively involved in advocacy 81 1.06 (.24) 29. Number of years of advocacy by administrative-level staff 76 8.82 (2.97) 31. Staff other than program staff actively contribute to operations 81 3.60 (15.28) 31. Number of years such staff have contributed 64 9.19 (3.14) Factor IV: Maintenance Niche Saturation 27. Program staff in permanent positions 81 3.17 (.92) 30. How active administrative- level individual has been in advocacy 81 3.26 (.80) 33. Proportion of organization staff other than program staff who actively contribute to operations 79 2.61 (.79) 72 Table 9. HTPCP Item Mean Scores (Continued) Item N X (SD) __________________________________________________________________ Factor V: Supportive Routine 34. Transition from pilot status to permanent status 81 1.09 (.28) 35. Number of years having permanent status 74 7.28 (3.54) 37. Permanent physical space within the organization 81 4.80 (18.59) 38. Number of years maintained permanent space 64 8.55 (3.46) 40. Source of funding similar to those for other programs 81 4.89 (18.58) 41. Number of years having similar funding source 57 7.09 (3.51) 43. Staff hired from stable funding source 81 3.64 (15.27) 45. Number of years staff hired from stable funding source 61 7.64 (3.53) Factor VI: Supportive Niche Saturation 36. Permanency of program 81 3.12 (.93) 39. Proportion of needed permanent space occupied 78 3.13 (1.07) 42. Permanent source of funding 78 2.76 (.90) 45. Permanency of funding for key staff 79 2.82 (.83) 73 Table 9. HTPCP Item Mean Scores (Continued) Item N X (SD) __________________________________________________________________ Factor VII: Managerial Routine 16. Supervisor formally assigned 81 1.04 (.19) 17. Number of years supervisor formally assigned 78 9.53 (2.68) 19. Written job descriptions 81 2.22 (10.89) 20. Number of years formalized job descriptions followed 79 9.30 (2.69) 23. Evaluation reports similar to other evaluation reports in organization 81 4.83 (18.59) 23. Number of years evaluation reports have been produced on a similar schedule as others 63 8.94 (2.99) Factor VIII: Managerial Niche Saturation 18. Proportion of program receiving supervision 81 3.37 (.64) 22. Number of program staff with written job descriptions 79 3.61 (.54) 25. Extent to which evaluation reports produced on a similar schedule as other programs 78 2.90 (.92) __________________________________________________________________ N = Number of responses to questionnaire item 74 Both the routine items (Factors I, III, V, and VII) and the niche saturation items (Factor II, IV, VI, and VIII) on the questionnaire were scored on a 4-point scale. Scores of 1, 2, 3, and 4 were equal to low, low to moderate, moderate to high, and high levels of institutionalization, respectively. Thus, the closer the score was to four, the higher the level of institutionalization demonstrated. Routine items 2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, and 44 (Factors I, III, V, and VII) had a raw score scale of 0 ? 14 before being categorized (Appendix J) on the 4-point scale. Table 10 provides the routine and niche saturation items score range. The routine and niche saturation mean scores ranged from 1.00 ? 9.53 and 2.49 ? 3.61, respectively. For the niche saturation items scored on the 4-point scale, the closer to four, the more sustainability demonstrated. Routine mean scores greater than five indicated a high level of sustainability. Table 10 Routine and Niche Saturation Items Mean Score Range Factor Scale Range Item Mean Score Range I: Production Routine 0 ? 14 1.00 ? 9.48 II: Production Niche Saturation 0 ? 4 2.49 ? 3.19 III: Maintenance Routine 0 ? 14 1.06 ? 9.19 IV: Maintenance Niche Saturation 0 ? 4 2.61 ? 3.26 V: Supportive Routine 0 ? 14 1.09 ? 8.55 VI: Supportive Niche Saturation 0 ? 4 2.76 ? 3.13 VII: Managerial Routine 0 ? 14 1.04 ? 9.53 VIII: Managerial Niche Saturation 0 ? 4 2.90 ? 3.61 75 The mean scores on the organizational sub-systems and institutionalization factors are shown in Table 11. In this table, the mean score was placed between two factors (such as Factor I and Factor II) because this score was produced by adding the mean score from the production routine organizational sub-system items (Factor I) and the mean score production niche saturation organizational sub-system items (Factor II) together. The means from the organizational sub- systems (routine production, managerial, maintenance, and support; niche saturation production, managerial, maintenance, and support) ranged from 2.80 to 3.54. Again, the closer to four, the more institutionalization is demonstrated. Because of the way the Goodman, et al. (1993) Instrument was scored, using only nominal data, chi-square nonparametric tests had to be employed to analyze hypotheses one through eight (Tables 12 ? 19). Hypotheses one, two, three, and four stated that there would be no significant differences in sustainability among HTPCP projects in PAC activity levels, pediatric involvement levels, matching funds levels, and hard money versus in-kind funds, respectively. There were no significant differences demonstrated for hypotheses one through four; each of these hypotheses was supported (Tables 12 - 15). Hypothesis five proposed there would be no significant difference among HTPCP projects that have PDs with disciplines in medicine as opposed to all other disciplines (Table 16). A significant difference (with a p value of .05) was found in the PD discipline niche saturation production subscale (Table 16). Thus, 76 Table 11 Sustainability Organizational Sub-system Variables and Institutionalization Factors Means __________________________________________________________________ Factor Variable I II III IV V VI VII VIII __________________________________________________________________ Routine Production Mean 3.42 SD .44 Managerial Mean 3.54 SD .77 Maintenance Mean 3.37 SD .99 Support Mean 2.80 SD 1.28 Niche Saturation Production Mean 2.87 SD .43 Managerial Mean 3.26 SD .61 Maintenance Mean 3.00 SD .65 Support Mean 2.90 SD .88 Key: Factor I = PR; Factor II = PNS; Factor III = MR; Factor IV = MNS; Factor V = MaiP; Factor VI = MaiNS; Factor VII = SR, and; Factor VIII = SNS 77 Table 12. Chi-square Analyses of Project Advisory Committee by Organizational Subsystem Project Advisory Committee End Year 1 End Year 3 End Year 5 Organizational Subsystem N (%) N (%) N (%) ?  __________________________________________________________________ Routine Project Advisory Committee Production .64 .73 Minimum-Moderate 6 (46) 4 (31) 3 (23) High 32 (47) 26 (38) 10 (15) Managerial 1.19 .55 Minimum-Moderate 13 (57) 7 (30) 3 (13) High 25 (43) 23 (40) 10 (17) Maintenance .32 .85 Minimum-Moderate 13 (48) 9 (33) 5 (19) High 25 (46) 21 (39) 8 (15) Support .77 .68 Minimum-Moderate 23 (51) 15 (33) 7 (16) High 15 (42) 15 (42) 6 (17) 78 Table 12. Chi-square Analyses of Project Advisory Committee by Organizational Subsystem (Continued) __________________________________________________________________ Project Advisory Committee End Year 1 End Year 3 End Year 5 Organizational Subsystem N (%) N (%) N (%) ?  __________________________________________________________________ Niche Saturation Project Advisory Committee Production .32 .85 Minimum-Moderate 30 (45) 25 (38) 11 (17) High 8 (54) 5 (33) 2 (13) Managerial 2.28 .32 Minimum-Moderate 17 (47) 11 (31) 8 (22) High 21 (47) 19 (42) 5 (11) Maintenance 5.77 .06 Minimum-Moderate 18 (41) 15 (34) 11 (25) High 20 (54) 15 (41) 2 (5) Support .91 .64 Minimum-Moderate 15 (42) 14 (39) 7 (19) High 23 (51) 16 (36) 6 (13) df = 2 79 Table 13. Chi-square Analyses of Pediatric Involvement by Organizational Sub- system Pediatric Involvement End of Year 1 End of Years 3-5 Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Routine Pediatric Involvement Production 2.27 .13 Minimum-Moderate 2 (15) 11 (85) High 3 (4) 65 (96) Managerial 2.62 .11 Minimum-Moderate 3 (13) 20 (87) High 2 (3) 56 (97) Maintenance .43 .51 Minimum-Moderate 1 (4) 26 (96) High 4 (7) 50 (93) Support .04 .84 Minimum-Moderate 3 (7) 42 (93) High 2 (6) 34 (94) 80 Table 13. Chi-square Analyses of Pediatric Involvement by Organizational Sub- system (Continued) Pediatric Involvement End of Year 1 End of Years 3-5 Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Niche Saturation Pediatric Involvement Production 1.21 .27 Minimum-Moderate 5 (8) 61 (92) High 0 (0) 15 (100) Managerial .52 .47 Minimum-Moderate 3 (8) 33 (92) High 2 (4) 43 (96) Maintenance 2.53 .11 Minimum-Moderate 1 (2) 43 (98) High 4 (11) 33 (89) Support 2.73 .10 Minimum-Moderate 4 (11) 32 (89) High 1 (2) 44 (98) __________________________________________________________________ df = 1 81 Table 14. Chi-square Analyses of Level of Matching Funds by Organizational Subsystem __________________________________________________________________ Level of Matching Funds Up to $100,000 Over $100,000 Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Routine Level of Matching Funds Production .58 .45 Minimum-Moderate 5 (38) 8 (62) High 34 (50) 34 (50) Managerial .28 .60 Minimum-Moderate 10 (43) 13 (57) High 29 (50) 29 (50) Maintenance .00 1.00 Minimum-Moderate 13 (48) 14 (52) High 26 (48) 28 (52) Support 2.69 .10 Minimum-Moderate 18 (40) 27 (60) High 21 (58) 15 (42) 82 Table 14. Chi-square Analyses of Level of Matching Funds by Organizational Subsystem (Continued) __________________________________________________________________ Level of Matching Funds Up to $100,000 Over $100,000 Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Niche Saturation Level of Matching Funds Production 1.04 .31 Minimum-Moderate 30 (45) 36 (55) High 9 (60) 6 (40) Managerial .56 .46 Minimum-Moderate 19 (53) 17 (47) High 20 (44) 25 (56) Maintenance .01 .93 Minimum-Moderate 21 (48) 23 (52) High 8 (49) 19 (51) Support .36 .55 Minimum-Moderate 16 (44) 20 (56) High 23 (51) 22 (27) __________________________________________________________________ df = 1 83 Table 15. Chi-square Analyses of Hard Money Versus In-kind Funding by Organizational Subsystem Hard Money Versus In-kind Funding Hard Money In-kind Funding Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Routine Hard Money Versus In-kind Funding Production .11 .74 Minimum-Moderate 3 (23) 10 (77) High 13 (19) 55 (81) Managerial .11 .74 Minimum-Moderate 4 (17) 19 (83) High 12 (21) 46 (79) Maintenance .16 .69 Minimum-Moderate 6 (22) 21 (78) High 10 (18) 44 (82) Support 1.13 .29 Minimum-Moderate 7 (16) 38 (84) High 9 (25) 27 (75) 84 Table 15. Chi-square Analyses of Hard Money Versus In-kind Funding by Organizational Subsystem (Continued) Hard Money Versus In-kind Funding Hard Money In-kind Funding Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Niche Saturation Hard Money Versus In-kind Funding Production .00 .98 Minimum-Moderate 13 (20) 53 (80) High 3 (20) 12 (80) Managerial .25 .62 Minimum-Moderate 8 (22) 28 (78) High 8 (18) 37 (82) Maintenance .03 .86 Minimum-Moderate 9 (20) 35 (80) High 7 (19) 30 (81) Support .39 .53 Minimum-Moderate 6 (17) 30 (83) High 10 (22) 35 (78) __________________________________________________________________ df = 1 85 Table16. Chi-square Analyses of Project Director Discipline by Organizational Subsystem _________________________________________________________________ Project Director Discipline Medicine All Others Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Routine Project Director Discipline Production .92 .34 Minimal-Moderate 8 (62) 35 (38) High 32 (47) 36 (53) Managerial .03 .86 Minimal-Moderate 11 (48) 12 (52) High 29 (50) 29 (50) Maintenance 1.21 .27 Minimal-Moderate 11 (41) 16 (59) High 29 (54) 25 (46) Support .63 .43 Minimal-Moderate 24 (53) 21 (47) High 16 (44) 20 (56) 86 Table 16. Chi-square Analyses of Project Director Discipline by Organizational Subsystem (Continued) __________________________________________________________________ Project Director Discipline Medicine All Others Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Niche Saturation Project Director Discipline Production 3.80 .05 Minimal-Moderate 36 (55) 30 (45) High 4 (27) 11 (73) Managerial .30 .59 Minimal-Moderate 19 (53) 17 (47) High 21 (47) 24 (53) Maintenance 1.03 .31 Minimal-Moderate 24 (55) 20 (45) High 16 (43) 21 (57) Support .30 .59 Minimal-Moderate 19 (53) 17 (47) High 21 (47) 24 (53) __________________________________________________________________ df = 1 87 Table 17. Chi-square Analyses of Project Director Education by Organizational Subsystem __________________________________________________________________ Project Director Education  8 years  9 years Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Routine Project Director Education Production .05 .83 Minimal-Moderate 5 (38) 8 (62) High 24 (35) 44 (65) Managerial .02 .90 Minimal-Moderate 8 (35) 15 (65) High 21 (36) 37 (64) Maintenance .03 .87 Minimal-Moderate 10 (37) 17 (63) High 19 (35) 35 (65) Support 2.11 .15 Minimal-Moderate 13 (29) 32 (71) High 16 (44) 20 (56) 88 Table 17. Chi-square Analyses of Project Director Education by Organizational Subsystem (Continued) __________________________________________________________________ Project Director Education  8 years  9 years Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Niche Saturation Project Director Education Production 2.46 .12 Minimal-Moderate 21 (32) 45 (68) High 8 (53) 7 (47) Managerial .00 .96 Minimal-Moderate 13 (36) 23 (64) High 16 (36) 29 (64) Maintenance .66 .42 Minimal-Moderate 14 (32) 30 (68) High 15 (40) 22 (60) Support .78 .38 Minimal-Moderate 11 (31) 25 (69) High 18 (40) 27 (60) _________________________________________________________________ df = 1 89 Table 18. Chi-square Analyses of Project Director Turnover by Organizational Subsystem __________________________________________________________________ Project Director Turnover 1 PD 2 PDs 3 PDs Organizational Subsystem N (%) N (%) N (%) ?  __________________________________________________________________ Routine Project Director Turnover Production .56 .76 Minimum-Moderate 2 (15) 3 (23) 8 (62) High 17 (25) 14 (21) 37 (54) Managerial 2.82 .24 Mimimum-Moderate 3 (13) 4 (17) 16 (70) High 16 (28) 13 (22) 29 (50) Maintenance 4.33 .12 Minimum-Moderate 4 (15) 9 (33) 14 (52) High 15 (28) 8 (15) 31 (57) Support 5.81 .06 Mimimum-Moderate 6 (13) 11 (24) 28 (62) High 13 (36) 6 (17) 17 (47) 90 Table 18. Chi-square Analyses of Project Director Turnover by Organizational Subsystem (Continued) Project Director Turnover 1 PD 2 PDs 3 PDs Organizational Subsystem N (%) N (%) N (%) ?  __________________________________________________________________ Niche Saturation Project Director Turnover Production 2.96 .23 Minimum-Moderate 13 (20) 14 (21) 39 (59) High 6 (40) 3 (20) 6 (40) Managerial 1.68 .43 Minimum-Moderate 6 17) 8 (22) 22 (61) High 13 (29) 9 (20) 23 (51) Maintenance .06 .97 Minimum-Moderate 10 (23) 9 (20) 25 (57) High 9 (24) 8 (22) 20 (54) Support 1.68 .43 Minimum-Moderate 6 (17) 8 (22) 22 (61) High 13 (29) 9 (20) 23 (51) __________________________________________________________________ df = 2 91 Table 19. Chi-square Analyses of Evaluation by Organizational Subsystem Evaluation Weak Years 1-3 Outcome Year 5 Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Routine Evaluation Production 1.04 .31 Minimum-Moderate 13 (100) 0 (0) High 62 (93) 5 (7) Managerial .33 .57 Minimum-Moderate 21 (91) 2 (9) High 54 (95) 3 (5) Maintenance .09 .76 Minimum-Moderate 25 (93) 2 (7) High 50 (94) 3 (6) Support 1.35 .25 Minimum-Moderate 40 (91) 4 (9) High 35 (97) 1 (3) 92 Table 19. Chi-square Analyses of Evaluation by Organizational Subsystem (Continued) Evaluation Weak Years 1-3 Outcome Year 5 Organizational Subsystem N (%) N (%) ?  __________________________________________________________________ Niche Saturation Evaluation Production .01 .94 Minimum-Moderate 61 (94) 4 (6) High 14 (93) 1 (7) Managerial 1.35 .25 Minimum-Moderate 35 (97) 1 (3) High 40 (91) 4 (9) Maintenance .49 .49 Minimum-Moderate 42 (96) 2 (4) High 33 (92) 3 (8) Support .49 .49 Minimum-Moderate 33 (92) 3 (8) High 42 (96) 2 (4) __________________________________________________________________ df = 1 93 hypothesis five was rejected. This meant that if the PD discipline was pediatrician, all program activities were less likely to be written and operationalized. Hypotheses six, seven, and eight stated there would be no significant differences in sustainability among HTPCP projects in PD education levels and turnover rates as well as evaluation levels. There were no significant differences determined for hypotheses six through eight; each of these hypotheses was supported (Tables 17 - 19). Sustainability was examined for each routine sub-system (production, managerial, maintenance, and support) and each niche saturation sub-system (production, managerial, maintenance, and support) across each dependent variable. Each dependent variable (PAC, pediatric involvement, level of matching funds, type of funding, PD discipline, PD educational level, PD turnover, and evaluation) was one out of eight variables examined for sustainability in four routine sub-systems and in four niche saturation sub- systems. Multiple regression was performed (using interval data) to analyze hypothesis nine using the forward regression procedure (Table 13 ? Appendix L). Hypothesis nine stated that matching funds would be the strongest predictor of sustainability when entered with all other predictor variables. Results of the multiple regression analyses did not demonstrate a predictive sustainability model as proposed; there was little to no variability in the independent variables as the 94 numbers were small in the various categories. However, the results revealed t-test values with significant differences in four sustainability subscales. These significant differences were demonstrated among HTPCP projects in both the routine and niche saturation production PD discipline and managerial PD turnover subscales. The significant difference in the routine and niche saturation production PD discipline subscale demonstrated that if the PD discipline was pediatrician that the program activities were less likely to have been written and operationalized. The routine and niche saturation managerial PD turnover subscale significant difference indicated that if a HTPCP project had three or more PDs in five years, this did not affect the extent of formal supervision, staff have written job descriptions, and program evaluation that occurred in the project. Summary This chapter described the study population characteristics and results of the hypotheses testing. The demographics were collected on each of the PD participants who completed the questionnaire and included age, gender, formal education after high school, discipline, job title, length of time in position, and years with the HTPCP. The results of the data analyses demonstrated a significant finding in the niche saturation production PD discipline subscale. This meant that if the PD discipline was medicine, the program activities were less likely to be written and implemented. Therefore, hypothesis five was rejected. There were 95 no significant differences revealed for hypotheses one, two, three, four, six, seven, and eight; each of these hypotheses was supported. The specific variables identified as predictive of sustainability (e.g. matching funds) did not hold up under the multiple regression analyses. 96 CHAPTER V: DISCUSSION, RECOMMENDATIONS, AND CONCLUSIONS Introduction This chapter includes a summary of results, discussion of findings, limitations, recommendations for future research, recommendations for health education practice, and conclusions. The purpose of this study was to determine sustainability and the main reason(s) for sustainability among the eighty-nine HTPCP projects. Eight null hypotheses were developed to determine whether the programmatic funding criteria (independent variables) for the HTPCP were the reason(s) for sustainability. Each of these eight independent variables (PAC, pediatric involvement, level of matching funds, type of matching funds, PD discipline, PD level of education, PD turnover, and evaluation) were used with the eight LoIn Instrument sustainability subscales (dependent variables) to determine sustainability of the HTPCP. The ninth hypothesis was proposed with the intent of developing a predictive sustainability model. There were 94 projects that received federal funding from 1989 through 1997 and completed their five-year federal project period by 2002. The researcher conducted a pilot-test with five of the 94 HTPCP project directors (PDs) eligible to participate in this study. 97 Following the pilot test, the LoIn Instrument was mailed to the remaining 89 project directors; 81 responded to the survey questionnaire (91%). The majority of respondents were female (80%) and had up to eight or more years of formal education after high school (59%). The mean age of respondents was 50 years of age (SD = 8.8). Chi-square analyses were conducted to answer hypotheses one through eight. A forward procedure for multiple regression was conducted for hypothesis nine. Hypothesis five was rejected. The sustainability predictor model could not be developed. Discussion Sustainability evaluation of preventive health care programs should be conducted for two reasons. First, sustainability needs to be measured because until this evaluation is conducted, there is no verification or validation that sustainability is occurring. Second, building and maintaining infrastructure in preventive health programs supports sustainability in these programs. Altman (1995) actually defines sustainability as the infrastructure that remains in a community after a research project ends. Although a great deal of federal funding is spent each year for infrastructure building, the federal government has no real proof of sustainability from these annual investments, yet it continues to fund these same programs and others like them. 98 The HTPCP projects examined in this study demonstrated sustainability. Statistical means from the organizational sub-systems (routine production, managerial, maintenance, and support; niche saturation production, managerial, maintenance, and support) ranged from 2.80 to 3.54 on a 4-point scale (4 = high level of sustainability), supporting the conclusion that some of the criteria MCHB/HRSA required for the original HTPCP grant applications are important in terms of sustainability. This conclusion is substantiated by the finding that most of these programs are still in existence in 2004, one to eight years after completion of their five-year federal funding. Ninety-one percent of the HTPCP projects were continuing their program at the time of this study. Based on data analyses, the high rate of sustainability of these projects was due to the infrastructure that was built with these community- based grants. The majority of the MCHB program requirements for the HTPCP projects laid a foundation for infrastructure and continued throughout (and beyond) the five-year life of the grant. For example, written goals and objectives and an implementation plan were basic requirements for these projects. The matching funds requirements also contributed to these small projects having had a written financial plan. It is common for HTPCP projects to use this small amount of federal funds for leveraging other funding streams. In addition, PAC members represented community organizations and their individual ?buy-in? to the scope of work as well as the financial plan for these projects influenced others in the community. 99 Five of the eight variables examined were ones the researcher believed may have had an impact on sustainability, although little to no information about their effects on sustainability was found in the literature. These five independent variables were: 1) matching fund levels; 2) type of matching funds; 3) PD discipline; 4) PD educational level; and 5) PD turnover. A significant difference found in this study was in the niche saturation production PD discipline subscale which meant that hypothesis five was rejected. Niche saturation production represented the extent to which all possible program activities had written documentation in place and had been implemented. This finding demonstrated that if the PD discipline was medicine, program activities and operationalization procedures were less likely to be written. It is further suggested that physicians do not personally take care of written documentation and that this task is left to other program staff. The discipline from which the PD comes does make a difference to sustainability of the program in terms of whether or not program activities are actually written down and operationalized. However, if there is a preponderance of other variables/HTPCP requirements present (i.e. matching funds, PAC, pediatric involvement, PD education level, PD turnover, etc.), having a physician as the PD in and of itself will not cause a project to end. The one significant finding in this study, which supported the rejection of hypothesis five, was consistent with both the theory (Kaluzny, Warner, Warren, and Zelman, 1982) used in this study and the experiences of Goodman and his 100 colleagues (1993). Their findings indicated that health promotion programs usually became institutionalized in the following order of organizational subsystems: production, managerial, maintenance, and support (Goodman, et al., 1993; Kaluzny, et al., 1982). This study?s one significant difference was in the niche saturation production subscale/subsystem. Again, this finding meant that if the PD discipline was pediatrician, program activities were less likely to have been written and operationalized. The question arises as to why hypothesis five was the only one out of the first eight hypotheses that was rejected. Why weren?t there more significant differences found? One reason may be that the LoIn Instrument was developed to measure sustainability and not the reason(s) for sustainability. The LoIn Instrument actually did measure sustainability of the HTPCP projects. In addition, all of the HTPCP program requirements used as independent variables in this study may not have been correct requirements (reasons) that this program was sustained. Although it seemed that there would have been a significant difference in sustainability and the level of program evaluation conducted, it was not a finding in this study. However, only 7% of the projects demonstrated outcome evaluations and it seemed that process evaluations (93%) were important to sustainability as the projects were sustained. Another potential reason why more significant differences were not found in this study was the lack of variance in the independent variables. Most of these variables were categorized into three groups (minimum, moderate, high) but most 101 espondents fell into one or two categories, making statistical comparisons difficult. In future studies, if larger sample sizes are obtained, it is possible there will be more variability across response categories. Other independent variables (program requirements for the HTPCP) such as AAP program support could have been used. A multiple regression was performed to analyze hypothesis nine. Results of the multiple regression analyses did not demonstrate a predictive sustainability model as proposed. Thus, hypothesis nine was also rejected. However, the results revealed t-test values with significant differences among HTPCP projects in both the routine and niche saturation production PD discipline and managerial PD turnover subscales. Again, the routine and niche saturation production PD discipline findings indicated that if the PD discipline was medicine, the program activities and implementation were less likely to be written and operationalized. The routine and niche saturation managerial PD turnover findings demonstrated that program supervision, written job descriptions for staff, and program evaluation were not greatly affected by PD turnover. In other words, if a HTPCP project had one PD or three or more PDs over the five year project period, program supervision, written job descriptions for staff, and program evaluation were still likely to occur. In fact, the HTPCP projects were more likely to have high institutionalization if PD turnover was high. This was an unexpected finding. It may be that the high institutionalization of these projects with a high PD turnover rate is due to the infrastructure that has been built and maintained 102 because of the individual projects continuing to meet the HTPCP program requirements. The PD turnover rate was quite high; 56% of the projects had three or more PDs during the five-years of federal funding. The majority of this PD turnover was likely to have occurred within the pediatrician PD discipline. The high turnover rate among pediatrician PDs was far more than in all other disciplines combined (34% versus 22%). The successor of a pediatrician PD is more often another pediatrician. Again, it appears that the sustainability or success of the projects is correlated more closely with how well the grant and program requirements are met as opposed to the actual number of PDs in a five-year project period. Although pediatricians currently hold approximately one-third (35%) of the PD positions, their participation as PDs decreased by 14% since the five-year funding cycle was completed by these 89 HTPCP projects. At the end of the five years, 49% (data taken from project records by the researcher) of the PDs were pediatricians. Eisen, et al. (1999) found that 47% of PDs who served in the HTPCP were pediatricians. Based on the decline in the number of pediatrician PDs in these projects and the sustainability of the HTPCP projects one to eight years after completion of the federal funding, a pediatrician PD did not greatly affect the sustainability of these projects. In addition, the sustainability or success of these projects seems to be correlated with how well the PD or project leadership met all grant and program requirements. 103 Ninety-three percent of the HTPCP evaluations were process evaluations of program effectiveness evaluations completed by the third year of federal funding. Seven percent of the HTPCP projects completed an outcome evaluation by the end of their fifth year of federal funding. This number was too small to allow any determination regarding whether the projects with an outcome evaluation demonstrated more sustainability than those with only process evaluations. An outcome evaluation is a requirement for HTPCP projects in the initial application. However, the majority of the projects that are approved for funding by the ORC reviewers and funded actually have only a process evaluation component. This occurs because the ORC approves the best overall applications out of the total number submitted and reviewed and only a few of all applications have outcome evaluation components at the onset. Consequently, the ORC makes funding recommendations to MCHB which consists of an approved project list; MCHB staff prioritize the list of projects and both of these tasks are accomplished without regard to any one funding requirement, except program conditions are stipulated by requirements not being fully met in the application. For example, if the ORC identified that a potential grantee had only a process evaluation described in their grant, and if this applicant is approved by the ORC, the ORC should automatically develop the condition (i.e. outcome evaluation component must be developed/strengthened) in preparation for funding. 104 Limitations  Self-administration of the survey questionnaire was a limitation. Dependence upon self-administration of the survey questionnaires and self- reported data potentially biased the data collected from the questionnaires. Respondents may have given more subjective responses to the questions than they might have given had the questions been asked by an interviewer. Also, the objectivity of the responses by participants may have been further influenced by the length of time between the time the individual HTPCP project was being reported on for the study and when its five-year funding period had been completed. For example, the number of years after completion of federal funding for these projects ranged from one to eight years and the recall of a respondent is likely to have been better in year one than in year eight. In addition, some of the PDs who originally worked with HTPCP projects had left and other PDs had taken her/his place sometime after the five-year federal funding was completed. The PDs who were relatively new to the HTPCP projects did not have the recall of her/his PD predecessor and may not have been as committed to responding to the questionnaires as she/he had not been involved with the project from the beginning.  There was little to no variance among the independent variables. The lack of variation across the individual independent variable categories meant there was an insufficient number of responses in the different cells, resulting in an inability to meet the assumptions of the statistical test. Most respondents fell into one or 105 two of the three available categories. For example, 80% of the HTPCP projects had a majority of in-kind funding while 20% had a majority of hard dollar funding. Recommendations for Further Research  This study should be replicated. The current study included the total population of project directors with a HTPCP project that received federal funding for five years from October 1, 1989 through September 31, 1997. Another study could be conducted which would include projects funded from 1998 through 2005. With a larger number of projects, there may be an increased number of respondents in each cell and thus, an increase in variability among independent variables.  This study should be repeated with a change in some of the independent variables. While levels of matching funds needs to be repeated in any follow-up study, some other variables should change. For example, the AAP-MCHB/HRSA partnership and the consultation and technical assistance grantees receive from one or both partners could be an independent variable(s) in a repeat study.  Sustainability evaluation research should be promoted across MCHB and HRSA, with both discretionary and legislatively mandated effective preventive health care programs. A responsibility of HRSA and the MCHB is to award federal grant funds for preventive health care programs to build infrastructure and sustain programs after the federal funds are gone. However, infrastructure and 106 sustainability are not currently being evaluated. The reason for this is that such evaluations are not a priority for funding and Congress wants to see new projects funded as opposed to evaluating ongoing programs already funded by HRSA and MCHB. For example, evaluation of sustainability is a requirement in Healthy Start (HS) programs. In fact, when HS was funded initially, there was $5,000,000 set aside to pay for program evaluation over five years which indicated that HS was handled differently (both in its evaluation prioritization and the amount of funding) in this aspect than other MCHB grants. While sustainability is mentioned in HS evaluations, it is included as a descriptive research evaluation component.  The federally funded community and migrant health centers (C/MHC) program, located in the Bureau of Primary Health Care (BPHC), should evaluate sustainability in effective primary care programs. The main measurement of sustainability occurring in 2004 is the monitoring of federal funds drawdown which is usually equated with financial viability. Once again, the main intent of these federal funds going to a grantee is to build infrastructure and promote the capability of the C/MHC to become self-supporting and maintain financial viability without an increase in federal funds or a request for supplemental funds.  The Centers for Disease Control and Prevention (CDC) should measure sustainability in numerous effective preventive health care programs, nationally and internationally. For example, CDC funds broad community, state, and international preventive health care programs in areas such as child health, 107 immunizations, breast and cervical cancer, prenatal care, etc., many of which focus on building infrastructure to maintain sustainability. Therefore, programmatic measurement of sustainability would provide information about whether a program was sustained. Once the most efficacious programs have been established, proven sustainability constructs can be applied to ensure such programs are maintained. Recommendations for Health Education Practice  Clearly there is a need for health educators to conduct research to evaluate sustainability, develop a predictive sustainability model, and contribute to the literature in this area. Sustainability of effective health promotion programs is important. Further understanding of the components that contribute to sustainability is worthwhile and could contribute to improved health outcomes related to whatever the particular program was addressing. Health educators, located in administrative, service, and academic programs as well as clinical practice, are in ideal positions and environments to conduct and participate in this research and application. Health educators are also in the position to stimulate this research among others. For example, the LoIn Instrument was developed by health education researchers and has been shared and used by clinicians, educators, academicians, epidemiologists, evaluators, and researchers in the broad areas of community and ambulatory preventive (medical) health clinics and public health practice. The impetus for this study was the Goodman et al. (1993) article. 108 Conclusions Based on the data analyses, several conclusions can be drawn. The HTPCP projects examined in this study demonstrated sustainability. It looks as if the program criteria/requirements set up by HRSA-MCHB/AAP are working as far as maintaining the programs over time is concerned. Because of methodological limitations, the study did not identify the variables most important to sustainability. The only statistically significant variable that was identified in this study was the niche saturation production PD discipline subscale. Niche saturation of program production represents the extent to which all program activities had written documentation and had been implemented. In this study, HTPCP projects were less likely to have written program activities and to have been implemented if the PD was a pediatrician. There was no relationship shown between sustainability and matching funds, as well as the extent of overmatch as there were no significant differences shown in this independent variable in the study. The HTPCP projects have developed infrastructure that supports sustainability based on the findings from this study. However, the proposed predictive sustainability evaluation model could not be developed. Further research is recommended to evaluate sustainability across MCHB and HRSA preventive health care programs, as well as those funded by CDC. The intent of these programs is to build infrastructure which maintains sustainability. However, sustainability is not currently being measured and the 109 results of such evaluations would yield invaluable information for organizational policy and decision makers. For example, sustainability, infrastructure building and maintenance had not been measured in the HTPCP program prior to this study. The MCHB Strategic Plan : FY 2003 ? 2007 includes a performance measure which specifies: ?The percent of MCHB supported projects that are sustained in the community after the federal grant project period is completed. 2007 Target: 50%.? This is a new indicator and there are no baseline data. The primary organizational policy and decision makers for this program consist of AAP/MCHB-HRSA leaders, with ongoing congressional input. The HTPCP is already in its seventeenth funding year and the annual funding level for each project is $50,000. It seems unlikely that t he AAP/MCHB-HRSA policy and decision makers will be able to find sustainability, infrastructure building and maintenance demonstrated in preventive health programs for the Nation?s mothers and children anywhere else other than the HTPCP at this price. The findings of this study will be shared with the MCHB staff and hopefully used by AAP/MCHB-HRSA policy and decision makers to support the continuation of future funding for the HTPCP. In addition, it is hoped these findings will be used as baseline data for the sustainability performance measure in the MCHB Strategic Plan. This is the very best outcome possible for this study. According to Basch and Gold (1986, p. 303), ?..regardless of what type of research approaches are applied and what conclusions are drawn based on the 110 results of the evaluation, the evaluation effort will be inconsequential if it is not used by decision-makers.? 111 APPENDIX A: HTPCP Requirements/Criteria (Page 1 of 4) LIST OF CRITERION SETS I. CRITERIA BASED ON INVOLVEMENT OF STATE MCH/CSHN AGENCIES 1. Contains sufficient information to document collaboration with the State MCH/CSHN agency, or there is sufficient documentation of attempts to solicit their involvement. (See p. 7 and p. 21, applicants' guidance.) II. CRITERIA BASED ON SPECIFIC REQUIREMENTS INDICATED IN EACH FUNDING PRIORITY 1. Definition of the health problem. (See pp. 25-27, applicants?guidance.) 2. Contributing factors for the specific health problem that is the focus of this application. (See pp. 27-28, applicants? guidance.) 3. Project goals. (See p. 28, applicants? guidance.) 4. Outcome objectives as positive changes to be achieved for each identified health problem. (See p. 29, applicants? guidance.) 5. Methodology or approach for bringing about improvements in specific health problems. (See pp. 29-31, applicants? guidance.) 6. Tracking of proposed project activities. (See p. 31, applicants? guidance.) 7. Monitoring of outcome objectives and evaluation of changes in specific health problems. (See pp. 32-33, applicants? guidance.) 8. Use of project information. (See p. 33, applicants? guidance.) 112 (Page 2 of 4) 9. Capabilities of the applicant. (See pp. 34-35, applicants? guidance.) 10. Budget and justification. (See pp. 35-36, applicants? guidance.) III. ELEMENTS BASED ON THE SPECIFIC PROGRAM REQUIREMENTS Project goals, objectives, and methodology meet the project requirements. (See pp. 28-31, applicants? guidance for program requirements.) Each project supported by the Healthy Tomorrows Partnership for Children Program must: o represent a local initiative that is community-based, family- centered, comprehensive and culturally relevant. o employ an innovative approach to improve access to health services by making health care arrangements more effective and affordable. o improve the health status, functional ability and developmental capability of infants, children, adolescents, or children with special health care needs. o represent a new initiative or a new component of an existing activity that will build upon, expand and enhance the family and community ability to meet the needs of its children. o provide evidence of a capability to meet cost participation ?matching? requirements by securing funds required for the second and sequential years in an amount not less than 66.7 percent of the total budget. (Within this document the term ?matching? will be used interchangeably with the term ?cost participation.?) o analyze both the existing health status conditions and the health services currently available for children and their families in its particular community. 113 (Page 3 of 4) o define clearly a high priority child health or health systems problem(s) or opportunity within the target area to be addressed by the project. o define clearly goals and measurable objectives that address the identified problem, and relate each project activity to a specific objective. o develop an effective plan to meet objectives through the application of existing knowledge and mobilization of local resources. o provide evidence that Bright Futures: Guidelines for Health Supervision of Infants,Children, and Adolescents are implemented as appropriate. o provide evidence that clinical services will be the responsibility of a qualified health professional, i.e., pediatrician, family practitioner, obstetrician, nurse, etc. o develop an evaluation component that utilizes the program information system(s) for measuring changes in child health status and monitoring progress in meeting the objectives and achieving the goals. o agree to participate in an independent evaluation by the HTPCP or a designee. o be initiated by a local group, public or private, that is recognized locally as most suited to this role. Each project should demonstrate evidence of agreement within the community that the applicant is recognized as the appropriate entity to assume leadership for initiating and managing the project. Priority consideration will be given to those proposals that come from organizations that have a record of involvement with child and adolescent health issues and the local pediatric provider community. 114 (Page 4 of 4) o demonstrate an ability to identify sources and amounts of community, foundation and State funds and in-kind contributions expected to support continuation of the program once the grant period has ended. A final plan for continuity of funding must be submitted with the third year project continuation application report. o provide evidence of a system of planning and governance involving the participation of a community and wide coalition of pertinent groups including parents, and representatives from the local pediatric provider community and parents. IV. ELEMENTS BASED ON THE OVERALL SIGNIFICANCE OF THE PROJECT In its entirety, the proposal is of regional or national significance. 115 APPENDIX B: LoIn Instrument LEVEL OF INSTITUTIONALIZATION (LOIN) SCALES FOR HEALTH PROMOTION PROGRAMS From Goodman, R.M., McLeroy, K.R, Steckler, A., & Hoyle, R.H. (1993). ?Development of Level of Institutionalization (LoIn) Scales for Health Promotion Programs.? Health Education Quarterly, 20 (2), 161-178. 116 PRODUCTION SUBSYSTEM 1a. Have the program?s goals and/or objectives been put into writing? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 2 1b. If yes, for how many years have written goals & objectives actually been followed? ______Year(s)   1c. Of all the aspects of this program that could have written goals and objectives, what is your best estimate of the proportion which actually have written goals and objectives? No aspects of Few aspects of Most aspects of All aspects of this program this program this program this program have written have written have written have written goals & goals & goals & goals & objectives objectives objectives objectives 1 2 3 4 2a. Have any of the plans or procedures used for implementing this program been put in writing? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 3 2b. If yes, for how many years have such written plans or procedures actually been followed? ______Year(s)   2c. Of all the aspects of this program that could have written plans or procedures, what is your best estimate of the proportion which actually have written plans or procedures? No aspects of Few aspects of Most aspects of All aspects of the program the program the program the program have written have written have written have written plans or plans or plans or plans or procedures. procedures. procedures. procedures. 1 2 3 4 117 3a. Has a schedule (e.g., timetable, plan of action) used for implementing program activities been put in writing? (1)______Yes (2)______No (3)______Not sure/not applicable   Go to Question 4 3b. If yes, for how many year have such written schedules actually been followed? ______Year(s)   3c. Of all the aspects of this program that could have written schedules, what is your best estimate of the proportion which actually have written schedules? No aspects of Few aspects of Most aspects of All aspects of this program this program this program this program have written have written have written have written schedules. schedules. schedules. schedules. 1 2 3 4 4a. Have the strategies for implementing this program been adapted to fit local circumstances? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 5 4b. If yes, for how many years have locally adapted strategies actually been followed? ______Year(s)   4c. Of all the aspects of this program that could be adapted to fit local circumstances, what is your best estimate of the proportion which have actually been adapted? No aspects of Few aspects of Most aspects of All aspects of this program this program this program this program have been have been have been have been adapted. adapted. adapted. adapted. 1 2 3 4 118 5a. Has a formal evaluation of the program been conducted? (1)_____Yes (2)______No (3) Not sure/not applicable   Go to Question 6 5b. If yes, for how many times has the program been formally evaluated? ______Years(s)   5c. Of all the aspects of this program that could be formally evaluated, what is your best estimate of the proportion which have been formally evaluated. No aspects of Few aspects of Most aspects of All aspects of this program this program this program this program have been have been have been have been evaluated. evaluated. evaluated. evaluated. 1 2 3 4 MANAGERIAL SUBSYSTEM 6a. Has a supervisor (e.g., section chief, department head) been formally assigned to oversee this program? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 7 6b. If yes, for how many years has such a supervisor actually been formally assigned to oversee the program? ______Year(s)   6c. Of all the aspects of this program that could receive supervision, what is your best estimate of the proportion which actually receives such supervision? No aspects of Few aspects of Most aspects of All aspects of this program this program this program this program receive receive receive receive supervision. supervision. supervision. supervision. 1 2 3 4 119 7a. Have formalized job descriptions been written for staff involved with this program? (1)_____Yes (2)______No (3) Not sure/not applicable   Go to Question 8 7b. If yes, for how many years have formalized job descriptions actually been followed? ______Year(s)   7c. What is your best estimate of the number of staff involved with this program who have written job descriptions? None of the staff Few of the staff Most of the staff All of the staff involved with involved with involved with involved with this program this program this program this program have written have written have written have written job descriptions. job descriptions. job descriptions. job descriptions. 1 2 3 4 8a. Are evaluation reports of this program done on a schedule similar to evaluation reports for most other programs in your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 9 8b. If yes, for how many years have evaluation reports actually been produced on a schedule similar to such reports for most other programs in your organization? _____Year(s)   8c. What is your best estimate of the extent that evaluation reports for this program are produced on a schedule similar to evaluation reports for most other programs in your organization? No evaluation Few evaluation Most evaluation All evaluation reports are reports are reports are reports are produced on a produced on a produced on a produced on a similar schedule. similar schedule. similar schedule. similar schedule. 1 2 3 4 120 MAINTENANCE SUBSYSTEM 9a. Have any permanent staff been assigned to implement this program? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 10 9b. If yes, for how many years have permanent staff been assigned to implement the program? ______Year(s)   9c. What is your best estimate of the number of staff who implement the program that are in permanent positions? No staff Few Staff Most staff All staff involved are involved are involved are involved are in permanent in permanent in permanent in permanent positions. positions. positions. positions. 1 2 3 4 10a. Has an administrative-level individual within your organization been actively involved in advocating for this program?s continuation? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 11 10b. If yes, for how many years has this administrative-level individual actively advocated for this program?s continuation? ______Year(s)   10c. What is your best estimate of how active this administrative-level individual has been in advocating for the program?s continuation? Not active Minimally Moderately Very at all active active active 1 2 3 4 121 11a. Do staff in your organization, other than those actually implementing the program, actively contribute to the program?s operations? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 12 11b. If yes, for how many years have such staff in your organization actively contributed to the program?s operation? ______Year(s)   11c. Of all the staff in your organization who could contribute to the operation of this program, what is your best estimate of the proportion that actually contribute to it? None of the staff Few of the staff Most of the staff All of the staff contribute to the contribute to the contribute to the contribute to the program?s program?s program?s program?s operation. operation. operation. operation. 1 2 3 4 SUPPORTIVE SUBSYSTEM 12a. Has the program made a transition from trial or pilot status to permanent status in your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 13 12b. If yes, for how many years has this program had permanent status? ______Year(s)   12c. What is your best estimate of how permanent this program is in your organization? Not permanent Minimally Moderately Very at all permanent permanent permanent 1 2 3 4 122 13a. Has the program been assigned permanent physical space within your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 14 13b. If yes, for how many years has it maintained such permanent space? ______Year(s)   13c. Of all the permanent space that this program needs, what is your best estimate of the proportion of permanent space it currently occupies? This program This program This program This program does not occupy occupies only occupies most occupies all any permanent a small amount of the permanent of the permanent space. of the permanent space that it needs. space that it needs. space that it needs. 1 2 3 4 14a. Is this program?s source of funding similar to the funding sources for other established programs within your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 15 14b. If yes, for how many years has this program?s funding sources been similar to those for other established programs within your organization? ______Year(s)   14c. In your best estimate, how permanent is the program?s source of funding? Not permanent Minimally Moderately Very at all permanent permanent permanent 1 2 3 4 123 15a. Is the staff most closely associated with this program?s implementation hired from a stable funding source? (1)_____Yes (2)______No (3)______Not sure/not applicable  15b. If yes, for how many years has the staff most closely associated with this program?s implementation been hired from a stable funding source? ______Year(s)   15c. What is your best estimate of how permanent the funding is for the staff most closely associated with this program?s implementation? Not permanent Minimally Moderately Very at all permanent permanent permanent 1 2 3 4 124 Definitions for Organizational Subsystems and Degrees of Program Penetration Subsystems* Production: Concerned with ?throughput,? or those activities which are product directed. Managerial Concerned with coordinating the operations of the other subsystems. Maintenance: Concerned with personnel issues and continuity of production in areas such as recruitment, indoctrination or socialization, rewarding, sanctioning and procurement of resources. Supportive: Concerned with hospitable environmental conditions by establishing legitimacy and favorable organizational relationships. Degrees+ Passages: The first degree of program institutionalization which is signified by one-time sentinel events such as the formalization of program plans, the shift from soft to hard sources of funding, and the program?s inclusion on the organizational chart. Routines: The second degree of program institutionalization which is signified by the habituation, or routinization of program passages, such as the continued inclusion of the program in the organization?s formal plans, annual renewal of stable funding, and continued inclusion of the program in new versions of the organizational chart. Niche The third degree of program institutionalization which is saturation: signified by the maximum feasible expansion of the program within the host organizations?s subsystems, such as the optimum realization of the programs plans, the achievement of optimum levels of funding, and the inclusion of the program in a core (versus peripheral) location on the organizational chart. *The definitions for the subsystems are adapted from Katz and Kahn (1978). +The definitions for the degrees are adapted from Yin (1979). 125 Scoring the LoIn Scale The grid on the next page can be used to score the LoIn Scale in conjunction with the following directions: Each question has three sub-questions (a,b, and c). Sub-questions ?a? and ?b? are scored together, resulting in one score for the two sub-items, and sub-question ?c? forms are scored separately. For all ?a? and ?b? sub-questions, score as follows:  If you checked ?No? or ?Not sure/not applicable? for ?a?, then the score for the sub-item = 0;  If you checked ?Yes? for ?a? and wrote ?0? or ?1? for ?b?, then the score for the sub-item = 1;  If you checked ?Yes? for ?a? and wrote ?2? or ?3? for ?b?, then the score for the sub-item = 2;  If you checked ?Yes? for ?a? and wrote ?4? or ?5? for ?b?, then the score for the sub-item = 3;  if you checked ?Yes? for ?a? and wrote ?6? or more for ?b?, then the score for the sub-item = 4. For all ?c? sub-questions, score them as the number that you circled for that item (e.g., if you circled a ?2? then the score for that item = 2). Each three-part item represents one of the following organizational sub-systems: production (items 1-5), managerial (items 6-8), maintenance (items 9-11), supportive (items 12-15). Using the grid on the next page, add the scores for all sub-items ?a? and ?b? as indicated and divide by the number listed on the grid. Follow the same procedure for all ?c? sub-items. For sub-items ?a? and ?b?:  If the mean score is ?1? or ?2? then institutionalization is low to moderate;  If the mean score is ?3? then institutionalization is moderate to high;  If the mean score ?4? then institutionalization is high. For sub-items ?c?:  If the mean score is less than or equal to ?2? then insitutionalization is low;  If the mean score is greater than ?2? but less than or equal to ?3? then institutionalization is moderate;  If the mean score is greater than ?3? then institutionalization is high. 126 SCORE SHEET FOR PROGRAM INSTITUTIONALIZATION ? ITEMS ?A? AND ?B? Subsystem Item Item Score Mean Score PRODUCTION 1 ?a? and ?b? 2 ?a? and ?b? 3 ?a? and ?b? 4 ?a? and ?b? 5 ?a? and ?b? Item sum = Item sum/5 = MANAGERIAL 6 ?a? and ?b? 7 ?a? and ?b? 8 ?a? and ?b? Item sum = Item sum/3= MAINTENANCE 9 ?a? and ?b? 10 ?a? and ?b? 11 ?a? and ?b? Item sum = Item sum/3 = SUPPORT 12 ?a? and ?b? 13 ?a? and ?b? 14 ?a? and ?b? 15 ?a? and ?b? Item sum = Item sum/4= SCORE SHEET FOR PROGRAM INSTITUTIONALIZATION - ITEM ?C? Subsystem Item Item Score Mean Score PRODUCTION 1c 2c 3c 4c 5c Item sum = Item sum/5 = MANAGERIAL 6c 7c 8c Item sum = Item sum/3= MAINTENANCE 9c 10c 11c Item sum = Item sum/3 = SUPPORT 12c 13c 14c 15c Item sum = Item sum/4= 127 APPENDIX C: Location of HTPCP Projects Location of HTPCP Projects Funded October 1, 1989 ? September 30, 1997 State Number of Projects State Number of Projects Alabama 1 Missouri 1 Alaska 2 Nebraska 1 California 12 New Hampshire 1 Colorado 2 New Jersey 1 Connecticut 2 New Mexico 3 Georgia 3 New York 5 Hawaii 2 North Carolina 1 Idaho 2 Ohio 3 Illinois 4 Oregon 2 Kansas 1 Pennsylvania 4 Kentucky 3 Puerto Rico 1 Louisiana 1 Rhode Island 1 Maine 2 South Carolina 1 Maryland 4 Texas 7 Massachusetts 7 Virginia 1 Michigan 4 Washington 1 Minnesota 4 Washington, D.C. 2 Mississippi 1 Wisconsin 1 Total Number of Projects = 94 128 APPENDIX D: Institutional Review Board 2100 Lee Building College Park, Maryland 20742-5121 301.405.4212 TEL 301.314.9305 FAX INSTITUTIONAL REVIEW BOARD Reference: IRB HSR Identification Number 03-0235 MEMORANDUM Notice of Results of Final Review by IRB on HSR Application TO: Dr. Sharon M. Desmond Ms. Latricia C. Robertson Public and Community Health FROM: Dr. Phylis Moser-Veillon, Co-Chairperson Dr. Joan A. Lieber, Co-Chairperson Institutional Review Board PROJECT ENTITLED: ?Sustainability of Healthy Tomorrows Partnership for Children Program? The Institutional Review Board (IRB) concurs with the departmental Human Subjects Review Committee?s (HSRC?s) preliminary review of the application concerning the above referenced project. The IRB has approved the application and the research involving human subjects described therein. We ask that any future communications with our office regarding this research reference the IRB HSR identification number indicated above. We ask that you not make any changes to the approved protocol without first notifying and obtaining the approval of the IRB. Also, please report any deviations from the approved protocol to the Chairperson of your departmental HSRC. If you have any questions or concerns, please do not hesitate to contact either of us at irb@deans.umd.edu. Thank you. _______________________________________________________________________ ADDITIONAL INFORMATION REGARDING IRB/HSRC APPROVALS EXPIRATION OF IRB APPROVAL?Approval of non-exempt projects expires one year after the official date of IRB approval; approval of exempt projects expires three years after that date. If you expect to be collecting or analyzing data after the expiration of IRB approval, please contact the HSRC Chairperson in your department about submitting a renewal application. (PLEASE NOTE: If you are not collecting data from human subjects and any on-going data analysis does not increase the risk to subjects, a renewal application would not be necessary.) STUDENT RESEARCHERS?Unless otherwise requested, the IRB will send copies of approval paperwork to the supervising faculty researcher (or advisor) of a project. We ask that suck persons pass on that paperwork or a copy to any student researchers working on that project. That paperwork may be needed by students in order to apply for graduation. PLEASE BE ADVISED THAT THE IRB MAY NOT BE ABLE TO PROVIDE COPIES OF THAT PAPERWORK, particularly if several years have passed since the date of the original approval. 129 APPENDIX E: Cover Letter for HTPCP Survey Hello, this is Latricia Robertson, the Maternal and Child Health Bureau, Director of the Healthy Tomorrows Partnership for Children Program (HTPCP) from June 1991 ? 2000. I am currently working on my doctoral dissertation, studying sustainability of the HTPCP projects funded in 1989 ? 1997. I am writing to ask for your help with this study by completing the enclosed questionnaire which should take 15 minutes of your time. The purpose of my study is to determine sustainability of the HTPCP projects. Hopefully, it will provide an indication of the extent to which the projects have developed infrastructure geared toward sustainability. Demonstration of sustainability of the HTPCP projects can influence future SPRANS grant policy and funding decisions in MCHB-HRSA, as well as other federal funding sources for health promotion programs. For the study to be successful, I need your help. Please complete the enclosed questionnaire. Before beginning, please read and sign the attached Consent Form. Your responses are GUARANTEED to be anonymous. As a sign of my appreciation for your help, you will have the opportunity to win a $100 lottery. Please provide your name, phone number, and mailing address on the enclosed self-addressed, stamped postcard, and mail separately from the survey questionnaire. The postcard will be destroyed after the random drawing. The winner will receive a cashier?s check in the mail approximately eight weeks after the study survey instruments are mailed out. An announcement of the winner will be mailed to all participants. After reading the enclosed cover letter, please complete this questionnaire. Depending on other staff roles in your HTPCP project, you may want to invite other project staff to participate in the completion of the survey. (If other staff participate, please provide this information at the end of the questionnaire in the space provided.) Please complete and return the survey questionnaire in the enclosed self-addressed envelope by -------Date-Within 3 weeks----. Should you have questions, Dr. Sharon Desmond, the Principal Investigator and my doctoral advisor, can be reached per phone at (301) 405-2526 and e-mail address SD47@umail.umd.edu. My phone number is (214) 767-3078 and e-mail address is lrobertson@hrsa.gov. If you would like to have the study results, please write me at: Latricia Robertson, PO Box 501214, Dallas, TX, 75250 and request the final report. Thank you for your participation! 130 APPENDIX F: Consent Form (Page 1 of 2) Initials ____Date____ INFORMED CONSENT FORM Identification of Project/ Sustainability of Healthy Tomorrows Title Partnership for Children Program (HTPCP) Statement of Age of I state that I am over 18 years of age, in good Subject physical health, and wish to participate in a program of research being conducted by Dr. Sharon M. Desmond in the Department of Public and Community Health at the University of Maryland, College Park. Purpose The purpose of this research is to determine institutionalization/sustainability of a federally initiated maternal and child health promotion program. Procedure The procedure involves a questionnaire that I will self-administer and answer questions regarding my HTPCP project. Confidentiality All information collected in this study is confidential to the extent permitted by law. I understand that the data I provide will be grouped with data others provide for reporting and presentation and that my name will not be used. Risks None 131 (Page 2 of 2) Initials____Date____ Benefits, Freedom to The experiment is not designed to help me Withdraw, & Ability personally, but to help the investigator to Ask Questions learn more about sustainability in the HTPCP and to potentially improve future Federal funding and policy. I am free to ask questions or withdraw from participation at any time and without penalty. Medical Care Not Applicable Contact Information Dr. Sharon M. Desmond of Investigators Room 2387, HHP Bldg, Valley Drive University of Maryland, College Park, MD 20741 (301) 405-2526 (301) 314-9167 Fax SD47@umail.umd.edu Latricia C. Robertson, MSN, MPH PO Box 501214 Dallas, TX, 75250 (214) 767-3078 (214) 767-3902 Fax LRobertson@hrsa.gov NAME OF SUBJECT ___________________________________ (PRINT) SIGNATURE OF SUBJECT ____________________________________ DATE ____________________ 132 APPENDIX G: HTPCP Record Data Collection Form Source Matching Funds In Kind Hard Money Year 1 Amount ____________ ____________ ____________ ____________ ____________ ____________ Year 2 Amount ____________ ____________ ____________ ____________ ____________ ____________ Year 3 Amount ____________ ____________ ____________ ____________ ____________ ____________ Year 4 Amount ____________ ____________ ____________ ____________ ____________ ____________ Year 5 Amount ____________ ____________ ____________ ____________ ____________ ____________ 133 Levels of Matching Funds High - Over $150,000 = 3 _______________ Moderate ? Up to $150,000 = 2 _______________ Minimal - Meeting requirement of $100,000 = 1 _______________ Types of Matching Funds Hard Money Funding = 1 _______________ In-kind (Soft) Money Funding = 2 _______________ PAC High - In place & functioning at the end of the 1st funding year = 3 _________ Moderate ? In place & functioning at the end of the 3 rd funding year = 2 _____ Minimal ? In place & functioning at the end of the 5 th funding year = 1 ______ Pediatric Involvement High ? Occurring at the end of the 1 st funding year = 3 _______________ Moderate ? Occurring at the end of the 3 rd funding year = 2 ___________ Minimal ? Occurring at the end of the 5 th funding year = 1 ____________ PD Discipline High ? Medicine (Predominantly pediatricians) = 1 ______ Moderate ? All other disciplines = 2 __________________ 134 PD Level of Education High ? 9 or more years of education completed after high school = 3 ________ Moderate ? Up to 8 years of education completed after high school = 2 ______ Minimal ? Up to 4 years of education completed after high school = 1 _______ PD Turnover High ? 3 or more PDs during 5-year project = 1 ___________ Moderate ? 2 PDs during the 5-year project = 2 ___________ Minimal ? 1 PD during the 5-year project = 3 _____________ Evaluation High ? Outcome evaluation was conducted by the end of funding year 5 = 3 ____ Moderate ? Evaluation continued to be weak through funding year 3 = 2 _______ Minimal ? Evaluation was identified as weak during 1 st year of funding = 1 _____ 135 APPENDIX H: Pilot HTPCP Survey Instrument Administration Form Name of Individual Completing Questionnaire: (1) ______________________________ Title of Individual: (1) ______________________________ Length of Time in Position: (1) ______________________________ Name of Individual Completing Questionnaire: (2) ______________________________ Title of Individual: (2) ______________________________ Length of Time in Position: (2) ______________________________ How long did it take to complete the questionnaire? Were any questionnaire items unclear? If so, which one(s)? Do you have questions regarding interpretation of questions? If so, what are your questions? Other questions? Other comments: 136 APPENDIX I: HTPCP Survey Instrument This survey is about the Healthy Tomorrows Partnership for Children Program (HTPCP). Please read each of the 45 questions. There are no right or wrong responses.  If you are not sure about an item, just respond to the best of your ability.  Please complete the entire survey.  Please remember not to write your name on any page.  Thank you for participating. 1. Have the program?s goals and/or objectives been put into writing? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 4 2. If yes, for how many years have written goals & objectives actually been followed? ______Year(s)   3. Of all the aspects of this program that could have written goals and objectives, what is your best estimate of the proportion which actually have written goals and objectives? No aspects Few aspects Most aspects All aspects 1 2 3 4 137 4. Have any of the plans or procedures used for implementing this program been put in writing? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 7 5. If yes, for how many years have such written plans or procedures actually been followed? ______Year(s)   6. Of all the aspects of this program that could have written plans or procedures, what is your best estimate of the proportion which actually have written plans or procedures? No aspects Few aspects Most aspects All aspects 1 2 3 4 7. Has a schedule (e.g., timetable, plan of action) used for implementing program activities been put in writing? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 10 8. If yes, for how many years have such written schedules actually been followed? ______Year(s)   9. Of all the aspects of this program that could have written schedules, what is your best estimate of the proportion which actually have written schedules? No aspects Few aspects Most aspects All aspects 1 2 3 4 138 10. Have the strategies for implementing this program been adapted to fit local circumstances? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 13 11. If yes, for how many years have locally adapted strategies actually been followed? ______Year(s)   12. Of all the aspects of this program that could be adapted to fit local circumstances, what is your best estimate of the proportion which have actually been adapted? No aspects Few aspects Most aspects All aspects 1 2 3 4 13. Has a formal evaluation of the program been conducted? (1)_____Yes (2)______No (3) Not sure/not applicable   Go to Question 16 14. If yes, for how many times times has the program been formally evaluated? ______Years(s)   15. Of all the aspects of this program that could be formally evaluated, what is your best estimate of the proportion which have been formally evaluated. No aspects Few aspects Most aspects All aspects 1 2 3 4 139 16. Has a supervisor (e.g., section chief, department head) been formally assigned to oversee this program? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 19 17. If yes, for how many years has such a supervisor actually been formally assigned to oversee the program? ______Year(s)   18. Of all the aspects of this program that could receive supervision, what is your best estimate of the proportion which actually receives such supervision? No aspects Few aspects Most aspects All aspects 1 2 3 4 19. Have formalized job descriptions been written for staff involved with this program? (1)_____Yes (2)______No (3) Not sure/not applicable   Go to Question 22 20. If yes, for how many years have formalized job descriptions actually been followed? ______Year(s)   21. What is your best estimate of the number of staff involved with this program who have written job descriptions? None of the staff Few of the staff Most of the staff All of the staff 1 2 3 4 140 22. Are evaluation reports of this program done on a schedule similar to evaluation reports for most other programs in your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 25 23. If yes, for how many years have evaluation reports actually been produced on a schedule similar to such reports for most other programs in your organization? ______Year(s)   24. What is your best estimate of the extent that evaluation reports for this program are produced on a schedule similar to evaluation reports for most other programs in your organization? No evaluation Few evaluation Most evaluation All evaluation reports reports reports reports 1 2 3 4 25. Have any permanent staff been assigned to implement this program? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 28 26. If yes, for how many years have permanent staff been assigned to implement the program? ______Year(s)   27. What is your best estimate of the number of staff who implement the program that are in permanent positions? No staff Few Staff Most staff All staff 1 2 3 4 141 28. Has an administrative-level individual within your organization been actively involved in advocating for this program?s continuation? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 31 29. If yes, for how many years has this administrative-level individual actively advocated for this program?s continuation? ______Year(s)   30. What is your best estimate of how active this administrative-level individual has been in advocating for the program?s continuation? Not active Minimally Moderately Very at all active active active 1 2 3 4 31. Do staff in your organization, other than those actually implementing the program, actively contribute to the program?s operations? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 34 32. If yes, for how many years have such staff in your organization actively contributed to the program?s operation? ______Year(s)   33. Of all the staff in your organization who could contribute to the operation of this program, what is your best estimate of the proportion that actually contribute to it? None of the staff Few of the staff Most of the staff All of the staff 1 2 3 4 142 34. Has the program made a transition from trial or pilot status to permanent status in your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 37 35. If yes, for how many years has this program had permanent status? ______Year(s)   36. What is your best estimate of how permanent this program is in your organization? Not permanent Minimally Moderately Very at all permanent permanent permanent 1 2 3 4 37. Has the program been assigned permanent physical space within your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 40 38. If yes, for how many years has it maintained such permanent space? _______Year(s)   39. Of all the permanent space that this program needs, what is your best estimate of the proportion of permanent space it currently occupies? This program This program This program This program does not occupy occupies only occupies most occupies all any permanent a small amount of the permanent of the permanent space. of the permanent space that it needs. space that it needs. space that it needs. 1 2 3 4 143 40. Is this program?s source of funding similar to the funding sources for other established programs within your organization? (1)_____Yes (2)______No (3)______Not sure/not applicable   Go to Question 43 41. If yes, for how many years has this program?s funding sources been similar to those for other for other established programs within your organization? _____Year(s)   42. In your best estimate, how permanent is the program?s source of funding? Not permanent Minimally Moderately Very at all permanent permanent permanent 1 2 3 4 43. Is the staff most closely associated with this program?s implementation hired from a stable funding source? (1)_____Yes (2)______No (3)______Not sure/not applicable  44. If yes, for how many years has the staff most closely associated with this program?s implementation been hired from a stable funding source? ______Year(s)   45. What is your best estimate of how permanent the funding is for the staff most closely associated with this program?s implementation? Not permanent Minimally Moderately Very at all permanent permanent permanent 1 2 3 4 144 Demographics 46. Please provide the following information for each participant (if any) in completing this questionnaire. Age ____________ Gender _________ Years of formal education after high school_______________ Discipline/Occupation____________ Job Title_______________________ Length of time in position_________ Years with the HTPCP____________ Age ____________ Gender _________ Years of formal education after high school_______________ Discipline/Occupation____________ Job Title_______________________ Length of time in position_________ Years with the HTPCP____________ 145 APPENDIX J: Scoring the HTPCP Survey Instrument Scale The grid on the next page can be used to score the Survey Instrument Scale in conjunction with the following directions: Questions (routine items) 1 & 2; 4 & 5; 7 & 8; 10 & 11; 13 & 14; 16 & 17; 19 & 20, 22 & 23; 25 & 26; 28 & 29; 31 & 32; 34 & 35; 37 & 38; 40 & 41; and 43 & 44 are scored together. Niche saturation items 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, and 45 are scored separately. Score as follows:  If you checked ?No? or ?Not sure/not applicable? for items 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, and 43, then the score for the item = 0;  If you checked ?Yes? for items 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, and 43, and wrote ?0? or ?1? for items 2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, and 44, then the score for the item = 1;  If you checked ?Yes? for items 1, 4,7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, and 43 and wrote ?2? or ?3? for 2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, and 44, then the score for the item = 2;  If you checked ?Yes? for items 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, and 43 and wrote ?4? or ?5? for items 2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, and 44, then the score for the item = 3;  If you checked ?Yes? for 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, and 43 and wrote ?6? or more for 2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, and 44, then the score for the item = 4. Niche saturation items, score them as the number that you circled for that item (e.g., if you circled a ?2? then the score for that item = 2). Each set of three items represent one of the following organizational sub-systems: production (items 1-15), managerial (items 16-24), maintenance (items 25-33), supportive (items 34-45). Using the grid on the next page, add the scores for all routine items as indicated and divide by the number listed on the grid. Follow the same procedure for all niche saturation items. For routine items:  If the mean score is ?1? or ?2? then institutionalization is low to moderate;  If the mean score is ?3? then institutionalization is moderate to high;  If the mean score is ?4? then institutionalization is high. 146 For niche saturation items:  If the mean score is less than or equal to ?2? then insitutionalization is low;  If the mean score is greater than ?2? but less than or equal to ?3? then institutionalization is moderate;  If the mean score is greater than ?3? then institutionalization is high. 147 APPENDIX K: HTPCP Score Sheet Grid SCORE SHEET FOR PROGRAM INSTITUTIONALIZATION ? HTPCP Routine ITEMS Subsystem Item Item Score Mean Score PRODUCTION 1 and 2 4 and 5 7 and 8 10 and 11 13 and 14 Item sum = Item sum/5 = MANAGERIAL 16 and 17 19 and 20 22 and 23 Item sum = Item sum/3= MAINTENANCE 25 and 26 28 and 29 31 and 32 Item sum = Item sum/3 = SUPPORT 34 and 35 37 and 38 40 and 41 43 and 44 Item sum = Item sum/4= SCORE SHEET FOR PROGRAM INSTITUTIONALIZATION ? Niche Saturation ITEMS Subsystem Item Item Score Mean Score PRODUCTION 3 6 9 12 15 Item sum = Item sum/5 = MANAGERIAL 18 21 24 Item sum = Item sum/3= MAINTENANCE 27 30 33 Item sum = Item sum/3 = SUPPORT 36 39 42 45 Item sum = Item sum/4= 148 APPENDIX L: Table 13. Multiple Regression Analyses of Hypothesis Nine Sub-system Variables t p F R R  __________________________________________________________________ Routine Production 1.48 .38 .14 .18 PAC -.26 .24 PI 1.54 .98 Matching Funds -.81 .14 Type Funds .14 .28 PD Discipline 1.77 .02 PD Education .22 .14 PD Turnover 1.24 .20 Evaluation .82 .26 Managerial .88 .30 .09 .54 PAC -1.70 .09 PI .99 .33 Matching Funds .65 .52 Type Funds .49 .63 PD Discipline .71 .48 PD Education .48 .63 PD Turnover -1.97 .05 Evaluation .01 1.00 149 Table 13. Multiple Regression Analyses of Hypothesis Nine (Continued) Sub-system Variables t p F R R  __________________________________________________________________ Routine Maintenance .50 .23 .05 .85 PAC -.28 .78 PI .45 .65 Matching Funds -.12 .90 Type Funds -.18 .85 PD Discipline -1.56 .12 PD Education -1.18 .24 PD Turnover -.68 .50 Evaluation -.47 .64 Support .88 .30 .09 .53 PAC .43 .67 PI .26 .80 Matching Funds -1.67 .10 Type Funds -.48 .64 PD Discipline .56 .58 PD Education -.34 .74 PD Turnover -1.16 .25 Evaluation .30 .76 150 Table 13. Multiple Regression Analyses of Hypothesis Nine (Continued) Sub-system Variables t p F R R  __________________________________________________________________ Niche Saturation Production 1.01 .32 .10 .44 PAC -.53 .60 PI .97 .34 Matching Funds -.71 .48 Type Funds 1.23 .22 PD Discipline 1.96 .05 PD Education 1.26 .21 PD Turnover -1.45 .15 Evaluation .05 .96 Managerial .98 .32 .10 .46 PAC -.41 .68 PI .39 .70 Matching Funds 1.30 .20 Type Funds .11 .91 PD Discipline 1.07 .29 PD Education .72 .48 PD Turnover -2.02 .05 Evaluation .57 .57 151 Table 13. Multiple Regression Analyses of Hypothesis Nine (Continued) Sub-system Variables t p F R R  __________________________________________________________________ Niche Saturation Maintenance .49 .23 -.05 .86 PAC .75 .46 PI -.38 .71 Matching Funds .12 .91 Type Funds -.31 .76 PD Discipline -.01 .99 PD Education -.75 .46 PD Turnover -.55 .59 Evaluation .93 .36 Support 1.18 .34 .12 .33 PAC 1.30 .20 PI 1.70 .09 Matching Funds -.30 .76 Type Funds -.64 .53 PD Discipline .15 .88 PD Education -.87 .39 PD Turnover -.96 .34 Evaluation .58 .56 __________________________________________________________________ Key: PAC = Project Advisory Committee; PI = Pediatric Involvement 152 GLOSSARY Community-based: The degree to which the program is developed and implemented by individuals from the community as a result of a community defined need and provides services at sites easily accessible to individuals living in the targeted community (USDHHS/HRSA/MCHB, 1999). Cultural competence: A set of values, behaviors, attitudes, and practices within a system, organization, program or among individuals and which enables them to work effectively cross culturally. Further, it refers to the ability to honor and respect the beliefs, language, interpersonal styles and behaviors of individuals and families receiving services, as well as staff who are providing such services (USDHHS/HRSA/MCHB, 1999). Degrees Passages: The first degree of program institutionalization which is signified by one-time sentinel events such as the formalization of program plans, the shift from soft to hard sources of funding, and the program?s inclusion on the organization chart (Yin, 1979). Routines: The second degree of program institutionalization which is signified by the habituation, or routinization of program passages, such as the continued inclusion of the program in the organization?s formal plans, annual renewal of stable funding, and continued inclusion of the program in new versions of the organizational chart (Yin, 1979). 153 Niche The third degree of program institutionalization saturation: which is signified by the maximum feasible expansion of the program within the host organization/s subsystems, such as the optimum realization of the programs plans, the achievement of optimum levels of funding, and the inclusion of the program in a core (versus peripheral) location on the organizational chart (Yin, 1979). Family-centered: Recognize that the family is the constant in the Child?s life while the service systems and personnel within those systems fluctuate; facilitate parent/professional collaboration at all levels of health care (USDHHS/HRSA/MCHB, 1999). Institutionalization: Refers to the long-term viability and integration of a new program within an organization (Steckler and Goodman, 1989). Synonymous with sustainability as used in this study. Leveraging: Refers to the use of initial investments and commitments to draw larger investments and commitments (Green and Kreuter, 1991). Matching funds: Used synonymously with cost participation requirements met by securing funds required for the second and sequential years in an amount not less than 66.7 percent of the total budget (USDHHS/HRSA/MCHB, 1999). Pediatric involvement: The degree to which community pediatricians and other pediatric health professionals participate in community-based health service program activities of planning, governance, implementation, and evaluation (USDHHS/HRSA/MCHB, 1999). Project Advisory Made up of a group of representatives from the Committee: community participating in a system of governance for a HTPCP project that plans, implements, and evaluates it (USDHHS/HRSA/MCHB, 1999). 154 Subsystems Production: Concerned with ?throughput,? or those activities which are product directed (Katz and Kahn, 1978). Managerial: Concerned with coordinating the operations of the other subsystems (Katz and Kahn, 1978). Maintenance: Concerned with personnel issues and continuity of production in areas such as recruitment, indoctrination or socialization, rewarding, sanctioning and procurement of resources (Katz and Kahn, 1978). Supportive: Concerned with hospitable environmental conditions by establishing legitimacy and favorable organizational relationships (Katz and Kahn, 1978). Sustainability: The continuation of programmatic services in the community after the federal funding is completed (USDHHS/HRSA/MCHB, 1999). As used in this study, synonymous with institutionalization. 155 REFERENCES Altman, D.G. (1995). Sustaining Interventions in Community Systems: On the Relationship Between Researchers and Communities. Health Psychology. 14(6), 526-636). American Academy of Pediatrics. (1999). The Pediatrician?s Role in Community Pediatrics. Pediatrics, 103: 1304-1306. Bamberger, M. & Cheema, S. (1990). Case Studies of Project Sustainability: Implications for Policy and Operations from Asian Experience. Washington, DC: The World Bank. Basch, C.E. (1984). Research On Disseminating and Implementing Health Education Programs in School. Journal of School Health, 54, 57-66. Basch, C.E. & Gold, R.S. (1986). The dubious effects of type V errors in hypothesis on Health education practice and theory. Health Education Research, 1(4), 299-305. Bentler, P.M. (1985). Theory and Implementation of EQS, a Structural Equations Program Version 2.0. Los Angeles, CA: BMDP Statistical Software. Bentler, P.M. & Bonnett, D.G. (1980). Significant tests and goodness-of-fit in the analysis of covariance structures. Psy chology Bulletin, 88, 588-606. Beyer, J.M. & Trice, H.M. (1978). Implementing Change: Alcoholism Policies in Work Organizations. New York: The Free Press. Bracht, N. & Kingsbury, L. (1990). Community organization principles in Health promotion: a five-stage model. In N. Bracht, (Ed.), Health Promotion at the Community Level. Newbury Park, CA: Sage. Bracht, N., Finnegan, J.R., Rissel, C., Weisbrod, R., Gleason, J. Corbett, J. & Veblen-Mortenson,, S. (1994). Community ownership and program continuation following a health demonstration project. Health Education Research, 9, 243-255. Bruckner, M., Mangan, M., Godin, S., & Pogach, L. (1999). Project LEAP of New Jersey. American Journal of Managing Care, 5(5), 609-616. Buller, P.F. & McEvoy, G.M. (1989). Determinants of the institutionalization of planned organizational change. Group Organizational Studies, 14, 33-50. 156 Chalmers, M.L., Housemann, R.A., Wiggs, I, Newcomb-Hagood, L., Malone, B., Brownson, R.C. (2003). American Journal of Health Promotion, 27(3), 190-196. Chapman, D.J., Damio, G, Young, S, Perez-Escamilla, R. (2004). Effectiveness of Breastfeeding Peer Counseling in a Low-Income, Predominantly Latina Population: A Randomized Controlled Trial. 158(9), 897-903. Claquin, P. (1989). Sustainability of EPI: Utopia or Sine Qua Non Condition of Child Survival. Arlington, VA: REACH. Comrey, A. L. (1988). Factor analytic methods of scale development in personality and Clinical psychology. Journal of Consultative Clinical Psychology, 56: 754-761. Cook, T.D. & Campbell D.T. (1979). Quasi-Experimentation: Design and Analysis Issues forField Settings. Boston, MA: Houghton Mifflin. Crist, J.D. & Escandon-Dominguez, S. (2003). Identifying and recruiting Mexican American partners and sustaining community partnerships. Journal of Transcultural Nursing. Volume 14(3), 266-271. Dasgupta, R. & Priya, R. (2002). The sustainability of hepatitis B. immunization within the Universal Immunization Programme in India. Health Policy Planning, 17(1), 99-105. Devaney, B., Howell, E.M., McCormich, M.C., & Moreno, B. (2000). Reducing Infant Mortality: Lessons Learned from Healthy Start. Washington, D.C.: Mathematica Policy Research, Inc. Donabedian, A. (1980). Explorations in Quality Assessment and Monitoring: The Definition of Quality and Approaches to its Assessment. Volume I. Ann Arbor, MI: Health Administration Press. Downs, G.W. & Mohr, L.G. (1976). Toward a Theory of Innovation. Administration and Society, 10(4), 379-408. Eisen, N., Evans, J., Kavanagh, L., Athey, J. & Schwab, J. (1999). The Healthy Tomorrows Partnership for Children Program in Review: Analysis and Findings of a Descriptive Survey. Arlington, VA: National Center for Education in Maternal and Child Health, Georgetown University. Eliott, M.M. (1960). The Children?s Titles in the Social Security Act. Children. 7, 135-140. 157 Glaser, E. (1981). Durability of Innovations In Human Services Organizations. Knowledge, 3, 167-185. Goodman, P.S. & Dean, J.W. (1982). Creating long-term organization change, In Goodman P.S. Associates: Change in Organizations. San Francisco, CA: Jossey Bass. Goodman, R.M. (1987). Factors Affecting the Long-Term Viability of Health Promotion Programs: An Institutionalization Perspective. Dissertation Abstracts International. (UMI No. 8728446) Goodman, R.M., Burdine, J.N., Meehan, E. & McLeroy, K.R. (Eds.). (1993). Special issue: Community coalitions for health promotion. Health Education Research, 8(3). Goodman, R.M., McLeroy, K.R., Steckler, A.B. & Hoyle, R.H. (1993). Development of Level of Institutionalization Scales for Health Promotion Programs. Health Education Quarterly, 20, 161-178. Goodman, R.M. & Steckler, A.B. (1987/88). The life and death of a health promotion program: an institutionalization case study. International Quarterly of Community Health Education, 8, 5-21. Goodman, R.M. & Steckler, A.B. (1989a). A Framework for Assessing Program Institutionalization. Knowledge in Society: The International Journal of Knowledge Transfer, 2, 57-71. Goodman, R.M. & Steckler, A.B. (1987). A model for the institutionalization of health promotion programs. Family and Community Health, 11, 63-78. Goodson, P., Murphy, S.M., Evans, A., Meyer, B., & Gottlieb, N.H. (2001). Maintaining Prevention in practice: survival of PPIP in primary care settings. American Journal of Preventive Medicine, 20(3), 184-189. Grason, H. & Morreale, M. (1997). Health Services for Children and Adolescents: A ?Non-System? of Care. In Stein, R.E.K., (Ed.), Health Care For Children. New York, NY: United Hospital Fund of New York. Graziani, C. (1996). Defining ?effectiveness? in health development. Synergy: Canadian Initiatives for International Health. 8(3), 1-5. Greer, A.L. (1977). Advances in the Study of Diffusion of Innovation in Health Care Organizations. Milbank Memorial Fund Quarterly, 55(4), 505-532. 158 Green, L.W. (1989). Is institutionalization the proper goal of grant-making? American Journal of Health Promotion, 3, 44. Green, L.W. & Kreuter, M.W. (1991). Health Promotion Planning: An Education And Environmental Approach. Second Edition. Mountain View, CA: Mayfield. Green, L.W. & Lewis, F.M. (1986). Measurement and Evaluation in Health Education and Health Promotion. Palo Alto, CA: Mayfield. Green, L.W., Wilson, R.W. & Bauer, K.G. (1983). Data requirements to measure progress on the objectives for the nation in health promotion and disease prevention. American Journal of Public Health, 73, 18-24. Greer, A.L. (1977). Advances in the Study of Diffusion of Innovation in Health Care Organizations. Milbank Memorial Fund Quarterly, 55(4), 505-532. Hall, G.E., & Loucks, S.F. (1977). A Developmental Model for Determining Whether the Treatment is Actually Implemented. American Educational Research Journal, 14(3), 263-276. Howell, E.M., Devaney, B., Foot, B., Harrington, M., Schettini, M., McCormick, M.C., Hill, I., Schwalberg, R. & Zimmerman, B. (1997). The Implementation of Healthy Start. Washington D.C.: Mathematica Policy Research, Inc. Hutchins, V. L. (1997). A History of Child Health and Pediatrics in the United States. In Stein, R.E.K. (Ed.), Health Care For Children. New York, NY: United Hospital Fund of New York. Hutchins, V.L. (1994). Maternal and Child Health Bureau: Roots. Pediatrics. 94:695-699. Hutchins, V. L. (1997). A History of Child Health and Pediatrics in the United States. In Stein, R.E.K. (Ed.), Health Care For Children. New York, NY: United Hospital Fund of New York. Hutchins, V.L. (2001). Maternal and Child Health at the Millennium: Looking Back, Moving Forward. Washington, D.C.: Center for Health Policy Research, The George Washington University. Jackson, C., Fortmann, S.P., Flora, J.A., Melton, R.J., Snider, J.P. & Littlefield, D. (1994). The capacity-building approach to intervention maintenance implemented by the Stanford Five-City Project. Health Education Research, 9, 385-396. 159 Joreskog, K.G. (1979). A general approach to confirmatory maximum likelihood factor analysis. In K. G.Joreskog, D. Sorbom (Eds.), Advances in Factor Analysis and Structural Equations Models. Cambridge, MA: Abbott Associates. Kaluzny, A.D. (1974). Innovation in Health Services: Theoretical Framework and Review of Research. Health Services Research, 9(2), 101-120. Kaluzny, A.D., Warner, D.M., Warren, D.G., Zelman, W.N. (1982). Management of Health Services. Englewood Cliffs, NJ: Prentice-Hall. Katz, K. & Kahn, R.L. (1978). The Social Psychology of Organizations. Second Edition. New York, NY: Wiley. Kleinbaum, D.G., Kupper, L.L. & Muller, K.E. (1988). Applied Regression Analysis and Other Multivariable Methods, Second Edition. Belmont, CA: Duxbury Press. Ledford, G.E. (1984). The Persistence of Planned Organizational Change: A Process Theory Perspective. Ann Arbor, MI: Unpublished dissertation. Lee, B.C., Westaby, J.D., & Berg, R.L. (2004). Impact of a National Rural Youth Health and Safety Initiative: Results from a Randomized Controlled Trial. American Journal of Public Health. 94(10), 1743-1750. Lefebyre, R.C. (1990). Strategies to maintain and institutionalize successful programs: a marketing framework. In N. Bracht, (Ed.), Health Promotion at the Community Level. Newbury Park, CA: Sage. Lesser, A.J. (1985). The Origin and Development of Maternal and Child Health Programs in the United States. American Journal of Public Health, 75(6), 590-598. Lienhardt, C. & Ogden, J.A. (2004). Tuberculosis control in resource-poor countries: have we reached the limits of the universal paradigm? Tropical Medicine and International Health, 9(7), 833-841. McQueen, D.V. (2003). Judging the success of the Global Programme on Health Promotion Effectiveness. Promotion & Education, 10(3), 117. Miles, M.B. (1983). Unraveling the mystery of institutionalization. Education Leadership, 41, 14-19. Miller, R.L., Bedney, B.J., Guenther-Grey, C. (2003). 30(5), 582-600. Assessing Organizational capacity to deliver HIV prevention services collaboratively: tales from the field. Health Education Behavior, 30(5), 582-600. 160 Mohr, L.B. (1969). Determinants of Innovation in Organizations. American Political Science Review, 63(1), 111-126. Newman, E.L., & Tejeda, M.J. (1996). The need for research that is designed to support decisions in the delivery of mental health services. American Psychologist. 51, 1040-1049. Nunnally, J.C. (1978). Psychometric Theory. Second Edition. New York, NY: McGraw-Hill. Olds, D.L., Robinson, J.A., O?Brien, R., Luckey, D.W. (2002). Home visiting by paraprofessionals and by nurses: A randomized, controlled trial. Pediatrics. 110 (3), 486-497. Polit, D.F. & Hungler, B.P. (1978). Nursing Research: Principles And Methods. Philadelphia, PA: J.B. Lippincott Company. Publication Manual of the American Psychological Association. (2002). Fifth Edition. Washington, DC: American Psychological Association. Rifkin, S.B. (1986). Lessons from community participation in health programmes. Health Policy and Planning. 1(3), 240-249. Rogers, B., Winslow, B., & Higgins, S. (1993). Employee satisfaction with occupational health services: Results of a survey. AAOHN Journal. 41(2), 58-65. Rogers, E.M. & Shoemaker, F.F. (1971). Communications of Innovations: A Cross Cultural Approach. New York: The Free Press. Rubardt, M., Chikoko, A., Glik, D., Jere, S., Nwayanwu, O., Zhang, W., Nkhoma,W., Ziba, C. (1999). Implementing a malaria curtains projects in rural Malawi. Health Policy Planning. 14(4):313-321. Runyon, R.P. & Haber, A. (1991). Fundamentals of Behavioral Statistics. Seventh Edition. New York, NY: McGraw-Hill, Inc. Sarvela, P.D. & McDermott, R.J. (1993). Health Education Evaluation and Measurement. Madison, WI: WBC Brown & Benchmark. Scheirer, M.A. & Rezmovic, E.L. (1983). Measuring the Degree of Implementation ? A Methodological Review. Evaluation Review. 7(5), 599-633. Schwartz, J.S., Ball, J.R., & Moser, R.H. (1982). Safety, efficacy, and effectiveness of Clinical practices: A new initiative. Annals of Internal Medicine. (65, 246-247. 161 Shediac-Rizkallah, M.C. & Bone, L.R. (1998). Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Education Research, 13(1), 87-108. Skillen, D.L., Anderson, M.C., Seglie, J.A., & Gilbert, J. (2002). Toward a model for Effectiveness: what Alberta occupational health nurses think. AAOHN Journal. 50(2), 75-83. Solloway, M., Gotschall, C.S., Barta, L.J. & Avery, A. (1996). Emergency Medical Services for Children: An Evaluation Of Sustainability In Seven States. Washington, D.C.: Center for Health Policy Research, The George Washington University. Steckler, A. & Goodman, R.M. (1989). How to institutionalize health promotion programs. American Journal of Health Promotion, 3, 34-44. Stein, R.E.K. (Ed.). (1997). Health Care for Children. New York, NY: United Hospital Fund of New York. Teitelbaum, M., Irvin, C., Mason, T., Foster, S. & Thomas, L. (1998). Evaluability Assessment and Evaluation of the Community Integrated Service System (CISS) Program. Bethesda, MD: Abbott Associates Inc. Thesis and Dissertation Manual. (1995). College Park, MD: The Graduate School, University of Maryland. Thompson, M., Minkler, M., Allen, Z., Bell, J.D., Bell, J., Blackwell, A.G., Carpenter, M., Rose, K., & Tamir, H.B. (2000). Community Involvement in the Federal Healthy Start Program. Oakland, CA: PolicyLink. Thesis and Dissertation Style Guide 2004-2005. (Fall 2004). College Park, MD: The Graduate School, University of Maryland. Tucker, L.R. & Lewis, C.A. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10. US Agency for International Development. (1988). Sustainability of Development Programs: A Compendium of Donor Experience. Washington, DC: USAID. US Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Bureau. (February 2001). Celebrating 65 Years of Title V: The Maternal and Child Health Program. US Department of Health and Human Services, Health Resources and Services Administration, Maternal and Child Health Bureau. (Spring 1999). Healthy Tomorrows Partnership for Children Grant Application Guidance. 162 US Department of Health and Human Services, Public Health Service, National Institutes of Health, Office of Cancer Communications, National Cancer Institute. April 1992. NIH Publication No. 92-1493. US Statutes, 62 Congress, Second Session. (1911-1912). Part 1, Chapter 73, 79-80. Waters, E. & Doyle, J. (2002). Evidence-based public health practice: improving the quality and quantity of the evidence. Journal of Public Health Medicine. 24(3), 227-229. Weitzman, M. (1997). The role of the Individual Child Health Care Practitioner. In Stein, R.E.K. (Ed.), Health Care for Children. New York, NY: United Hospital Fund of New York. Whittemore, R. & Grey, M. (2002). The systematic development of nursing interventions. Journal of Nursing Scholarship. 34(2), 115-121. Yin, R.K. (1979). Changing Urban Bureaucracies: How New Practices Become Routinized. Lexington, MA: Lexington Books.