“Evaluation” is the collection of data before a program begins and again once it is underway or finished to identify and quantify how effectively processes are working. Public programs compete for taxpayer resources, making demonstrations of effectiveness important to the future of any programming. Evaluation also allows participants to see progress, defend future funding, drive program improvements and share successes to encourage participation. There are no widely recognized evaluation standards, so it has been difficult for researchers and communities to compare across programs and thereby secure initial or continued funding from outside sources. The only state and national standard measurements for wellness are the Healthy People 2010 objectives specified by 10 leading health indicators, including physical activity and overweight and obesity. These are coordinated by the U.S. Department of Health and Human Services through the National Center for Health Statistics. Healthy New Jersey 2010 has a similar set of 67 objectives in 11 health categories. Both the state and national Healthy People 2010 initiatives can be invaluable to mayors when planning programs and activities. The indicators can serve as goals and the data as a starting point from which to measure progress by identifying at-risk populations.
Define Success in Terms the Participants Value
As a component of program planning, brainstorm with participants to determine a list of areas that will be influenced by the program. This can include cognitive measures of knowledge and attitude; emotional measures of belonging, self-esteem, stress reduction and spiritual well-being; and physical measures such as BMI, fitness, flexibility and endurance. It is important not to base success solely on body mass index (BMI) as a change in weight may be less valued than improved health or lifestyle change. It may be necessary to address a number of factors to bring about real changes. A redefinition of success can lay out realistic and sustainable goals upon which community stakeholders can agree and funding agencies can view as legitimate outcomes. The common definition of success should be disseminated to each stakeholder group for input.
Determine Culturally Sensitive Evaluations Methods Built on Definition of Success
Once consensus is reached on a definition for success, the community can collaboratively develop a methodology for evaluation. It is important to remember when developing evaluations that they are realistic and proportional to the level/duration of the intervention. Provide measures that facilitate short-term milestones and help participants reach long-term goals. Short-term goals allow participants to focus and maintain motivation towards achieving long-term goals. Evaluation tools should be developed in ways that are culturally appropriate. Once selected, they should be provided to program managers for use in measurement strategies and to support the use of specific program measures. These measurement strategies can enhance program implementation and create an environment in which cyclic evaluation and improvement are expected and thus drive sustainability. Having a strategy that supports sustainability will increase the number of programs that benefit participants. In addition, a measure of cultural sensitivity will aid philanthropies in determining how to adapt a program to its target community.
A useful tool in this area is provided by the W.K. Foundation Logic Model Development Guide. A link is provided at the end of this document.
Different Types of Evaluations
As you decide how to evaluate your programs, there are several types of evaluations to consider:
Process evaluations can occur throughout the project cycle and guide managers to make changes so as to maximize effectiveness. Examples of process measures are the number of community partners or number of students participating. Output measures, commonly used in process evaluations, describe a program’s activities, such as how many students participated in a walk or how many classes were convened, rather than the ultimate effect of the program (e.g., changes in health.) Program planners can also use outputs to identify a need to better tailor programs to a target population (e.g., if students are not joining a wellness class) or to monitor changes in program performance, such as fewer students in the class than before.
Outcome evaluations determine if desired changes to attitudes, behavior, or knowledge have been attained as a result of the intervention. Outcome metrics are usually measured at the beginning and end of a program. Examples of outcomes include positive changes to health status or a quantifiable increase in walking seniors.
Impact evaluations seek to isolate a program’s effect on participants and communities, by filtering out effects from other potential sources (e.g., weather, other programs). Although impact evaluations require a higher level of technical expertise, they are considered the “gold standard” of evaluation. Impact evaluations compare a group receiving services against a group that is not.
For youth programs, the following evaluations should be considered:
Process Evaluations |
|
| Metric |
Means of Measurement
|
| Participation, Attrition |
|
| Expert, Community and Task Force Involvement |
|
| Communication Plan Implementation |
|
| Program Completion |
|
| Budgetary issues |
|
Outcome Evaluations |
|
| Metric |
Means of Measurement
|
Diet/activity changes, including:
|
|
Cognitive & psychosocial changes, including:
|
|
| Student satisfaction with suggestions |
|
| Parent satisfaction with suggestions |
|
| Program staff /teacher satisfaction with suggestions |
|
| Financial outcomes |
|
Impact Evaluations |
|
| Metric |
Means of Measurement
|
To determine scientific significance of program, comparison to a control group receiving no or different interventions. Look for changes in:
|
|
Resources
www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf
W.K. Kellogg Foundation Logic Model Development Guide
A comprehensive, easy-to-understand guide on evaluation that can be downloaded from the internet.
www.healthierus.gov/
HealthierUS.gov
www.state.nj.us./health/chs/hnj2010u05/index.shtml
Healthy NJ 2010: Update 2005
References
Joint Committee on Standards for Educational Evaluation. (1999). Program evaluation standards: How to assess evaluations of educational programs. 2nd ed. Thousand Oaks, CA: Sage Publications, as cited in Centers for Disease Control and Prevention. (1999). Framework for Program Evaluation in Public Health. Morbidity and Mortality Weekly Report, 48 (N. RR-11), 26-31.
Pyramid Communications. (2004). Equal Rights to Health: Moving Forward Together. Funded by The Robert Wood Johnson Foundation, The California Endowment and Casey Family Programs.
Pyramid Communications. (2004). Communities Helping Children Be Healthy: A Guide to Reducing Childhood Obesity in Low-Income African American, Latino and Native American Communities. Funded by The Robert Wood Johnson Foundation, The California Endowment and Casey Family Programs.
Rosenberg, Morris. (1989). Society and the Adolescent Self-Image. Revised edition. Middletown, CT: Wesleyan University Press. Retrieved January 31, 2005 from: http://www.bsos.umd.edu/socy/grad/socpsy_rosenberg.html.
W.K. Kellogg Foundation. (2004, January). Logic Model Development Guide. Retrieved January 31, 2005 from: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf.
