Define and explain each of the types of evaluation research. When is each appropriate?

Question

What will be an ideal response?

Answer 1

There are five primary types of program evaluation: needs assessment, evaluability assessment, process evaluation (including formative evaluation), impact evaluation, and efficiency (cost-benefit) analysis. Today, the field of evaluation research remains somewhat of a dichotomy. There are those who conduct evaluation research for the sake of knowledge alone, whereas others strive to make their products useful for some action (the more specific program evaluation research, such as that performed by Sherman and Berk, 1984. Regardless of the emphasis, however, all evaluation is empirical and data driven. Both program evaluation and evaluation research in general offer empirical answers to questions of policy and policy effectiveness. Objective and empirical assessments of policies and programs are the cornerstone of the evaluation field.
Evaluation research may be undertaken for a variety of reasons: for management and administrative purposes, to test hypotheses derived from theory, to identify ways to improve the delivery of services, or to decide whether to continue, cut, or modify a particular program. The goal of evaluation research, however, is primarily the same as the goal for all social science research: to design and implement a study that is objective and grounded in the rules of scientific methodology. These methods run the gamut of the methods we have discussed in this text. They can range from the strictly quantitative experimental and quasi-experimental designs to the qualitative methodologies of observation and intensive interviewing.
Evaluation projects can focus on several questions related to the operation of social programs and the impact they have:
• Is the program needed? (evaluation of need)
• Can the program be evaluated? (evaluability assessment)
• How does the program operate? (evaluation of process)
• What is the program’s impact? (evaluation of impact)
• How efficient is the program? (evaluation of efficiency)
The specific methods used in an evaluation research project depend, in part, on which of these questions is being addressed.
Do We Need the Program? Is a new program needed or is an old one still required? Is there a need at all? A needs assessment attempts to answer these questions with systematic, credible evidence. The initial impetus for implementing programs to alleviate social problems and other societal ailments typically comes from a variety of sources, including advocacy groups, moral leaders, community advocates, and political figures. Before a program is designed and implemented, however, it is essential to obtain reliable information on the nature and the scope of the problem as well as the target population in need of the intervention. Evaluation researchers often contribute to these efforts by applying research tools to answer such questions as “What is the magnitude of this problem in this community?” “How many people in this community are in need of this program?” “What are the demographic characteristics of these people (e.g., age, gender, and race or ethnicity)?” and “Is the proposed program or intervention appropriate for this population?”
Needs assessment is a type of evaluation research that attempts to determine the needs of some population that might be met with a social program.
Needs assessment is not as easy as it sounds (Posavac & Carey, 1997). Whose definitions or perceptions should be used to shape our description of the level of need? How will we deal with ignorance of need? How can we understand the level of need without understanding the social context from which that level of need emerges? (Short answer to that one: We can’t!) What, after all, does need mean in the abstract? We won’t really understand what the level of need is until we develop plans for implementing a program in response to the identified needs.
Can the Program Be Evaluated? Evaluation research will be pointless if the program itself cannot be evaluated. Yes, some type of study is always possible, but a study conducted specifically to identify the effects of a particular program may not be possible within the available time and resources. So researchers may carry out an evaluability assessment to learn this in advance rather than expend time and effort on a fruitless project.
Evaluability assessment is a type of evaluation research conducted to determine whether it is feasible to evaluate a program’s effects within the available time and resources.
Knowledge about the program gleaned through the evaluability assessment can be used to refine evaluation plans. Because they are preliminary studies to check things out, evaluability assessments often rely on qualitative methods. Program managers and key staff may be interviewed in depth, or program sponsors may be asked about the importance they attach to different goals. These assessments also may have an “action research” aspect, because the researcher presents the findings to program managers and encourages changes in program operations.
Is the Program Working as Planned? What actually happens in a program? Once a program has been started, evaluators are often called on to document the extent to which implementation has taken place, whether the program is reaching the target individuals or groups, whether the program is actually operating as expected, and what resources are being expended in the conduct of the program. This is often called process evaluation or program monitoring. Rossi and Freeman (1989) define program monitoring as the systematic attempt by evaluation researchers to examine program coverage and delivery. Assessing program coverage consists of estimating the extent to which a program is reaching its intended target population; evaluating program delivery consists of measuring the degree of congruence between the plan for providing services and treatments and the ways they are actually provided.
Process evaluation (program monitoring) is evaluation research that investigates the process of service delivery.
Process evaluations are extremely important, primarily because there is no way to reliably determine whether the intended outcomes have occurred without being certain the program is working according to plan. For example, imagine you are responsible for determining whether an anti-bullying curriculum implemented in a school has been successful in decreasing the amount of bullying behavior by the students. You conduct a survey of the students both before and after the curriculum began and determine that rates of bullying have not significantly changed in the school since the curriculum started. After you write your report, however, you find out that, instead of being given in a five-day series of one-hour sessions as intended, the curriculum was actually crammed into a two-hour format delivered on a Friday afternoon. A process evaluation would have revealed this implementation problem. If a program has not been implemented as intended, there is obviously no need to ask whether it had the intended outcomes.
A process evaluation can take many forms. Because most government and private organizations inherently monitor their activities through such things as application forms, receipts, and stock inventories, it should be relatively easy to obtain quantitative data for monitoring the delivery of services. This information can be summarized to describe things such as the clients served and the services provided. In addition to this quantitative information, a process evaluation will also likely benefit from qualitative methodologies such as unstructured interviews with people using the service or program. Interviews can also be conducted with staff to illuminate what they perceive to be obstacles to their delivery of services.
Process evaluation can employ a wide range of indicators. Program coverage can be monitored through program records, participant surveys, community surveys, or number of utilizers versus dropouts and ineligibles. Service delivery can be monitored through service records completed by program staff, a management information system maintained by program administrators, or reports by program recipients (Rossi & Freeman, 1989).
Qualitative methods are often a key component of process evaluation studies because they can be used to understand internal program dynamics, even those that were not anticipated (Patton, 2002; Posavac & Carey, 1997). Qualitative researchers may develop detailed descriptions of how program participants engage with each other, how the program experience varies for different people, and how the program changes and evolves over time.