How does a researcher conduct an evaluation? What are the elements to be considered in designing an evaluation?

Question

What will be an ideal response?

Answer 1

There are many ways to conduct evaluation research. Generally evaluation consists of the following steps: determine the scope of evaluation, get acquainted with the program, choose the methodology for evaluation, collect the data, analyze the data, and report the findings.

To determine the scope of an evaluation, the evaluator first reviews all pertinent information, and then a background investigation is conducted to find out more about the assignment. Finally, the evaluator negotiates and reaches agreements with the sponsor about their mutual expectations.

Once the contract has been signed, the evaluator officially proceeds with the evaluation. The first task is to get fully acquainted with the program or project to be evaluated, and includes becoming acquainted with the meaning of all terms to be used. Specifically, the evaluator should learn about the program’s goals and objectives, main activities, organizational structure and functions, personnel roles and responsibilities, operations and outcomes and their relationships, clients and services, financial performance, and important problems. If feasible, the evaluator may wish to personally experience the program components and activities. At a minimum, he or she should conduct one field site visit to obtain firsthand impressions of the actual program operations. It is often useful to create a logic model during this stage to connect the program’s goals to its specific objectives or activities and desired outcomes. The completed logic model acts as a blueprint of sorts for the rest of the evaluation.

After the evaluator has become acquainted with the program, it is time to finalize the methodology. This involves deciding upon the evaluation design, instruments and measures, and data collection and data analysis methods. The evaluator should involve the primary users in the process of evaluation planning, and a detailed schedule should also be drawn up.

Evaluation design is concerned with choosing the appropriate research methods, both quantitative and qualitative, and selecting the unit of analysis and sampling techniques. Research methods may include surveys, experiments, and/or case studies, depending on the objectives of the evaluation and constraints of the situation. The design can include both quantitative and qualitative methods. The type of evaluation determines whether the program should be measured prospectively (before and after the program) or retrospectively (during and after the program). In choosing evaluation methods, the evaluator may incorporate both quantitative and qualitative approaches.

Multiple sources of evidence in an evaluation may provide an opportunity to enhance the evaluation’s comprehensiveness and credibility. The criteria used to select data source and procedures used to collect data in an evaluation should be described clearly and in detail to help researchers interpret the evidence accurately. The design of an evaluation also considers the unit of analysis and sampling procedures. The choice of unit of analysis depends on the program objectives and is related to the target that the program or intervention is designed to affect. A sampling plan specifies how the unit of analysis will be selected for study. If interviews or surveys are to be conducted, a random sampling method should be used to select subjects. If a focus group is to be employed, participants should be selected based on their representativeness of organizational units and responsibilities as well as demographics.

Once the evaluation method has been chosen, the evaluator selects or designs instruments containing measures of the variables to be studied. The instrument should include all important measures for the evaluation objectives and, if possible, for the development of associated scientific theories. The program sponsor and staff should be invited to participate in the selection and/or design of the research instrument, the complete instrument should be pilot-tested and revised, and the validity and reliability of the measure documented. The following measures are usually examined as part of an evaluation: participant characteristics; characteristics of the program structure or context that might affect the intervention; characteristics of the program implementation or processes; characteristics of the program outcomes, both long-term and short-term; costs associated with the program; and benefits associated with program outcomes.

Data for evaluation research may be obtained through surveys, interviews, administrative records, observations, or content analysis. The evaluator should first try to find useful information that has been or is being collected routinely in the setting. When there is no record-keeping system, the evaluator may suggest that one be established to collect needed information.

The types of data analyses conducted depend on the evaluation design. When designs consist of control groups or nonbiased comparison groups, the analysis is straightforward. Comparisons can be made between the experimental and control groups, and statistical procedures determine whether the observed differences are significant. For programs with nonuniform coverage, evaluators may take advantage of variations in intervention in order to approximate quasi-experimental designs, and the program effect can be assessed in the same way as with experimental designs. However, variables that measure program intensity should be included in the analysis. Designs without control groups generally are evaluated using before-and-after time-series designs, in which multiple measurements before and after the program are introduced. More sophisticated multivariate statistical procedures are available for causal analyses and for providing statistical controls.

The data collection stage usually involves different kinds of data and different ways of collecting them. Needs assessment and process evaluation generally take place during the program implementation; outcome evaluation focuses on the accomplishments and impact of the program, so data collection usually occurs after the completion of the program; policy evaluation is mostly conducted sometime after the policy implementation. Data may be collected cross-sectionally or longitudinally and from program staff and/or participants themselves. The evaluator should make sure that the instruments are administered, interviews and observations conducted and coded, secondary data gathered and processed, and scheduled deadlines met. If necessary, the investigator should see to it that proper training has been given to those responsible for data collection.

In the data analysis stage, the overall objective is to determine the net outcome of the program, which may be expressed as gross outcome subtracting the effects from unrelated extraneous confounding factors and designs. The net outcome may then be compared with that from other programs or some objective standard. Commonly used approaches to analyzing evaluation data include program coverage measurement, cost–benefit analysis, cost-effectiveness analysis, decision analysis, and modeling.

After the data has been analyzed, evaluators can convey program results through informal meetings with program and evaluation sponsors as well as by way of memos, newsletters, formal presentations, formal written reports, and scholarly publications. For program sponsors, a formal written report is the most important product of evaluation and is required from the evaluator.