Program Evaluation for the Practitioner (page 2)

How Is It Done?

Even when practitioners understand why program evaluation is important, they often do not feel they have the skills to do it. Although a complex or large-scale evaluation is best left to the professional evaluator, the following five steps can be used by practitioners in any setting to effectively gather and interpret useful data.

STEP 1. Set the stage and clarify goals.   Often, the best time to make decisions about evaluation is when a program is being designed. A staff meeting or plan-writing session can provide an opportunity to discuss the proposed program, why it is being initiated, and its intended effects. For example, a school staff might decide to start a voluntary afterschool tutoring program in mathematics and English. They would begin by discussing the target audience and how the program will operate, but they also would clarify what they hope to accomplish: improved passing rates, for instance, or a greater number of students proficient on state assessments. The clearer and more specific the goals for the program are from the outset, the more likely it is that the evaluation will be able to determine if those goals were achieved. The participation of the staff in the initial design will also help to ensure their full and active participation in the evaluation activities.   STEP 2. Create evaluation questions.   The designers of an informal evaluation can use their goals to create questions that the evaluation will answer. In the case of the tutoring program, the staff might decide that it wants to know the following:

• Was the tutoring program implemented as designed? Did implementation support the established goals?

• Did participation in the program have a positive effect on student achievement in mathematics and reading?

• How many students chose to participate in the tutoring program? Did the number of participants grow over time? Did participation improve participants’ outlook toward school?     

STEP 3. Decide what data are needed and how to collect them.   Once the evaluation questions have been formulated, the next step is to decide what data are needed to answer them. Practitioners should avoid designing elaborate data collection methods and focus instead on using data that are already available or easy to collect. Using team planning time or a portion of every staff meeting to discuss and informally summarize tutoring sessions will provide implementation data that address the first question. If the school already uses pretests and posttests or other assessment measures periodically throughout the year, gathering student achievement data for the participants in the afterschool program will be relatively easy. Answering the student participation questions might require new data collection tools, but here, too, the emphasis should be on simplicity and practicality. The first step would be to establish a system to track attendance. A sign-in sheet or computer logon screen would work. Participant interviews can provide data about students’ interest in the program, if and how the program has improved their attitude toward school, and even what changes they would recommend.   STEP 4. Analyze and interpret data.   By analyzing and interpreting the data collected, the staff can make an informal judgment about how well the goals of the tutoring program have been met. Notes and recorded comments on the structure of the tutoring sessions, lesson plans, and activities can be examined. Average gains in test scores can be computed and trend data examined. Percentages, averages, and other statistical measures can be calculated to determine patterns in daily attendance at tutoring sessions. Student interview data can be analyzed to determine why students participate and how the program can be improved.   The timing of the data analysis and interpretation should be driven by the evaluation questions. Staff members might decide to interpret some data formatively—that is, while the program is being conducted—so they can see what implementation modifications and adjustments are suggested. Other data are best analyzed at the program’s end (such as the end of the school year or semester) and used to assist in decision making about further intervention efforts.   STEP 5. Use the results.   After staff members have discussed the analyzed data from their afterschool program, they need to formulate one more important evaluation question: “What will we do with what we found out?” It is likely that the data collected provide useful information about what was accomplished but also highlight areas that need improvement. Armed with this information, the staff has what it needs to revise and adapt the program for the following year.