The incremental roll-out strategy proposed for the Institute offers a unique opportunity to build data-driven decision-making into the day-to-day operations of the Institute. As part of the model, external evaluators will assist in the development and implementation of a continuous improvement framework as well as the design of appropriate data collection strategies and data management structures.
Each year, indicators linked to the objectives of the Institute will be identified, and data will be collected to measure performance against these indicators. This continuous improvement framework will enable the Institute leaders to assess progress toward their objectives, identify areas in which revisions are needed or data suggest needs for further research, and devise program revisions suggested by evaluation data. MPR Associates, Inc. and FHI 360 will jointly design and execute the continuous improvement assessment process, in collaboration with the executive director and Institute partners.
During the first year, Institute evaluation activities will focus specifically on the identification of strategies and processes that contribute to the delivery of highquality training for instructional coaches, mentoring and other types of support for coaches, higher levels of regular one-on-one coaching in participating schools, greater student engagement—and ultimately to improved student outcomes. Questions that might be examined through the evaluation include:
Answers to questions such as these can help the Institute determine the efficacy of its processes and programs from year to year and suggest ways to rethink and recalibrate before new cohorts are added to the roster. Furthermore, assessment for continuous improvement will enable the Institute to begin to build the evidentiary base needed to attract additional schools, districts, and funders.
Once the major evaluation questions are identified for a given year, the evaluation partners will use a variety of assessment strategies to gather the data needed to gauge progress toward key objectives. These might include: field observations of mentors, coaches, teachers, classrooms, and training sessions; surveys; interviews and focus groups; document reviews; and analysis of grades and standardized test scores.
In most instances, qualitative data collection and analysis will be connected to quantitative data collection and analysis in an effort to answer the why and how questions typically suggested by initial tabulations of quantitative data. For example, the evaluators might want to interview teachers to determine why some participated in regular one-on-one coaching and others did not. Observations in the classrooms of the two groups might reveal other differences.
In-depth case studies will be conducted in all schools during the transition year, and cross-case analyses will be completed to identify emerging patterns and trends. During the case study visits, MPR and FHI 360 researchers will collect data on students, teachers, and coaches through interviews, focus groups, and observations. Additional data, primarily on student engagement and achievement, will be obtained from school records and other documents. From the beginning, the MPR and FHI 360 research team, in collaboration with the executive director, will build a strong partnership with the schools and districts around the critical need for evidence and its use in program improvement and program expansion across the state.