Program Evaluation

Introduction

As part of the Community Engagement Project for the Educational Technology Leadership module, I conducted a program evaluation for an English Immersion Program at a learning academy in Hong Kong. The program evaluation aimed to determine the viability and effectiveness of the program in fostering academic excellence while instilling the social-emotional and life skills needed for children to succeed in a 21st-century society.

Structured parent interviews, focus groups, and teacher diaries were deployed as methods for data collection. Parents with children enrolled in the pilot phase of the English Immersion Program were invited to participate in the focus group and interviews.

All data collected via the different methods were then consolidated and organized by categories. Similar topic responses were grouped together in the same category during the data consolidation exercise and color-coded based on whether the response was a suggestion, a positive response, or an opportunity for improvement.

Methodology

Research Design

The following approaches have been chosen to conduct the evaluation:

Participatory Evaluation – this approach involves engaging the stakeholders in the evaluation process from the evaluation design to data analysis stages. According to Gujit (2014, p.2), this approach leads to “better data, better understanding of the data, more appropriate recommendations, [and] better uptake of findings”.
Most Significant Change – this approach will enable the capturing of personal accounts of change firsthand from the key stakeholders for us to assess the program’s impact over time.
Developmental Evaluation – the dynamic nature of this approach is particularly suited for this program evaluation as the program is a newly designed and launched course. This approach allows for the measurement and tracking of outcomes quickly as they emerge, providing insights for key learnings through rapid feedback and enables the academy to respond and fine tune the course accordingly.

The following data collection methods have been selected to gather data:

Structured Parent interviews and focus groups – parents are invited to participate in a focus group interview or one-on-one interview based on their availability. Parents are asked a set of structured questions and their responses captured in a spreadsheet.
Teacher Diaries – Teachers were asked to keep a diary about each student and to capture data and insights immediately after each lesson. In particular, teachers were asked to observe significant changes in students’ learning as well as capture instant feedback gathered from parents when they came to pick up their children.

Participants

Parents whose children were enrolled in the program were invited to participate in the focus group interviews. For those who were not able to attend the focus group due to scheduling issues, informal conversations were conducted instead to gauge their sentiment toward the Program. Teachers were also key stakeholders in the data collection exercise. Teachers were asked to record student observations and parent feedback in the Teacher Diaries.

Procedures

Parent Interviews

For the structured parent interviews, invitations were first sent out to all parents to check their participation response and availability. They were offered a choice of 2 focus group sessions to cater to their busy schedule. All interviews were conducted in Cantonese language and notes were later translated into English for this evaluation report.

Teacher Diaries

For the Teacher Diaries, a collaborative Google Sheet was created with a matrix organized by students and teachers. Since there may be multiple subject teachers teaching the same student, the matrix helped to map and provide a more holistic view of each student’s progress by subject matter.

Data Consolidation

Interview Design & Analysis

A series of questions were developed for the focus group interviews to collect data across teaching quality, teacher communication, class activities, curriculum, campus facilities, logistics, and others. Once all the collected data has been categorized and sorted, the data were analyzed to derive insights and actions. As the data collected was qualitative data, sentiment analysis was used as an indicator to determine if the sentiment behind each response was positive or negative. A further layer analysis was applied to all the responses to determine if the response was a suggestion for something new, a praise for something done well, or a critique for something not done well and needs improvement. The “Start-Keep-Stop” analysis can be summarized as follows:

Start – A suggestion to begin doing something that was not thought of previously. Identifies a gap in student needs that has not been previously addressed and hence an opportunity to “start doing.”
Keep – A positive response indicating that something was done well or right and should be kept and continued.
Stop/Improve – A negative response indicating that something was not working well and that the practice should be discontinued or improved.

Findings and Limitations

After analyzing the data collected, the findings were tabulated and synthesized. Recommendations were then made based on the findings of each category.

In reviewing the program evaluation execution, there were areas that could have been improved especially for future studies. In facilitating the focus group and parent interviews, better probing techniques can be utilized to draw out the quieter subjects while the subjects who tend to speak more than others can be reined in to allow for the quieter ones to speak up.

Better instructions initially would have also helped with the quality of the data collected from the Teacher Diaries. Some teachers who were more experienced made and captured more comprehensive observations while some could have benefitted from more regular check-ins and guidance on what to capture.

The voice of the Administration team was not initially included in the Teacher Diaries but was later added as a valuable data source as they have frequent contact with parents and are on the frontlines of parent communication.

Final Recommendations

Based on the overall positive feedback from parents, it is concluded that the English Immersion Program is a commercially viable product with a unique value proposition, market position, and potentially strong reception by parents. The multi-subject program designed to provide frequent immersive language exposure to students is also effective in fostering academic excellence while instilling the social-emotional and life skills needed for children to succeed in a 21st-century society. It is recommended that the program should be continued, strengthened, and officially launched to the market.

References

Guijt, I. (2014). Participatory Approaches, Methodological Briefs: Impact Evaluation 5, UNICEF Office of Research, Florence. Retrieved from: http://devinfolive.info/impact_evaluation/img/downloads/Participatory_Approaches_ENG.pdf
Robson, C. (2017). Small-scale evaluation: Principles and practice. Sage Publications.