Research Rationale
Teaching for Mastery first began in 2015, and since then, the NCETM’s Maths Hubs programme has
continued to grow. At NEHS Maths Hub, we have been working with 67% of all primary schools and
56% of all secondary schools to implement a Teaching for Mastery approach to Mathematics
education. We aim to work with at least 75% of all primary schools and 65% of all secondary schools
by 2025.
Recently, Maths Hub Lead, Becky Shepherd, has carried out some research looking at local secondary
schools who have implemented Teaching for Mastery. The aim of the research was to trial non-experimental methods to address whether Teaching for Mastery is associated with improved
assessment scores for students.
When planning this research, the aim was to use GCSE Maths results for schools that implemented
Teaching for Mastery in 2015-16 or 2016-17 with year 7 students and thus, investigate the long-term
impact. These year 7 students were due to sit their GCSEs in 2019-20 and 2020-21 respectively.
However, the global pandemic prevented any analysis of GCSE Maths results – but this can come in
the future as the effects of the pandemic on education continue to waver.
Research Methods
As it was not possible to use GCSE Maths results, the research analysed internal school assessments for different year groups using an observational approach. Within educational research, randomised controlled trials (RCTs) are usually deemed the ‘gold standard’ but as an experimental approach, they often lead to narrow foci, ethical concerns and trial effects.
By adopting an observational approach, schools were able to provide data that already existed and therefore there was no risk of ethical concerns and trial effects. Schools were encouraged to provide data for cohorts of students that had been taught using Teaching for Mastery approaches for two or
more years so that a long-term effect could be explored to assess if Teaching for Mastery can be associated with improved assessment outcomes. With the data, propensity score matching methods were employed.
Propensity Score Matching (PSM), first introduced by Rosenbaum and Rubin in 1983, is an observational approach which analyses existing data in seeking to estimate the effect of a policy or intervention. When random assignment is not feasible, PSM creates matched groups that are equal on propensity scores (Robinson et al., 2014, Henson, Hull, and Williams, 2010). According to Robinson et al., (2014), “a propensity score is defined as the conditional probability that an individual would be assigned to the treatment condition, given a set of relevant covariates” (p. 344).
Covariates are variables that may otherwise affect student achievement and group allocation, such as pupil premium status, gender and prior attainment.
By matching on a composite score, PSM accounts for the covariates that predict the treatment group and also relate to the outcome. Therefore, the method endeavours to reduce bias by balancing confounding variables across the two groups. Results obtained from experiments using PSM
methods have been found to be closely aligned to those from RCTs (Becker & Ichino, 2002). Full details on how propensity score matching was used for the research can be found in the published thesis (Shepherd, 2023).
Findings
Three secondary schools that are in the ‘sustaining’ phase of Teaching for Mastery participated in the research. Of these, two of them were able to provide data that preceded the global pandemic, so that conclusions could be drawn as to whether adopted a Teaching for Mastery approach to mathematics teaching and learning impacts on student results.
For the purposes of this article, these two schools will be called school Y and school Z. Within school Y and school Z, internal assessment scores of two different cohorts of students were compared using propensity score matching methods. The assessments were consistent for the two cohorts, and this was an important criterion for a school to be involved in the research.
The schools provided data on the following key covariates on each student for both the Teaching for
Mastery and non- Teaching for Mastery cohorts: prior assessment score, gender, ethnicity, pupil premium, SEND and EAL status. Propensity scores were generated for each student and through rigorous matching, two ‘matched’ groups were created that were equal on propensity scores and therefore balanced in their covariates which could otherwise explain a difference in assessment scores. The matching process followed the steps set out by Leite (2016).
Findings - continued
For school Y, following propensity score matching and analysis of a year 10 Maths assessment, it was found that the students who had been taught using Teaching for Mastery approaches outperformed the cohort who had not been taught using Teaching for Mastery approaches by 5.3%.
For school Z, two analyses were conducted: one on year 7 assessment data, and one on year 10
assessment data. The treatment effect of Teaching for Mastery was found to be 6.5% following the
propensity score matching and analysis on the year 7 assessment data. For the year 10 assessment
data, the students who had been taught using Teaching for Mastery approaches outperformed the
students who were not taught using Teaching for Mastery approaches by 4%.
Where Next?
It was difficult to find schools to participate in the research who had transitioned to a Teaching for Mastery approach to mathematics education but had kept their assessments constant. To adopt an evidence-based approach to Teaching for Mastery, it’s important to try to keep assessments the same so that the impact can be captured.
The results outlined above must be viewed with caution. School data is incredibly noisy, and whilst the listed key covariates above were controlled and balanced between the two groups, there may have been other factors that influenced the difference in assessment scores. That said, the covariates used have all been found to be highly influential on student achievement (Sammons et al., 2014).
The research paper concluded that propensity score matching can provide a useful way to analyse the effect of changes in curricula or new initiatives, and therefore for any schools about to embark upon the Teaching for Mastery journey through their local Maths Hub, it is worth considering if internal maths assessments can remain in their current form so that the impact can be quantitively assessed.
References
Henson, R. K., Hull, D. M., & Williams, C. S. (2010). Methodology in our education research culture toward a stronger collective quantitative proficiency. Educational Researcher, 39(3), 229–240.
Leite, W. L. (2016). Practical Propensity Score Methods Using R (1st ed.). SAGE Publications, Inc.
Robinson, J.T., Fischer, L., Wiley, D., & Hilton, J. (2014). The Impact of Open Textbooks on Secondary
Science Learning Outcomes. Educational Researcher 43(7), 341-351.
Rosenbaum P.R. and Rubin D. B. (1983). The central role of the propensity score in observational
studies for causal effects. Biometrika. 70(1), 41-55. https://doi.org/10.1093/biomet/70.1.41
Sammons, P., Sylva, K., Melhuish, E., Siraj, I., Taggart, B., Toth, K., & Smees, R. (2014). Influences on
students’ GCSE attainment and progress at age 16 Effective Pre-School, Primary & Secondary
Education Project (EPPSE). Department for Education.
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_
data/file/373286/RR352_-
_Influences_on_Students_GCSE_Attainment_and_Progress_at_Age_16.pdf
Shepherd, R. (2023). Approaches to Evaluating Teaching for Mastery in Secondary Mathematics
Education. Masters thesis, Durham University.