Methodology matters

We know that educational inequalities set in before children start school. At age five, there is already a 19-month difference in school readiness between the very richest and the poorest children, a gap that only widens and becomes more entrenched throughout school and later in life.

The Sutton Trust is one of several organisations to argue that investing in high-quality early years education is one of the best ways to close the attainment gap before it takes hold. Their position is rooted in evidence; research has consistently shown how important good early years’ experiences are for later academic outcomes, especially for the poorest children.

But in February, academics at the London School of Economics (LSE) published an investigation into the effects of pre-school on children’s development. Their findings contradicted much of the existing research that has shown quality pre-school and early experiences have significant positive effects on outcomes. It was disheartening to see the results seized upon by some as an excuse to reduce spending in a chronically under-funded sector.

If the weight of evidence tells us that early years education from qualified practitioners does make a difference, why has this new piece of research drawn such very different conclusions?

The devil is in the detail of the research methods.

The LSE paper analyses information on about 1.8 million children born between September 2003 and August 2006 and uses data to link the development of children (aged five and seven) with the quality of their pre-schools, measured by the child’s access to a degree-level teacher and by Ofsted inspection ratings.

This large number of children analysed is the paper’s unique selling point, but it comes at the price of weak methodology.

Here are the four main reasons why:

• The researchers used the teacher-assessed EYFS Profile (EYFSP) at the end of Reception to measure outcomes.Despite attempts to ensure consistency, there are inevitable problems when considering assessments made by thousands of teachers. Other research uses more precise measures, including age appropriate tests, administered by highly trained assessors.
• The researchers only looked at each child’s assessment at the end of Reception and not their characteristics when entering the system. This means it’s impossible to calculate the ‘value added’ by their early years education. Other studies avoid this by collecting data at the start of the preschool period and using it as baseline for measures at the end.
• The Ofsted measures of quality have the potential to be up to six years adrift. Given this time lapse and interim changes, Ofsted ratings may inaccurately reflect quality or vice versa. Other studies use more differentiated measures (like the Early Childhood Environment Rating Scales (ECERS)) that were taken when the children in the statistical analysis were present in the settings.
• For decades, research has shown that family background has the largest influence on children’s development. This paper uses only two measures for this: free school meals and neighbourhood disadvantage via post codes. Other research uses more nuanced measures, including parents’ qualifications, salary, family size and the home learning environment (HLE). Studies that consider the effect of background before estimating the effects of pre-school quality are therefore better at demonstrating pre-school effects ‘net’ of background.

For these reasons, we think the limitations of the LSE papers’ data and design mean it is unable to provide reliable results on pre-school effects and underestimates the effects of pre-school quality and staffing.

By contrast a recent review of the international literature found consistent, although modest, effects of pre-school quality on children’s developmental outcomes. The observations reported in the review required highly trained researchers using validated research instruments. Moreover, the child outcomes were finely differentiated tests and rating scales, not broad brush teacher assessments.

Another recent study found an association between teacher qualifications and systematically observed quality in England and Finland, but not in other countries.  Again, the measure of quality was systematically observed by trained researchers using research validated instruments.

I would urge anyone involved in policymaking or the early years to put the LSE study in the context of the wider research base – a research base that overwhelmingly tells us that good quality early years education matters. If we are to improve social mobility, that evidence says we must find ways to achieve high quality. There is only one thing more expensive than investing in high quality early education – and that is NOT investing in it!

Kathy Sylva is Professor of Education at the Department of Education at the University of Oxford.