Judging the SNP on Their Education Record

*************************

Note: this blog post is now more neatly summarised here: Scottish Education: A Generation, Failed

*************************

"Let me be clear – I want to be judged on this. If you are not, as First Minister, prepared to put your neck on the line on the education of our young people then what are you prepared to. It really matters." 
Nicola Sturgeon, August 2015

"the statistics also show a drop in writing performance for S2 pupils which is of particular concern [..] the SSLN statistics are disappointing"
John Swinney, May 2016

"The figures for Scotland do not make comfortable reading [..] compared to 2012, our performance in science and reading has fallen.  In science and maths we are now below the levels at which we performed in 2006, and more countries have outperformed Scotland in all three areas than at any time since PISA began."
John Swinney December 2016

Nicola Sturgeon has asked to be judged on her record on education and the SNP has been in charge of our fully devolved Scottish education system for roughly 10 years. Responsibility for the educational outcomes we're now seeing must lie with the SNP - and the trend in those outcomes is simply terrible, as Cabinet Secretary for Education and Skills John Swinney has had to admit.

The most recent international benchmark data provided by PISA¹ covers the period from 2006 to 2015 (it's a survey that happens only once every three years). PISA surveys 15-year-old pupils, so the most recent data shows results for pupils who had been in the Scottish education system for 7 years under an SNP administration.

The graphs below are pretty clear. Since the SNP came to power, in each of the three main disciplines as measured by PISA (mathematics, reading & science), Scotland's performance has declined significantly. This is true in absolute terms, relative to the whole of the UK and relative to the average for all OECD countries.

So in maths and reading, where Scottish pupils used to perform significantly better than rUK students, latest available data shows we've dropped back to average at best. In science, where Scottish pupils used to perform in line with the UK average, their performance in the most recent survey is very significantly worse. In all three disciplines, we've gone from out-performing the OECD average to being - well - distinctly average.

Looking at our performance in international league-table format: in all three subject areas we've dropped below the OECD average and into the bottom half of the table.

As the Scottish Government themselves make clear, "PISA is the major international study of pupil performance in which Scotland participates" - it is their chosen method of benchmarking our educational attainment.

In fact, having withdrawn from the other globally recognised assessment body (TIMSS & PIRLS) in 2011, it's now the only data we have to compare our performance on an international basis. It's good quality data¹ and the results are a damning indictment of the SNP's performance as custodians of our education system.

***

There is one other way we can objectively assess the performance over time of the Scottish education system and that's the Scottish Government run Scottish Survey of Literacy and Numeracy² (SSLN). This doesn't offer international comparisons, but it does allow us to observe trends over time on a like-for-like basis and - among other things - look at the "Attainment Gap" (the extent by which pupils from the most deprived backgrounds under-perform those from the least deprived).

SSLN focuses on numeracy and literacy in alternate years, so we have numeracy data for 2011, 2013 & 2015 and literacy data for 2012, 2014 & 2016. Pupils are assessed at three different stages of education: P4 (age 7-8), P7 (age 10-11) and S2 (age 13-14). For each of these groups the graphs below show trend in the percentage of pupils assessed as performing "well" or "very well" in the subject, and splits these by deprivation status (most deprived, middle, least deprived).

Taking numeracy first:

---

To read these graphs: the dotted black line shows the overall performance over time, the gap between the red (least deprived) and green (most deprived) lines illustrates the Attainment Gap.

---

So the available data for numeracy shows overall standards declining slightly, although stabilised (at a very poor level) for S2. The largest Attainment Gap exists at the S2 level and there's no evidence of it closing. The Attainment Gaps at both P4 and P7 are growing significantly, and the decrease in attainment for the most deprived pupils at P4 level (the earliest measured stage) is dramatic and concerning.

Now looking at literacy performance:

---

**
[03/06/2017] NB: as has been helpfully pointed out by James McEnaney, the graphs below are for the "reading" element of literacy only (result of me not paying enough attention to the data table structure, my bad: assumed would be a single measure as the numeracy data is). When I have time I'll crunch the "writing" data ["listening & talking" data only exists for two years]. Full correctly labelled set of graphs is now >here
**

---

Overall attainment levels are broadly stable, although there is some evidence of decline at P4. The Attainment Gaps for literacy are less severe than for numeracy. There is no evidence of any sustained reduction in the Attainment Gap at S2 or P7 - and at P4 the Attainment Gap is widening (primarily due to a decline in performance of pupils from the most deprived areas).

Having looked at all these graphs now, the poor level of overall numeracy attainment at S2 level is striking: is our education system producing a generation soon to vote who aren't great at adding up?

Combine the SSLN findings with the PISA results and a clear picture emerges: on the SNP's watch, overall standards in Scottish education have dropped and pupils from the most deprived areas have been the ones to suffer the most.

You might think it's going to be interesting to see what the next SSLN survey data shows, particularly as it would be on numeracy again, the area where the most worrying trend has been observed. Well it turns out we'll be unable to objectively judge Sturgeon's record on this because - with a spectacular disregard for the value of consistently audit-trailed performance data - her government have chosen to stop the SSLN survey altogether. To coin a rather poignant phrase in this context: you do the math.

***

In her recent grilling by Andrew Neil on BBC1, Nicola Sturgeon delivered a masterclass in obfuscation and misdirection when pressed on this topic. She brushed off references to the latest available international benchmark data as "from two years ago" and airily dismissed her own Government's survey of over 10,000 pupils as somehow irrelevant "because it's a sample survey". She even had the brass neck to go on to argue that "there's real progress been made".

---

The Spectator's Fraser Nelson has written an excellent blog unpicking the detail of Sturgeon's blustering in that interview: "Fact-checking what Nicola Sturgeon told Andrew Neil about education" - I heartily recommend it.

---

Sturgeon's argument for dismissing the SSLN data and discontinuing the survey was that it doesn't give meaningful data at a school-by-school level. That is a frankly ridiculous argument that would only make sense if anybody was trying to use the data to judge specific schools or local authorities - they're not.

SSLN was designed to give an overall assessment of the quality of literacy and numeracy education in Scotland, and it does that well. The (Scottish Government commissioned) OECD report "Improving Schools in Scotland: An OECD Perspective"cites the SSLN data fully 27 times and at no point suggests the data is in any way not fit-for-purpose³.

It's hard not to conclude that the SNP's decision to stop the SSLN is motivated by the fact that it exposes failings on their part - and that they're fearful of what the next ones would show.  Put it like this: do you think they'd scrap it if they thought the upcoming surveys were going to show how successful their reforms have been?

It gets worse. The Scottish Government states "New statistics on literacy and numeracy performance will be available annually from the teacher professional judgement data collection". So not only are we losing the ability to track performance versus prior years on a like-for-like basis, we're moving from largely objective test-based measurements to subjective measures "based on teachers' professional judgements". It's almost as if the SNP are going out of the way to avoid being judged objectively on their performance.

***

So how does the SNP attempt to put a positive spin on this?

In that Andrew Neil interview Sturgeon used the following three arguments, none of which withstand scrutiny

1. "If you take Level-5 qualifications which are broadly equivalent to the old O-Grades and Standard Grades [..] we've seen the gap between the richest pupils and the poorest pupils almost half"

I assume here she's referring to the National Improvement Framework: 2016 Evidence Report which includes this analysis

If I'm right, then she's dismissing robust PISA data from 2 years ago but seizing on a snippet of data from - er - 2 years ago. Either way, the measure of "one or more qualifications as SCQF Level 5 or better" is laughably crude compared to the SSLN data.


2. "If you take the tariff scores in Scottish education which measure not just the quantity of qualifications that young people get but the quality as well, it shows that performance in the top 20% has improved by about 9% but the performance in the bottom 20% has improved by 26%"

The data she quotes clearly comes from the National Benchmarking Overview Report 2015/16:

This is certainly a measure of something, and quite possibly of something to be applauded. But the report itself offers some pretty heavy caveats about the crudeness of the measure: "measures of both attainment and outcome for education are presently crude and need developed [..] “Total tariff score” needs controlled for the year in which young people left school so that we compare like-with-like". So this crude measure that confesses not to be offering like-for-like figures is in no way an alternative to the SSLN data, which it appears to directly contradict.

The Tariff Score is explained as "a summary measure of the number, level and grade of qualifications children had achieved during the senior phase". A bit more digging tells us its based on "the attainment of all pupils who left school that year after either S4 S5 or S6 by level of deprivation. Each qualification attained by a pupil is awarded tariff points based on its SCQF level and credit points. Points are also based on the grade of award achieved."

There are two glaringly obvious weaknesses with this measure compared to the SSLN data

• It only offers a crude measure of what happens at the end of the education pipeline, it offers no insight into issues in the pipeline. Take the P4 numeracy issues exposed by the SSLN survey: these won't translate into "tariff points" for another 8 years or so!
• The impacts of mix (for example if students are encouraged to take easier subjects) and grade inflation (which may differ by grade within subject as well as across subjects) will be near impossible to allow for

To seize on this tariff data as proof of something good happening (which it might be) while simultaneously dismissing the transparently more robust and strategically valuable  SSLN data is simply indefensible.

3. "we're seeing more young people coming out with Highers and Advanced Highers, we're seeing more young people going in to University, we're seeing the positive destinations of young people continuing to increase"

Claims on numbers of Higher and admissions to University (which I don't have the energy to check, but believe) in no way counter the core issue of declining educational standards highlighted by PISA and the chronic Attainment Gap problem as shown in the SSLN data. Those surveys provide like-for-like comparisons over time, crudely counting number of Higher passes or number of people getting in to University obviously doesn't.

As for "positive destinations", this appears to be little more than a proxy for "not being unemployed" given the term is defined as including "higher education, further education, training, voluntary work, employment and activity agreements"

***

So Sturgeon did a good job of spinning her way through the interview, but the underlying issue can't be spun away so easily.

Don't get me wrong: I believe that Sturgeon and her colleagues care about education and I believe that after nearly 10 years in power they've finally really started trying to improve things. But I also believe that it is not and never will be their main priority because - as the party's constitution and Sturgeon herself has made very clear - for them, independence transcends all.

If you want proof of the SNP's culpability (beyond that provided by the PISA and SSNL data), read this report on John Swinney's appearance in front of Holyrood's Education Committee, by Angus Howarth in the Scotsman:

Cuts in the number of teacher training places in Scotland “probably” went too far, the Education Secretary has conceded. John Swinney said that with hindsight, the target intake for student teachers was “probably over-corrected too far” in 2011.

In 2005-06, before the SNP came to power, the target number was 4,437, but it then went from 3,857 in 2009-10 to 2,307 in 2010-11 - a drop of 1,550 places.

Alternatively take a look at our old friend the Scottish Government's GERS figures. As I've pointed out before, the SNP relatively reduced per capita spending on education (relative to the rest of the UK) in their early years. This is all about priorities: in other areas like policing they chose to spend more.

Of course this graph shows that we've continued to spend more per capita on education than the rest of the UK - but our population density is 80% lower than the rest of the UK and we have extensive island communities to serve, so we'd expect that. What's undeniable is that the SNP reduced spend relative to the rest of the UK (so "Tory austerity" can't be blamed) and have only very recently looked to return to the levels of spend they inherited (in a move some of us applauded at the time).

[A more compete analysis of spend, teacher numbers, class sizes etc. can now be found >here]

Ultimately its about priorities and education simply hasn't been a priority for the SNP until very recently. I fear that hollow sound you can hear is a stable door being shut on an empty stable.

---

It would be unfair to suggest the SNP have neglected our children entirely. Children are future voters after all, so when it comes to marketing "brand SNP" in schools, we should give them their due: they do a great job

---

***

There can be few responsibilities greater for a party of government than to ensure a country's children receive the best possible education, to provide the next generation with the best chances to succeed in life.

Tony Blair swept to power in 1997 promising to make "education, education, education" his top priorities in office; the Scottish education system appears to be suffering the effects of having an SNP government in power with "independence, independence, independence" as theirs.

Young Scots emerging from education with reduced life-chances are the ones paying the price for the SNP's failings today. The question is, will the SNP start paying the price at the ballot box tomorrow?

***

---

Notes

---

1. Programme for International Student Assessment (PISA)

The global benchmark standard for education is provided by the OECD's PISA:

The Programme for International Student Assessment (PISA) is a triennial international survey which aims to evaluate education systems worldwide by testing the skills and knowledge of 15-year-old students. In 2015 over half a million students, representing 28 million 15-year-olds in 72 countries and economies, took the internationally agreed two-hour test.

As the Scottish Government website explains:

PISA seeks to measure skills which are necessary for participation in society. Accordingly, it assesses how students apply the skills they have gained to the types of problem they may encounter in work or elsewhere. Pupils are assessed at the age of 15 as this is regarded as a reasonable point at which to test the impact of compulsory education throughout the developed world (most PISA 2012 participants in Scotland were attending S4).

I have used the PISA "data explorer" tool to download and play with the data that's available as a consistent time-series. Fortuitously when it comes to judging the SNP's record, this means our data set starts in 2006, the year before the SNP came to power.

The data is survey based and provided as score averages. As with all good statistical analysis, standard error figures are given. This means our graphs can include vertical bars to show the range within which we can be 95% confident the true figure lies.

A minor irritation is that I've not been able to clarify if the UK data includes Scotland - by implication it does, but either way it doesn't change the overall message (in fact at worst if UK does include Scotland then it dampens the severity of the observed relative trend).

More detail on Scotland's PISA results can be found on the Scottish Government website.

2. Scottish Survey of Literacy & Numeracy (SSLN)

As the Scottish Government website explains:

The SSLN was a sample-based survey which monitored national performance over time in literacy and numeracy at P4, P7 and S2. It provided a snapshot of Scotland's achievement in literacy and numeracy at a specific point in time and allowed for comparisons over time to be made. The information from the survey informed the development of dedicated resources to facilitate improvements in learning and teaching.

Whilst the data is survey based, it qualifies for National Statistics certification and rigorous statistical techniques are used to to ensure the data is presented with meaningful confidence intervals. As the latest report explains in helpful detail

As in all sample surveys, as the SSLN is based on a sample of pupils rather than on the
whole population, the results shown are estimates. Therefore there is an element of
uncertainty within the results because the pupils sampled may not reflect the population
exactly.

Uncertainty around the results is estimated using standard errors. Standard errors are a
measure of the variation in the data i.e. how each observation differs from the mean. As
the SSLN sample design is not a simple random sample - pupils at small schools have a
higher probability of being selected than pupils at large schools - this means that standard formulae used to calculate the standard error from a simple random sample would not be appropriate. Standard errors are therefore calculated empirically using the jackknife procedure.

Standard errors are in turn used to produce confidence intervals around the estimates.
Confidence intervals show the range of values within which one can be reasonably
confident that the actual value would lie if all pupils were assessed. Ninety-five per cent confidence intervals for the main national estimates were calculated and were around ± two percentage points. This means that the true value of each estimate is likely to lie within two percentage points either side of the given estimate.

Where appropriate, confidence intervals are represented on charts by error bars to help
demonstrate this level of uncertainty. Where the estimates are different but the error bars
overlap we cannot be sure that the true values of each estimate are statistically
significantly different from each other. Significance tests (t-tests) are used to assess the
statistical significance of comparisons made.

3. The only observation the OECD make relating to limitations of the SSLN data is "The SSLN sample size is too small to be disaggregated to the local authority level." . This is true, but a reason to gather more comprehensive data as well as the SSLN, not instead of.