It turns out the author matter
An exploration into AI, authenticity and who really wrote that essay.
It's Friday evening, the research digest has landed, and I'm on my second cup of tea. Top of the list this week is a proper randomised trial of AI tutoring in real UK classrooms. Ten state schools, nearly 1,200 pupils, a whole term. Written up by Google's own AI education team and the maths platform Eedi. I'll be honest, my first reaction was "oh, that's good news". AI tutors helping pupils fix their mistakes about as well as an expert human tutor? In a profession where a decent human tutor costs more per hour than my car insurance costs per month, that's quite a headline.
So I made a note, moved on to the second item, and that's where my evening went a bit sideways.
The second item was an audit of the research that sits behind all those "AI boosts learning" claims. And by the time I'd finished reading it, I had to go back and read the first study again, with a very different pair of glasses on. This post is about that second read, and about why I think it matters far more for universities than it does for a maths app.
Study one: AI tutors in UK maths lessons
Here's what Eedi and Google did. Pupils in Years 8 to 10 practised maths on Eedi's platform for one lesson a week, with their teacher in the room. Every time a pupil got a question wrong, one of four things happened, depending on which group they'd been randomly put in:
- the platform's normal "here's where you went wrong" content
- a live text chat with an expert human tutor
- an AI tutor, with a human checking and approving every message before the pupil saw it
- the same AI tutor, but given a summary of the pupil's past mistakes and habits
And the results? Pupils who got any kind of tutoring, human or AI, answered the follow-up questions correctly about 56 to 58% of the time, compared with 43% for the normal content. AI and humans were basically level. Human tutors approved about 95% of the AI's messages without changing a word. Lovely.
Then there's the bit you have to scroll down for. The study's main measure, the one they registered in advance before the trial started, was a standardised maths test (Star Maths) taken at the start and end of term. On that test, none of the tutoring groups did any better than the pupils who just got the normal content. Not the AI. Not the humans either. A term of tutoring after every mistake, and the bigger test didn't move.
Study two: 39 meta-analyses, 39 positive findings
Now the audit. Patrick O'Neill, a community college professor in Indiana, went through 14 of the 39 published meta-analyses on AI in education. If you've not come across them, meta-analyses are the studies of studies. They pool lots of smaller experiments into one big number, and they're the thing policy-makers, senior leaders and conference keynotes love to quote. All 39 of these found a substantial benefit from AI. Every single one. That alone should make a scientist's eyebrow twitch.
What O'Neill found, as reported by Jill Barshay at the Hechinger Report, was a bit of a mess:
- Impossible statistics. Numbers that simply can't come out of the data they claim to come from.
- Made-up references. Some of the underlying studies cited papers about generative AI that were apparently published before generative AI was publicly available. Some reference links led to completely unrelated studies. Hallucinated references, in other words, in papers arguing that AI is good for learning. You couldn't write it.
- Comparisons that don't measure AI at all. One study compared nursing students who had three hours a week of study sessions (with an instructor who used AI) against students who had no study sessions. Shockingly, extra teaching helped. - A stacked deck. About two-thirds of journal calls for papers on AI in education specifically asked for research on its benefits. O'Neill's phrase for this is "manufacturing evidence", which is about as polite as an academic gets.
These 14 papers were published by Springer Nature, Wiley, SAGE and Taylor & Francis. They've been cited more than 3,000 times. One has already been retracted.
O'Neill is now running his own meta-analysis using only well-designed experiments. There are 44 of them so far, and the results range from very helpful to very harmful. The preliminary average? Roughly zero.
A quick health warning. The audit is a preprint, so it hasn't been through peer review yet (which, given what it's saying about peer review, is a little ironic). The re-analysis is preliminary. And "roughly zero on average" still leaves room for AI helping in some situations and hurting in others, which is exactly what you'd expect from a tool.
So, back to study one
Reading the Eedi study a second time, I noticed things I'd skated over the first time.
It's a company evaluating its own product, with Google's own team as co-authors. It isn't peer reviewed. The positive results are on the "secondary" measures: answering questions just like the one you got wrong, straight after a tutor has walked you through it. The measure they chose in advance as the real test of learning came back flat. And when you think about it, that sits rather neatly with O'Neill's "roughly zero" for well-designed experiments.
Now, to be fair to Eedi, and I do want to be fair, they published the null result. They registered the study in advance, they reported the test that didn't go their way, and they're running a much bigger trial now. That's more honesty than a lot of the 39 meta-analyses managed. If anything, the first study comes out of this looking quite good. The second study simply taught me how to read it. Look past the headline, find the outcome they said mattered before they started, and check whether that one moved.
Which is a skill. A slow, effortful, slightly boring skill that you only build by doing it.
Anyone can write an essay now
And this is where my mind went straight to universities.
If peer reviewers at major academic publishers can wave through references to papers that don't exist, what chance does a marker have with 200 scripts and a Sunday evening deadline? We already know the answer. In 2024 a team at the University of Reading quietly submitted exam answers written entirely by GPT-4 under fake student names. Markers didn't spot 94% of them, and on average the AI answers got better grades than the real students. That was GPT-4. The models have moved on a fair bit since.
Meanwhile HEPI's 2026 survey found that 95% of UK undergraduates now use AI, and 94% use it to help with assessed work. So the essay, the thing universities have used for well over a century as proof that a student can think, now proves that a student has access to a laptop.
I spend a fair bit of my working life running workshops on AI and academic misconduct, and I can tell you the honest version: detection is not the answer. AI detectors are unreliable, they flag the wrong people, and every hour we spend trying to catch students is an hour we're not spending on the actual problem. Students producing a polished essay without thinking is a symptom. The real problem is that we built an assessment system where the polished essay was the only thing we ever looked at.
Assess the thinking, not the essay
So what do we do? I think the answer is the same lesson I learnt from reading that first study twice. Stop judging the shiny final product and start looking at the process that produced it.
That's what portfolio assessment does. Instead of one essay handed in at the end, students build up a body of evidence over time: early drafts, notes on the sources they found (and the ones they rejected), reflections on what changed their mind, feedback they received and what they did with it, and a conversation where they explain and defend their choices. You can't fake a semester of thinking the night before a deadline. Well, you can try, but it's a lot more work than just doing the thinking.
It also lines up nicely with what we saw in my last couple of posts. AI that does the thinking for you makes you worse once it's gone. AI that asks you questions and makes you have a go first doesn't seem to do the same damage. Portfolio assessment rewards the second kind of use and makes the first kind pointless.
The catch, and every teacher reading this is already thinking it, is workload. Portfolios are lovely in theory and a nightmare to mark at scale. Which is exactly the problem I've been working on.
Where Rheton comes in
This is where we can help. At Obline Education, we've built a tool called Rheton that captures the whole process of writing an essay, from the first ideas right through to the finished piece. Students discuss the papers they've found in an online forum, link ideas together, plan their essay and then write the final piece, all in one place. Once the process of creating the essay is out in the open, the final product becomes a lot harder to fake. If you'd like to see it in action, you can [try the demo](https://obline.org/demo).
I'm not pretending it fixes everything. But I'd much rather build tools that make the thinking visible than spend another year arguing about whether a paragraph "sounds like ChatGPT".
Back to that Friday evening
So here's where I ended up. The first study looked like good news. The second study made me read the first one properly, and once I did, I found a careful trial with an honest null result hiding under a positive headline. Nobody handed me that reading. I had to go and do it.
That's the thing I want our students to be able to do, and it's exactly the thing an essay can no longer prove they can do. So let's stop marking the essay and start marking the reading. Go on, have a look at both papers yourself, and see whether you read the first one differently the second time.
Read more: - [Measuring the impact of AI tutoring, from near transfer to standardised tests: an expanded RCT in UK mathematics classrooms](https://services.google.com/fh/files/misc/learnlm_oct26.pdf) (Google LearnLM team and Eedi, October 2026) - [39 meta-analyses, 39 positive findings: an audit finds serious flaws in research on AI in education](https://hechingerreport.org/proof-points-ai-education-meta-analysis/) (Jill Barshay, Hechinger Report, 26 October 2026), and the [audit preprint](https://osf.io/preprints/edarxiv/n29zd_v3) by Patrick O'Neill - [A real-world test of artificial intelligence infiltration of a university examinations system](https://doi.org/10.1371/journal.pone.0305354) (Scarfe et al., 2024, PLOS ONE) - [Student Generative AI Survey 2026](https://www.hepi.ac.uk/reports/student-generative-ai-survey-2026/) (HEPI and Kortext)