Can AI actually help us learn, or is it just making us worse at asking questions?
A new randomised trial at the University of Maryland gave 2,379 students access to a GPT-4o study assistant built into their LMS. The results weren’t what the AI evangelists would hope for.
Once, as a kid, I asked our retired neighbour, a man who had spent forty years as an engineer, how our kettle knew when to switch itself off. I expected a sentence like ‘it has a temperature sensor in it’ so that I could rush home and show off my newfound knowledge. He put down his tea, fetched a screwdriver and said, “Right, let’s have a look.”
An hour later we were sitting at his kitchen table with the kettle in pieces between us. He had shown me a tube that could register pressure differences, the heating element, and all kinds of magic electronics, and suddenly I was being told about stuff that I hadn't even considered. Why does water turn into steam? What is heat, really? Why does steam do the job better than a thermometer would? I left with a kettle I couldn’t put back together and a head full of better questions than the one I arrived with. It is, hopefully, an experience we can all relate to: a simple question turns into a much more complicated one almost the moment you ask it, and you walk away with far more than you bargained for.
But now we have AI, and if you believe Sam Altman and his fellow CEO crew, AI will have the answers to all of these questions (despite getting the kettle explanation wrong when I asked it for help writing this article). One of the jobs it will supposedly get rid of is the tutor, all to be replaced with AI chatbots. But is that actually the case?
I’ve been looking into a brand new working paper from the University of Maryland that claims access to an AI chatbot actually reduced students’ grades. It was released this month through Brown University’s EdWorkingPapers series and hasn’t been peer-reviewed yet, so we need to take these findings with a massive fistful of salt. Oh, and these findings aren't hurtful to my personal brand at all, so I’m fine with these results…
What did they actually do?
This is a big study by education research standards. During the autumn 2025 semester, the researchers ran a randomised controlled trial with 2,379 undergraduates and 30 instructors across 34 course sections. The courses came from 11 of the university’s 12 colleges and covered everything from biology, computer science and maths to English, communication, education and public policy, from first-year intro modules up to final-year courses.
The clever bit is how they randomised. Rather than giving the chatbot to some students in a class and not others (who would then just share it with their mates), they randomised whole instructors. Instructors teaching the same or similar courses were grouped together, and within each group some were randomly given the tool and some weren’t. If your lecturer got it, your whole class got it. For the cleanest comparison, they also looked separately at an “exact-match” group of 22 instructors and 1,353 students, where the two sides were teaching sections of literally the same course with the same curriculum and assessments.
The tool itself, which they called a Virtual Study Assistant, ran on GPT-4o and sat in the sidebar of the university’s LMS. It used a RAG (or 'fact lookup') system to answer questions from the course materials the instructor chose: pages, PDFs, slides and even lecture video transcripts. Instructors could see their students’ anonymised questions, got a weekly email summarising what students were asking about, and could switch between a “direct instruction” mode that gives explanations and a “tutor” mode that guides students towards the answer.
I’ll be honest, that is a more thoughtful tool than most of what gets bolted onto an LMS. Which makes the results all the more interesting.
What did they find?
Grades went down. In the exact-match courses, students who had access to the chatbot finished with final grades about 4 points lower out of 100 than students on the same course without it. Across the full sample the drop was smaller (just under 3 points) and only weakly significant.
Engagement with the course fell off a cliff. This was the most consistent finding. In the exact-match courses, students with the chatbot viewed about 720 fewer LMS pages over the semester, were active on about 7 fewer days, and their participation in the activities instructors had designed (discussion posts, quizzes, online tasks) dropped by 0.9 standard deviations. To put that in plainer terms, control students averaged around 43 recorded activities across the semester; students in the chatbot classes averaged somewhere between 16 and 21. Interestingly, they still submitted their homework, and on time. They did what was required and apparently ONLY what was required.
Hardly anyone used it. Surprisingly, only about 15% of students who had access ever used the chatbot at all, and of those, around 40% only had a single session. Use spiked around exam dates and dropped off in between. Even more strangely, the students in the chatbot classes who never touched it did just as badly as the ones who did. Weirdly, something about having the chatbot in the course seems to have changed things for the whole class.
People asked it for answers. The researchers sorted all 3,460 questions students sent to the chatbot. Nearly three-quarters (73.8%) were requests for a fact, an explanation or a solution. About 11% were for practice questions or revision. Asking for feedback on their own work? 0.7% (I think this is the stat that shocked me the most). Students treated it like a vending machine: question in, answer out, learning done.
They asked their tutors fewer questions. In the end-of-term survey, 57% of students in the chatbot classes agreed they had asked their instructor fewer questions about course content, compared with 44% of the others. And that brings me back to my original point: without an expert in the room you really don't ever get to see the bigger picture and your learning becomes shallow.
So what is this telling us?
The study suggests that having an AI to ask makes students less likely to ask their tutor, and less likely to take part in the messy, social bits of a course, and so they miss out on the wider context surrounding the question they actually asked. This is why you always need an expert in the room who will suck air through their teeth when you ask a question, proclaim ‘well, that’s really interesting’, and then launch into a 30-minute lecture on the specific locations and densities of dopamine receptors in the basal ganglia. Not talking from experience here.
How does this inform our learning design?
At Obline Education, we specialise in bespoke learning experiences built around learning objectives. A big part of that is integrating bespoke AI tools that support our students (you can read my research into an AI chatbot to improve academic writing on my website).
The authors make a distinction I think is spot on: technical integration and instructional integration are two different things. This chatbot was technically well integrated. It lived in the LMS, it was grounded in the course materials and it cited its sources. What it lacked was any instructional design. It defaulted to handing out answers; nobody built it into a learning activity, and it ended up competing with the tutor and the course activities instead of supporting them. Oh, if only they had consulted an expert Learning Designer by emailing info@obline.org before they conducted this study.
So if you’re thinking about putting AI into your courses, here’s what I’d take from this paper:
- Make it ask, as well as answer. Default to a tutoring mode that guides learners towards the answer and prompts them to have a go first.
- Build it into the learning activities. A chatbot floating in a sidebar is an optional extra that students will use as a shortcut the night before the exam. Design tasks where the AI has a specific job to do.
- Design for feedback on attempts. Only 0.7% of requests were students asking for feedback on their own work. That’s where AI can genuinely help, and students need to be shown how.
- Protect the humans. Make sure the AI points students towards their tutor, their peers and the discussion boards, rather than quietly replacing them.
- Watch who is losing out. Check how different groups of learners are affected, especially first-generation students, who appear to have the most to lose.
In other words, what you need is something bespoke, designed around how your learners actually learn and the outcomes you want them to reach. Turns out this research paper is amazing for my company!
As always, if you need any help developing AI systems that can make a real impact on how your students learn, please contact me.
The paper: Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Masters, M. C., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J. K., and Rice, J. K. (2026). The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial. EdWorkingPaper 26-1598, Annenberg Institute at Brown University. https://doi.org/10.26300/y3f8-vh05