AI Tutoring and Durable Understanding: What Learning Gains Must Show

Adaptive tutoring should be assessed through retention and transfer as well as scores
Evidence from a bounded trial does not validate an entire school model
Human interaction and source quality remain important parts of learning design

AI tutoring promises more individual attention at a scale conventional teaching struggles to provide. Alpha School's expansion makes that promise concrete, while raising a question that applies far beyond one provider: do improved scores represent understanding students can retain and use elsewhere? Published accounts describe a model centred on adaptive software and ambitious plans for wider reach. Public concern about absenteeism and uneven educational outcomes explains interest in alternatives, but does not establish their effectiveness. Scientific evaluation must examine the learning produced, the students served and the conditions under which gains occur. Durable understanding involves more than obtaining correct answers within a familiar sequence. It includes explaining a concept, recognising its limits and applying it when the wording or circumstances change.

An Expanding Model with an Unfinished Evidence Base

Adaptive systems can respond to observed mistakes, adjust difficulty and provide repeated practice. Alpha School's learning scientists compare the process with training systems to recognise unusual cases, an analogy that highlights the value of learning from errors. However, educational evaluation needs direct evidence about learners rather than confidence in the analogy. Published accounts report that underlying data supporting some public claims were not available for independent examination. That limits what outsiders can conclude about the model's contribution. Researchers need to understand the selection of students, the comparison group, the curriculum and the assessments. Established adaptive technologies offer relevant evidence, but findings about them cannot automatically validate every new implementation.

A Harvard randomised study published in 2025 provides evidence for a specific, carefully designed AI tutoring intervention in undergraduate physics. Students receiving the intervention achieved stronger immediate learning gains than those in the active-learning comparison. The study is a reason to investigate tutoring seriously, not proof that an automated school model is superior across ages and subjects. Its design and population differ from those of Alpha School. The authors also caution against assuming that the approach always outperforms classroom teaching, including for complex synthesis and higher-order thinking. The scientific task is to identify which instructional features produce benefits and whether they persist beyond the conditions measured.

From Immediate Gains to Transfer

Fragile knowledge is a useful description of performance that depends too heavily on familiar cues. A student may succeed on a practiced format yet struggle to recognise the same underlying concept in a different problem. This is a possibility to test, not a learning loss established simply because an assessment resembles instruction. Researchers should examine delayed retention, transfer to unfamiliar questions and the ability to explain reasoning. These outcomes can coexist with strong immediate gains. A programme that improves short-term performance may still need additional teaching to support broader application. Evaluation should reveal that pattern rather than force a choice between celebrating the technology and dismissing its results.

The review of 28 studies examined here finds generally positive effects of intelligent tutoring systems, while reporting less clear advantages against stronger comparison approaches. This underscores the importance of the baseline. A system may outperform limited instruction without outperforming well-designed active learning. Differences in age, subject, implementation and study quality also affect what a pooled result means. Good tutoring and a complete educational environment are related but not identical. A school provides social relationships, opportunities for discussion and support that may not appear in a narrow subject test. Claims about replacing those functions need evidence addressing them directly.

Questioning as a Learning Process

Questioning can help learners move beyond a remembered response. A teacher or peer may ask why a method works, what would happen if an assumption changed or how an example differs from a familiar one. Such interaction can expose misunderstanding and support revision. It is not exclusive to one cultural tradition, and automated systems can also be designed to ask useful questions. The issue is whether the learner genuinely attempts and evaluates the reasoning. A conversational exchange that merely supplies another answer may add little practice. Educational design should therefore examine the function of dialogue and the student's engagement, rather than equate human conversation with understanding or machine interaction with memorisation.

Language models can be uneven on unfamiliar problems, but their architecture should not be reduced to memorising question-and-answer pairs. Nor does a failure on an unusual example demonstrate that they cannot generalise. The educational concern is more specific: a tutor may fail to recognise a misconception, provide an inappropriate explanation or confidently endorse an incorrect step. Students may lack the expertise to notice. Structured content, testing and escalation to a knowledgeable person can help address those risks. Learners also need occasions to solve problems without assistance so that educators can see what has been retained. The system's performance and the student's learning are distinct objects of evaluation.

Models, Evidence and Unfamiliar Problems

Bayesian analogies for adaptive learning should be treated cautiously. Adaptive teaching involves many possible mechanisms, and human learning cannot be ranked above model learning simply by describing one as better than Bayesian updating. A more useful comparison concerns observable flexibility. Can the student revise an explanation when evidence changes? Can the tutor identify an exception and supply accurate guidance? These questions can be investigated without a sweeping claim about either architecture. A good learning environment creates opportunities to test and revise understanding, including when a familiar answer is no longer appropriate. The quality of those opportunities matters more than a metaphor about how minds and models update.

Source quality is another requirement, especially in scientific subjects. The HERA example examined here concerns a tool intended to rely on peer-reviewed literature that instead returned substantial secondary material. This illustrates a retrieval and verification problem; it does not prove that all scientific weakness comes from paywalls or that all models lack access to reliable research. A tutor should identify the evidence behind consequential claims and distinguish established material from uncertain findings. Students learning science need to examine sources as well as answers. Open educational resources can help widen access, but accessibility alone does not establish quality. Content selection and review remain part of the teaching design.

What a Credible Evaluation Should Follow

Evaluation should compare the model with credible alternatives and track which students benefit. Selection, family resources, prior attainment, time spent and additional support may contribute to observed gains. A school's published scores cannot isolate those effects without an appropriate design. Transfer into more varied public-school settings may provide useful evidence, but should be followed with transparent methods and attention to students who struggle. Researchers should also examine teacher roles, access needs and the cost of maintaining effective support. A scalable programme is valuable when it preserves benefits as conditions change, rather than merely extending the same software to more learners.

AI tutoring should be judged through a sequence of outcomes: immediate understanding, later retention, application to new problems and development of independent judgment. Strong performance at one stage is encouraging without settling the others. The merged evidence from these articles supports investigation of adaptive assistance alongside careful limits on claims about whole-school transformation. For Gordon Institute's Science Forum, the humanitarian ambition is to make useful education available more widely. That requires learning environments whose effects can be examined and improved. The strongest model will be one that helps students continue reasoning when the prompt, the interface or the familiar example is no longer there.

This Science Forum article is adapted from material published within Gordon Institute’s network. It presents editorial analysis and does not constitute professional advice or an official institutional policy.

References

Coyle, T. (2025) The AI Knowledge Crisis: Why Artificial Intelligence Is Struggling to Learn from Science. Auckland: HERA.
Kestin, G., Miller, K., Klales, A., Milbourne, T. and Ponti, G. (2025) AI Tutoring Outperforms In-Class Active Learning. Scientific Reports, 15, 17458.
Létourneau, A., Deslandes Martineau, M., Charland, P., Karran, J.A., Boasen, J. and Léger, P.M. (2025) A Systematic Review of AI-Driven Intelligent Tutoring Systems (ITS) in K-12 Education. npj Science of Learning, 10(1), 29.
Malkus, N. (2026) Return to Learn Tracker: Chronic Absenteeism 2017-2026. Washington, DC: American Enterprise Institute.
Mineo, L. (2025) Is AI Dulling Our Minds? Cambridge, MA: Harvard Gazette.
Randolph, M. (2026) Alpha School Wants AI to Teach a Billion Kids. Should It? New York: Scientific American.