AI Tutoring and the Scientific Problem of Fragile Knowledge

High performance inside a tutoring system may conceal weak transfer outside it
Scientific reasoning depends on exceptions, source quality and changes of representation
AI tutoring should be evaluated as an epistemic instrument, not only an instructional product

Artificial intelligence can make instruction adaptive at a scale that no individual teacher can match. A system can observe thousands of responses, identify recurring errors and select a new explanation or exercise within seconds. The expansion of Alpha School, a private-school network that assigns much of its morning instruction to software, offers a prominent test of this proposition. Its founder has spoken of reaching one billion children, and early studies of related adaptive tools report substantial learning gains. The scientific question is not whether these systems can improve measured performance. It is what kind of knowledge they produce, how that knowledge transfers and whether the evidence used to validate the system is independent of the environment the system created.

Performance Is Not Yet Understanding

A randomized study in an introductory physics course at Harvard University found that students using an adaptive digital assistant achieved more than twice the average improvement of students receiving active classroom instruction. This is meaningful evidence that a carefully designed digital tutor can accelerate progress on a defined curriculum. Yet teaching and assessment were closely aligned, leaving open whether students could reproduce the reasoning under a different representation, explain it without the system’s prompts or apply it to a problem whose surface features had changed. The authors described the risk as fragile knowledge: capability that appears robust under familiar conditions but does not survive a shift in context.

The distinction is fundamental in science. A learner may manipulate an equation correctly after repeated practice without understanding the assumptions under which it holds. A student may classify examples accurately while failing to recognize a case that violates the classification. Scientific competence becomes visible when the learner can move between verbal, graphical and mathematical representations; distinguish a measurement error from a theoretical anomaly; and revise an explanation when new evidence conflicts with it. Assessments drawn from the same distribution as the instructional examples can reward fluency without testing these capacities.

The Benchmark Determines the Result

A 2025 systematic review of 28 studies involving nearly 5,000 primary and lower-secondary pupils found broadly positive effects from AI-driven intelligent tutoring systems. The apparent advantage changed, however, when the comparator changed. Against conventional provision, intelligent systems performed well; against active learning without artificial intelligence, the incremental benefit became negligible. This does not show that AI tutoring is ineffective. It shows that part of the measured gain may arise from structured practice, frequent feedback and learner engagement rather than from the model’s intelligence as such.

For scientific evaluation, the comparator should represent the best plausible alternative mechanism. A trial that compares an intensively engineered platform with passive lectures tests a bundle of differences at once. It cannot reveal whether adaptive sequencing, immediate feedback, interface design, additional time on task or the language model caused the gain. Factorial designs, preregistered outcomes and common assessments administered outside the platform would allow stronger causal claims. Longitudinal follow-up is equally important because fragile knowledge may remain invisible immediately after instruction and emerge only when the learner must retrieve and transform the idea months later.

Why Exceptions Matter More Than More Examples

AI tutoring is sometimes compared with autonomous driving: both systems learn from large numbers of cases and become better at recognizing unusual events. The comparison is useful up to a point. More examples can improve coverage of a known distribution, including rare cases within it. Scientific learning also requires recognition that the distribution itself may be wrong. An anomalous observation may be noise, but it may instead reveal a missing variable, a failed assumption or the boundary of an established theory. Treating every exception as another case to be absorbed can preserve a model that should be reconsidered.

This is where Bayesian language can mislead if used too loosely. Bayesian updating is not intrinsically incapable of accommodating anomalies or model uncertainty. The limitation arises when the hypothesis space, prior structure or accessible evidence excludes the alternative that the learner needs to consider. A tutoring system can update its estimate of which exercise a student should receive while remaining unable to determine that its own representation of the concept is inadequate. Human scientists face the same problem, but a skilled teacher can deliberately step outside the established sequence, reformulate the question and ask whether the learner’s explanation remains coherent.

Socratic questioning is valuable in this context not because human dialogue is automatically profound, but because it can function as adversarial evaluation. The teacher requests a justification, changes an assumption, introduces a counterexample and asks the learner to locate the point at which the conclusion fails. An AI system can generate similar prompts, and future systems may do this well. The scientific requirement is to demonstrate that the prompts diagnose and repair conceptual structure rather than simply lead the learner toward another answer pattern contained in the training data.

Scientific Literature Creates a Distinct Data Constraint

The evidentiary environment of science creates another limitation. Large language models learn disproportionately from material that is digitally accessible, whereas much peer-reviewed literature remains behind subscription barriers or inside databases with restricted machine access. A New Zealand research organisation reported that a tool instructed to rely only on peer-reviewed work returned predominantly secondary web content because the primary literature was inaccessible. The failure was not merely a poor prompt. It reflected a mismatch between the stated epistemic standard and the sources available to the system.

Source scarcity affects fields differently. In a domain with stable concepts and low consequences of error, a broad corpus of textbooks, public documentation and repeated explanations may support useful instruction. At the research frontier, small differences in sample construction, experimental design or model assumptions can reverse an inference. Abstracts and secondary summaries often omit precisely those details. A tutor that sounds equally confident across these domains obscures the changing quality of its evidence and may teach students that scientific authority is a property of fluent language rather than a traceable relationship between claims, methods and observations.

An Evaluation Agenda for AI-Assisted Science Learning

Research should separate at least three outcomes: procedural performance, conceptual transfer and epistemic judgment. Procedural performance concerns whether students can complete the task they were taught. Conceptual transfer asks whether they can use the idea in a novel representation or domain. Epistemic judgment concerns whether they can evaluate evidence, identify uncertainty and decide when an apparent exception warrants revising the model. An AI tutor may improve one outcome while leaving another unchanged, and a single aggregate test score will conceal that pattern.

Evaluation must also consider interaction effects with teachers. The relevant intervention may not be AI tutoring alone, but AI tutoring embedded in a classroom where a qualified teacher reviews misconceptions, organizes discussion and introduces deliberately adversarial problems. Studies should compare substitution and complementarity directly: the same platform used to replace teacher time may produce different outcomes from the platform used to redirect teacher time toward higher-order reasoning. Reporting only the software’s average effect makes those institutional designs appear equivalent when they are not.

Keeping the Instrument Open to Scrutiny

AI tutoring should be approached as a promising scientific instrument whose measurement properties remain under study. Its scale, consistency and capacity to personalize practice are genuine strengths. They do not eliminate the need to test construct validity: whether the system’s metrics correspond to the understanding educators and scientists claim to value. Independent access to trial data, clear documentation of model and curriculum changes, and assessments designed outside the vendor platform are essential if researchers are to distinguish instructional progress from adaptation to the instrument.

The most productive research question is therefore not whether AI can teach science. It is which components of scientific learning it can support reliably, under what source conditions, and which forms of judgment still require a different kind of interaction. Systems capable of delivering correct explanations at enormous scale may improve education substantially. But if they also scale knowledge that fails under unfamiliar evidence, the apparent efficiency will have concealed an epistemic cost. The task for researchers is to make that cost measurable before educational ambition turns an unresolved hypothesis into infrastructure.

References

Coyle, T. (2025) The AI Knowledge Crisis: Why Artificial Intelligence Is Struggling to Learn from Science. Auckland: HERA.
Kestin, G., Miller, K., Klales, A., Milbourne, T. and Ponti, G. (2025) ‘AI Tutoring Outperforms In-Class Active Learning’, Scientific Reports, 15, 17458.
Létourneau, A., Deslandes Martineau, M., Charland, P., Karran, J.A., Boasen, J. and Léger, P.M. (2025) ‘A Systematic Review of AI-Driven Intelligent Tutoring Systems in K–12 Education’, npj Science of Learning, 10(1), 29.
Mineo, L. (2025) ‘Is AI Dulling Our Minds?’, Harvard Gazette.
Randolph, M. (2026) ‘Alpha School Wants AI to Teach a Billion Kids. Should It?’, Scientific American.