A tutor that keeps a lab notebook

Here is a note from week twelve of a simulated school year, written by an AI agent about a student it has never spoken to directly. Over the previous weeks, the student made the same mistake five times: distributing a negative, they lost the sign. Then, on a new problem, they got it right.

Week 12 of 36Simplifying expressions

Working hypothesis

Possible persistent sign error when distributing a negative.

What the evidence does and does not show

All five errors were on the exact same expression, so a pattern that holds across different problems is not established. The one changed instance was correct.

Confidence

0.48

Plan for next time

Give two or three different expressions with negative factors and varied signs. Ask for the expansion and one independent check, then see whether the correct sign transfers without prompting.

A weaker system would have written “struggles with negative signs” after the second mistake and never looked back. This one noticed that five errors on the same expression are less informative than they look, recognized that the most recent evidence cut the other way, and decided what it would need to see next. That is not memory in the sense most AI products mean it. It is closer to the way a scientist keeps a notebook.

This is the core idea behind Adaptive, and it is the system we are building for every student who uses Honorly: an agent that studies each learner over time, keeps careful notes, forms hypotheses about how they think, plans the next conversation around what it most needs to learn, and gets sharper every week it works with them.

Remembering is not understanding

One way to give a tutor memory is to save what happened. Keep the transcripts, summarize them, put the summary in the next prompt. That produces a tutor that can say “last time we talked about fractions.”

It does not produce a tutor that knows what to do about fractions. A transcript records what was said. A summary compresses it. Neither one records what is still uncertain, what evidence would settle it, or what to try next. A summary of what a student did does not say why they did it.

A randomized trial published the same day as this piece tested the summary approach in real classrooms. Over ten weeks with 1,179 UK secondary students, Google’s LearnLM team and Eedi compared two versions of an AI maths tutor, alongside human tutors and the platform’s standard support.¹ Both AI versions saw the question and the misconception behind the student’s wrong answer. One also received a profile of the student’s history: recent unit scores, predicted proficiency, past errors on earlier topics. The history changed the tutor’s private plan for each session, which mentioned the student’s prior proficiency 70% of the time instead of 3%. It barely changed the tutoring. Supervising tutors approved drafts unedited at nearly the same rate, 94.7% with the history and 94.8% without, and students learned about as much and rated the two tutors about the same. The authors offer two explanations: once a diagnostic question has identified the misconception, a summary may add little, or their implementation “fell short of encoding the important aspects of personalisation.” One trial cannot tell those apart. Either way, it shows what a summary does on its own: the history gave the model things to mention, in 18% of conversations instead of 1%, without changing how it taught.

So we made the unit of memory something different: not the conversation, but the claim. A claim is a specific, checkable belief about a student. It names a skill, says what the agent thinks is going on, carries the evidence for it, and comes with a falsifier: the observation that would prove it wrong.

The falsifier is the part that matters. “Has trouble with equations” cannot be wrong, so it can never be corrected. “Keeps a term’s sign when moving it across the equals sign” can be, and the student solving a problem with the negative on the right-hand side, unprompted, would count against it. A claim with a falsifier is a claim the system can test.

None of this is new to the field. Brown and Burton modeled children’s subtraction mistakes as specific procedural bugs and generated the problems that would tell one bug from another.² Assessment design formalized the same discipline: state a claim about what a student knows, say what evidence would support it, and build tasks that produce that evidence.³ We apply the same discipline to open-ended work in natural language.

An agent that studies one student

Between sessions, when something genuinely new has happened, the student’s agent gets to work. It runs as its own isolated session with a narrow view of the world: this one learner’s recent work, its own earlier notes, and a library of teaching knowledge about how students tend to misunderstand particular ideas and how to tell those misunderstandings apart.

It reads the way a teacher would. It looks at the evidence that is new, pulls up original conversations when the summary is not enough, consults what is known about the skill in question, and revisits its own past conclusions. Then it updates its notes: what is secure, what is still breaking, which explanations remain plausible, and what it plans to check. Finally it writes a report a teacher can read in a minute, with every finding traced to the work that supports it.

The agent stays quiet when there is nothing to learn. If a student spends a week practicing things the agent already understands, there is no reason to spend money re-deriving the same conclusions. It runs when the evidence changes.

Beliefs that can change

Because claims are explicit, they can change, and how they change is where the intelligence shows. A first observation is held loosely. It becomes a confident belief only when independent evidence agrees, and we learned the hard way that “independent” has to mean a genuinely different problem, not the same worksheet seen twice. A confident concern can then be tested, and if fresh work shows the difficulty is gone, the claim closes.

1. ObservationWhat the student actually did
2. Possible explanationsHeld loosely, each with what would disprove it
3. A distinguishing questionPlanned into a future lesson
4. Later workThe student answers it as part of normal tutoring
5. Revised explanationStrengthened, closed, or replaced

The other ways a belief can change matter just as much. A difficulty that was resolved can come back after a long break, and when it does, the agent reopens the old claim and links the two, rather than starting a fresh one and forgetting the history. A related difficulty in a different skill, rooted in the same kind of misunderstanding, gets connected to its predecessor. A teacher who disagrees with a claim can overrule it, and the agent cannot quietly bring it back.

And beliefs fade. A concern from October with no new evidence by March should not be presented with October’s confidence. Just as important, the absence of evidence is not evidence of decline. A student who stopped practicing fractions because the class moved on to geometry has not forgotten fractions. A system that read silence as regression would paint an increasingly bleak picture of every student simply because the curriculum moved.

The plan becomes the next lesson

This is where the notebook stops being a record and starts being a tutor.

When the agent has a confirmed concern and a falsifier for it, that falsifier becomes a plan. The next time the student is working on something related, the tutor has a quiet instruction: here is a good opportunity to check whether this difficulty is still real. The check arrives as ordinary teaching, a well-chosen practice problem, a request to show the step, a question that happens to be the exact one the hypothesis predicts they will miss. From the student’s side, it is simply a good tutor asking a good question.

What happens in that moment flows back into the claim. A clean result counts toward closing it. A repeat of the error strengthens it. An ambiguous result is recorded as ambiguous, not rounded to whichever answer is more convenient. The tutor is also free to skip the check if the moment is wrong; a student in the middle of something else does not need to be interrupted for the agent’s benefit.

The loop matters because it closes. A system that only writes down what it sees is observant, not adaptive. Here, what the agent believes changes what the tutor does, and what the tutor sees changes what the agent believes. That is the difference between a record and a model. Knowledge tracing built a loop like this decades ago for a fixed set of procedural skills, updating its estimates from each response and using them to choose the next practice.⁴ Ours has to interpret open-ended student work: a misconception has to be recognized in an explanation or a worked problem, not read off a scored item.

Others have started on that problem with language models. Scarlatos, Baker and Lan used a model to tag each turn of a tutoring dialogue with the skills it involved and whether the student was right, then traced knowledge through the dialogue. They could not follow a student from one conversation to the next, because no open dataset of tutoring dialogues links the same student across conversations.⁵ Berthon and van der Schaar’s Language Bottleneck Models have one model write a student’s knowledge state in plain language, misconceptions included, and a second model predict the student’s answers from that text alone.⁶ Those summaries are judged mainly by how well they predict, and they assume the student’s knowledge holds still within a session. Our claims are judged by whether they survive a planned check, and they have to change over a year.

Testing it on a year that never happened

A system like this is hard to evaluate. Its value only appears over months, and real students cannot be rerun with a different configuration. So we built a simulated school year.

The simulation follows synthetic students across thirty-six weeks, and each student has a planted story the agent does not know. One has a real misconception that resolves partway through the year. One has a difficulty that persists no matter what. One improves, takes a break, and relapses. One has a skill that decays from disuse. And one, the control, has no real misconception at all.

The control is the most important student in the simulation. Any system that hunts for problems will find some. The question is whether it finds problems that are not there. Berthon and van der Schaar measured the same risk in their synthetic students and found a trade-off: the models that caught more real misconceptions mostly also named more that the student did not have, and only the strongest did better on both.⁶

Run	Misconception claims about the control student at year end	Real misconceptions correctly closed	Relapse detected and linked
First full year	7	0 of 2	—
Second	0	0 of 2	—
Third	0	1 of 2	Yes
Fourth	0	0 of 2	No
Fifth	0	2 of 2	Yes

Read top to bottom, this table is the research. In the first full simulated year, the agent ended the year holding seven misconception claims about a student who had none. Together they described a struggling student who did not exist. Every later run ended with zero such claims, and by the fifth the agent closed both real misconceptions when the evidence said they were gone, and detected the relapse and connected it to its history.

Auditing those runs taught us things the scores did not. When we traced each closure back to the work that justified it, several turned out to rest on the student solving the same problem again, sometimes in a different conversation, but the same problem. Two conversations are not two problems. That finding changed how the system decides what counts as independent evidence, and it is the kind of error that only shows up when you read the agent’s notebook instead of its scorecard.

The cost is the other number we watch. The best simulated year ran at about six cents per student for the entire year, and the agent operates under a hard annual budget per student that it cannot exceed. Understanding a student well should not be a premium feature.

What we are building toward

Everything in this piece points at one goal: a tutor that gets measurably better at teaching a particular student the longer it works with them. Not because it has more transcript to search, but because it has spent the year forming and testing ideas about how that student thinks, keeping what held up and discarding what did not.

Two questions will decide how far this goes, and both are where our research is focused now. The first is calibration: when the agent says it is eighty percent sure, is it right eighty percent of the time?⁷ We record claim outcomes and confidence snapshots so that question can be answered. The second is whether teachers find the agent’s conclusions useful enough to act on, which only real classrooms can tell us.

The individual agent is also only half the system. Each one writes, alongside its private notes, a set of observations meant to be read in aggregate. What happens when a school has thousands of these agents, each with a year of careful notes, and you point another layer of agents at all of them at once is the subject of our next piece.

Footnotes

1. LearnLM Team, Google, and Eedi, “Measuring the impact of AI tutoring, from near transfer to standardised tests: An expanded RCT in UK mathematics classrooms”, technical report, October 29, 2026. A platform bug kept summaries of earlier tutoring chats out of the history, so it covered past performance only. ↩

2. John Seely Brown and Richard R. Burton, “Diagnostic Models for Procedural Bugs in Basic Mathematical Skills”, Cognitive Science, 1978. ↩

3. Robert J. Mislevy, Linda S. Steinberg and Russell G. Almond, “On the Structure of Educational Assessments”, Measurement: Interdisciplinary Research and Perspectives, 2003. An institutional report version is freely available. ↩

4. Albert T. Corbett and John R. Anderson, “Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge”, User Modeling and User-Adapted Interaction, 1994. ↩

5. Alexander Scarlatos, Ryan S. Baker and Andrew Lan, “Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs”, Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK ’25), 2025. ↩

6. Antonin Berthon and Mihaela van der Schaar, “Language Bottleneck Models for Qualitative Knowledge State Modeling”, arXiv preprint, 2025. ↩ ↩²

7. Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, “On Calibration of Modern Neural Networks”, Proceedings of the 34th International Conference on Machine Learning, 2017. ↩

@misc{honorly_a_tutor_that_keeps_a_notebook_2026,
  author = {Rosato, Trevor},
  title = {A tutor that keeps a lab notebook},
  year = {2026},
  howpublished = {\url{https://research.honorlyai.com/a-tutor-that-keeps-a-notebook/}}
}