# A team of data scientists for every school

https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/

Trevor Rosato · 2026-11-03

A school using Honorly for a year would accumulate careful, evidence-backed notes on how every student thinks. We are building layers of AI agents that read those notes the way a team of analysts would. The hard part is not finding patterns. It is not fooling yourself.

Schools are full of data and short on insight. Gradebooks record scores. Benchmarks arrive a few times a year. Dashboards turn both into percentages. What almost no school has is an account of _how_ its students think: which misunderstandings are spreading, where they started, and what the teachers who have fixed them did differently.

That account exists, in principle, in the work itself. Every time a student explains a step, makes a mistake, or asks a question, they leave evidence of their thinking. Reading it all by hand is impractical, because there is far too much and it takes expertise to interpret. A good data scientist could spend a semester on one school’s work and find things that would change how it teaches. Few schools have a semester of a data scientist to spare.

We think that is about to change. Language models are approaching the point where they can do a meaningful share of a junior analyst’s work: read evidence carefully, propose explanations, notice when an explanation does not fit, and write up what they found with the sources attached. One analyst is useful. A team of them, running continuously across every classroom, pointed at a detailed record of how students think, is something new. That is what Adaptive Insights is.

## The data that makes it possible

The raw material is not chat logs. It is the notes written by each student’s own agent, which we describe in [the previous piece](https://research.honorlyai.com/a-tutor-that-keeps-a-notebook): specific, checkable beliefs about how a student thinks, each one with its evidence, its uncertainty, and the observation that would prove it wrong.

That distinction is the foundation of everything else. An analyst reading transcripts has to do the interpretation from scratch. An analyst reading thousands of carefully kept notebooks can spend its effort on the question that matters at its own level: what do these students have in common, and why?

## Agents in layers

Insights is organized the way a school is. Each class has an agent that reads the observations its students’ agents prepared for aggregate use. Broader agents, for courses, grades, departments, schools and the district, each read reports from the relevant groups beneath them.

**District**Questions for leaders to investigate

**School**Coverage and patterns across rooms

**Course, grade, department**Shared materials, pacing and sequence

**Class**What to change in the next lesson

**Student**One agent per learner, with private notes

Reports flow up to the groups above. Raw conversations and private notes never leave the student layer.

Figure 1

Each layer of agents works at the scale where its conclusions are useful. A class agent aims at tomorrow’s lesson. A grade or department agent looks at shared materials and pacing. A district agent produces questions worth investigating, not verdicts.

Two properties of this shape matter. First, each agent only reads the layer directly beneath it. A district agent never sees a student’s conversation, or even an individual student’s private notes; it sees the reports of the schools. That keeps both privacy and cost bounded as the system grows. Second, each layer asks a different question. The same pattern means “reteach this on Thursday” to a class agent and “check whether this unit’s materials introduce the idea too early” to a department agent.

## Summarizing summaries is how you fool yourself

The first version of any system like this finds patterns everywhere. Models are fluent, summaries are easy to write, and a summary of summaries sounds even more authoritative than the original. Fluency is not fidelity: research on abstractive summarization has documented fluent summaries that include content the source never supported.[1](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fn-faith) Most of our work on Insights has gone into the specific ways this goes wrong, because each one produces a conclusion that looks well supported and is not.

**Coverage is not support.** A school report might cover twenty-four students while a particular finding in it rests on twelve. Both numbers are true, but a reader who sees “24 students” next to a finding will assume all twenty-four support it. So the system does not let a model state how many students back a finding. It derives that count from the sources the finding actually cites.

**Class A**

**Class B**

*   Supports this finding
*   Covered by the report, but not by this finding

Figure 2

A report on two classes covers twenty-four students. The finding it contains is supported by twelve. Reporting the first number beside the finding would double its apparent support.

**Repetition is not corroboration.** One classroom’s report can reach the school through more than one route: through the grade-level agent and through the department agent. If the school agent counts both, a single classroom looks like two independent confirmations. The same happens when one student is enrolled in two classes, or when a report is regenerated. In each case, the system traces findings back to their original classrooms and counts each source once. A pattern only counts as widespread when it appears in genuinely separate rooms. Ecologists named the underlying mistake pseudoreplication: treating observations that are not independent as independent evidence.[2](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fn-pseudo)

One classroom's report reaches the school through both the grade-level agent and the department agent. The school agent counts it once.

Figure 3

Organizational structure is not evidence. The same observation can arrive by several paths, and every additional path makes it look better supported without adding any new information.

**Compression destroys uncertainty.** An early version of the pipeline flattened each student’s structured observations into a short block of text before passing them up. The caveats were the first thing to go. A student-level note like “possible, but only seen on one problem type” could arrive at the class level as a plain fact, with no way for the class agent to recover the doubt. Keeping observations structured, with their support and their caveats attached, gave every layer above something honest to reason from. Compression is a reasoning decision. It determines what the next model is still able to notice.

**The same label can hide different stories.** In one of our test cases, eleven students across four classes showed the same misunderstanding. A naive summary would call it one pattern and look for one cause. When the system grouped the observations by the work each one was recorded in, the eleven split apart.

**Assignment A**titled "Worksheet 4B"

**Assignment B**also titled "Worksheet 4B"

**Assignment C**also titled "Worksheet 4B"

**No shared work found**cause still open

*   Student showing the same misunderstanding

Figure 4

Eleven students, one label. Six cluster in pairs around three different assignments that happen to share a title, which a system matching on names would have merged into one. For five, a shared work context has not been established. The right conclusion is three small investigations and an open question, not one big finding.

## Rules for publishing a finding

These failure modes led to a short list of conditions a finding has to meet before a teacher or administrator ever sees it. It has to cite the specific evidence it rests on. It has to be supported by enough distinct students, with the bar rising as the scope widens. At school and district scale, it has to appear in independent classrooms. And it has to come with a competing explanation: another account that would fit the same evidence, and what an educator could look at to tell the two apart.

That last requirement does more work than it seems to. A finding that says “students are confusing slope and intercept” is a claim. A finding that adds “or the unit’s worked examples always put the negative in the intercept, which students are copying; compare their work on examples with a positive intercept” is an investigation someone can actually run. Shared material is a textbook confound: when students who used the same worksheet also share a teacher and a schedule, the data cannot say which one mattered.[3](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fn-wwc) So the system treats shared material as a candidate cause until it is tested. The competing explanation turns a report into a next step.

Findings that fall short do not disappear. They become things to watch, with the reason recorded: too few students so far, only one classroom, no alternative considered. Sometimes the right output is nothing. In one test, a department-level agent was asked for findings before any of the class reports beneath it existed. Rather than reach down and reread the students’ raw observations itself, it reported that it had no evidence-bound conclusion to offer. We count that as a success.

![A class Insights view split into 'What's holding' and 'What's breaking'. One breaking finding reads 'The negative sign moves from the slope to the intercept', supported by three students across thirteen chats, noting where it showed up, how it could be wrong, and one way to answer it: give two equations with the same intercept and opposite slopes, and have students predict which line falls before plotting.](https://research.honorlyai.com/media/adaptive/class-insights-detail-crop.webp)

A class report. Each finding names who and how much supports it, where it showed up and where it did not, how it could be wrong, and one concrete way for the teacher to find out. Shown with synthetic preview data.

## What the agents found

On synthetic school data, the cohort agents produced findings of exactly the kind we hoped for. In one biology class, eight students could identify the maximum and the intercepts on a graph but never said what those features meant biologically. The agent proposed two explanations, a gap in connecting graph features to the science, or a writing barrier that hid understanding the students had, and suggested comparing oral or multiple-choice answers with written ones to tell them apart. In a math class, it noticed students fixing a sign error only after being prompted, and designed a comparison with negatives on opposite sides of the equation and a positive-only control.

We test the system against data with known answers. Fixture schools contain planted patterns in some subjects and none in a control subject. Across our cohort runs, the agents published no findings at all in the control subject, and every published finding cited its evidence. In our largest run, covering forty-eight students in six classes, four of the five scored reports outside the control subject matched the planted pattern. The class-level and higher agents’ share of that run cost less than a tenth of a cent per student.

## Toward a learning nervous system

The version of Insights we are building runs continuously. Every week, student agents update their notes, class agents read them, and the layers above look for what is spreading, what is stuck, and what is working. A misconception that appears in three rooms in October can be traced, before winter break, to shared material that may have introduced it, ready for an educator to test. A teacher whose students recovered from a common difficulty unusually fast can have what they did surfaced to colleagues who are still fighting it.

Each of these is a job a human analyst could do, given time. The difference is that the agents never stop, never skim, and cost almost nothing, so the attention a school could never afford becomes something every school can have. As models improve, we expect these agents to go from careful readers to genuine investigators, forming hypotheses across a whole district and designing the comparisons that would test them.

Work outside education shows what that step would take. POPPER, from Huang and colleagues at Stanford, has language-model agents turn a hypothesis stated in plain language into falsification experiments, run them, and combine the results under sequential statistics that bound how often a false hypothesis gets accepted.[4](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fn-popper) Our rules for publishing a finding are an informal version of the same discipline, and the control subject is our check on them. An agent that designs comparisons across a district would need the formal version.

Everything depends on the discipline described here. A system that reads a school’s learning data at this scale will find patterns whether or not they are real. The value is entirely in its ability to tell the difference, and in being honest with educators when it cannot.

## Methods and scope

All schools, students and findings in this article are synthetic. Cohort results come from development runs on fixture schools with planted patterns and a control subject. A report “matched” when its findings contained the planted pattern’s expected terms under an automated scorer; the largest run had seven scored reports, plus two runs that errored. The cost figure is modeled from token usage at the prices in effect at the time and covers the cohort agents only. Interface images are from a local preview of the teacher dashboard with synthetic data. We have not yet measured how useful educators find these findings; that study is in preparation. The cited work on independence, summarization, confounding and automated falsification supplies the problems and the vocabulary; it is not evidence that our system gets them right.

We describe the principles here and keep the specific thresholds, rules and prompts private.

1.  Joshua Maynez, Shashi Narayan, Bernd Bohnet and Ryan McDonald, [“On Faithfulness and Factuality in Abstractive Summarization”](https://aclanthology.org/2020.acl-main.173/), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, 2020. [↩](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fnref-faith)
    
2.  Stuart H. Hurlbert, [“Pseudoreplication and the Design of Ecological Field Experiments”](https://doi.org/10.2307/1942661), _Ecological Monographs_, 1984. [↩](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fnref-pseudo)
    
3.  What Works Clearinghouse, [“Procedures and Standards Handbook, Version 5.0”](https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf), U.S. Department of Education, Institute of Education Sciences, 2022. [↩](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fnref-wwc)
    
4.  Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candès and Jure Leskovec, [“Automated Hypothesis Validation with Agentic Sequential Falsifications”](https://proceedings.mlr.press/v267/huang25n.html), _Proceedings of the 42nd International Conference on Machine Learning_, 2025. [↩](https://research.honorlyai.com/a-team-of-data-scientists-for-every-school/#user-content-fnref-popper)