Two weeks ago, a new AI tutoring company called Bloomy launched out of Y Combinator with a number attached: students in an early pilot grew 1.8 times faster than expected on the NWEA MAP assessment, a widely used test of academic growth. The founder has been careful — he’s called it an observational pilot, not a randomized study, and the sample was about 150 middle schoolers at one Massachusetts charter school.
That caveat matters, and other outlets have already covered it well: no control group, no randomization, one school. If you want the stats critique, it’s out there.
We want to talk about something the coverage has missed — because it’s sitting in Bloomy’s own product design, not in a footnote.
The part of Bloomy that looks a lot like us
Bloomy’s platform moves students through three stages for every skill: Base Camp (worked examples), Climb (guided practice with an AI tutor), and Summit — an unaided, ten-question test with the tutor deliberately absent. Students don’t advance until they clear roughly 90% on Summit.
If that structure sounds familiar, it should. It’s the same logic behind our Mastery Gates: Explain, Apply, Sustain. A student doesn’t move forward because they got through the material. They move forward because they can perform without support, cold, when nobody’s coaching them through it.
Here’s the question that matters: why build that gate at all?
Because a growth number doesn’t tell you what you think it tells you
NWEA MAP growth is a well-respected, widely used measure — but it measures performance on a norm-referenced test. It tells you a student answered more items correctly, or harder items correctly, than before. It does not, by itself, tell you why — whether a student built durable understanding, or got faster at pattern-matching the kind of problems that show up on that kind of test
That gap between “the score went up” and “the understanding is real” is what we call the illusion of mastery: the condition where a student looks like they’ve learned something, but the underlying thinking never actually happened. Good grades without reasoning. A rising RIT score without a student who can explain what they did.
The interesting thing is that Bloomy’s own architecture seems to already know this. If a rising test score were sufficient proof of learning, you wouldn’t need a Summit gate. You’d just assign more problems and let the score climb. The fact that Bloomy built an unaided, high-bar checkpoint before letting a student move on suggests someone on that team understands the difference between “the number went up” and “the student can actually do it alone” — even while the marketing leans on the number.
What this means for you
If you’re evaluating any AI tutoring tool — Bloomy or otherwise — for your family or your classroom, the growth-percentage headline is the least useful thing to ask about. The better questions:
- Does the tool require a student to perform without help before it lets them move forward?
- Can you see why a student got something wrong — a specific misconception — not just that they got it wrong?
- Does “mastery” mean “answered correctly,” or does it mean “can explain the reasoning, apply it somewhere new, and still have it three weeks later”?
That’s the standard we hold our own Mastery Gates to, and it’s the standard worth holding any tool to — including ours.
Leave a Reply