Google released Gemini 3.5 Flash-Lite this morning. By tonight we had run it through the same validation suite we use on every model that grades student work in ClassLens: agreement with human raters, grade reproducibility, and a demographic bias check. Here is everything we found, including the result that surprised us, and why ClassLens is staying on its current models for now.
Key findings (tested July 21, 2026, the day of release)
- On agreement with trained human raters, Gemini 3.5 Flash-Lite scored a Quadratic Weighted Kappa of 0.574 in our validation harness, using the same grading instructions and temperature ClassLens runs in production. The model ClassLens uses today, Gemini 2.5 Flash-Lite, scored 0.623 on the identical test, above the 0.612 agreement level of two trained human raters.
- Gemini 3.5 Flash-Lite did not produce reproducible grades. On 13 of 27 real student submissions, rerunning the exact same grading request changed the score. The two models ClassLens uses returned identical scores on all 27 submissions across six runs each.
- No statistically detectable gender bias in any of the three model generations we tested.
What Google shipped today
Gemini 3.5 Flash-Lite is the newest entry in Google's cheapest, fastest model line, the same family ClassLens builds on. Google positions it as the high-throughput workhorse of the 3.5 series: a one-million-token context window (independent benchmarks clock it at roughly 490 output tokens per second), and API pricing of $0.30 per million input tokens and $2.50 per million output tokens. For comparison, Gemini 2.5 Flash-Lite costs $0.10 and $0.40, and Gemini 3.1 Flash-Lite costs $0.25 and $1.50. For developers and anyone reproducing our tests: the exact API model IDs are gemini-3.5-flash-lite, gemini-2.5-flash-lite, and gemini-3.1-flash-lite, all called at temperature 0.
A new model in this family matters to us in a way it might not matter to most software teams. The model is the part of an AI-assistive grading and teaching tool that actually reads student work. When it changes, everything we have published about accuracy and bias has to be re-earned. So we don't adopt new models based on release notes. We test.
How we tested it (same day, same conditions)
We ran all three current flash-lite generations, 2.5, 3.1, and 3.5, through three studies, each model under identical conditions. The reproducibility study ran through our benchmark tool, which uses the same grading engine as ClassLens production. The agreement and bias studies ran through our validation harness with the same grading instructions ClassLens sends in production, against the same fixed, seeded work samples for every model. Everything at temperature 0.
| Study | Dataset | Size |
|---|---|---|
| Human-rater agreement (QWK) | ELLIPSE Corpus essays with published human scores | 5 replications of 200 essays per model (3,000 gradings) |
| Grade reproducibility | Real classroom drafting assignment: scanned hand-drawn projection drawings, graded as images (10-point rubric) | 27 submissions, graded 6 times per model |
| Gender bias check | Fixed sample of 150 female-author and 150 male-author essays | 300 essays per model, permutation test |
A detail that surprised even us: the Gemini 2.5 Flash-Lite control run reproduced our May 2026 results to three decimal places, replication for replication. Same seeds, same prompts, same model, two months apart, identical numbers. If our pipeline were sloppy, that could not happen, so when the new model produces different numbers, the numbers are about the model.
Does Gemini 3.5 Flash-Lite agree with human graders?
Quadratic Weighted Kappa (QWK) measures how closely two graders agree, weighted by how far apart their scores are. This study is essays because that is where human ground truth exists: public datasets with trained-human scores are text-only, so agreement on drawings, graphs, and screenshots cannot be benchmarked the same way. Visual work is covered by the reproducibility study below, which uses real scanned student drawings. On the ELLIPSE corpus, two trained human raters agree at QWK 0.612. That is the line that matters: a grading model is useful when it agrees with a trained human about as often as a second trained human does.
| Model | Mean QWK vs human raters | Range across 5 replications |
|---|---|---|
| Gemini 2.5 Flash-Lite (ClassLens Standard) | 0.623 | 0.546 to 0.656 |
| Gemini 3.5 Flash-Lite (released today) | 0.574 | 0.526 to 0.624 |
| Gemini 3.1 Flash-Lite (ClassLens Advanced) | 0.385 | 0.324 to 0.441 |
| Two trained human raters (reference) | 0.612 |
The new model is good. It isn't better. Under our production configuration it trails the model ClassLens already uses by about five QWK points and sits below the human-human agreement line. It also under-scored essays by more on average (about 2.2 points low on a 30-point scale, versus about 1.7 for 2.5 Flash-Lite). Credit where it is due: 3.5 Flash-Lite returned cleanly parseable output on all 1,000 of its gradings, while 2.5 Flash-Lite dropped 8, so the new model is more reliable at following output instructions even though it agrees with humans less.
An honest caveat: our calibration prompt was tuned against 2.5 Flash-Lite. The 3.1 number in this table is a good illustration of how much that matters. 3.1 Flash-Lite is a stronger model than its 0.385 essay score here suggests. In our May comparison it wrote noticeably better feedback than 2.5 on visual and technical work, which is why it powers our Advanced tier. Today's drawing test backs that up with numbers: scored against grades the teacher verified to within one point, 3.1 landed within a point of the teacher on 85 percent of the drawings, versus 77 percent for both 2.5 and the new 3.5. Different models are good at different kinds of student work, and a single essay benchmark does not capture that. But calibration language tuned for one model's scoring habits does not automatically transfer to another's. With model-specific retuning, 3.5 Flash-Lite might close some of the gap. We'll run that experiment. But "might match our current model after more tuning, at three to more than six times the price" is not a switching argument.
Can Gemini 3.5 Flash-Lite produce reproducible grades?
This is the result that settles the question by itself.
ClassLens grades at temperature 0, which for the current models means the same submission gets the same score every time. We consider that non-negotiable. A grade a teacher signs their name to has to be defensible, and a score that changes when you re-run it is not. When we re-graded 27 real submissions six times with each model, both models ClassLens uses returned identical scores on every submission, every run. Gemini 3.5 Flash-Lite changed its score on 13 of the 27. The worst case was a single student whose identical submission, a hand-drawn projection drawing, scored anywhere from 6 out of 10 to 9 out of 10 depending on the run. Harder-to-read work is where the variance bites hardest: this test set is images, not text, and a three-point swing on a ten-point rubric is, proportionally, the widest variation we measured anywhere in today's testing.
We dug into why. The 3.5 series has reasoning built in, and reasoning is a common source of run-to-run variation, but the API reported zero thinking tokens on these requests. In the cleanest version of the probe, we sent the exact same essay-grading request six times and got 15 out of 30 three times and 18 out of 30 three times. Identical request, temperature 0, no reasoning tokens, a three-point spread. The variation appears to live in how the model is served, and as of today there is no API setting that removes it.
One more check, because it matters where ClassLens actually runs: our tests above used the Gemini API, but production grades through Vertex AI. So we repeated the probe on Vertex with our exact production request shape. Same story, worse spread: one essay scored anywhere from 13 to 18 out of 30 across six identical calls to gemini-3.5-flash-lite, while gemini-2.5-flash-lite and gemini-3.1-flash-lite stayed deterministic on both serving surfaces.
Google will very likely tighten this up. Early-release serving behavior changes fast, and we will re-test when it does. But today, a model that returns different scores for the same student work is not a model we will let near a gradebook, whatever its benchmark numbers say.
Is Gemini 3.5 Flash-Lite biased?
Good news here. Our release-day check covers gender; the fuller demographic audit lives in our published bias post linked below. We graded a fixed sample of 150 essays by female authors and 150 by male authors with each model and compared the scoring gap between groups (one grading call out of the 900 failed to return a parseable score). All three generations scored the two groups within 0.3 points of each other on a 30-point scale, and none of the differences came close to statistical significance (permutation-test p values from 0.40 to 0.66). This matches the result of our published bias audit: across model generations, we have yet to detect demographic bias in rubric-based grading with these models.
So ClassLens is staying put
ClassLens has one operating principle: maximum grading context, minimum teacher effort. Chasing model version numbers serves neither. A model switch has to clear three bars: agree with human raters at least as well as what we run today, produce reproducible scores, and justify its cost. Today, Gemini 3.5 Flash-Lite clears none of the three.
- It agrees with human raters less than the model we already run: 0.574 versus 0.623, under identical conditions.
- It changed its score on nearly half the submissions we re-graded, and nothing in the API fixes that today.
- It costs three times as much for input and over six times as much for output as our Standard model, a cost that could ultimately reach teachers as higher subscription prices.
The uncomfortable marketing truth is that "now powered by the newest model" is an easy line to write, and several tools in every AI category will write it this week. Nobody wants to publish "we tested the new model and kept the old one." But that is what the data says, so that is what we are doing, the same way we published our full accuracy results, including the experiments that failed.
We'll keep re-testing as Google updates the 3.5 line, and the day a new model beats what your students' work is graded with today, on the evidence, we'll switch and tell you why. Until then, the boring, reproducible model keeps the job.
FAQ
Is Gemini 3.5 Flash-Lite better at grading than Gemini 2.5 Flash-Lite?
Not in our testing. On agreement with trained human raters across 1,000 gradings per model, Gemini 2.5 Flash-Lite scored QWK 0.623 and Gemini 3.5 Flash-Lite scored 0.574 under identical production grading conditions. Gemini 3.5 Flash-Lite also returned non-reproducible scores on 13 of 27 real submissions at temperature 0, which rules it out for gradebook use today.
Does ClassLens automatically use Google's newest Gemini model?
No. Every candidate model is validated against human-scored academic datasets, checked for score reproducibility, and screened for demographic bias before it grades any student work. ClassLens currently runs Gemini 2.5 Flash-Lite as its Standard tier and Gemini 3.1 Flash-Lite as its Advanced tier, because those are the models that pass.
Why does grade reproducibility matter for AI-assisted grading?
A grade must be defensible to students, parents, and administrators. If re-running the same submission produces a different score, the grade reflects chance, not the student's work. ClassLens grades at temperature 0 with models verified to return identical scores on identical submissions, so a teacher reviewing a suggested grade can trust it is stable.