We tested our AI on 41,000 student submissions. Here's everything we found.

I spent the last three weeks running experiments.

Not on the product. Not on features. On the question underneath all of it: when ClassLens grades a student's work, does it grade fairly?

I did not start this project expecting the answer to be complicated. It was.

The problem nobody talks about

Every AI grading tool uses a large language model. What most of them do not tell you is that these models grade too harshly. Consistently, measurably, across every model family we tested. The AI reads student work and scores it lower than a trained human rater would.

On our first test run, the gap was 6.2 points on a 30-point scale. Every criterion. Every student. The AI was too strict.

The standard fix in the research literature is to pre-grade 20 or more student papers by hand and use those as calibration examples for the AI. That works for a researcher studying one assignment. It does not work for a teacher grading 30 different assignments across 5 classes.

So I set out to fix the scoring without requiring any calibration at all. Zero pre-graded examples. Zero setup per assignment.

What did I test ClassLens grading on?

I ran 200+ separate experiments on 41,000+ individual student submissions across 8 published academic datasets. The primary benchmark was the ELLIPSE Corpus (Crossley et al., 2022), a set of 3,911 argumentative essays written by English Language Learners in grades 8-12, scored independently by two trained human raters.

The metric is Quadratic Weighted Kappa (QWK). It measures how closely two sets of scores agree while accounting for chance and penalizing larger disagreements more heavily. It is the standard metric in automated essay scoring research. On the ELLIPSE dataset, the agreement between the two human raters is QWK = 0.612.

Here is the sample-size breakdown for the datasets with published counts:

These four itemized datasets (37,535 submissions) are a transparent subset of the 8 published academic datasets that make up the 41,000+ total; the remaining data includes the SPRAG Python Q&A set, an ASAP reading-and-science short-answer set, and two additional science datasets. The complete methodology and the full per-dataset breakdown are available to school districts under NDA during procurement.

All told, the research consumed over 300 million tokens of AI processing. Tokens are the unit these services bill on, and 300 million is the scale of testing needed to validate a grading engine across this many experiments and datasets.

How accurate is ClassLens grading?

During optimization, I reached a QWK of 0.696 on the ELLIPSE benchmark. Best single run was 0.742. I was ready to ship it.

Then I tested it on a rubric with a different scale. It overscored by 2.4 points.

The scoring instructions I had written were calibrated specifically for a 1-5 rubric totaling 30 points. They worked well on that exact format. On a 1-6 scale, they broke. On a 0-10 scale, they would break worse. ClassLens teachers create rubrics with whatever scale makes sense for their assignment. A scoring engine that only works on one rubric format is not useful.

So I went back and rebuilt the scoring instructions to be scale-agnostic. No reference to specific point values. No assumptions about how many criteria or what the scale is. The universal configuration scores 0.623 on the ELLIPSE benchmark across five independent replications. Lower than the overfit version. Still above human-to-human agreement (0.612). And it works on any rubric format a teacher creates.

I chose reliability over peak performance. A grading engine that scores 0.623 on every rubric a teacher throws at it is more useful than one that scores 0.696 on one rubric format and breaks on the next.

Not just essays

A grading engine that only works on essays is not useful to most teachers. I tested the same core architecture on other assignment types. The only thing I changed was the domain-specific references in the system-level instructions. The prompt structure and scoring logic stayed identical.

• Essays (ELLIPSE): QWK = 0.696 (optimized config) / 0.623 (universal config)
• Python programming Q&A (SPRAG): QWK = 0.730
• Reading and science short answers (ASAP): QWK = 0.580

Programming scored the highest. Short answers passed. The same core architecture works across formats with only minor domain-specific adjustments.

Does ClassLens grading show demographic bias?

Accuracy alone is not enough. A grading engine that agrees with human raters on average but scores one group of students differently than another is worse than useless.

I tested for demographic bias using the PERSUADE 2.0 corpus (Crossley et al., 2024), which includes demographic labels for ELL status, gender, and race. I sampled 800 essays, half ELL and half non-ELL, and measured effect sizes using Hedges' g.

Hedges' g is the standard way researchers measure how big a gap is between two groups, scaled so the same number means the same thing across studies. A g of 0 means no difference at all. Researchers treat anything above 0.20 as a "small but real" effect worth investigating. Anything below 0.20 is in the range of normal noise.

No bias was detected on any dimension.

• ELL status: g = -0.049
• Gender: g = +0.020
• Hispanic vs White: g = +0.032
• Black vs White: g = +0.080
• Asian vs White: g = -0.107

All well below the 0.20 threshold that researchers use to flag a concern. The AI scores ELL and non-ELL students, male and female students, and students of every racial background with equivalent accuracy relative to human raters.

The honest limitations

Score compression. The AI compresses the scoring range slightly. Weak essays get scored a little higher than they should. Strong essays get scored a little lower. The best students are the most affected.

Most of the QWK loss in moving from the overfit configuration to the universal one, about 0.07 points, lives in this compression. The middle of the scoring range is accurate. The errors concentrate at the outer edges, where the strongest and weakest work drifts toward the mean. We know where the loss lives.

I found it, reduced it significantly, but did not fully solve it. Telling the AI to avoid compression made it worse, which I did not expect.

One thing worth knowing about compression: it is a documented phenomenon in human rater behavior too. Two trained human raters scoring the same papers will show some of this same drift toward the middle, because extreme scores feel risky to a rater. So ClassLens's compression may not be meaningfully worse than what two humans would produce on the same papers. I did not run enough head-to-head human-to-human compression analysis to validate that claim conclusively. But the human-human QWK ceiling on this benchmark (0.612) sits in the same range where our errors concentrate, which is at least consistent with the interpretation that we are not worse than human raters on this dimension.

The universal scoring configuration has higher variance than the benchmark-optimized one (SD = 0.041 vs 0.028). One of five replications came in at 0.546, below the 0.612 human-human line. The other four exceeded it. A more targeted instruction scores more consistently but breaks on different rubric formats. That is a tradeoff I am comfortable making.

This is why every grade in ClassLens goes to the teacher for review before students see it.

What did not work

Here are 9 approaches that didn't work, despite looking promising on paper.

1. Bigger, more capable AI models scored more harshly, not less.
2. Chain-of-thought reasoning made the AI more analytical and more conservative.
3. Defining what each score level means in detail made the AI pickier.
4. Giving it example essays to calibrate against made it fixate on flaws.
5. Changing the temperature (randomness) setting had zero measurable effect.
6. Telling it to expect a certain score distribution hurt performance every time.
7. Framing rubric criteria positively produced harsher scores on grammar and conventions.
8. Telling it to avoid score compression made compression worse.
9. Requiring the AI to cite evidence before scoring made it more conservative.

Each of these seemed reasonable before testing. None survived contact with the data.

Four grading modes

Standard is the research-validated configuration. This is what the numbers above describe.

Lenient shifts borderline rubric scores up and gives the benefit of the doubt. For assignments where you want to reward effort on close calls.

Understanding uses the rubric as a guide but grades primarily on whether the student demonstrates understanding of the material. It does not focus on individual criteria or nitpick semantics or spelling.

Effort grades purely on whether the student gave a genuine effort in the assignment. No rubric criteria are applied.

Standard is the only mode with published benchmark validation. The other three are teacher tools for different classroom situations. Comment style and length settings do not affect scoring accuracy. Those settings control feedback presentation only.

The full report

The complete research report, including methodology, stability tables, per-criterion calibration data, cross-validation results, and the full overfitting analysis, is available to school districts under NDA during procurement. The report contains everything you would need to evaluate the claims in this post independently.

If you are evaluating AI grading tools for your district, ask every vendor you are considering for their QWK data on a published benchmark. Ask for their bias testing results. Ask for their sample sizes. Then compare.

Contact us at steven.swanson@evolvedacademics.com

Try it

ClassLens works with Google Classroom. Sign in, pick an assignment, set your rubric, and let it grade in Standard mode. You review every grade before it reaches a student.

Try ClassLens at classlens.com

ClassLens is an AI-assistive grading and teaching tool built by Evolved Academics, LLC. Steven Swanson is a 22-year classroom teacher in California. He teaches engineering (design/drafting, mechatronics, and senior capstone) at Whittier High School, and AP Computer Science and AP Physics online. He ran 200+ experiments on 41,000+ student submissions to validate ClassLens's grading engine because no one else in the category has published theirs.