How accurate is ArticScore? The benchmark, with its caveats
ArticScore's per-phoneme scores were measured against five human phoneticians on children's speech, and against the same engine before it was fine-tuned on child audio. Everything below is a single internal evaluation run, reported with its sample sizes, its confidence intervals, and the rows where the result went against us. It is a ranking engine that agrees usefully with human judges about which sounds are worth a second listen — and, at every operating point we measured, one that gets most of its individual flags wrong.
The evaluation set
Every figure on this page comes from one sample, scored once.
Read these before the numbers
These limitations are reproduced from our internal evaluation reports rather than written for this page, and they sit above the tables on purpose. Every figure below is our own measurement on a limited sample, not an independently verified result. If a number from this page is going into a slide deck or a procurement document, the corresponding caveat should travel with it.
The benchmark children are English learners, not children with speech sound disorders
speechocean762's child speakers are Mandarin-native English learners aged 6–15. Their error patterns — TH-fronting, gliding of /l/ to /y/, final-consonant deletion — overlap usefully with paediatric speech sound disorders, but this benchmark validates phoneme scoring on child voices. It does not validate disorder detection. Nothing in the headline tables tells you how either system behaves on a five-year-old with a lateral lisp.
Every F1 on this page rests on 74 labelled errors
Only 1.45% of the 5,112 phones were labelled incorrect by the human judges. A system that flags nothing scores 98.6% "accurate" on this data, which is why we report precision, recall, F1 and ROC-AUC on the error class rather than raw accuracy. It is also why the confidence intervals around every difference on this page are wide enough that small gaps should not be read as results.
Threshold sweeps are in-sample
The best-F1 thresholds reported were chosen on the same phones they are reported on. That inflates every F1 and precision figure on this page relative to what you would see on unseen audio. The threshold-free metrics, ROC-AUC and average precision, are not affected by this.
Distortions are invisible to the model
ArticScore scores against a closed set of phonemes. A distorted production that is not cleanly any other phoneme — a lateral /s/, a partially rhotacised /r/, the "undifferentiated lingual gesture" clinicians describe — has no symbol to be scored as, and the engine will often score it as correct. This is a structural limitation of the approach, not a tuning problem, and it applies to every commercial phoneme-scoring API we are aware of.
The acoustic model is the ceiling, not the score formula
Refitting the calibration curve on held-out data moved phone correlation by +0.013 and nothing else. The loss is in how well the underlying acoustic model hears child speech, which is why fine-tuning on child audio — not tuning the scoring maths — is where the gains came from.
These are our own measurements
Every figure comes from evaluation runs we conducted ourselves, against publicly documented corpora, with the code and the caveats written down. They are not third-party certified and they are not a standardized assessment. We publish the caveats alongside the numbers because a benchmark without its limitations is marketing, not evidence.
Phone-level agreement with human judges
The human phoneticians' scores are the ground truth. PCC and Spearman measure how closely the engine's per-phone score tracks the humans'; ROC-AUC measures how well it separates the phones humans called wrong from the phones they called right, independently of any threshold. The baseline row is the same engine before fine-tuning on child speech, so the two rows isolate what the child-speech training actually bought.
| System | Phone PCC | Phone Spearman | Error-detection ROC-AUC |
|---|---|---|---|
| ArticScore (fine-tuned on child speech) | 0.4329 | 0.2915 | 0.9079 |
| ArticScore (baseline, no child fine-tuning) | 0.4020 | 0.3039 | 0.8909 |
Word-level and utterance-level correlation
Phone scores roll up into word and utterance scores. Fine-tuning on child speech helped most at word level and least at utterance level, which is consistent with where it changes the acoustic model's behaviour.
| Level | ArticScore baseline | ArticScore fine-tuned |
|---|---|---|
| Phone PCC (0–2 scale) | 0.401 | 0.422 |
| Phone Spearman | 0.304 | 0.287 |
| Word PCC (0–10 scale) | 0.346 | 0.402 |
| Utterance PCC (0–10 scale) | 0.438 | 0.454 |
Error detection — flagging the phones humans called wrong
The task here is binary: does the engine flag the phone that the human judges scored below 0.5 out of 2? Each row is reported at its own swept best-F1 threshold, so the precision/recall trade-offs are not directly comparable between rows — the threshold-free ROC-AUC and average-precision columns are.
| System | ROC-AUC | Avg. precision | Best F1 | Precision | Recall | Flag rate |
|---|---|---|---|---|---|---|
| ArticScore (fine-tuned) | 0.901 | 0.210 | 0.325 | 0.312 | 0.338 | 1.6% |
| ArticScore (baseline) | 0.889 | 0.161 | 0.270 | 0.196 | 0.432 | 3.2% |
False flags on phones the humans scored correct
This is the number a child actually feels: they said the sound correctly and the app told them otherwise. Measured on the 4,855 phones the human judges scored at least 1.5 out of 2.
| System | Operating point | Correct phones | False flags | Rate |
|---|---|---|---|---|
| ArticScore (fine-tuned) | score < 50 | 4,855 | 317 | 6.5% |
| ArticScore (baseline) | score < 50 | 4,855 | 388 | 8.0% |
| ArticScore (fine-tuned) | best-F1 threshold (score < 3.4) | 4,855 | 38 | 0.8% |
| ArticScore (baseline) | best-F1 threshold (score < 7.4) | 4,855 | 102 | 2.1% |
By age band
The whole reason for building a child-specific engine is that performance on children should not be an afterthought. Broken out by the age of the speaker:
| Age band | Phones | Labelled errors | Phone PCC | Best F1 |
|---|---|---|---|---|
| 6–8 | 2,066 | 37 | 0.429 | 0.358 |
| 9–11 | 1,527 | 9 | 0.388 | 0.286 |
| 12–15 | 1,519 | 28 | 0.431 | 0.391 |
Detection by clinical error type
Human judges labelled what kind of error each mistake was. This breakdown is the most clinically interesting table on the page and also the least statistically solid — every cell has between 7 and 35 samples.
| Error type | n | ArticScore baseline | ArticScore fine-tuned |
|---|---|---|---|
| Omission | 35 | 17/35 (48.6%) | 15/35 (42.9%) |
| Substitution | 19 | 6/19 (31.6%) | 3/19 (15.8%) |
| Unclassified | 13 | 7/13 (53.8%) | 5/13 (38.5%) |
| Distortion | 7 | 2/7 (28.6%) | 2/7 (28.6%) |
The explanation layer, and how well it holds up
A scoring API that tells a clinician a sound was wrong has done part of the job; one that also names what it heard instead gives the clinician something to act on. That second layer is what this engine was built around, so it deserves its own measurements rather than a claim. Coverage is structural and total. Correctness is where it gets thin, and the samples below are small enough that we report them as directions rather than findings.
ArticScore names a candidate substitute on every phone it flags. Coverage is 100% by construction, not by tuning.
The decode always produces a most-likely phone over the aligned span, so there is always something to report. This is a property of how the engine works rather than a measured rate that could drift between releases.
On a labelled subset, the engine classified substitution versus omission correctly 81.8% of the time.
That typology figure rests on 22 phones. It is directional and nothing more. We report it with its n rather than as a headline percentage.
Naming the correct substitute — not just naming one — is the weakest measurement we have.
On the phones that carried a clean single-phone substitution label, the engine named the right substitute 4 times out of 8. n = 8. Our own report says a sample that size cannot support a conclusion. It is here because leaving it out would be selective, not because it proves anything.
On children with actual speech sound disorders
The benchmark above measures phoneme scoring on child voices. It does not measure disorder detection, because its speakers are English learners rather than children in therapy. That gap is real, and we have not left it unexamined.
Separately from the benchmark above, the engine is evaluated against a speech and language therapist's judgements on recordings of children in active therapy for speech sound disorders. That work is ongoing research on a corpus licensed for non-commercial use, so we do not publish its figures here. Two findings from it do shape what we tell developers, and both are stated below.
What we think this does and does not show
It shows that the engine ranks well and decides badly. Its agreement with human judges about which sounds are worth a second listen is strong enough to build a worklist on, and fine-tuning on children 's voices lowered the rate at which it interrupts a child who did nothing wrong. Its individual flags are another matter: at the operating points we measured, most of them are false positives, and that is a statement about the state of this technology rather than about any one product.
It does not show that ArticScore is more accurate than anything else on the market. We have not published a comparison, we are not going to infer one, and a single 300-utterance sample with 74 labelled errors could not settle it anyway. What the engine is designed to do differently is name a candidate for what the child produced instead on every sound it flags, and type that error the way a clinical note does. Whether that matters is a product question, and it is one you can answer faster by running fifty of your own recordings through it than by reading this page.
One limit applies whatever you choose: keep a person in the loop.
Benchmark questions
How accurate is ArticScore at detecting mispronounced sounds in children's speech?
How does ArticScore compare with other pronunciation scoring APIs?
What data was the benchmark run on?
Why is the F1 score so low?
Has ArticScore been tested on children with actual speech sound disorders?
Are these numbers independently verified?
Want to run your own audio through it?
ArticScore is in limited early access, rolled out case by case. Tell us your use case and expected volume and we will come back to you.
Request access