Buying live captions for local news? Make vendors take the same blind test

Build a blind live-caption vendor trial for local news. Compare critical names, corrections and viewer delay with word error rate before signing a contract.

Amira Labs3 min read

Share

Share on LinkedIn
Interpretation headphones rest on a reserved press placard at the European Parliament

Imagine two caption vendors each miss ten words in a thousand-word newscast. One repeatedly drops “the.” The other changes an evacuation order from “west of the river” to “east of the river.” A procurement spreadsheet can give them the same word error rate. A viewer deciding where to go has an excellent reason to prefer one.

A station can expose that difference before signing a contract. Give every finalist the same audio, the same preparation time and the same delivery path to a viewer screen. Put the purchase criteria in writing before anyone sees which vendor produced which captions.

Build the trial from a real shift

Ask the assignment desk for a recently aired, rights-cleared rundown with a studio read, a noisy live shot, an unfamiliar local name and a correction from the anchor. Keep a second rundown in reserve. A vendor may tune its vocabulary after the first round. The reserved material tells you whether that work carries over to tomorrow’s newscast.

Prepare a reference transcript against the audio that actually aired. A producer’s script can contain an earlier road name or an instruction the anchor changed at the last second. If the anchor says the wrong town and corrects it, keep both in the reference. Mark the audible start of the correction. Share the same pronunciation list with each finalist and record which entries the station would have to maintain after launch.

Score the words and the decisions separately

Strip vendor names from the recordings. Have an engineer align each output to the reference. Word error rate is the sum of substitutions, deletions and insertions divided by the number of reference words. Agree on how to normalize numerals, contractions and punctuation before scoring. NIST describes the metric and publishes a toolkit that can reproduce the alignment. NIST metric definition and NIST Speech Recognition Scoring Toolkit

Then hand the recorded viewer output to a caption user and a newsroom editor. Ask each to write down the evacuation zone, the speaker and the corrected time without seeing the script. Keep disagreements in the record. When someone gives the wrong answer, save the clip and identify whether the word was missing, arrived too late, or appeared behind a lower third. The FCC’s television caption quality framework addresses accuracy, synchronicity, completeness and placement. It gives the panel concrete categories for those observations. 47 CFR § 79.1(j)(2)

Make the last line of the bid a decision

For the anchor correction, measure from the spoken cue to the first correct, readable text on the station’s return monitor. Keep an early wrong guess and its later correction as separate events. Record the receiver, the audio path and the playout delay so the station can run the same comparison after a vendor update.

Give each decisive moment one row in the buying memo. Include the expected meaning, viewer answer, elapsed time, clip and owner of the fix. Set tolerances with the newsroom and accessibility lead before unblinding. A vendor that wins on aggregate accuracy but repeatedly reverses a public instruction has given the station a specific contract question: who can intervene during a live bulletin, and can the vendor demonstrate that intervention on the held-back rundown?

Archival cover: Interpretation headphones at the European Parliament, European Parliament, June 2025. Cropped and JPEG-encoded under CC BY 4.0.