Translation Quality Assurance That Actually Catches Bugs

Ask most teams how they know their translations are good and the answer is some version of "a native speaker looked at it and said it reads fine". That is not a quality process. It is not repeatable, it cannot be compared between locales or across releases, and it systematically misses the errors that actually hurt users — because the errors that hurt users are rarely the ones that read badly.

A translation QA process worth having does three things: it classifies errors by type and severity, it produces a number you can compare, and it catches the specific failure modes that turn a translation bug into a support ticket or a data-loss incident.

What actually goes wrong

From our review data, the errors that reach production and cause real harm cluster in a few places, and elegance is not among them:

• Inverted destructive actions. A confirmation dialog where "Discard changes" and "Keep editing" were swapped. Every user who trusted their language lost work.
• Broken placeholders. %(count)d translated as %(cont)d. Crash on render, in one locale only, discovered by users.
• Inverted negations. "Do not share this data" becoming "Share this data". Reads perfectly; means the opposite.
• Terminology drift. The same concept translated four different ways across one interface, so the search function and the help documentation no longer agree.
• Truncation. A correct translation that does not fit, silently clipped to something that reads as an entirely different instruction.
• Untranslated safety-critical text. The one string a user most needs to understand still in English, because it was added after the freeze.

Notice that a fluent-reading review catches almost none of these. Severity has to be part of the model from the start.

Build on MQM rather than inventing a scheme

Multidimensional Quality Metrics is an open framework for classifying translation errors. Use a subset — seven categories is plenty for software — and resist the temptation to invent your own taxonomy.

Category	Covers	Example
Accuracy	Meaning changed, added or omitted	"Delete" rendered as "Archive"
Fluency	Grammar, spelling, register, punctuation	Wrong case ending after a numeral
Terminology	Departs from the glossary	"Repository" translated three ways
Locale convention	Formats, units, addresses, quotation marks	US date order in a European locale
Markup	Placeholders, tags, entities, accelerators	Dropped <strong>, mangled %s
Design	Fits the space, wraps sensibly	Button label clipped at 12 characters
Style	Matches the style guide's voice	Informal register in a formal product

Two properties make MQM worth adopting rather than rolling your own: the categories are stable enough that two reviewers usually agree on classification, and because it is an open standard, an external vendor's report can be compared with your internal one.

Severity is where the value is

Category alone is not enough — a missing comma and an inverted warning are both "fluency" and "accuracy" respectively, but they are not remotely comparable. Weight them.

Severity	Definition	Weight
Critical	Causes data loss, security exposure, legal breach, or a crash	25
Major	User misunderstands and takes the wrong action	5
Minor	Noticeable and wrong, but the user still succeeds	1
Neutral	Preference, recorded but not scored	0

The neutral bucket earns its place. Reviewers need somewhere to put "I would have said it differently" that does not inflate the error count, and translators need to see that such notes are not being counted against them. Without it, reviewers either suppress the observation or score it as an error; both are bad.

The score itself is straightforward:

penalty      = Σ (errors × severity weight)
score        = 100 × (1 − penalty / word_count)
pass         = score ≥ 95 and critical_errors == 0

The second condition matters more than the first. A single critical error fails the sample regardless of the score, because one inverted confirmation dialog is not offset by nine hundred correct strings.

Sampling: review the risky strings, not a random slice

Reviewing everything is unaffordable for most projects. Random sampling is affordable and nearly worthless, because the strings that matter are a small, identifiable minority. Stratify instead:

• Everything destructive. Delete, discard, revoke, overwrite, permanently, cannot be undone. 100% review, no exceptions, every release.
• Everything legal or financial. Consent text, terms, prices, tax language, refund policy. 100%.
• Error and warning messages. Users read these under stress and act on them immediately. 100% for the top fifty by frequency.
• Onboarding and first-run. Highest-visibility strings in the product; a bad impression here is expensive. 100%.
• Everything else. A 10–20% random sample, weighted towards strings shown most often.

This typically brings a large catalogue down to a few hundred strings for full review — affordable, and covering essentially all the risk.

Automate everything a machine can judge

A significant share of real translation defects are mechanically detectable. Anything on this list should never reach a human reviewer, because a machine catches it faster and never gets bored:

• Placeholder mismatch between source and target — the single highest-value automated check.
• Markup and tag imbalance: unclosed tags, dropped attributes, mangled entities.
• Untranslated strings identical to source in a locale where that is implausible.
• Glossary violations — the source contains a glossary term, the target does not contain its approved rendering.
• Double spaces, leading and trailing whitespace differences, wrong quotation marks for the locale.
• Length overruns against declared UI limits.
• Accelerator and shortcut key conflicts within one screen.
• Number and unit consistency: the source says 30 days, the target says 30 — not 60.

# Translate Toolkit — placeholder and variable checks across all locales
pofilter --gnome \
  -t printf -t variables -t xmltags -t escapes -t doublespacing \
  po/ review/errors

# Fail CI if anything landed in the error directory
find review/errors -name '*.po' -size +0 | grep . && exit 1 || true

Human review time is expensive and finite. Spending it on things pofilter would have caught in two seconds is the most common waste we see in translation QA programmes.

Review in context, not in a spreadsheet

A reviewer looking at a list of strings cannot see that "Open" is a column header, that the button is 90 pixels wide, or that the confirmation dialog has its buttons in the reverse order from the source. In-context review catches a category of error that out-of-context review structurally cannot.

In descending order of usefulness:

1. A running build in the target locale. Best, and worth the setup cost for any product with a UI. The reviewer navigates as a user would.
2. Automated screenshots per string. A CI job that captures each screen in each locale and attaches images to the catalogue. Very good, and reusable across releases.
3. Manual screenshots for key screens. Better than nothing; goes stale quickly.
4. Source comments and character limits. The bare minimum. Every string should have at least this.

Nearly every catastrophic translation bug we have investigated shared one property: the string was correct in isolation and wrong in place.

— from a post-mortem on a swapped confirmation dialog

Write a report the team will act on

A quality report that lists two hundred errors of equal weight gets skimmed and filed. One that is structured by what to do first gets fixed. Ours have five sections, in this order:

1. Verdict. Pass or fail, the score, and the count of critical errors. One paragraph.
2. Criticals in full. Every one, with the string ID, the source, the translation, a plain-English explanation of the consequence, and the suggested fix. This section is the report; everything else is supporting material.
3. Patterns. Systematic issues rather than instances — "the glossary term for repository is inconsistently applied across 34 strings" is one finding, not 34.
4. The full error table. Sortable, with category, severity, string ID and comment.
5. Recommendations. Process changes, glossary additions, automated checks to add. What stops this recurring.

Send it to the team lead before it goes anywhere else. A translator who first learns their locale failed by seeing it in a management summary will not enjoy the next review, and their cooperation is what makes the next one accurate.

Close the loop, or stop bothering

QA that produces reports nobody acts on is theatre. Three mechanisms convert findings into changed outcomes:

• Every critical error becomes a test. A swapped confirmation becomes an automated check that the confirm and cancel labels are not transposed. The class of bug is closed, not just the instance.
• Every terminology finding updates the glossary. With the reasoning recorded. If a term was ambiguous enough to be got wrong once, it will be got wrong again.
• Every pattern finding changes the process. If truncation keeps appearing, the fix is character limits in the catalogue and a pseudo-locale screenshot job — not a note asking translators to be more careful.

And track the score over time. A locale improving from 91 to 97 across three releases tells you the process is working. A locale flat at 96 tells you that you are sampling the same easy strings every time.

A programme you can start on Monday

You do not need a QA department. In roughly a week, with one person:

1. Day one. Run pofilter across every catalogue. Fix what it finds. This alone typically removes a large fraction of the real defects.
2. Day two. List your destructive, legal and top-fifty error strings. That list is your permanent full-review set.
3. Day three. Write the glossary for the twenty terms that appear most often. Twenty is enough to start; a hundred is a project.
4. Day four. Review the full-review set for one locale using the MQM categories and the severity table above. Score it.
5. Day five. Write the report in the five-section structure. Send it to the team lead. Agree on one process change.

Then repeat every release, rotating which locale gets the deep review. Within a year you have trend data, a glossary that reflects real decisions, an automated check suite that grows with every incident, and — most valuably — translators who know precisely what "good" means in your project, because someone finally wrote it down.

---