Machine Translation and Post-Editing in Open Source Projects

Machine translation is genuinely good now, in a way it was not a decade ago, and pretending otherwise does nobody any favours. It is also an unusually effective way to destroy a volunteer translation community's trust in a single release. Both things are true, and the difference between them is almost entirely about how it is introduced — not about the quality of the model.

This is our working position, developed over several projects where we got it wrong before we got it right.

Where machine translation earns its place

MT is a good fit where the content is high-volume, low-risk, and where being roughly right quickly beats being exactly right slowly:

• Documentation backlogs. Ten thousand words of API reference nobody will ever translate by hand. A post-edited machine pass is far better than nothing.
• Changelogs and release notes. High volume, short shelf life, low consequence if the phrasing is stiff.
• Community-generated content. Forum posts and issue comments, where the reader wants gist and knows they are reading a translation.
• First drafts for post-editing. A translator editing decent MT output is often meaningfully faster than one starting from an empty field, particularly for long, repetitive text.
• Triage. Understanding a bug report written in a language nobody on the team reads.

And where it does not belong:

• Interface strings. Short, context-free, high-consequence — the worst possible input for MT. "Open" with no context is a coin flip in most target languages.
• Error and warning messages. Read under stress, acted on immediately.
• Legal, safety and consent text. The consequences of an inverted negation are not linguistic.
• Anything destructive. Confirmation dialogs are where a mistranslation costs a user their work.
• Brand and marketing voice. MT produces competent, generic prose. That is exactly the wrong register for copy meant to sound like someone.

Post-editing is a defined level, not an attitude

"We will have someone check it" is not a specification. The industry distinguishes two levels, and agreeing which one applies before work starts prevents most disputes about whether the result was acceptable.

Level	Target	Editor does	Suits
Light PE	Accurate and understandable	Fixes meaning errors only; leaves clumsy but correct phrasing alone	Internal docs, archives, gist translation
Full PE	Indistinguishable from human translation	Fixes meaning, fluency, terminology, register and style	Published documentation, help centres

An honest observation from our own timesheets: full post-editing of poor MT output is frequently slower than translating from scratch, because the editor must first understand what the machine was attempting before deciding what to keep. Measure this on your own content before assuming MT saves money. For some language pairs and domains it plainly does; for others it plainly does not.

Use quality estimation to route work

Modern MT systems can estimate their own confidence per segment, and that signal is more useful than the translation itself for planning purposes. Route by score rather than treating all output identically:

• High confidence, low-risk content → light post-edit, or publish with a machine-translated label.
• Low confidence → discard the output and translate from scratch. Do not make an editor fight bad output.
• Any risk-flagged string — destructive, legal, safety — → human translation regardless of confidence.
• Numbers, placeholders, glossary terms present → automated check first, then human review. MT drops and mangles these more often than its fluency suggests.

Quality estimation is imperfect and correlates loosely with actual adequacy. Treat it as a triage signal, never as a substitute for review of anything consequential.

Constrain the machine with your terminology

Most serious MT systems support glossary or terminology constraints — forcing specific source terms to specific target renderings. Use it. Without constraints, MT will translate your product name, invent three different renderings of your central domain term across one document, and cheerfully translate a placeholder.

# Terminology constraints passed to the MT engine
glossary:
  - source: "repository"
    target: "репозиторій"
  - source: "OpenTranslators"
    target: "OpenTranslators"   # do not translate
  - source: "pull request"
    target: "запит на злиття"

# Always validate afterwards — constraints are honoured, not guaranteed
pofilter -t variables -t printf -t terminology po/uk.po errors/

And always run placeholder validation on MT output specifically. Machine systems reorder and occasionally drop format specifiers, producing translations that are fluent, plausible and crash on render.

The licensing and privacy questions people skip

Two questions that are easy to postpone and awkward to answer afterwards.

Where does your text go? Sending strings to a hosted MT service means sending them to a third party. For an open source project whose strings are public anyway, that is usually fine. For a client's pre-release product, unreleased feature names or anything under NDA, it may be a contract breach. Check the provider's data retention and training terms — several use submitted content for model improvement by default, with an opt-out you have to find. Self-hosted models exist and are worth the operational cost when confidentiality matters.

What is the licence of the output? Terms vary between providers and have changed over time. If your project is under a copyleft licence, you need reasonable confidence that MT output can be distributed under it. Read the current terms rather than relying on what was true two years ago, and record the decision so nobody has to re-litigate it.

The community question, which is the real one

Every technical point above is manageable. The part that actually breaks projects is social, and it is worth being blunt about it.

Volunteers translate because the work matters and because their contribution is theirs. Introduce MT badly and both premises are undermined at once: the work appears to have been automated away, and their remaining role looks like cleaning up after a machine. We have watched a healthy eleven-person team lose seven members in two months after MT was switched on without discussion. The output quality was not the issue — nobody was consulted, and people concluded the project no longer needed them.

What has worked, consistently:

• Ask before, not after. Bring it to the language teams as a proposal with a specific scope. Let them say no for their locale. Several will; some will ask for it.
• Scope it explicitly and hold the line. "Documentation only, never interface strings" is a commitment. Quietly expanding it later costs more trust than never starting would have.
• Label machine-translated content. In the catalogue and, where it is published, to readers. Hiding it is what turns a tool into a betrayal.
• Pay for post-editing where you paid for translation. If MT reduces cost, that is a saving for the project, not a reason to stop compensating the humans who make the output publishable.
• Give teams the switch. A locale that wants to opt out entirely should be able to, without justifying itself.
• Report on it. Share the QA scores of post-edited content alongside human translation. If it holds up, the data makes the argument; if it does not, you needed to know.

The question is never whether the machine is good enough. It is whether the people who have given you years of unpaid work were treated as though their judgement mattered.

— from a retrospective after losing most of a language team

A workable policy in six lines

This is close to what we now propose on every project. It fits on one page deliberately.

1. Machine translation is never used for interface strings, error messages, legal, safety or destructive-action text.
2. It may be used for documentation, changelogs and community content, with full post-editing before publication.
3. Any locale team may opt out entirely, for any reason, without discussion.
4. All machine-assisted content is labelled as such in the catalogue.
5. Terminology constraints and automated placeholder validation are mandatory on all MT output.
6. Post-editing is compensated at the same rate as translation until measurement shows it is genuinely faster — then the rate is renegotiated openly with the people doing it.

None of this is anti-machine. We use MT ourselves, daily, and it saves real money on the right content. The policy exists because the technology is the easy part: the hard part is introducing a labour-saving tool into a volunteer community without making the volunteers feel like the thing being saved on.

---