Skip to content

MQM: Measuring Translation Quality

August 21, 2014

Shamus Dermody

For many years within the Global labeling/marketing/UI/Content departments of Enterprises Worldwide and indeed within the Translation and localization industry the quality of delivered translations has been very difficult to measure and define.   Choosing translation providers or individual translators that claim the highest quality process’s or have the strongest reputations has not always proved to deliver the desired and expected results.    

Variable requirements and complexity make it difficult to consistently evaluate a language deliveries.    

The ability to rate quality in an effective and uniform manner has long been a requirement in general in the industry but of crucial importance especially for high risk life science translation projects and translations for high risk important patient safety medical device labeling or User Interface Strings..

For example, a translation project, may have a negative reaction from an end in-country recipient client.    But there can be several reasons;

  • Low quality of the source content
  • Translation process issues
  • Subjective views of End Country reviewer
  • Incomplete or low quality source or target terminology
  • Software Internationalization issues ( Often caused by lack of a complete Pseudo Localization process)
  • Low quality translation

Process owners need to be able to rate the performance of various translation resources at their disposal for a given language in a consistent and uniform manner.

Another example a Translation Process Owner needs to assess the relative quality of a number of translators for a given language pair. A particularly high sensitivity project may demand the highest level translation resources.   A historical record from a pool of translators assessed in a uniform and managed methodology would significantly help to choose those with a consistently high quality of output.

QTLaunchPad is an EU FP7 funded project designed to help improve translation quality. The key to any form of translation (be it human or machine translation or machine translation post-editing) quality is the ability to provide consistent and measurable metrics. In the past initiatives such as the LISA QA model or JAE2450 have tried to approach this, but without a great deal of success in terms of industry acceptance, mainly due to the fact that they adopted a ‘one size fits all’ approach to all translation projects, whereas in reality there is a very wide spectrum of diversity within the type of translation task being undertaken. In addition these models have not kept up with the rapid pace of adoption of machine translation and subsequent post-editing nature of translation in this field.

The main goal of the QTLaunchPad project is establishing the concept of Measurable Quality Metrics (MQM). MQM is at the heart of QTLaunchPad and is a completely open standard.  These standards were designed by a committee of industry experts and are open to public scrutiny as part of their development.

In terms of the QTLaunchPad the key participants were the German Research Center for Artificial Intelligence (DFKI), Dublin City University, the University of Sheffield and the Athena Institute for Language and Speech Processing. All leading academic institutions with access to world renowned experts in the filed including Arle Lommel who was one of the major architects of the LISA QA model.

MQM principles

MQM is based on the following key principles:

  1. Flexibility: MQM eschews the ‘one size fits all’ model of previous QA initiatives. It allows adapting metrics to the specific nature of the translation project to hand. Translation projects do not all have the same quality requirements or complexity. Documentation destined for automotive technicians in the workshop may have a quality requirement of ‘fit for purpose’ (technicians never read the manuals anyway, well only as a matter of last resort when all other attempts including hitting the affected item hard with a hammer have failed), whereas owner glove box instructions for a car must have no errors whatsoever. Metrics must be ‘tuneable’ along different dimensions, e.g. domain, purpose, audience. MQM provides a flexible catalogue of issues types that can be extended and adapted as required for the project to hand.
  2. Fairness: MQM allows for the defining the true cause of the problem. If the source text is badly and ambiguous, then the blame should not be laid at the feet of the translator. If the source text is of a low quality then it must be possible to determine this.
  3. Suitability: MQM is suitable for all types of translation, production methods and technologies, as well as allowing extensibility for new categories of checks.
  4. Comparability: MQM results must be comparable with each other even if the assessments tasks are not checking the same thing.
  5. Standards: MQM is based on ISO 11669 for defining QA check dimensions.
  6. Granularity: MQM supports various levels of granularity as required by the given project to hand.

MQM features

MQM offers the following features:

  1. Open Standard: MQM is a completely open and unencumbered standard.
  2. Flexibility: MQM provides a comprehensive and extensible list of issues types.
  3. Separation: MQM provides for identifying both source and target issues.
  4. Granularity: MQM allows for a hierarchical view of issues from high level to finely grained.
  5. Dimensions: MQM provides for up to eleven dimensions of QA issues, based on ISO/TS-11669, to guide users in choosing the appropriate types of issue for QA tasks.
  6. Backwards Compatibility: MQM supports previous QA models such as the LISA QA model or SAE J2450.
  7. Extensibility: MQM allows a project manager to add additional or more finely grained issue types.

The importance of MQM

Using MQM, a project manager can now provide an unambiguous, non-subjective and systematic quality assessment of the work of a translator. This takes into account any shortcomings of the original source text and can be used over time to build up a complete and comprehensive objective assessment of a linguist’s quality of work.

The functionality offers benefits for both – Alex and John alike: one being able to work with only the best, and the other one being fully appreciated for all the hard work he’s done.

MQM provides a significant improvement based on a fair and open standard of the quality of the work of individual translators.

XTM Cloud and MQM

MQM provides a significant improvement based on a fair and open standard of the quality of translation deliveries. The functionality so far has only been introduced into one commercial translation management technology – the latest release of XTM Cloud the Enterprise Translation Management System from XTM.   It is available under the LQA (Linguist Quality Assurance) section and is a predefined optional workflow step in XTM.

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: