The MT Eval Harness is a language-agnostic evaluation framework for machine translation, purpose-built for low-resource languages where commercial metrics fall short. It produces standardized JSON reports with chrF++, BLEU, exact match, and semantic validation scores.

Designed as a companion to i18n-rosetta: translation methods developed and validated inside the harness can be exported as rosetta-compatible plugins, creating a direct pipeline from research to production i18n.

Zero language-specific dependencies. Bring your own corpus, bring your own translation provider — the harness evaluates anything that produces text.

Feed the harness a parallel corpus (source + reference translations) and a translation provider. It runs each entry through the provider, then scores the output against the reference using multiple complementary metrics — chrF++ for character-level similarity, BLEU for n-gram precision, and optional FST validity for morphologically complex languages.

The plugin export pipeline packages validated translation methods — including prompts, preprocessing gates, and coaching data — as i18n-rosetta compatible bundles with full provenance metadata.