How well do machines translate into Oshiwambo?
Nobody could say — there was no way to measure it. Ongiini-Eval-OW is an open benchmark for English → Oshindonga and English → Oshikwanyama translation, with references written by native speakers and sentences in the English Namibians actually write.
Why this exists
Half of Namibia's homes speak it. No benchmark did.
Oshiwambo is the home language of roughly half of Namibian households. Yet its two written standards, Oshindonga and Oshikwanyama, are missing from FLORES-200, NLLB-200, MADLAD-400 and the major commercial translation services. Large language models will happily produce text that looks like Oshiwambo — and there was no public way to check whether it is.
We run Ongiini AI, a free AI assistant for people in Namibia. Our users keep asking for Oshiwambo. Before we can offer it responsibly, we have to be able to measure it. So we built the yardstick, and we are making it open for everyone who works on these languages.
180 of the benchmark sentences are paraphrased from real questions people ask Ongiini AI. How people use it, and what for, is public on our statistics page.
The benchmark
600 sentences, built to find where translation breaks.
The English is plain second-language English, the way Namibians write on WhatsApp — questions about school, jobs, health, government services and family life. The difficulty sits in the target language, not in the English. Sentences inspired by real conversations are rewritten in full: no user's words, names or details are in the dataset.
Where the sentences come from
Table view
| Source | Items |
|---|---|
| Paraphrased from real conversations | 180 |
| Written for a phenomenon | 210 |
| Everyday phrasebook | 150 |
| Formal and institutional | 60 |
Linguistic phenomena
Table view
| Phenomenon | Development | Blind | Total |
|---|---|---|---|
| Negation | 26 | 12 | 38 |
| Noun-class agreement | 22 | 9 | 31 |
| Pronoun coreference | 19 | 11 | 30 |
| Tense and aspect | 24 | 10 | 34 |
| Numbers and dates | 40 | 18 | 58 |
| Named entities | 28 | 11 | 39 |
| Loanwords | 22 | 9 | 31 |
| Politeness register | 24 | 12 | 36 |
| Figurative language | 21 | 9 | 30 |
| Polysemy | 21 | 9 | 30 |
| Multi-sentence | 26 | 10 | 36 |
Register
Table view
| Domain | Items |
|---|---|
| Everyday chat | 213 |
| Phenomenon items | 181 |
| Formal | 138 |
| Family and community | 50 |
| Faith | 18 |
Lengths follow machine-translation practice: 24% short (up to six words), 51% medium, 25% long (19 words or more). 420 sentences form the development split — use them freely for prompts and tuning. 180 (30%) form the blind split, stratified by phenomenon, length and register, and are only for final reporting.
What it tests
One sentence per phenomenon.
Each item is tagged with the phenomena that make it hard. A few English sources from the set:
“I never received the confirmation message you mentioned.”
single and double negation, scope
“Those two old men sat under the tree near the river.”
concord across noun, verb, adjective and numeral
“The nurse phoned the mother because she was worried.”
which person or thing a pronoun points to
“It has been raining since morning.”
recent past, perfect, habitual, progressive
“Visiting hours are from 14:00 to 16:00 daily.”
amounts in N$, dates, times
“Is the combi to Ondangwa full?”
Namibian places, ministries, institutions
“My data bundle is finished.”
English words that stay English in everyday Oshiwambo
“Kuku, did you sleep well?”
Tate, Meme, Kuku — elder, peer and child address
“Money is tight this month, so we cannot buy new clothes.”
everyday non-literal expressions
“Do you have a match to light the fire?”
one English word, several Oshiwambo words
“The bus broke down halfway. We had to walk for almost two hours in the heat. By the time we arrived, the meeting was already over.”
cohesion across two to four sentences
Leaderboard
Scores arrive with v1.0.
The first public leaderboard launches together with the full dataset. Headline scores are chrF++ on the blind split, broken down by phenomenon, length and register, with human ratings from native speakers alongside.
Quality control
Native speakers check the references.
A benchmark is only as good as its references. Before v1.0, native-speaker volunteers check them on ongiini.ai/rate: one English sentence, one translation, one question — does it say the same? Yes, almost, or no, with an optional better translation.
Raters see our references and machine translations mixed, never which is which. Planted errors show whether a rater compares carefully, and the same person also judges a strong machine translation of sentences whose reference they rated — so we learn whether the references are at least as good as the systems they will measure. Raters join through WhatsApp; we store a scrambled ID, never the phone number.
The protocol — samples, qualification rule and analysis — was fixed before the first rating: docs/rating-protocol.md. The same raters then judge the systems on the leaderboard, so the automatic scores can be checked against people.
Take part
Three ways to join in.
Built a translation model? Get it scored
For teams with a machine-translation system or a language model: let it translate the 600 public English sentences into both dialects and send us the output files. We score them privately against the references and send the results back. Submissions made before the release are, by construction, free of test-set contamination. How to submit →
Speak Oshiwambo? Check translations
Most needed right now: tell us whether translations say the same as the English — five sentences, about two minutes, started from WhatsApp. You can also add sentence pairs, one at a time. Contribute →
Work with us
Researchers, universities and teams building for Namibian languages — we would like to hear from you. hello@ongiini.ai
Roadmap
Where we are.
September 2026 Done
Concept paper and first 423 references
Design published on arXiv; the first 423 sentences are translated into both dialects.
Autumn 2026 Now
600-sentence source set, translation under way
All 600 English sources are final and public; the new sentences are with our translator.
Since October 2026 Now
Human evaluation by native speakers
Volunteers rate translations on ongiini.ai/rate. The first round checks our references side by side with Claude Opus 5 and Gemma 4, and confirmed errors go back to the translator. Then the baseline systems, so the leaderboard launches with human scores next to chrF++.
End of 2026
Dataset v1.0 and leaderboard
Checked references, automatic and human scores published on Hugging Face under CC BY 4.0. Systems submitted later are rated the same way.
People
Who makes it.
Kaarina Shoozi
Reference translator, Oshindonga and Oshikwanyama
Elizabeth Hamukwaya
Reference translator, Oshindonga and Oshikwanyama
Rauna Awene
Quality control · community management · social media
Sebastian Küpers
Benchmark design · Common Intelligence Foundation
Cite
If you use the benchmark.
The dataset will be released under CC BY 4.0, the code is MIT-licensed.
Measure it, then build it.
An open yardstick is the first step towards AI that really speaks Oshiwambo — for Ongiini AI and for everyone after us.