Where Off-the-Shelf Falls Apart
Our internal CX team's review-monitoring dashboard got garbage classifications the moment a reviewer wrote in Saudi dialect Arabic instead of Modern Standard Arabic. Fine-tuning XLM-RoBERTa on a general multilingual dataset holds ~82% F1 on English and collapses to 60–65% on Urdu and Saudi dialect Arabic — a gap no amount of extra fine-tuning epochs closes, because the pretraining corpus skews toward formal, Wikipedia-style text that customer reviews never resemble.
A brand-perception dashboard built on a model that scores dialect complaints as neutral is worse than no dashboard at all — it tells product and marketing teams that unhappy customers are fine. Saudi dialect compounds this: colloquial vocabulary, code-switched English loanwords, and orthographic variants for the same word show up constantly in review text, none of which resemble Modern Standard Arabic's formal register.
Routing By Language, Not One Model For All
AraBERT, not a general multilingual model
Pretrained on Arabic news and social text — a much closer match to customer reviews than XLM-RoBERTa's Wikipedia-heavy corpus. Picks up root-pattern morphology and cliticization a general model treats as noise.
Attention heads tuned for morphology
Sentiment in Arabic and Urdu often lives in a prefix or suffix, not the root word. Standard fine-tuning pools everything into the [CLS] token and loses that signal.
A dedicated classifier for Saudi dialect vs. MSA
Saudi dialect and Modern Standard Arabic diverge enough in vocabulary and morphology that a single AraBERT head fine-tuned on one underperforms on the other. Detecting register first, then routing, avoids splitting the difference.
One backend per language family
A single model tuned to satisfy all three languages ends up satisfying none of them particularly well. Routing by detected language adds sub-millisecond overhead.
Ablation — What The Custom Attention Heads Actually Buy You
Same 120K+ review dataset, same AraBERT/RoBERTa-large backbones, standard [CLS]-token fine-tuning swapped in for the custom attention layer. The gain is concentrated exactly where the morphology argument predicts it should be — negligible in English, largest in Urdu and Saudi dialect Arabic.
| Language | Standard fine-tune | Custom attention heads | Δ F1 |
|---|---|---|---|
| English | 97% | 97% | +0 |
| Arabic (MSA + Saudi dialect) | 90% | 95% | +5 |
| Urdu | 88% | 95% | +7 |
| Aggregate | 92% | 96% | +4 |
What The Aggregate Number Hides
Short texts break language detection in ways longer texts don't. A five-word review mixing English and Urdu stumps every detector tried — the fix was a fallback hierarchy, script analysis then langdetect then default to English, rather than trusting any single one.
Custom attention heads need careful initialization. Random initialization introduced noise into the pretrained representations early on; zero-initializing the new heads until the base model settled fixed most of the instability.
96% F1 hides where the errors actually live. Breaking the metric out by language, and by dialect within Arabic, was what made the remaining Urdu gap visible at all.