The Diacritic Blindspot: Robustness in Arabic BERT Models

This research paper presents the first systematic robustness evaluation of Arabic BERT models against adversarial text attacks, with a focus on the unique linguistic challenges of Arabic — particularly diacritics (Tashkeel).

Key Findings:
1. High-Performance Fragility: Models with the highest baseline accuracy (CAMeLBERT: 61.2%, AraBERT: 65.2%) exhibit catastrophic degradation under diacritical attacks — performance drops exceeding 22 percentage points.

1. Lower-Performance Resilience: MARBERT, despite lower baseline accuracy (57.9%), maintains remarkable stability under diacritical attacks (retention >92%), representing an explicit accuracy-robustness trade-off.

1. Tokenizer-Driven Mechanism: The forensic analysis reveals that robustness correlates strongly with tokenizer behavior — MARBERT shows high token stability (Jaccard similarity = 0.73) and minimal UNK explosion (Δ = 2,844 tokens), while CAMeLBERT shows complete token fragmentation.

Study Design:
- Total N = 2,000 samples (1,000 Tweets + 1,000 Reviews)
- 5 Arabic BERT models evaluated
- 5 adversarial attack types tested
- Cross-domain validation (social media vs. product reviews)

Keywords: Adversarial Robustness, Arabic NLP, Tokenization, BERT, Sentiment Analysis