Less is More: Benchmarking LLMs for Arabic Sentiment Analysis

This research paper presents a rigorous benchmark comparing generalist large language models with specialized encoder-based models for the task of Arabic sentiment analysis on noisy, real-world data.

Models Evaluated:
- Llama-3 (Generalist LLM)
- Qwen-2.5 (Generalist LLM)
- CAMeLBERT (Specialized Arabic Encoder — Baseline)

Dataset:
- 3,822 samples of Arabic company reviews
- Noisy, colloquial Arabic from real-world sources
- Binary sentiment classification (positive/negative)

Key Results

| Model | Accuracy | F1-Score | Avg Latency |
|------------|----------|----------|-------------|
| Qwen-2.5 | 91.52% | 91.54% | 0.405s |
| CAMeLBERT | 89.32% | 89.42% | N/A (base) |
| Llama-3 | 87.86% | 87.92% | 0.290s |

Findings:
- Qwen-2.5 achieves the highest accuracy (91.5%) but is 28.3% slower than Llama-3
- Llama-3 offers the best accuracy-speed trade-off for production systems
- Statistical significance confirmed (McNemar test, p < 0.00001)
- Specialized encoders remain competitive for resource-constrained deployments

Practical Recommendation: For Arabic sentiment pipelines, Llama-3 provides the optimal balance of speed and accuracy.