Backbone Matters More Than Architectural Complexity: A Controlled Empirical Study of Hinglish Sentiment Classification

Abstract

Hindi–English code-mixed (Hinglish) text is still a challenging problem for sentiment classification because of the inconsistent transliteration, frequent code switching and linguistic variability. A number of recent works have introduced ever more intricate neural architectures, yet the impact of each architectural element and how it contributes to the finetuned backbone, language model, is not fully understood. In this paper we provide a controlled empirical analysis of the backbone selection and downstream architectural complexity on the SemEval-2020 Task 9 Hinglish sentiment dataset. Four pretrained transformer models are fine-tuned and evaluated with exactly the same settings for preprocessing, optimization, and evaluation. For every backbone, this is done by comparing a plain fine-tuning version to an architecture-enhanced version that includes CNN, BiGRU, attention, language-tag embeddings, and gated feature fusion. The effect of model robustness is then investigated in terms of the Code-Mixing Index (CMI) at various linguistic complexity levels. Experimental results demonstrate that plain MuRIL outperforms its architecture enhanced variant with 84.11% Macro-F1 score, while the latter has a Macro-F1 of 71.20%. McNemar's test and bootstrap resampling verify that the difference in performances is statistically significant at the p < 0.0001 level. The findings further illustrate the benefit of using a code-mix-aware pretrained backbone as compared to architectural richness in the downstream layers for the Hinglish sentiment task. Through the study, the authors offer useful insights and practical recommendations for building effective and effective sentiment classification systems for multilingual code-mixed text.