Adaptive Dual-Script Fusion for Low-Resource Text Classification with Selective State-Space Modelling

Abstract

Kashmiri is a low-resource language where the amount of labelled data is lower compared to other natural language processing tasks and the language is mainly written in Perso-Arabic (Nastalik) script. In this paper, a proposed model with two classifiers using pooled dual-script transformer embeddings with two state-of-the-art models: LightGBM and CatBoost, as well as a custom sequence model called DSF-Mamba, that combines the Perso-Arabic and Devanagari-transliterated representations using a gated fusion layer followed by a selective state-space encoder is evaluated for multiclass Kashmiri text classification. The three models use pre-trained frozen non-fine-tuned embeddings of XLM-RoBERTa (Perso-Arabic branch) and MuRIL (Devanagari-transliterated branch) to save computation for fine-tuning the end-to-end transformer, while still taking advantage of the pre-trained multi-lingual representations. The self-created dataset consists of 51,000 sentences, which upon cleaning process yielded 50,966 sentences. The proposed DSF-Mamba classifier achieves the highest accuracy (89.25%) on a held-out test set with 7,645 sentences, which comes from an 85/15 stratified split in comparison with LightGBM (87.30%) and CatBoost (83.96%). A theoretical analysis of the dual-script fusion mechanism is also provided, as part of an expanded version of a class-level error analysis, along with discussion of the model architectural and training differences of the three models. The results indicate that simple, frozen embedding sequence models can provide a desirable accuracy compute ratio for low-resource language classification tasks, while still allowing for hybrid and ensemble methods.