Open Problems and Future Directions in Multimodal Affective Computing: A Confidence-Aware Fusion Case Study

Abstract

Affective computing has moved from single-channel emotion classification toward deep multimodal systems that jointly exploit speech, text, facial cues, and physiological signals. This paper's primary contribution is a structured review of that shift, organised around unimodal recognition methods, multimodal fusion strategies, and the deep architectures now driving progress — convolutional and recurrent backbones, transformers, graph neural networks, selective state-space models, and self-supervised learners. To test whether the review's central architectural claim holds empirically, and to illustrate, we then reimplement a confidence-aware fusion mechanism under a common training protocol on the MELD and CMU-MOSEI conversational benchmarks and compare it against our own reimplementations of two recent reliability-aware and graph-based methods from the literature. Under clean conditions the confidence-aware variant improves weighted-F1 by 1.6 points over the stronger reliability-aware baseline and by 13.0 points over the static-graph baseline; under 50% single-modality corruption, its weighted-F1 falls by only 6.3 points, against 18–20 points for the reimplemented baselines. These results support a claim the review otherwise makes on architectural grounds: reliability estimation is not a peripheral add-on but a load-bearing component of robust multimodal fusion. The case study also exposes what reliability modelling alone does not solve, synthetic corruption does not fully represent real sensor failure, graph construction scales poorly with dialogue length, and fixed hyperparameters limit generalisation across deployment settings. Six open problems are analysed in this light: missing or unreliable modalities, cross-domain and cross-subject generalisation, data scarcity, privacy constraints, limited explainability, and the real-time deployment costs that reliability-aware methods themselves introduce. Comparative tables summarise representative unimodal methods, fusion strategies, and architectures drawn from recent literature, and each open problem is mapped to a corresponding research direction.