An Attention-driven BiLSTM Framework towards Computation of SEP in 6G Networks operating over Nakagami-m Fading Channels

Abstract

The recent emergence of 6G wireless networks has given rise to the demand for accurate channel characterization across the terahertz (THz) frequency band to capture the rapid fluctuations in signal propagation. Symbol error probability (SEP) acts as a crucial performance metric for communication systems operating over 6G fading environments. Analytical and approximate approaches to SEP computation are based on numerous mathematical simplifications, complex integrals, and models that are specific to the channel conditions, thus making SEP calculations intensive and less flexible to meet the requirements of new 6G paradigms as they evolve with nonlinear THz channel fading, high-density interference, and changing noise environments. This paper presents an intelligent, data-driven SEP estimator for Nakagami-m fading channels that utilizes an attention-guided bi-directional long short-term memory (AG-BiLSTM). This supervised AI-based estimation method reduces computational complexity and improves estimation accuracy compared to conventional Q-function-based estimators. Experimental results demonstrate that the AI-based estimation model has a Mean Absolute Error (MAE) of approximately 1.25 × 10⁻³, averaged over all tested cases, and an R² value of 0.9707. Further, SEP performance for M-QAM signals over the AG-BiLSTM model is verified using Monte-Carlo simulations of order O(10⁶).