Multimodal Emotion Recognition From Images and Text Using Deep Learning and Explainable AI

Why It Matters

Integrating text and images boosts emotion‑recognition performance while explainable AI ensures trust, advancing affective computing and human‑computer interaction.

Key Takeaways

• ITMER fuses text and images for emotion classification.
• Multimodal model reaches 92% accuracy, beating unimodal baselines.
• Image-only best accuracy 83.81%; text-only best 71%.
• Grad‑CAM and LIME provide visual and textual explanations.
• Systematic comparison shows complementary strengths of each modality.

Pulse Analysis

Emotion recognition has long been a cornerstone of affective computing, yet most commercial systems rely on a single data stream—either speech, video, or text. This siloed approach limits the richness of cues available to infer nuanced affective states, especially in contexts where one modality may be ambiguous or noisy. By acknowledging that humans express feelings through both language and facial cues, researchers are pushing toward more holistic models that can capture the full spectrum of emotional expression.

The ITMER framework addresses this gap by deploying state‑of‑the‑art transformer encoders for textual inputs alongside deep convolutional networks for visual inputs, then merging their latent representations through a learned fusion layer. Empirical results on benchmark datasets reveal a consistent performance edge: the multimodal model tops out at 92 % accuracy and a macro F1 of 92 %, outpacing the strongest unimodal contenders by double‑digit margins. Such gains underscore the complementary nature of linguistic sentiment and facial micro‑expressions, confirming that each modality contributes unique, non‑redundant information that strengthens overall predictive power.

Beyond raw performance, ITMER incorporates explainable AI tools—Grad‑CAM highlights image regions influencing decisions, while LIME surfaces pivotal words in text. This transparency is critical for deployment in sensitive domains like mental‑health monitoring, customer experience analytics, and adaptive user interfaces, where stakeholders demand insight into algorithmic reasoning. As businesses seek more empathetic AI, the combination of multimodal accuracy and interpretability positions ITMER as a blueprint for next‑generation emotion‑aware technologies.