VINAYAKA: Multilingual Audio-Visual Hate Speech Detection via Cross-Modal Fusion in Hyperbolic Space

Abstract

In this work, we investigate the robustness of audio–visual (AV) cues for Multilingual Audio–Visual Hate Speech Detection (M-AVHSD). We propose VINAYAKA, a novel framework for M-AVHSD that relies solely on AV cues for decision-making. Along with WavLM and ImageBind, it employs Cross-modal Fusion in Hyperbolic Space (CFHS), which models the effective alignment between paralinguistic and behavioral AV cues to better capture hateful intent in temporal audio signals and corresponding visual scenes. Empirical results demonstrate that the proposed framework achieves state-of-the-art performance in both in-distribution and out-of-distribution settings across cross-lingual and cross-dataset evaluations. Furthermore, the results show that the framework remains robust to ASR error propagation.

Method

VINAYAKA (Multilingual Audio-Visual Hate Speech Detection via Cross-Modal Fusion in Hyperbolic Space) detects hateful intent directly from audio and visual signals without relying on textual transcriptions. The framework combines pretrained audio and visual representations with hyperbolic cross-modal reasoning to model interactions between paralinguistic speech cues and behavioral visual cues. By performing multimodal fusion in hyperbolic space, VINAYAKA captures hierarchical relationships between modalities and improves generalization across languages and datasets.

Citation

@inproceedings{vinayaka2026,
    title={Multilingual Audio-Visual Hate Speech Detection via Cross-Modal Fusion in Hyperbolic Space},
    author={Bhavinkumar Vinodbhai Kuwar and Orchic Chetia Phukan and Rajesh Sharma},
    booktitle={Interspeech},
    year={2026}
}