Aligning Sound, Sight and Sense: Multimodal Semantic Understanding as a Driver of Cross-Modal Communication Effectiveness on Douyin
Short-video platforms have made tri-modal production—moving images, voice, and on-screen text—the default grammar of digital communication, yet much of this content suffers from a paradox we call modality surplus with semantic deficit: channels multiply while meanings drift apart. Drawing on multimodal discourse analysis and the cognitive theory of multimedia learning, this study conceptualizes multimodal semantic understanding (MSU) not merely as a recognition technique but as a measurable production resource, and asks whether semantic alignment across modalities improves communication outcomes. We constructed a Multimodal Semantic Consistency Index (MSCI) with a CLIP-style audio-visual-text alignment pipeline and applied it to a stratified sample of 600 Douyin videos across six content categories. Communication effectiveness was measured as a composite of completion rate, likes, comments, and shares. Hierarchical regression shows that semantic consistency is the strongest single predictor of effectiveness (β = 0.243, p < .001), raising explained variance by 23.8 percentage points over account- and format-level controls; the effect is significantly amplified in knowledge-intensive categories (β = 0.083, p < .001), and tri-modal production pays off only when consistency is high. The findings reframe platform-era communication as a problem of cross-modal meaning governance, and yield concrete strategies for creators, MCNs, and platform governance.
References
- Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. Arnold.
- Mayer, R. E. (Ed.). (2014). The Cambridge handbook of multimedia learning (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139547369
- Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML) (pp. 8748-8763). PMLR.
- Li, J., Li, D., Xiong, C., & Hoi, S. (2022). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning (ICML) (pp. 12888-12900). PMLR.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998-6008.
- Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. Springer. https://doi.org/10.1007/978-1-4612-4964-1
- Sweller, J. (2011). Cognitive load theory. In J. P. Mestre & B. H. Ross (Eds.), The psychology of learning and motivation (Vol. 55, pp. 37-76). Academic Press. https://doi.org/10.1016/B978-0-12-385527-5.00002-4
- Baltrušaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423-443. https://doi.org/10.1109/TPAMI.2018.2798607
- Zhang, Y., & Chen, L. (2024). Short-form video and the attention economy: Evidence from a natural experiment on Douyin. Journal of Digital Media Studies, 12(3), 211-234.
- Nguyen, T., & Ortiz, M. (2023). Cross-modal coherence and viewer retention in vertical video. New Media & Society, 25(9), 2401-2422.
- Wang, H., Liu, F., & Zhao, M. (2025). Algorithmic recommendation and content homogenization on Chinese short-video platforms. Communication Research, 52(4), 455-478.
- Jaakkola, T., & Haussler, D. (1999). Exploiting generative models in discriminative classifiers. Advances in Neural Information Processing Systems, 11, 487-493.