How Well Do Generative LLMs Translate Chinese Political Texts into English? A Multi-Metric Evaluation with Human Judgment
Taking the 2026 New Year Address by China’s president as its research corpus, this study selects ten representative sentences to evaluate English translations produced by four generative large‑language models: Gemini, Doubao, DeepSeek and ChatGPT. It adopts a mixed-method evaluation framework that integrates three widely‑used automatic evaluation metrics, namely METEOR, BERTScore and COMET, together with expert human scoring. On this basis, the paper makes a comparative analysis of the consistency between scores yielded by different automatic evaluation metrics and human expert judgments. The experimental results indicate that these automated metrics differ substantially in their respective evaluation priorities, and neural‑network‑based evaluation metrics achieve relatively higher consistency with human ratings. Within the scope of the present corpus, the comprehensive translation performance of the four large‑language models is ranked in descending order as follows: ChatGPT, Doubao, DeepSeek, and Gemini.
References
- Yuan, X. Y. (2024). Translation studies from the perspective of international communication. Journal of Foreign Languages, 47(2), 74–81.
- Wang, H. S., & Liu, S. J. (2022). Exploring post-editing competence in machine translation in the era of artificial intelligence. Chinese Science & Technology Translators Journal, 35(4), 21–24.
- Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Michigan.
- Zhang, T., et al. (2020). BERTScore: Evaluating text generation with BERT. ICLR.
- Rei, R., et al. (2020). COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.