Evaluation Pitfalls in Post-Training Quantization of Diffusion Transformers: An Empirical Audit

Zeming Jin

doi:10.54097/v6881m05

Evaluation Pitfalls in Post-Training Quantization of Diffusion Transformers: An Empirical Audit

Zeming Jin

Frontiers in Computing and Intelligent Systems, (2026), Vol.18, No.2, pp.87-96

Published: October 30, 2026

DOI: 10.54097/v6881m05

PDF: Download Full Text PDF

Abstract

Diffusion transformers are moving toward practical deployment, and post-training quantization (PTQ) has emerged as a standard, retraining-free compression route. The conventions used to evaluate PTQ for diffusion transformers—the Fréchet Inception Distance (FID) of images produced by a fixed sampling pipeline, with a cheap calibration-error statistic as a configuration-search proxy—are typically inherited from prior codebases rather than audited. Auditing this pipeline through a DiT-XL/2 (ImageNet 256×256) case study of 209 controlled experimental records, this paper identifies three systematic pitfalls. First, sampling-pipeline configuration: two latent defects in the legacy pipeline manufactured spurious method-level differences, and controlled A/B reproduction (deviation below 1%) shows that correcting them reverses earlier conclusions on channel equalization. Second, proxy-metric failure: 32 records sharing a bit-identical calibration error span an FID range of 98.74, and its rank correlation with FID is statistically indistinguishable from zero in every experiment family tested. Third, calibration-protocol sensitivity: under the corrected protocol, an error-aware dynamic-programming grouping strategy is statistically indistinguishable from a fixed-interval baseline (273.62±1.78 vs. 274.32±1.80), while group-wise equalization turns from apparently beneficial to harmful (273.62→313.23). The only method-level conclusion that survives the protocol correction—under three seeds, a single model, and the 8-bit weight-and-activation (W8A8) configuration studied—is that grouped quantization outperforms no grouping (388.96→274.32). This work closes with a corrected evaluation-protocol checklist.

Keywords: Post-training Quantization, Diffusion Transformer, Evaluation Audit, Fréchet Inception Distance, Calibration

APA Citation: Zeming Jin (2026). Evaluation Pitfalls in Post-Training Quantization of Diffusion Transformers: An Empirical Audit. Frontiers in Computing and Intelligent Systems, 18(2), 87-96. https://doi.org/10.54097/v6881m05

References

Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV51070.2023.00449.
Wu, J., Wang, H., Shang, Y., Shah, M., & Yan, Y. (2024). PTQ4DiT: Post-training Quantization for Diffusion Transformers. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
Chen, L., Meng, Y., Tang, C., Ma, X., Jiang, J., Wang, X., Wang, Z., & Zhu, W. (2025). Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Zhao, T., Fang, T., Liu, E., Rui, W., Soedarmadji, W., Li, S., Lin, Z., Dai, G., Yan, S., Yang, H., Ning, X., & Wang, Y. (2024). ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation. arXiv preprint arXiv:2406.02540.
Ding, N., Han, J., Tian, Y., Xu, C., Han, K., & Tang, Y. (2025). Post-Training Quantization for Diffusion Transformer via Hierarchical Timestep Grouping. arXiv preprint arXiv:2503.06930.
Hwang, Y., Lee, H., & Kang, J. (2025). TQ-DiT: Efficient Time-Aware Quantization for Diffusion Transformers. arXiv preprint arXiv:2502.04056.
Liu, W., & Zhang, S. Q. (2024). HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization. arXiv preprint arXiv:2405.19751.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the International Conference on Machine Learning (ICML). https://doi.org/10.48550/arXiv.2211.10438.
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., & Keutzer, K. (2023). Q-Diffusion: Quantizing Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV51070.2023.00244.
Parmar, G., Zhang, R., & Zhu, J.-Y. (2022). On Aliased Resizing and Surprising Subtleties in GAN Evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR52688.2022.01137.
Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y., Ross, B. L., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, J. E. T., & Loaiza-Ganem, G. (2023). Exposing Flaws of Generative Model Evaluation Metrics and their Unfair Treatment of Diffusion Models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
Obukhov, A., Seitzer, M., Lin, E. Y.-J., Kyl, J., Zhydenko, S., & willylulu. (2021). torch-fidelity (Version 0.3.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.4957738.
Lee, D., Hur, J., Shon, H., Lee, J. Y., & Kim, J. (2025). DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Feng, W., Qin, H., Yang, C., Li, X., Yang, H., Li, Y., An, Z., Huang, L., Magno, M., & Xu, Y. (2025). S²Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
Ryu, S., Salzmann, M., & Javed, S. (2026). Q-Drift: Quantization-Aware Drift Correction for Diffusion Model Sampling. arXiv preprint arXiv:2603.18095.
Sørensen, M., Holm, F., Kristensen, A., Jensen, S., Madsen, L., Rasmussen, E., & Aline, C. (2025). Quantization as a Foundation for Deployable High Performance Diffusion Models within the Landscape of Large Scale Generative AI. TechRxiv preprint. https://doi.org/10.36227/techrxiv.175624440.05705133.
Yang, K., Yang, K., Wu, B., Zhang, X., Yang, Q., Qin, H., Zhang, H., & Zhang, Y. (2025). TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search. arXiv preprint arXiv:2512.06353.
Sun, R., Yang, T., Niu, W., & Sun, J. (2025). RSTR: Reducing SpatioTemporal Redundancy in Diffusion Transformers. arXiv preprint arXiv:2512.14096.
Dufour, N., Efros, A. A., & Pérez, P. (2026). The FID Lottery: Quantifying Hidden Randomness in Generative Model Evaluation. arXiv preprint arXiv:2606.20536.
Yang, L., Lin, H., Zhao, T., et al. (2025). LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation. arXiv preprint arXiv:2508.03485.
Kim, D., Hwang, J., Oh, C., & Park, J. (2025). MixDiT: Accelerating Image Diffusion Transformer Inference with Mixed-Precision MX Quantization. arXiv preprint arXiv:2504.08398.