Answer Distribution Bias in OmniBench: How Answer-Position Skew Affects Multimodal Large Language Model Evaluation

© 2026 by the Authors. Licensee Whioce Publishing, Singapore. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution 4.0 International License ( https://creativecommons.org/licenses/by/4.0/ )

Download PDF

Cite

XML

HTML

Abstract

OmniBench is a widely used tri-modal (image–audio–text) benchmark containing 1,142 four-choice multiple-choice questions. We discover a severe answer-position skew in OmniBench: option D is correct 48.6% of the time (χ² = 384.34, p = 5.46×10⁻⁸³), nearly twice the expected 25%. To test whether this skew distorts evaluation outcomes, we design an option-shuffling experiment: keeping all question content unchanged, we randomly reassign letter labels so that the correct answer is uniformly distributed (D ≈ 25%), then re-evaluate the same models. Results show that accuracy changes significantly in two of three tested models after shuffling (up to 4.20%, p < 0.01), demonstrating that unequal answer distribution can significantly bias model evaluation outcomes. Furthermore, we propose a label-free content-scoring evaluation method based on conditional log-probability, which achieves distribution-invariant evaluation (accuracy difference ≤ 0.18%, p > 0.4).

References

Li Y, Wu B, Zhao F, et al., 2024, OmniBench: Towards the Future of Universal Omni-Language Models, arXiv preprint arXiv:2409.15272.

Yue X, Ni Y, Zhang K, et al., 2023, MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, Proceedings of CVPR 2024. arXiv:2311.16502.

Liu Y, Duan H, Zhang Y, et al., 2023, MMBench: Is Your Multi-modal Model an All-around Player? Proceedings of ECCV 2024. arXiv:2307.06281.

Fu C, Chen P, Shen Y, et al., 2023, MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, arXiv preprint arXiv:2306.13394.

Wang B, Zou X, Lin G, et al., 2024, AudioBench: A Universal Benchmark for Audio Large Language Models, arXiv preprint arXiv:2406.16020.

Pezeshkpour P, Hruschka E, 2023, Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions, Findings of NAACL-HLT 2024. arXiv:2308.11483.

Zheng C, Zhou H, Meng F, et al., 2023, Large Language Models Are Not Robust Multiple Choice Selectors, Proceedings of ICLR 2024 (Spotlight). arXiv:2309.03882

Zhao Z, Wallace E, Feng S, et al., 2021, Calibrate Before Use: Improving Few-Shot Performance of Language Models, Proceedings of ICML 2021. arXiv:2102.09690.

Zhou H, Wan X, Proleev L, et al., 2023, Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering, Proceedings of ICLR 2024. arXiv:2309.17249.

Yue X, Zheng T, Zhang K, et al., 2024, MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark, arXiv preprint arXiv:2409.02813.

Previous article in this issue

Next article in this issue