Video-MME Benchmark Explained: What Long Video Really Tests
Video-MME benchmark: 900 videos, 2,700 QA pairs, 11 seconds to 1 hour. Check the score's duration bucket, subtitle setting and whether it is lab-reported.
Video-MME is a benchmark of 900 videos (254 hours in total) with 2,700 question-answer pairs, built to test multimodal large language models on video analysis. The paper, by Chaoyou Fu and 20 other authors, was first submitted on 31 May 2024 (arXiv:2405.21075). Its videos run from 11 seconds to 1 hour, so a "long video" score covers only one of three duration buckets.
What is Video-MME and who built it?
Video-MME is an evaluation benchmark for multimodal large language models (MLLMs) that analyze video. The authors describe it as "the first-ever comprehensive evaluation benchmark" of MLLMs in video analysis (Video-MME project page). That is the authors' own framing. It is not an independent ranking, and this article does not compare Video-MME against other video benchmarks.
What does Video-MME measure?
It measures video understanding across varied content, length and input types. The project page says the videos span 6 primary visual domains and 30 subfields, which the authors say ensures broad scenario coverage (Video-MME home page).
| Axis | What Video-MME covers | Source |
|---|---|---|
| Visual domains | 6 primary domains, 30 subfields | Project page |
| Duration | 11 seconds to 1 hour | arXiv |
| Inputs | Video frames, optionally with subtitles and audio | arXiv, Epoch AI |
| Scale | 900 videos, 254 hours, 2,700 QA pairs | arXiv, project page |
The input axis matters most for interpreting scores. The arXiv abstract lists subtitles and audio as inputs beyond video frames, so the same video can be tested with different information available to the model.
How is the dataset built?
The authors selected and annotated the 900 videos by hand. According to the project page, annotators watched all of each video repeatedly before writing the 2,700 question-answer pairs (Video-MME home page). Epoch AI and Emergent Mind both describe the questions as multiple-choice (Epoch AI, Emergent Mind). The arXiv paper and project page do not state the answer format.
What are the duration buckets?
Video-MME sorts videos into three groups. Epoch AI and the project page both give the same ranges: short videos are under 2 minutes, medium videos run 4 to 15 minutes, and long videos run 30 to 60 minutes (Epoch AI, project page).
| Bucket | Stated duration | Gap in the sources |
|---|---|---|
| Short | Under 2 minutes | None stated |
| Medium | 4 to 15 minutes | 2 to 4 minutes not assigned |
| Long | 30 to 60 minutes | 15 to 30 minutes not assigned |
The buckets are not contiguous. A video of 3 minutes or 20 minutes does not appear in any stated range, so a reader cannot tell from the sources which bucket such a video falls into.
How is a Video-MME score calculated?
The leaderboard reports accuracy percentages, broken down by video duration and by subtitle setting (Epoch AI). The project page presents accuracy for short, medium and long videos, with and without the corresponding subtitles as input (project page). Accuracy is the only metric the cited sources name.
Subtitles are a separate input, not a cosmetic setting. Inference: a model given subtitles can answer some questions from the text, so a subtitle score and a frames-only score measure partly different things. The sources do not give a verified size for that effect.
Which model led Video-MME in the original paper?
The paper states that Gemini 1.5 Pro was the best-performing commercial model in its experiments (arXiv). That statement is from the first version of the paper, submitted on 31 May 2024. It describes a May 2024 result. The sources used here do not establish a current leaderboard position, so do not carry that ranking forward to newer models.
What does a long-video score tell you?
A long-bucket score tells you how a model handles videos of 30 to 60 minutes. It does not tell you how the model handles a 90-second clip, and a headline average blends the buckets together. Inference: a model can do well on short videos and poorly on long ones, and an overall average would hide the difference.
When you compare models on Video-MME, record these five fields with every score:
- Duration bucket (short, medium or long)
- Subtitle setting (with or without subtitles)
- Benchmark version and the date the score was captured
- Harness or evaluation setup, since results can shift with it (see our harness effect analysis)
- Whether the number is lab-reported or independently measured
For a general method, see how to read AI leaderboards.
What's still unclear?
- Subtitle protocol. The sources cited here do not state how subtitles are passed to each model, so the exact comparison between settings is not verified.
- Bucket gaps. Videos of 2 to 4 minutes and 15 to 30 minutes have no stated bucket.
- Answer format. Epoch AI and Emergent Mind describe the questions as multiple-choice. The arXiv paper and project page do not state it.
- Age of the standings. The one named leader comes from May 2024. Any newer ranking needs its own dated source.
- Authors' claims. "First comprehensive" is the authors' own label, and the cited sources do not include independent confirmation of it.
- Statistical detail. The cited sources do not give confidence intervals or the rule used to combine scores across buckets.
The weakest point of the benchmark, as described in these sources, is the mismatch between its 2024 launch and any current model comparison. A table that mixes a 2024 Video-MME result with a 2026 result needs a version and date flag on every row.
Last checked: 26 October 2026, against the sources listed in the front matter.
Frequently asked questions
What is the Video-MME benchmark?
Video-MME is an evaluation benchmark for multimodal large language models that analyze video. Its authors, Chaoyou Fu and 20 other authors, describe it as the first comprehensive benchmark of this kind. It covers 6 primary visual domains and 30 subfields, and accepts video frames alone or with subtitles and audio. The first version of the paper was submitted on 31 May 2024.
How many videos and questions does Video-MME contain?
Video-MME contains 900 manually selected and annotated videos, totaling 254 hours, and 2,700 question-answer pairs. The videos range from 11 seconds to 1 hour. The project page says annotators watched each video repeatedly before writing its questions. Epoch AI and Emergent Mind describe the questions as multiple-choice. The arXiv paper does not state the answer format.
What are the Video-MME duration buckets?
Video-MME sorts videos into short (under 2 minutes), medium (4 to 15 minutes) and long (30 to 60 minutes) groups. The sources cited here do not assign videos of 2 to 4 minutes or 15 to 30 minutes to a bucket. A long-bucket score therefore describes only 30 to 60 minute videos.
Does a Video-MME score say whether subtitles were used?
It should. The leaderboard reports accuracy by duration and by subtitle setting, so a score without the setting is incomplete. Models can be evaluated on visual frames alone or with subtitles and audio. Record the setting alongside the bucket, the benchmark version and the date the score was captured.
Which model led Video-MME in the original paper?
The paper states that Gemini 1.5 Pro was the best-performing commercial model in its experiments. That result comes from the first version of the paper, submitted on 31 May 2024. It is a May 2024 result, not a current leaderboard position, and newer models are outside the sources used for this article.
Sources
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis, arXiv (2024-05-31)
- Video-MME project page, Video-MME (2026-10-26)
- Video-MME home page, Video-MME (2026-10-26)
- Video-MME benchmark, Epoch AI (2025-06-18)
- Video-MME topic page, Emergent Mind (2025-07-01)