Current evaluation metrics lack the granularity to assess whether models truly understand what they see. We decompose anomaly understanding into What happened, Who was involved, and Where it occurred — and propose FV-Score, the first metric that penalizes models for missing what matters.
Existing metrics focus on lexical overlap or language quality, not on whether the model captured the actual anomaly.
LVLM Generated Response:
"A person approaches a residential property and enters through a door. Two individuals are present in the scene and appear to be engaged in some form of physical interaction near the entrance. One person is seen moving around the property, and there are objects visible including a door and window. The scene takes place at a residential location during what appears to be daytime."
Reference (Ground Truth):
"A man climbs over a wall and enters a residential property through a small door. The homeowner confronts the intruder. The robber stabs the defender multiple times with a knife. The defender fights back using an iron rod. The robber escapes by jumping through a window."
LVLM Response:
"A person approaches a residential property and enters through a door. Two individuals are present in the scene and appear to be engaged in some form of physical interaction near the entrance. One person is seen moving around the property, and there are objects visible including a door and window. The scene takes place at a residential location during what appears to be daytime."
ROUGE-L
0.22
Measures longest common subsequence between response and reference. Shared words like "door" and "property" inflate the score, despite completely missing the anomaly.
BLEU
0.15
N-gram precision against the reference. Low score, but provides no diagnostic — is it wrong about the events, the entities, or the scene? No way to tell.
VAU-EVAL
6.8/10
LLM judge scoring fluency, informativeness, key concepts, and factual consistency. Holistic criteria lack the granularity to pinpoint which anomaly-specific visual elements are missing.
AnomEVAL
6.1/10
Evaluates causal reasoning — cause and result of anomalies. But the response omits rather than fabricates, so the hallucination check passes while critical violence is never mentioned.
A human rates this response 2/10. These metrics range from 0.15 to 6.8 — none explain why the response fails.
Correctly identifies this as a catastrophic failure (Human rating: 2/10)
Actionable insight: FV-Score pinpoints exactly which visual elements were hallucinated or omitted — the model sanitized a life-threatening stabbing into "physical interaction." Traditional metrics cannot detect this.
1,544 videos with structured What/Who/Where annotations. Select a clip to explore its ground-truth events.
Zero-shot evaluation of state-of-the-art LVLMs. Click headers to sort, expand rows for failure analysis.
| # | Model | All ▲ | Event ▲ | Entity ▲ | Location ▲ | Attribute ▲ |
|---|
Higher scores indicate better fine-grained understanding. Click any row to see detailed failure analysis.
@inproceedings{pereira2026finevau,
author = {Pereira, Jo\~{a}o and Lopes, Vasco and Neves, Jo\~{a}o and Semedo, David},
title = {FineVAU: A Novel Human-Aligned Benchmark for Fine-Grained Video Anomaly Understanding},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
}