Remote-sensing video understanding · CVPR 2026 public version

RSVideo: Are Your Vision-Language Models Ready for Remote Sensing Videos?

A continuous-video benchmark and evidence-aware training framework for finding the right target, frame, and scene context before answering.

Hongjie Zhou · Shiqin Wang · Haoyang Chen · Haonan Guo

Di Wang · Juhua Liu · Fu Lin · Yong Luo

Wuhan University · Zhongguancun Academy

Corresponding authors: d_wang@whu.edu.cn, luoyong@whu.edu.cn

Paper Code Dataset BibTeX

Comparison of vision-language model accuracy on natural videos and remote-sensing videos — Remote-sensing video understanding remains substantially harder than natural-video understanding.

small targets

temporal evidence

The question

Can current vision-language models understand what happens in remote-sensing videos?

RSVideo studies continuous video question answering from UAV and satellite viewpoints. The benchmark asks models to identify small targets, follow short-lived events, and reason over spatial and temporal relationships that are easy to miss when the evidence occupies only a few frames or pixels.

Our central observation is simple: a correct answer depends on knowing where and when the evidence appears, not only on recognizing the scene category.

10,773QA instances

4,629evidence clips

1,473,150decoded frames

17.02 hvideo duration

17fine-grained subtasks

Why this is different

Remote-sensing videos compress the evidence.

Overhead viewpoints, repetitive textures, and fast-changing targets create a gap between seeing a scene and explaining what happened inside it.

Small Targets

Vehicles, people, and other relevant objects may occupy only a small fraction of the frame, making global visual summaries unreliable.

spatial focus

Short-lived Events

Actions and state changes can last for only a brief temporal window. A few sampled frames may erase the event that answers the question.

temporal focus

Scene-constrained Relations

Answers often depend on roads, buildings, boundaries, nearby targets, movement direction, and the order of multiple events.

spatiotemporal reasoning

Dataset

RSVideo-10K turns continuous video into auditable questions.

Every item is a five-choice question grounded in a released video clip. RSVideo-Instruct supports training and validation, while RSVideo-Bench is a locked test set for controlled evaluation.

Overview of the RSVideo-10K dataset composition, source distribution, and two core capabilities — RSVideo-10K combines eight public sources, audited evidence clips, and a taxonomy with two core capabilities and 17 fine-grained subtasks.

5,651train

2,391validation

2,731fixed test

All questions use one shared annotation protocol. Evidence fields identify the relevant temporal window and, when needed, the spatial target region. The fixed test set contains 653 Perception items and 2,078 Reasoning items.

RSVideo-Instruct: 8,042 train/validation instances

RSVideo-Bench: 2,731 held-out instances

RSVideo data construction pipeline from public videos to expert-reviewed release — Public videos → decoding and screening → evidence grounding → QA construction → expert review → release.

Task taxonomy

Two core capabilities, 17 subtasks.

The taxonomy separates what a model can perceive from how it reasons about change, relations, and evidence across time. The groups below summarize the two capabilities; the full benchmark contains 17 fine-grained subtasks.

01PerceptionWhat is visible?+

Scene Perception Target Perception Action Perception

02ReasoningWhat happened, where, and how did it change?+

Complex-Environment Reasoning Spatiotemporal Evolution Reasoning Spatiotemporal Localization Reasoning Spatiotemporal Consistency Reasoning

RSVideo-Bench

A fixed protocol for evidence-grounded evaluation.

All models receive the same video question and five answer choices. Scores are computed over the fixed 2,731-item test set, making it possible to compare backbones and training strategies under a shared input and evaluation budget.

Perception

What is visible?

Scene, target, and action perception tasks test whether the model can identify the relevant content in overhead video.

653 items

Reasoning

What happened, where, and when?

Reasoning tasks connect targets, relations, trajectories, state changes, and temporal order across multiple frames.

2,078 items

01 same video input 02 one question 03 five choices 04 fixed test set 05 accuracy

Method

RSVideo learns to keep the evidence that matters.

Evidence-aware initialization and reinforcement learning work together to select informative spatiotemporal tokens while compressing redundant background context under a fixed visual-token budget.

01Video + QuestionStart with continuous visual context and the task query.

02Evidence ScoringCombine saliency, question relevance, change, and region priors.

03Spatiotemporal FocusingPreserve high-value tokens from the right places and moments.

04Background CompressionReduce redundant context into compact question-related slots.

05Evidence-grounded AnswerGenerate the final answer from a sparse visual sequence.

Heatmap-style visualization of evidence focusing in the RSVideo method — Evidence focusing is designed to preserve target- and question-relevant visual tokens while keeping scene context compact.

Results

The bottleneck is evidence, not just model scale.

Across a diverse set of vision-language backbones, evidence-aware training consistently improves performance on the fixed RSVideo-Bench test set.

+9.01%maximum absolute improvementInternVL3.5-14B

40.63%best RSVideo-Bench accuracyQwen3.6-27B

26evaluated open-source backbones1B → 241B parameters

Training strategyrelative trend

Direct Inferencebaseline

Evidence-aware SFT+ supervision

RSVideobest across backbones

Video understanding gapaverage accuracy

69.7%natural videos

− 40.7 pts →

29.0%remote sensing

Models that perform well on natural videos still struggle when relevant evidence is small, distant, and distributed across time.

Qualitative examples

Six questions that make the challenge concrete.

Each example highlights a different way the answer depends on localized visual evidence instead of a single global scene impression.

Target perception

Target identification

Which object is the question referring to?

Evidence: local appearance and scene context.

Target perception

Target counting

How many relevant targets are visible?

Evidence: small-object coverage across frames.

Action perception

Action recognition

What action is taking place?

Evidence: target state and short-lived motion.

Evolution reasoning

Action transition

How does the target state change?

Evidence: ordered frames around the transition.

Temporal reasoning

Temporal ordering

Which event happens first?

Evidence: event sequence rather than appearance alone.

Consistency reasoning

Re-identification after occlusion

Is the target still the same object?

Evidence: identity continuity across missing views.

Citation

Bring RSVideo into your next project.

Cite the paper from arXiv:2608.02039 or copy the BibTeX entry below.

@article{zhou2026RSVideo,
  title   = {RSVideo: Are Your Vision-Language Models Ready for Remote Sensing Videos?},
  author  = {Zhou, Hongjie and Wang, Shiqin and Chen, Haoyang and Guo, Haonan and Wang, Di and Liu, Juhua and Lin, Fu and Luo, Yong},
  journal = {arXiv preprint arXiv:2608.02039},
  year    = {2026}
}