SpatialVID: A Large-Scale Video Dataset with Spatial Annotations

SpatialVID

Directory Structure

DATA_PATH/
├─ annotations/
│  ├─ group_0001/
│  │  ├─ 0a00f99d-9d9a-5265-9548-e97a34c1302c/
│  │  │  ├─ caption.json
│  │  │  ├─ dyn_masks.npz
│  │  │  ├─ indexes.txt
│  │  │  ├─ instructions.json
│  │  │  ├─ intrinsics.npy
│  │  │  └─ poses.npy
│  │  ├─ 0a7b25d8-41fd-563f-bec3-9fb2b675be3e/
│  │  │  ├─ caption.json
│  │  │  ├─ dyn_masks.npz
│  │  │  ├─ indexes.txt
│  │  │  ├─ instructions.json
│  │  │  ├─ intrinsics.npy
│  │  │  └─ poses.npy
│  │  └─ ...
│  ├─ group_0002/
│  └─ ...
├─ data/
│  ├─ train/
│  │  └─ SpatialVID_metadata.csv
├─ depths/
│  ├─ group_0001/
│  │  ├─ 0a00f99d-9d9a-5265-9548-e97a34c1302c.zip
│  │  ├─ 0a7b25d8-41fd-563f-bec3-9fb2b675be3e.zip
│  │  └─ ...
│  ├─ group_0002/
│  └─ ...
├─ videos/
│  ├─ group_0001/
│  │  ├─ 0a00f99d-9d9a-5265-9548-e97a34c1302c.mp4
│  │  ├─ 0a7b25d8-41fd-563f-bec3-9fb2b675be3e.mp4
│  │  └─ ...
│  ├─ group_0002/
│  └─ ...
└─ README.md

Dataset Download

You can download the entire SpatialVID dataset using the following command:

hf download SpatialVID/SpatialVID --repo-type dataset

The whole dataset is approximately 7.67TB in size. We have split the dataset into 545 groups for easier management. Each group contains approximately 14GB of video data and 1.5GB of annotation data, with naming conventions following the format group_0*** (e.g., group_0001, group_0002).

For downloading specific files (instead of the full dataset), please refer to the download_SpatialVID.py provided in our GitHub repository.

Usage Guide

1. Unzipping Group Files

After downloading the group files (in .tar.gz format), use the tar command to extract their contents. For example:

# Extract annotation groups
tar -xzvf annotations/group_0001.tar.gz
tar -xzvf annotations/group_0002.tar.gz
tar -xzvf annotations/group_0003.tar.gz
...

# Extract video groups (if downloaded separately)
tar -xzvf videos/group_0001.tar.gz
tar -xzvf videos/group_0002.tar.gz
...

2. Using the Metadata File

The SpatialVID_HQ_metadata.csv file contains comprehensive metadata for all video clips in the dataset. It can be loaded with pandas for easy filtering and analysis based on your research needs.

Metadata Fields Explained

Field Name	Description
id	Unique identifier for each video clip (matches the annotation folder name).
group id	Identifier of the group the video clip belongs to (e.g., group_0001).
video path	Relative path to the video file (e.g., videos/group_0001/xxx.mp4).
annotation path	Relative path to the annotation folder (e.g., annotations/group_0001/xxx/).
num frames	Total number of frames in the video clip.
fps	Frames per second (FPS) of the video clip.
resolution	Resolution of the video clip (e.g., 1920x1080).
aesthetic score	Subjective score evaluating the video’s visual aesthetics (0–10).
luminance score	Score measuring the video’s overall brightness.
motion score	Score quantifying the intensity of motion in the video (0–20).
ocr score	Score indicating the proportion of text occupying the screen (0–1).
moveDist	Total distance of camera movement (in meters) during the clip.
distLevel	Categorized camera movement level: 1(low) → 5(extreme).
rotAngle	Total camera rotation angle (in degrees) during the clip.
trajTurns	Number of direction changes in the camera’s movement trajectory.
dynamicRatio	Proportion of dynamic content (e.g., moving objects) in the video (0–1).
motionTags	Tags describing the type of motion (e.g., "up", "right").
sceneType	Scene category (e.g., interior, urban, natural).
brightness	Brightness category (e.g., Dark, Dim, Bright).
timeOfDay	Time of day when the video was recorded (e.g., Daytime, Night).
weather	Weather conditions during recording (e.g., Sunny, Cloudy, Rainy).
crowdDensity	Density of people in the scene (e.g., Sparse, Moderate, Dense).

Example: Loading Metadata with Pandas

import pandas as pd

# Load the metadata file
metadata = pd.read_csv('data/train/SpatialVID_metadata.csv')

# Example: Filter videos with "urban" scene type and FPS ≥ 24
filtered_metadata = metadata[(metadata['sceneType'] == 'urban') & (metadata['fps'] >= 24)]
print(filtered_metadata.head())

3. Detailed Explanation of Annotation Files

Each video clip is paired with a dedicated annotation folder (named after the video’s id). The folder contains 5 key files, each serving a specific purpose:

1. caption.json

A structured JSON file with descriptive text for the video clip, covering scene details, camera motion, and dynamic elements. Its structure includes:

• SceneSummary: A concise overview of the entire scene.
• SceneDescription: A detailed, granular description of the scene (e.g., objects, backgrounds, lighting).
• CameraMotion: A clear account of how the camera moves throughout the clip (e.g., "slow pan to the right").
• ShotImmersion: An immersive narrative that simulates the viewer’s experience of the shot.
• CategoryTags: A list of tags aligning with metadata fields (e.g., sceneType, brightness, weather).

2. dyn_masks.npz

A compressed NumPy file storing dynamic masks for each annotated frame. These binary masks (0 = static, 1 = dynamic) highlight regions with moving objects (e.g., cars, pedestrians). To decompress and use these masks, refer to the utility script expand_npz.py in our GitHub repo.

3. intrinsics.npy

A NumPy file containing the camera’s intrinsic parameters (critical for 3D reconstruction). It stores a (n, 4) array, where n is the number of annotated frames. Each row [fx, fy, cx, cy] represents normalized intrinsics:

• fx, fy: Normalized focal lengths (scaled by image width/height).
• cx, cy: Normalized coordinates of the principal point (scaled by image width/height).

To convert these to pixel-space intrinsics (for practical use), apply the following formulas:

# Assume image_width and image_height are from the video's resolution
fx_pixel = fx * image_width
fy_pixel = fy * image_height
cx_pixel = cx * image_width
cy_pixel = cy * image_height

4. poses.npy

A NumPy file recording the camera’s 3D pose for each annotated frame. It stores a (n, 7) array, where n is the number of annotated frames. Each row [tx, ty, tz, qx, qy, qz, qw] represents:

• tx, ty, tz: Translation vector (world-to-camera coordinates in the OpenCV coordinate system, in meters).
• qx, qy, qz, qw: Unit quaternion (camera orientation following the right-hand rule, in the OpenCV coordinate system).

For coordinate transformation related to the pose data, please refer to the utility script quat_to_mat.py.

5. instructions.json

A JSON file with motion instructions for transitions between consecutive annotated frames. The file uses key-value pairs, where keys follow the format [frame1]->[frame2] (e.g., 0->6) and values are motion commands. Supported instructions include:

• Dolly In/Out: Move the camera forward/backward along its viewing direction.
• Truck Left/Right: Move the camera left/right while maintaining orientation.
• Pedestal Up/Down: Move the camera up/down while maintaining orientation.
• Tilt Up/Down: Rotate the camera up/down around its horizontal axis.
• Pan Left/Right: Rotate the camera left/right around its vertical axis.
• Roll CW/CCW: Rotate the camera clockwise/counterclockwise around its viewing axis.
• Stay: Keep the camera stationary (no movement).

6. depths.zip (Optional)

A compressed file containing precomputed depth maps for each annotated frame. The depth maps are stored in inverse depth format. To decompress and use these depth maps, refer to the utility script read_depth.py in our GitHub repo.

Important Note on Annotated Frames

Annotations are generated for frames extracted at intervals of int(fps/5). For example:

• If a video has 30 FPS, frames are extracted at indices 0, 6, 12, 18, 24, ....
• If a video has 25 FPS, frames are extracted at indices 0, 5, 10, 15, ....

And the current instuctions are generated by get_instructions.py in our GitHub repo, which you can modify the interval and threshold parameters to customize the frame selection process based on your experimental needs.

License

SpatialVID is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC-BY-NC-SA 4.0). By using this dataset, you agree to the following terms:

1. Attribution: You must credit the original source of the dataset.
2. Non-Commercial Use: The dataset may not be used for commercial purposes.
3. ShareAlike: Any modified or derived works must be released under the same license.

For the full license text, visit: https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.

If you are the copyright owner of any video in our dataset and you need it to be removed, please contact us, and we will remove the video samples from our dataset / Github / project webpage / technical presentation as soon as possible.

Citation

If you use SpatialVID in your research, please cite our work using the following format (citation will be updated upon paper publication):

@article{wang2025spatialvid,
  title={Spatialvid: A large-scale video dataset with spatial annotations},
  author={Wang, Jiahao and Yuan, Yufeng and Zheng, Rujie and Lin, Youtian and Gao, Jian and Chen, Lin-Zhuo and Bao, Yajie and Zhang, Yi and Zeng, Chang and Zhou, Yanxi and others},
  journal={arXiv preprint arXiv:2509.09676},
  year={2025}
}