video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models

🚀🚀 Welcome to the repo of video-SALMONN 2!

video-SALMONN 2 is a powerful audio-visual large language model (LLM) that generates high-quality audio-visual video captions, which is developed by the Department of Electronic Engineering at Tsinghua University and ByteDance.

🔥 News

2026-09-03: We release the code and checkpoint of video-SALMONN 2 pro 4B, 8B, and 32B. Based on Qwen3-VL backbone, the model further improves on audio-visual QA benchmarks. With the support of vLLM, video-SALMONN 2 significantly improves caption generation throughput and latency.
2026-02-24: We release the minimal inference code for video-SALMONN 2+ 3B and 7B.
2026-02-15: We release audio-aligned video-SALMONN 2+ 3B checkpoint.
2026-01-28: We release the audio-aligned model of video-SALMONN 2+ 72B for finetuning larger audio-visual models.
2025-12-18: We have released the audio-aligned model of video-SALMONN 2+ 7B for further finetuning.
2025-09-26: A new version (Version-2509) of video-SALMONN 2+ is released, containing minor code revision, an update for 7B model and 72B model, as well as the addition of 3B model. The upgraded video-SALMONN 2+ further enhances audio-visual and visual-only understanding capability on various benchmarks.
2025-07-17: We release the code and checkpoint of video-SALMONN 2+ at video-SALMONN 2+ （Version-2507). video-SALMONN 2+ achieves SOTA results on Video-MME benchmark.
2025-07-08: We release the 7B version of video-SALMONN 2.
2025-06-18: We release the code of video-SALMONN 2.

⚡️ Results

We evaluate the models on audio-visual QA benchmarks including Video-MME, WorldSense, AVUT, Video-Holmes, and DailyOmni. Our 4B and 8B models achieve SOTA results at comparable scales, while the 32B model surpasses all other open-source systems.

Model	Video-MME	WorldSense	AVUT	Video-Holmes	DailyOmni	FutureOmni
VideoLLaMA3 (2B)	59.6	-	-	-	-	-
Qwen2.5-Omni (3B)*	62.0	-	-	-	40.5	38.9
Qwen2.5-VL (3B)	61.5	-	-	-	37.4	-
video-SALMONN 2+ (3B)*	68.3	48.3	66.2	42.2	67.7	50.5
video-SALMONN 2 Pro (4B)*	75.4	54.3	77.2	50.1	78.9	55.6

video-SALMONN (13B)*	43.3	-	38.3	-	-	-
LLaVA-Video (7B)	63.3	40.2	56.5	-	-	-
VideoLLaMA2 (7B)*	54.9	25.4	44.9	-	35.2	40.8
VideoLLaMA3 (7B)	66.2	-	-	-	-	46.8
Qwen2.5-Omni (7B)*	64.3	45.4	-	16.4	47.5	47.5
Qwen2.5-VL (7B)	65.1	-	-	27.8	40.7	43.7
video-SALMONN 2 (7B)*	67.4	48.6	65.6	40.7	66.3	48.7
video-SALMONN 2+ (7B)*	73.4	50.9	69.5	46.9	71.8	51.0
video-SALMONN 2 Pro (8B)*	77.5	55.0	78.3	55.1	80.0	58.0

GPT-4o	71.9	42.6	56.6	42.0	56.5	49.7
Gemini-1.5 Pro*	75.0	48.0	78.3	41.2	-	-
Qwen3-Omni-Flash*	71.4	54.1	-	57.3	76.2	-
LLaVA-Video (72B)	70.5	-	-	-	-	-
VideoLLaMA2 (72B)*	61.4	-	-	-	-	-
Qwen2.5-VL (72B)	73.3	-	-	50.2	61.8	-
video-SALMONN 2+ (72B)*	79.7	56.5	72.2	57.8	79.4	60.8
video-SALMONN 2 Pro (32B)*	82.7	59.4	81.1	61.7	81.7	61.4

* Models that use audio input.

🌈 How to Use

For video-SALMONN 2+, please refer to video_SALMONN2_plus

For video-SALMONN 2 Pro, please refer to video_SALMONN2_pro

How to train video-SALMONN 2

Prepare the dataset following scripts/example_sft.json and scripts/example_dpo.json.
Download LLaVA-OneVision Model from huggingface.
Modify the parameters in scripts/train_sft.sh and scripts/train_dpo.sh.
Run bash scripts/train_sft.sh or bash scripts/train_dpo.sh.

How to evaluate a checkpoint

Prepare the dataset following scripts/example_sft.json.
Modify the parameters in scripts/eval.sh.
Run bash scripts/eval.sh.

👀 Team

Team Tsinghua: Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Chao Zhang

Team ByteDance: Wei Li, Zejun Ma

✨ Citation

If you find video-SALMONN 2 useful, please cite the paper:

@article{tang2025video,
    title={{video-SALMONN 2: Captioning-Enhanced Audio-Visual Large Language Models}}, 
    author={Changli Tang and Yixuan Li and Yudong Yang and Jimin Zhuang and Guangzhi Sun and Wei Li and Zejun Ma and Chao Zhang},
    journal={arXiv preprint arXiv:2506.15220},
    year={2025},
}