[CVPR2024] 🥥COCONut: Crafting the Future of Segmentation Datasets with Exquisite Annotations in the Era of ✨Big Data✨

Xueqing Deng, Qihang Yu, Peng Wang, Xiaohui Shen, Liang-Chieh Chen

🚀 Contributions

🔥 1st large-scale human verified dataset for segmentation, more info can be found at our website.

🔥 COCONut is now available at Kaggle and huggingface, welcome to download!

teaser

📢 News!

2/12/2026: COCONut-Pancap is fully available at huggingface: https://huggingface.co/datasets/xdeng77/coconut_pancap. Feel free to try and leave any comments.
5/11/2025: COCONut-Pancap-50K panoptic grounding captions are vailable at huggingface. For the dense masks subset, we are still working on the final inspection.
3/28/2025: COCONut-Pancap region30k is released for the interest of region-level instruction data. More tutorials are coming!
3/22/2025: COCONut-XLarge is avaible! Please check 🔗preparing datasets.
3/7/2025: Releasing COCONut-Pancap! Stay tuned!
9/9: Relase a tutorial to prepare all dataset splits for training and evaluation.
6/24: Release COCONut-val and instance segmentation annotations.
5/6: Tutorial on semantic segmentation is out!
4/30: Tutorials on open-vocabulary segmentation and object detection are out!
4/28: COCONut is back to huggingface. relabeled COCO-Val, COCONut-S, and COCONut-B are available.
4/25: Tutorial on visualizing COCONut panoptic masks using detectron2. Turn the black mask image into overlayed colorful mask.
4/24: Collected FAQs are out, please check them before you leave any issue.
4/22: Tutorial on instance segmentation is out! More are coming!
4/19: Tutorial on panoptic segmentation is out!
4/16: COCONut is available at Kaggle! No need to merge COCONut-B from COCONut-S, update a version of ready-to-use.
4/15: COCONut is higlighted by AK’s daily paper!
4/15: Huggingface download links are temporarily closed.

TODO

Huggingface dataset preview on relabeled COCO-Val and COCONut-S
Huggingface preview on COCONut-B
Convert the annotation to semantic segmentation.
Release COCONut-val and instance segmentation annotations (no need to convert from the panoptic masks).
Release COCONut-L.

Dataset Splits

Splits	#images	#masks	images	kaggle	huggingface
COCONut-S	118K	1.54M	download	download	preview
COCONut-B	242K	2.78M	download	download	preview
COCONut-L	358K	4.75M	download	downoad	download
COCONut-XL	488K	6.43M	[invalid]	[invalid]	download
relabeled-COCO-val	5K	67K	download	download	preview
COCONut-val	25K	437K	download	download	download

Please refer to 🔗preparing datasets for exploring training and evaluation.

Get Started

We only provide the annotation, for those who are interested to use our annotation will need to download the images from the links: COCONut-S images, COCONut-B images and relabeled COCO-val images.

We provide two methods to download the dataset annotations, details are as below。

🔗Kaggle download link

You can use the web UI to download the dataset directly on Kaggle.

If you find our dataset useful, we really appreciate if you can upvote our dataset on Kaggle,

🔗Huggingface dataset preview

Directly download the data from huggingface or git clone the huggingface dataset repo will result in invalid data structure.

We recommend you to use our provided download script to download the dataset from huggingface.

pip install datasets tqdm
python download_coconut.py # default split: relabeled_coco_val

You can switch to download COCONut-S by adding “–split coconut_s” to the command.

python download_coconut.py --split coconut_s

The data will be saved at “./coconut_datasets” by default, you can change it to your preferred path by adding “–output_dir YOUR_DATA_PATH”.

To use COCONut-Large, you need to download the panoptic masks from huggingface and copy the images by the image list from the objects365 image folder. Then add them on top of COCONut-B, to consist the full COCONut-Large dataset.

Tutorials

FAQ

We summarize the common issues in FAQ.md, please check this out before you create any new issues.

More visualization on COCONut annotation

vis1

vis2

Terms of use

We follow the same license as COCO dataset for images. The dataset cannot be used for commercial purposes.
This is not an official ByteDance product. The dataset is created for research purposes.

Acknowledgement

Bibtex

If you find our dataset useful, please cite:

@inproceedings{coconut2024cvpr,
  author    = {Xueqing Deng, Qihang Yu, Peng Wang, Xiaohui Shen, Liang-Chieh Chen},
  title     = {COCONut: Modernizing COCO Segmentation},
  booktitle   = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year      = {2024},