EyeVLM-Data is the annotation release of "Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models" (NeurIPS 2026, Evaluations and Datasets track). It is used to evaluate and fine-tune vision-language models on gaze following and social gaze prediction, and has two parts:
The social gaze files of VQA/ are the question-answer form of Balanced-Subset/: one record per labelled pair, with the same frames, pairs and labels. Only annotations are released; no images are included. Croissant metadata (croissant.json) is provided.
If you use this dataset, please cite the following publication:
@inproceedings{wang2026eyevlm,
title = {Eyes on {VLM}: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models},
author = {Wang, Hengfei and Gupta, Anshul and Vuillecard, Pierre and Odobez, Jean-Marc},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
year = {2026}
}
Please also cite VSGaze and the source datasets:
@inproceedings{gupta2024mtgs,
title = {MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction},
author = {Gupta, Anshul and Tafasca, Samy and Farkhondeh, Arya and Vuillecard, Pierre and Odobez, Jean-Marc},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2024}
}
@inproceedings{recasens2015they,
title = {Where are they looking?},
author = {Recasens, Adria and Khosla, Aditya and Vondrick, Carl and Torralba, Antonio},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2015}
}
@inproceedings{chong2020detecting,
title = {Detecting Attended Visual Targets in Video},
author = {Chong, Eunji and Wang, Yongxin and Ruiz, Nataniel and Rehg, James M.},
booktitle = {CVPR},
year = {2020}
}