EyeVLM-Data

annotation used to evaluate and fine-tune vision-language models on gaze following and social gaze prediction

Description

EyeVLM-Data is the annotation release of "Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models" (NeurIPS 2026, Evaluations and Datasets track). It is used to evaluate and fine-tune vision-language models on gaze following and social gaze prediction, and has two parts:

  1. VQA/: 31 question-answer files (ShareGPT JSON format) built from the annotations of GazeFollow, VideoAttentionTarget (VAT) and ChildPlay, with train/val/test splits. They cover gaze following (gaze point and in/out-of-frame questions) and social gaze prediction between two people: Looking-At-Humans (LAH, directed), Looking-At-Each-Other (LAEO, undirected) and Shared Attention (SA, undirected). Each record holds an English question, a JSON answer, the frame path and the normalized ground truth. Coordinates in the text use Qwen3-VL's 0-1000 integer convention. The folder also holds the row indices of the paper's 1K evaluation subsets (Table 4).
  2. Balanced-Subset/: VAT-ChildPlay-Balanced-for-SocialGaze, 18 Parquet tables. This is a class-balanced, pair-level subset of the VSGaze social gaze annotations (Gupta et al., NeurIPS 2024) for VAT and ChildPlay. One frame out of every 3 is kept. LAH pairs with an invalid person id are dropped, as are all pairs involving a head box narrower or shorter than 0.05 (normalized). For each dataset, split and task, negative pairs are then randomly downsampled to match the positive pairs, which gives an exact 1:1 ratio. The original VSGaze labels are highly imbalanced (about 2-14% positives on test).

The social gaze files of VQA/ are the question-answer form of Balanced-Subset/: one record per labelled pair, with the same frames, pairs and labels. Only annotations are released; no images are included. Croissant metadata (croissant.json) is provided.

 

Reference

If you use this dataset, please cite the following publication:

@inproceedings{wang2026eyevlm,
  title     = {Eyes on {VLM}: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models},
  author    = {Wang, Hengfei and Gupta, Anshul and Vuillecard, Pierre and Odobez, Jean-Marc},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations and Datasets Track},
  year      = {2026}
}

Please also cite VSGaze and the source datasets:

@inproceedings{gupta2024mtgs,
  title     = {MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze Prediction},
  author    = {Gupta, Anshul and Tafasca, Samy and Farkhondeh, Arya and Vuillecard, Pierre and Odobez, Jean-Marc},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2024}
}
@inproceedings{recasens2015they,
  title     = {Where are they looking?},
  author    = {Recasens, Adria and Khosla, Aditya and Vondrick, Carl and Torralba, Antonio},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2015}
}
@inproceedings{chong2020detecting,
  title     = {Detecting Attended Visual Targets in Video},
  author    = {Chong, Eunji and Wang, Yongxin and Ruiz, Nataniel and Rehg, James M.},
  booktitle = {CVPR},
  year      = {2020}
}