FaceLLM: A Multimodal Large Language Model for Face Understanding

Aug 28, 2025·
Hatef Otroshi Shahreza
Prof. Sébastien Marcel
Prof. Sébastien Marcel
· 0 min read
Abstract
Multimodal large language models (MLLMs) have shown remarkable performance in vision-language tasks. However, existing MLLMs are primarily trained on generic datasets, limiting their ability to reason on domain-specific visual cues such as those in facial images. In particular, tasks that require detailed understanding of facial structure, expression, emotion, and demographic features remain underexplored by MLLMs due to the lack of large-scale annotated face image-text datasets. In this work, we introduce FaceLLM, a multimodal large language model trained specifically for facial image understanding. To construct the training data, we propose a novel weakly supervised pipeline that uses ChatGPT with attribute-aware prompts to generate high-quality question-answer pairs based on images from the FairFace dataset. The resulting corpus, called FairFaceGPT, covers a diverse set of attributes including expression, pose, skin texture, and forensic information. Our experiments demonstrate that FaceLLM improves the performance of MLLMs on various face-centric tasks and achieves state-of-the-art performance. This work highlights the potential of synthetic supervision via language models for building domain-specialized MLLMs, and sets a precedent for trustworthy, human-centric multimodal AI systems. FairFaceGPT dataset and pretrained FaceLLM models are publicly available.
Type
Publication
International Conference on Computer Vision Workshop
publications
Large Language Models
Prof. Sébastien Marcel
Authors
Prof. Sébastien Marcel (he)
Senior Research Scientist
Prof Sébastien Marcel (IEEE Fellow and IAPR Fellow) is a senior research scientist at the Idiap Research Institute (Switzerland), he heads the Biometrics Security and Privacy group and conducts research on face recognition, speaker recognition, vein recognition, attack detection (presentation attacks, morphing attacks, deepfakes) and template protection. He is also Professor at the University de Lausanne (UNIL) at the School of Criminal Justice. He is also the Director of the Swiss Center for Biometrics at Idiap, which conducts certifications of biometric products. He was Associate Editor and Guest Editor of IEEE journals (TBIOM, SPL, TIFS and SPM). He is also the lead Editor of the Springer Handbook of Biometrics Anti-Spoofing (Editions 1, 2 and 3). Since June 2025 he is a member of the Idiap Direction ad interim.