My research area is social computing, a domain focused on the automatic sensing, analysis, and interpretation of human and social behavior from sensor data. Through microphones and cameras in multi-sensor spaces, mobile phones, and the web, sensor data depicting human behavior can increasingly be obtained at large-scale - longitudinally and population-wise. With my research group, we are integrating models and methods from multimedia signal processing and information systems, statistical machine learning, ubiquitous computing, and applying knowledge from social sciences to address questions related to the discovery, recognition, and prediction of short-term and long-term behavior of individuals, groups, and communities in real life. This can range from people at work having meetings, users of social media sites, or people with mobile phones in urban environments. Our methods are aimed at creating ethical, personally and socially meaningful applications that support social interaction and communication, in the contexts of work, leisure, healthcare, and creative expression.

These are examples of our recent work.


Learning and Predicting Multimodal Daily Life Patterns from Cell Phones

In this work, we design a multimodal representation of daily human life. Representing the daily lives of individuals in terms of their locations and their interactions, we are able to discover rich human daily routines. We can determine whether individuals were predominantly at work alone or in small or large groups, and when large group interactions occur as well as where. We also use our methodology to predict missing location and proximity data. read more

Flickr Hypergroups

Analyzing Flickr Groups Web Page

In this work we extended our analysis of Flickr Groups looking at how similar groups can be found and clustered together using Latent Dirichlet Allocation and Affinity Propagation. A collaboration with Svetha Venkatesh's group from Curtin University, Australia. read more

Role Analysis in Competitive Meetings

Most of the existing work on automatic role analysis in groups has focused on cooperative situations. With this work we add a novel dimension to this problem, by addressing it in a highly competitive scenario from a popular reality TV american show. Our approach only relies on nonverbal cues, and shows that conversational dynamics do contain sufficient information to be useful for this task. read more

Characterizing Conversational Group Dynamics using Nonverbal Behaviour

This work addresses the novel problem of characterizing conversational group dynamics. It is well documented in social psychology that depending on the objectives a group, the dynamics are different. For example, a competitive meeting has a different objective from that of a colloborative meeting. We propose a method to characterize group dynamics based on the joint description of a group members' aggregated acoustical nonverbal behaviour to classify two meeting datasets (one being cooperative-type and the other being competitive-type). read more

Discovering Group Nonverbal Conversational Patterns with Topics

This work addresses the novel problem of discovering conversational group dynamics. from thin-slices of interaction. We first propose and analyze a novel thin-slice interaction descriptor - a bag of group nonverbal patterns - which robustly captures the turn-taking behavior of the members of a group while integrating its leader's position. We then rely on probabilistic topic modeling of the interaction descriptors which, in a fully unsupervised way, is able to discover group interaction patterns. read more

Predicting Remote Versus Collocated Group Interactions using Nonverbal Cues

We investigate the problems of classifying remote and collocated small-group working meetings, and of identifying the remote participant, using nonverbal behavioral cues. We hypothesize that the difference in the dynamics between collocated and remote meetings is significant and measurable using speech activity based nonverbal cues. We used the Augmented Multi-Party Interaction with Distance Access (AMIDA) corpus for experiments. read more

Retrieving Ancient Maya Glyphs with Shape Context

As part of our interdisciplinary work on the analysis of the ancient Maya writing system, we investigated the automatic retrieval of Maya syllabic glyphs using the Shape Context descriptor. We investigated the effect of several parameters to adapt the descriptor given the high complexity of the shapes and their diversity in our data. We propose an improvement in the cost function used to compute similarity between shapes, obtaining promising performance. Read more


Daily Routine Classification from Mobile Phone Data

This initial work on the Reality Mining dataset, investigates the classification of people's daily routines both from physical location information (derived from cell tower connections) and social context (given by person proximity information from Bluetooth). We look at single-and multi-modal routine representations varying in timescales, each highlighting differing features, to determine what best characterizes human daily routine structure. read more

Discovering Human Routine from Cell Phone Data with Topic Models

In this work, we investigate both the location-driven and proximity-driven daily routines of people in the Reality Mining dataset. We present a methodology for bag representations of both location and proximity data capturing transitions and both fine and coarse grain times. Probabilistic Latent Semantic Analysis (PLSA) is then applied for successful routine discovery. read more

What Did You Do Today? Discovering Daily Routines from Large-Scale Mobile Data

This work extends the previous work, by applying a similar approach to Hierarchical Bayesian Topic Models, more specifically Latent Dirichlet Allocation (LDA) and the Author Topic Model. In this work, we only consider location information. We formulate a bag representation which captures transitions in locations and both fine and coarse grain time considerations for routine discovery. read more

Analyzing Flickr Groups

Analyzing Flickr Groups Web Page

Based on a large Flickr dataset of photos, users, and existing groups, we extract relevant patterns of photo-to-group sharing practices in Flickr communities . We also define a new topic-based representation for Flickr groups that allows for neat modeling more

Topickr: Flickr Groups and Users Reloaded

Topickr: Flickr Groups and Users Reloaded Web Page

Based on the same Flickr data, we take the topic-based representation one logical step further, unifying groups and users under the same representation. Through topics, users and groups can then be compared, ranked, and discovered. read more

Characterizing Individual Behaviour in Groups

Dominance - a behavioral expression of power - is a fundamental mechanism of social interaction, expressed and perceived in conversations through spoken words and audio-visual nonverbal cues. Status- an ascribed or achieved quality implying respect or privilege, [but] does not necessarily include the ability to control others or their resources. The automatic modeling of dominance or status patterns from sensor data represents a relevant problem in social computing. read more

Estimating Dominant People in Meetings from Single Distant Microphones

This work aims to estimate the most dominant person from a single audio source in a meeting scenario. We first use a speaker diarization methodto find out 'who spoke and when'. We then investigate how changing conditions in the input audio source and also speed-optimized diarization strategies can affect the dominance estimation performance. read more

Audio-Visual Speaker Association and Dominance in Small Groups

This work looks at how the estimation of dominance in a group conversation can be enhanced by having both audio and video segments of the most dominant person. We take advantage of computations already performed during video compression, to generate a represent an individual's visual activity. This is then correlated with automatically extracted speech activity so that speakers and video streams can be associated. read more

Visual Attention and Speaking Activity for Dominance Estimation

This work investigates measures inspired by work in social pyschology to estimate dominance in small group meetings from automatically estimated visual attention and speaking activity, in scenarios where participants have access to artifacts in the room such as a whiteboard, slidescreen, and table. These are normal items to be found in a meeting room but can greatly affect where people look when they are in a discussion. read more

Exploiting Contextual Information for Speech/Non-speech Detection

We investigate the effect of temporal context for speech/non-speech detection (SND). It is shown that even a simple feature such as full-band energy, when employed with a large-enough context, shows promise for further investigation. Experimental evaluations on the test data set, with a state-of-the-art multi-layer perceptron based SND system and a simple energy threshold based SND method, using the F-measure, show an absolute performance gain of 4.4% and 5.4% respectively. The optimal contextual length was found to be 1000 ms. Further numerical optimizations yield an improvement (3.37% absolute), resulting in an absolute gain of 7.77% and 8.77% over the MLP based and energy based methods respectively. ROC based performance evaluation also reveals promising performance for the proposed method, particularly in low SNR conditions. read more

Analyzing Collections of Pre-Columbian Pictorial Documents

The aim of this project is to advance the state-of-the-art on categorization and matching of visual patterns in collections of cultural and historical images by developing probabilistic more

...and here are some links to research in previous years

Back to the group page