Probabilistic and Generative Modeling for Emotion Recognition and Synthesis

Abstract

Artificial Intelligence (AI) technologies have made remarkable progress in recent years, enabling systems that enhance human productivity, creativity, and decision-making across diverse domains. Despite these advances, current AI systems remain largely deficient in social intelligence: the ability to perceive, interpret, and express social signals in ways that support seamless human–agent interaction. This limitation is particularly critical in purposive social interactions, where success depends on interdependence, shared goals, coordinated action, and mutual trust. In such settings, a lack of social awareness can fundamentally compromise interaction quality and outcomes. Scenarios such as meetings, job interviews, collaborative design workshops, and healthcare consultations demand that agents accurately interpret social cues, adapt to interpersonal dynamics, and respond appropriately to sustain effective cooperation. For AI systems to be meaningfully integrated in these contexts, they must demonstrate the ability to understand and generate socially appropriate behavior. This dissertation adopts the framework of social signal processing (SSP), which distinguishes two essential components of social intelligence: recognition of social signals and their synthesis. Among the wide array of social signals—including personality, dominance, rapport, and regulation—affect plays a central role. Experienced as conscious feelings such as pleasure–displeasure or energy–tiredness, affect permeates social interactions and shapes outcomes at individual, group, and organizational levels. Accordingly, affect constitutes the focus of this thesis, with research contributions spanning both its recognition and synthesis.

The first line of work focuses on the recognition of affective expressions. A central challenge arises from the inherently subjective and ambiguous nature of affect labels—annotators frequently disagree, and collapsing their judgments into a single “ground truth” discards meaningful variation. To address this, we develop methods that explicitly model label uncertainty for individual-level emotion recognition. By combining Bayesian neural networks (BNN) with label distribution learning (LDL), our approach captures both the central tendencies and the variability in annotations, yielding improvements in predictive accuracy, calibrated uncertainty, and robustness to annotator disagreement. Furthermore, we investigate how uncertainty can be modeled effectively under limited annotation conditions. By replacing Gaussian assumptions with a Student’s t-distribution, we directly connect annotation sparsity to uncertainty, resulting in higher predictive accuracy, faster convergence, and improved cross-corpus robustness. Together, these findings reinforce a central aim of this thesis: rethinking affect modeling by treating uncertainty not as noise to be eliminated, but as an intrinsic property of affective data that provides valuable insight into emotional ambiguity. While individual-level affect recognition has been extensively explored, the collective level of affect—how group-level affect emerges and evolves within groups—remains under-researched. To extend affect modeling from individuals to the group level, we introduce an annotation protocol grounded in psychological theory to capture the dynamic ebb and flow of group affect. Building upon these annotations, we propose a graph-based multimodal framework that models collective affect at scale, capturing both convergence and divergence across group members and revealing systematic patterns of group dynamics.

The second line of work addresses the synthesis of affective expressions, with a particular emphasis on speech. Prior approaches largely rely on acted corpora with parallel data, which fail to capture the richness and spontaneity of natural emotional speech. We investigate generative modeling for speech emotion conversion in in-the-wild data using a disentanglement–resynthesis framework: by separating speaker, lexical, and affective embeddings and reconstructing them with a modified HiFiGAN vocoder, we achieve conversion without parallel corpora. This yields improvements in naturalness, controllability, and pitch modulation. We also explore diffusion-based decoders as an alternative to vocoder-based methods, showing that they broaden the range of emotions, improve expressivity, and better capture rare states, albeit sometimes at the expense of perceptual naturalness. Finally, we study the role of prosodic cues such as rhythm and stress, integrating a dedicated duration predictor (DP) that modulates speech rate with arousal and enhances overall quality and naturalness.

Together, we frame seven research questions that advance both the perception and response aspects of socially intelligent systems. On the recognition side, this thesis contributes probabilistic methods for modeling label uncertainty, introduces novel group-level annotation protocols, and develops scalable, multimodal graph-based frameworks. On the synthesis side, it proposes unsupervised generative models for in-the-wild emotion conversion, explores diffusion models for controllable affect synthesis, and demonstrates the importance of duration modeling. Beyond technical improvements, these contributions reconceptualize affect recognition and synthesis by treating ground truth as distributions rather than absolutes, group affect as dynamic and emergent, and affective speech as inherently diverse and ecologically grounded.

By addressing these challenges, this thesis lays the groundwork for AI systems that can both perceive and express affect appropriately in purposive social interactions. Such systems hold promise for enhancing trust, collaboration, and effectiveness across societal domains, including education, healthcare, and organizational teamwork.