Title:	Generative deep learning with human feedback for animating 3D characters in social settings
DNr:	Berzelius-2026-21
Project Type:	LiU Berzelius
Principal Investigator:	Gustav Eje Henter <ghe@kth.se>
Affiliation:	Kungliga Tekniska högskolan
Duration:	2026-01-30 – 2026-08-01
Classification:	10210
Keywords:

Abstract

Realistic 3D character animation for film and gaming is a technically demanding task, traditionally requiring expensive motion-capture equipment, trained actors, and extensive manual post-processing. This is especially true in interactive, social contexts, where human motion is in complex interplay with language and communication. While generative deep learning offers a promising path towards automating social behaviour for virtual characters, existing models often produce generic motion with little communicative value and weak alignment between movement, speech, and conversational context. This project explores how to integrate rich human preference data directly into the 3D generative pipeline. Our proposed methodology incorporates human preference data at three stages: 1) data curation, exploring learning-efficient representations of human opinion for generative models; 2) generative modelling, utilising reinforcement learning from human feedback and preference-based objective functions for flow matching and diffusion models; 3) automated evaluation, establishing new metrics that have strong alignment with human perception. Leveraging recent large-scale motion-capture datasets with 4000+ hours of conversational motion capture, we will use modern deep learning frameworks (e.g., PyTorch) to train state-of-the-art generative models (e.g., diffusion and flow matching) to enable immersive experiences with AI-driven virtual characters.