Title:	Embodied perception for interacting with the 3D world
DNr:	Berzelius-2026-291
Project Type:	LiU Berzelius
Principal Investigator:	Hedvig Kjellström <hedvig@kth.se>
Affiliation:	Kungliga Tekniska högskolan
Duration:	2026-11-01 – 2027-05-01
Classification:	20208
Homepage:	https://www.kth.se/profile/hedvig
Keywords:

Abstract

Embodied artificial intelligence is important for developing human assistive technologies to support our daily lives, and one key requirement is the ability to understand and interact with the 3D world. This project aims to develop embodied perception techniques that enable machines and virtual agents to model the 3D bodies of subjects, i.e., humans and animals, from 2D videos, and then to understand and interact with them in downstream applications. The project is organized into five stages of study. Firstly, we Model the real-world subjects, i.e., humans and animals, from the machines’ perspective. In particular, we develop improved 3D models that can encode fine details and fast motions of human bodies in different growth states, e.g., newborns, toddlers, teens and adults. For animals, we focus on quadruped animals, starting with specific species such as dogs and horses, and potentially extending to cattle and other species. Additionally, we Estimate the subjects’ 3D model representations from 2D videos while preserving the social signals, such as emotion, conveyed by the subjects as comprehensively as possible. On the one hand, the estimated 3D models capture the expressive social properties of the subjects, such as pose, shape, facial expressions and body behaviors. On the other hand, they also encode spontaneous cues revealed unconsciously by the subjects, such as head pose, gaze, facial micro-expressions and micro-gestures. Moreover, we Infer the semantic meaning of 3D models in application scenarios, e.g., the activity and emotion conveyed by the motion. This project develops methods for recognizing human actions from 3D skeleton data by leveraging large-scale vision-pretrained models, understanding drivers’ affect state and frustration level, analyzing facial expressions and body gestures of neonates for automated assessment of their neurodevelopment, investigating how well multimodal large language models understand the communication between conductors and orchestras, and analyzing animal behaviors to give humans insight into the minds of animals in their care. Furthermore, we Synthesize 3D pose and motion data, both to generate new training material for downstream tasks and as a goal in itself. Large-scale data is a key success factor in training deep learning models, yet collecting 3D data is extremely costly. This project develops efficient 3D synthesis techniques that can transfer existing poses and activities to new subjects or species, and generate new, unseen poses and activities from text descriptions using diffusion models and multimodal large language models. Lastly, we Evaluate generative models and generated samples. As generative models are transforming many industries and professions, it is crucial to understand the quality of the models and their samples. Since this quality is often not intuitive to ordinary users, measuring it usually requires experts. This part of the project designs reliable, interpretable and human-aligned automated metrics for evaluating generative models and generated samples.