Scalable Multimodal AI: Modeling, Security, and Privacy
Title: Scalable Multimodal AI: Modeling, Security, and Privacy
DNr: Berzelius-2026-323
Project Type: LiU Berzelius
Principal Investigator: Niklas Carlsson <niklas.carlsson@liu.se>
Affiliation: Linköpings universitet
Duration: 2026-11-07 – 2027-06-01
Classification: 10201
Homepage: https://www.ida.liu.se/~nikca89/publications.html
Keywords:

Abstract

Our primary goal is to develop scalable methods for multimodal modeling and analysis of AI systems, with a particular focus on robust identity tracking and identification across challenging real-world scenarios and the security and privacy implications of increasingly capable AI models. Building on recent advances in multi-object tracking and our prior work on zero-shot tracking and global identity fusion, we aim to generalize these approaches into a broader framework that integrates video, text, and geometric representations while supporting research on the robustness, security, and privacy of multimodal and agent-based AI systems. In the continuation phase, we extend our research beyond traditional tracking by incorporating human-centric UV texture modeling, 3D reconstruction, and cross-modal reasoning, enabling identity-consistent tracking across viewpoints, occlusions, and long temporal horizons. These methods support applications in domains such as sports analytics, UAV/CCTV-based surveillance, and large-scale video understanding, while also connecting to emerging research directions in privacy-aware multimodal learning and social media data analysis. In parallel, we are expanding into the security and reliability of LLM-based and agent-based AI systems, investigating how unreliable or adversarially manipulated information sources influence AI assistants, how persistent memory and agent interactions may introduce information leakage, and how such vulnerabilities can be detected and mitigated. Our approach leverages modern deep learning architectures, including Transformer-based models with customized memory and attention mechanisms, and is designed to scale across multiple tracking and generative modeling paradigms and emerging LLM-based and agent-based AI systems. We build on existing tracking frameworks to advance multimodal identity representations, while developing complementary methods for evaluating and mitigating security and privacy risks in emerging AI systems. The project relies heavily on large-scale GPU-based experimentation, including training on long video sequences and multimodal datasets combining visual, textual, and 3D information, as well as repeated inference with open-weight LLMs and systematic evaluations across AI models, information-source configurations, adversarial conditions, and defense mechanisms. Using the Berzelius supercomputing infrastructure, we will conduct extensive experiments to develop and evaluate these models at scale. This includes training advanced multimodal models, optimizing data pipelines, and supporting multi-GPU experimentation for increasingly complex architectures, together with large-scale evaluations of security, privacy, and robustness in emerging AI systems. The expected outcomes include improved robustness in identity tracking, new methods for multimodal representation learning, better understanding of AI security and privacy vulnerabilities and potential defenses, and a series of high-impact scientific publications. Overall, the project represents a transition from a task-specific tracking system to a broader, scalable research framework for multimodal AI modeling and analysis, providing a foundation for future work at the intersection of computer vision, machine learning, AI security, and privacy-aware data analysis.