Towards Detailed Visual Understanding via Large-scale Vision-Language Models
Title: Towards Detailed Visual Understanding via Large-scale Vision-Language Models
DNr: Berzelius-2024-6
Project Type: LiU Berzelius
Principal Investigator: Fahad Khan <>
Affiliation: Linköpings universitet
Duration: 2024-02-02 – 2024-09-01
Classification: 10207


Significant advancements in the field of computer vision have recently been observed due to the development of many foundational vision-language models. These models represent a significant leap towards creating general-purpose vision models capable of tackling various tasks simultaneously. To this end, conversation agents fueled by Large Language Models (LLMs) are providing a new way to interact with visual data. While there have been initial attempts for image-based conversation models, this project addresses the underexplored field of video-based multimodal foundation models by developing novel approaches that combine the capabilities of LLMs with a pretrained visual encoder adapted for multimodal spatial or spatiotemporal representations. The project further aims to explore a semi-automatic annotation framework for generation high quality instruction data for images, videos and text. Moreover, the aim is to look into developing novel and efficient transformers-based architectures that serve as a key building block behind these recent large vision-language foundation models. Such new novel proposed methods are expected to hava profound impact in many real-world applications ranging from healthcare to intelligent autnomous systems.