Title:	Growing and Looping for Vision Foundation Models
DNr:	Berzelius-2026-275
Project Type:	LiU Berzelius
Principal Investigator:	Karl Henrik Johansson <kallej@kth.se>
Affiliation:	Kungliga Tekniska högskolan
Duration:	2026-10-23 – 2027-05-01
Classification:	10201
Homepage:	https://www.kth.se/profile/amkm
Keywords:

Abstract

Understanding how vision foundation models use their depth is important for developing more effective and computationally efficient training methods. Growing for Vision studies whether gradually increasing the depth of a model during training can improve the quality of its learned representations while reducing the cost compared with training at full depth throughout. We investigate this question using self-supervised vision transformers, primarily DINOv3 and MAE, through comparisons with fixed-depth baselines and layer-wise evaluation on image classification, semantic segmentation, and depth estimation. Building on the experiments completed during the current allocation, the proposed continuation will provide the compute needed for final controlled comparisons, targeted ablations, and downstream evaluations to complete the study and support its review process. The paper is now in its final experimental and writing stages and is planned for submission to ICLR 2027 in late September 2026. Ongoing Collaboration Amir Mohammad Karimi Mamaghan joined the collaboration on Group Distributionally Robust Generative Optimization (GDRGO) in July/August 2026. This project addresses a limitation of standard generative model fine-tuning: maximizing average reward can leave some groups of prompts poorly served, even when overall performance improves. Performance can also be sensitive to changes in the proportions of these groups between training and deployment. GDRGO explicitly accounts for group-wise performance and optimizes against shifts in prompt-group proportions. The proposed method first estimates group weights using samples from a pretrained model, then uses these weights to guide a single fine-tuning stage. The aim is to improve worst-group performance while allowing control over the trade-off between average performance and robustness. The current manuscript includes theoretical analysis, a controlled 2D example, and preliminary text-to-image experiments. It is planned for submission to ICLR 2027 in late September 2026. During the extension, we will complete the remaining comparisons and ablations and carry out targeted additional experiments needed to finalize the paper and address reviewer feedback. Side Project We will also continue the existing protein–protein binder design collaboration with researchers in Munich/TUM. This project aims to guide existing protein foundation models toward designing binders with higher experimental binding affinity, through the curation of binding-affinity data and targeted model adaptation. The work will build on the initial model investigations and data preparation described in the previous proposal. During this extension, we plan to carry out a limited number of fine-tuning and candidate-evaluation runs using the existing models and pipeline. This remains a longer-term side project with modest compute requirements, and its scope during the extension will depend on the available binding-affinity data and progress of the collaboration. Purpose of the Extension WASP PhD student Amir Mohammad Karimi Mamaghan plans to complete his doctoral studies within the next six months. We therefore respectfully request a six-month continuation of the compute project to complete the current manuscripts and support the remaining doctoral research. The allocation will primarily support final experiments and analyses for Growing for Vision and GDRGO projects, additional experiments during the rebuttal periods, and the finalization of the manuscripts. A small share will support the ongoing binder-design work. The proposed extension is focused on completing these existing research contributions.