Title:	Robust ML via Signal Space Representations
DNr:	Berzelius-2025-297
Project Type:	LiU Berzelius
Principal Investigator:	Mikael Skoglund <skoglund@kth.se>
Affiliation:	Kungliga Tekniska högskolan
Duration:	2025-09-26 – 2026-04-01
Classification:	10210
Homepage:	https://people.kth.se/~skoglund/
Keywords:

Abstract

Machine learning and AI are quickly becoming ubiquitous, and yet the development of learning algorithms is often based on heuristics and empirically investigated best practices. Moreover, we often lack theory to explain the algorithms’ properties and performance, which makes designing and optimising the algorithms challenging. Recent attempts at explaining learning algorithms has included inductive bias, which are constraints induced either implicitly or explicitly, by optimisers, architecture, and loss functions. We have previously utilised well-developed mathematical tools from communication theory and related fields to design and analyse inductive biases. These tools have enabled both theoretical and empirical investigations of common inductive biases. In a previous NAISS project (2023/22-844), we induced an explicit inductive bias on the signal space representations: we characterised the optimal separation between signal space representations, as well as provided multiple approaches to achieve near-optimal separation in practice [1]. In subsequent work (NAISS projects 2024/22-1101 and 2025/22-665), we scaled this approach to large-scale computer vision benchmarks [2]. These benchmarks have called into question several design choices of existing algorithms. In this project, we aim to build on our previous work to induce and analyse inductive biases. Our goals are threefold. Firstly, we aim to run possible follow-up experiments in response to reviewer comments on our previous work. Secondly, we aim to extend our previous empirical results to self-supervised learning settings. Thirdly, we aim to align these inductive biases with theoretical guarantees like PAC-Bayesian bounds. During the autumn, we expect to fulfill the first aim, as well as conducting initial feasibility analyses on the other goals. These initial results will then inform our continued work. We will use PyTorch with DistributedDataParallel to efficiently parallelise workloads across GPUs. We also aim to utilise mixed precision training to utilise computational resources even more efficiently. We will mainly consider computer vision benchmarks on ImageNet, for which we have already developed efficient paralellised software. We will mainly use the resources to scale up methods to larger datasets, and do initial developent on smaller benchmarks like CIFAR-10/100 elsewhere since they do not require parallelisation. [1] M. Lindström, B. Rodríguez-Gálvez, R. Thobaben, and M. Skoglund, ‘A Coding-Theoretic Analysis of Hyperspherical Prototypical Learning Geometry’, in Proceedings of the Geometry-grounded Representation Learning and Generative Modeling Workshop (GRaM), PMLR vol. 251, pp. 78–91. [2] M. Lindström, R. Thobaben, and M. Skoglund, ‘On the Importance of Separation and Labelling in Protoypical Learning’, in IEEE Transactions on Pattern Analysis and Machine Intelligence, under review.