Title:	Structural Simplicity in Neural Networks
DNr:	NAISS 2024/22-1047
Project Type:	NAISS Small Compute
Principal Investigator:	Joel Nilsson <joel.nilsson@liu.se>
Affiliation:	Linköpings universitet
Duration:	2024-08-12 – 2025-09-01
Classification:	10202
Keywords:

Abstract

The aim of this project is to understand from a new point of view the ability of deep neural networks to successfully learn from data and generalize. There is a tendency of neural networks (trained in presence or absence of explicit regularization) to represent a given input-output relation in a structurally “simple” manner, in the sense that many parts of the network encode for the same, most informative feature(s) of the input. We speculate that the degree of simplicity, which is related to the ability of the models to learn the high-information modes of data and disregard the noise, can be quantified using graph theoretic measures on trained neural networks. So far we have observed the tendency to simplicity in the structure of small-scale networks of different architectures, e.g., ANNs and CNNs, trained on real data. To test the hypothesis further, we need to run experiments on medium to large-scale networks, which require significant computational resources.