High-Dimensional Bayesian Optimization
||High-Dimensional Bayesian Optimization|
||Leonard Papenmeier <email@example.com>|
||2023-02-08 – 2023-09-01|
Bayesian optimization (BO) is a popular technique for optimizing expensive-to-evaluate black-box functions (EBFs), i.e., functions whose value can only be observed at specific points and whose evaluation might take a considerable time.
While being paramount for applications such as neural architecture search (NAS), drug design, robotics, or engineering, EBFs with hundreds or thousands of parameters still pose a challenge. Current techniques make strong assumptions about the structure of the function, which might not hold in practice.
This project is targeting to push the boundaries of high-dimensional BO. More specifically, I am working on a follow-up project of BAxUS (1), a recent algorithm that gave a new state-of-the-art performance for high-dimensional BO.
BAxUS is restricted to problems with a continuous search space and does not allow for parallel function evaluations. To make it an off-the-shelf solver for practitioners, I aim to support more complex search spaces, such as a mix of continuous, categorical, and ordinal variables or hierarchical search spaces.
Mixed spaces are important for hyperparameter tuning and many engineering problems. Hierarchical spaces are spaces where the presence of a parameter depends on the presence of other parameters. For example, the kernel size of a layer in NAS only needs to be optimized if that layer is a convolutional layer and not, for example, a fully-connected layer.
High-dimensional BO usually requires a considerable amount of function evaluations, and the computation of a Gaussian process (GP) posterior grows cubically with the number of observations. Furthermore, the maximum-likelihood estimation for the GP hyperparameters grows with the number of function parameters.
Luckily, modern frameworks such as BoTorch (2) allow training GPs with GPUs. This is why I request access to the Berzelius cluster.