Bayesian Optimization with Large-scale samplinig
Abstract
Bayesian optimization( (BO) is used with large prevalence as a technique for the global optimization of expensive-to-evaluate black-box functions. It does so by sequentially querying points in a pre-defined search space and constructing a probabilistic surrogate model from the observations.
When selecting the next point to query, several approaches exist. These approaches range from simple heuristics to sophisticated information-theoretic methods that utilize different properties of the surrogate model for making their selection. We are currently investigating a new information-theoretic approach, which uses a Monte Carlo approach to simulate an expected information gain through simulating realizations of the optimum in all possible regions of the search space. Early results indicate that the approach is significantly more effective than the current state of the art, in terms of number of black-box queries to find (approximately) the global optimum.
However, to test out method for higher-dimensional problems and to explore its theoretical properties, we need to run larger-scale sampling rounds for our expectation. Our computational load mainly constitists of large matrix-matrix multiplications as well as Cholesky decompositions, which drives up both the computational cost an memory footprint. As such, we see a severe need to get access to "fatter" nodes (the fatter, the better), where experiments can be conducted with an order of magnitude more samples than we are currently using. Allowing such experiments would likely be crucial for the success of our work.
All experiments can (and have) been run on a single CPU, and one experiment typically takes between 2-12 hours.
