Add LFBO Pattern Search

[ethche]

Includes the LFBO Pattern Search autotuner, which modifies PatternSearch to search through configs using a learned acquisition function (RandomForestClassifier) according to the likelihood-free bayesian optimization framework [1].

• Similar to PatternSearch, we generate neighbors from search copies. But instead we generate random neighbors instead of exhaustive set.
• Filters a fraction of them to evaluate using a fitted RandomForestClassifier. This classifier is trained to learn which configs are the best x% of configs. As a result, the classifier learns which configs are likely to improve upon the best config seen so far.

This improves over PatternSearch in kernel latency and autotuning wall-clock time on B200 for a set of benchmark kernels. DifferentialEvolution can improve further upon this, but takes substantially longer. DESurrogate in #1096, has comparable performance. We also compare to UCBPatternSearch, a previous proposal which uses a Gaussian Process + UCB acquisition function.

Kernel latency:

Autotuning Wall-clock Speedup:

Autotuning Convergence Time:

Some example convergence plots:

[1] J. Song, et al. A General Recipe for Likelihood-free Bayesian Optimization

[jansel]

If you update the benchmarking CI job to install the extra deps we can also do a benchmark run for this.

[ethche]

Hi @jansel, I ran the benchmark CI job for LFBO Pattern Search: HUD.

Overall, we observe equivalent performance to the default tuner while wall-clock times are substantially reduced (0.8x to 0.5x across the board).

• PatternSearch CI
• LFBOPatternSearch CI

We even see slight performance improvements for AMD.