*27 February 2017*

notes on (Duvenaud et al., n.d.) and (Lloyd et al., 2014). these papers are part of the automatic statistician project.

understanding: 5/10

code: https://github.com/jamesrobertlloyd/gp-structure-search, https://github.com/jamesrobertlloyd/gpss-research

the problem: find good kernels for GPs.

kernels are closed under addition and multiplication.
we form a generative grammar for kernel construction:
\begin{align}
\mathcal S &\to \mathcal S + \mathcal B \\

\mathcal S &\to \mathcal S \times \mathcal B \\

\mathcal B &\to \mathcal B’
\end{align}
where \(\mathcal S\) is a subexpression (e.g. \(\mathrm{LIN} \times \mathrm{LIN}\)) and \(\mathcal B\) is a base kernel; one of \(\mathrm{SE}\) (exponentiated square difference), \(\mathrm{PER}\) (periodic), \(\mathrm{LIN}\) (linear), \(\mathrm{RQ}\) (rational quadratic).

how do we search over expressions? we use greedy search: at each stage, we choose the highest scoring kernel and expand it by applying all possible operators.

how do we score kernels? using the bayesian information criterion: \begin{align} \mathrm{BIC}(M) = -2 \log p(D \given M) + |M| \log n. \end{align} here, \(M\) is kernel, \(D\) is data, \(n\) is number of data points \(|M|\) is number of kernel parameters, \(p(D \given M)\) is the evidence.

? this is some sort of approximation to \(p(D) = \int p(D \given M) p(M) \,\mathrm dM\) if we had a prior on \(M\).

- Duvenaud, D., Lloyd, J. R., Grosse, R., Tenenbaum, J. B., & Ghahramani, Z. Structure Discovery in Nonparametric Regression through Compositional Kernel Search.
*30th International Conference on Machine Learning (June 2013)*.@inproceedings{duvenaud2013structure, title = {Structure Discovery in Nonparametric Regression through Compositional Kernel Search}, author = {Duvenaud, David and Lloyd, James Robert and Grosse, Roger and Tenenbaum, Joshua B and Ghahramani, Zoubin}, booktitle = {30th International Conference on Machine Learning (June 2013)}, organization = {International Machine Learning Society} }

- Lloyd, J. R., Duvenaud, D., Grosse, R., Tenenbaum, J., & Ghahramani, Z. (2014). Automatic Construction and Natural-Language Description of Nonparametric Regression Models.
*Twenty-Eighth AAAI Conference on Artificial Intelligence*.@inproceedings{lloyd2014automatic, title = {Automatic Construction and Natural-Language Description of Nonparametric Regression Models}, author = {Lloyd, James Robert and Duvenaud, David and Grosse, Roger and Tenenbaum, Joshua and Ghahramani, Zoubin}, booktitle = {Twenty-Eighth AAAI Conference on Artificial Intelligence}, year = {2014} }

[back]