# automatic statistician (wip)

27 February 2017

notes on (Duvenaud, Lloyd, Grosse, Tenenbaum, & Ghahramani, n.d.) and (Lloyd, Duvenaud, Grosse, Tenenbaum, & Ghahramani, 2014). these papers are part of the automatic statistician project.

## summary

the problem: find good kernels for GPs.

kernels are closed under addition and multiplication. we form a generative grammar for kernel construction: \begin{align} \mathcal S &\to \mathcal S + \mathcal B \\ \mathcal S &\to \mathcal S \times \mathcal B \\ \mathcal B &\to \mathcal B’ \end{align} where $\mathcal S$ is a subexpression (e.g. $\mathrm{LIN} \times \mathrm{LIN}$) and $\mathcal B$ is a base kernel; one of $\mathrm{SE}$ (exponentiated square difference), $\mathrm{PER}$ (periodic), $\mathrm{LIN}$ (linear), $\mathrm{RQ}$ (rational quadratic).

how do we search over expressions? we use greedy search: at each stage, we choose the highest scoring kernel and expand it by applying all possible operators.

how do we score kernels? using the bayesian information criterion: \begin{align} \mathrm{BIC}(M) = -2 \log p(D \given M) + |M| \log n. \end{align} here, $M$ is kernel, $D$ is data, $n$ is number of data points $|M|$ is number of kernel parameters, $p(D \given M)$ is the evidence.

? this is some sort of approximation to $p(D) = \int p(D \given M) p(M) \,\mathrm dM$ if we had a prior on $M$.

1. Duvenaud, D., Lloyd, J. R., Grosse, R., Tenenbaum, J. B., & Ghahramani, Z. Structure Discovery in Nonparametric Regression through Compositional Kernel Search. In 30th International Conference on Machine Learning (June 2013). International Machine Learning Society.
@inproceedings{duvenaud2013structure,
title = {Structure Discovery in Nonparametric Regression through Compositional Kernel Search},
author = {Duvenaud, David and Lloyd, James Robert and Grosse, Roger and Tenenbaum, Joshua B and Ghahramani, Zoubin},
booktitle = {30th International Conference on Machine Learning (June 2013)},
organization = {International Machine Learning Society}
}

2. Lloyd, J. R., Duvenaud, D., Grosse, R., Tenenbaum, J., & Ghahramani, Z. (2014). Automatic Construction and Natural-Language Description of Nonparametric Regression Models. In Twenty-Eighth AAAI Conference on Artificial Intelligence.
@inproceedings{lloyd2014automatic,
title = {Automatic Construction and Natural-Language Description of Nonparametric Regression Models},
author = {Lloyd, James Robert and Duvenaud, David and Grosse, Roger and Tenenbaum, Joshua and Ghahramani, Zoubin},
booktitle = {Twenty-Eighth AAAI Conference on Artificial Intelligence},
year = {2014}
}


[back]