*13 March 2017*

notes on (Krishnan et al., 2017).

- understanding: 7/10
- code: github.com/clinicalml/structuredinference

basically vae on state space models (SSMs): learn model parameters of SSMs and at the same time learn an inference network. the SSM under consideration is the standard SSM. but the transition is a neural net. emission is also a neural net. everything is gaussian.

the novelty lies in

- form of \(q\)
- the reformulation of the ELBO

\(q\) takes in the form \begin{align} q_{\phi}(x_{1:T} \given y_{1:T}) = q_{\phi}(x_1 \given x_1, \dotsc, x_T) \prod_{t = 2}^T q_{\phi}(x_t \given x_{t - 1}, y_t, \dotsc, y_T), \end{align} i.e. condition only on the last \(x\) and the future \(y_t\)s. this comes from considering the conditional independence structure of the posteriorâ€¦

other forms of \(q_{\phi}\) such as

- \(q_{\phi}(x_1 \given x_1, \dotsc, x_T) \prod_{t = 2}^T q_{\phi}(x_t \given y_1, \dotsc, y_T)\),
- \(q_{\phi}(x_1 \given x_1) \prod_{t = 2}^T q_{\phi}(x_t \given y_1, \dotsc, y_t)\),
- \(q_{\phi}(x_1 \given x_1, \dotsc, x_T) \prod_{t = 2}^T q_{\phi}(x_t \given x_{t - 1}, y_1, \dotsc, y_T)\),
- \(q_{\phi}(x_1 \given x_1) \prod_{t = 2}^T q_{\phi}(x_t \given x_{t - 1}, y_1, \dotsc, y_t)\),

but \(q_{\phi}(x_1 \given x_1, \dotsc, x_T) \prod_{t = 2}^T q_{\phi}(x_t \given x_{t - 1}, y_t, \dotsc, y_T)\) performs best.

the elbo has some sort of weird form because everything is gaussian. reparametrization trick is thus not neededâ€¦ check eq. 6.

experiments on

- 2d linear gaussian state space model:
- \(x\) is in \(\mathbb R^2\)
- \(y\) is in \(\mathbb R^2\)
- \(T\) is 25
- \(N\) (number of data sets) is 5000

- polyphonic music
- \(x\) is in ??
- \(y\) is in \(\{0, 1\}^{88}\)
- \(T\) is ??
- \(N\) is ??

- ehr patient data
- \(x\) is in \(\mathbb R^{200}\)
- \(y\) is in \(\{0, 1\}^{48}\)
- \(T\) is 18
- \(N\) is 15000

- Krishnan, R. G., Shalit, U., & Sontag, D. (2017). Structured Inference Networks for Nonlinear State Space Models.
*AAAI*.@inproceedings{krishnan2016structured, title = {Structured Inference Networks for Nonlinear State Space Models}, author = {Krishnan, Rahul G and Shalit, Uri and Sontag, David}, booktitle = {AAAI}, year = {2017} }

[back]