Tuan Anh Le

improved semantic representations from tree-structured long short-term memory networks

13 February 2017

notes on (Tai, Socher, & Manning, 2015).

summary

understanding: 7/10
code: https://github.com/stanfordnlp/treelstm

the tree-lstm paper. gives an original architecture in which lstms are not just sequential. hidden state is composed from arbitrarily many child units, not just previous. two suggested architectures:

normal lstm

\begin{align} i_t &= \sigma\left(W^{(i)} x_t + U^{(i)} h_{t - 1} + b^{(i)}\right) \\ f_t &= \sigma\left(W^{(f)} x_t + U^{(f)} h_{t - 1} + b^{(f)}\right) \\ o_t &= \sigma\left(W^{(o)} x_t + U^{(o)} h_{t - 1} + b^{(o)}\right) \\ u_t &= \tanh\left(W^{(u)} x_t + U^{(u)} h_{t - 1} + b^{(u)}\right) \\ c_t &= i_t \odot u_t + f_t \odot c_{t - 1} \\ h_t &= o_t \odot \tanh(c_t) \end{align}

child-sum tree-lstm

given a tree, let denote the set of children of node : \begin{align} \tilde h_j &= \sum_{k \in C(j)} h_k \\ i_j &= \sigma\left(W^{(i)} x_j + U^{(i)} \tilde h_j + b^{(i)}\right) \\ f_j &= \sigma\left(W^{(f)} x_j + U^{(f)} \tilde h_j + b^{(f)}\right) \\ o_j &= \sigma\left(W^{(o)} x_j + U^{(o)} \tilde h_j + b^{(o)}\right) \\ u_j &= \tanh\left(W^{(u)} x_j + U^{(u)} \tilde h_j + b^{(u)}\right) \\ c_j &= i_j \odot u_j + \sum_{k \in C(j)} f_{jk} \odot c_k \\ h_j &= o_j \odot \tanh(c_j) \end{align}

n-ary tree-lstm

tree structure must have at most children. children are ordered. for any node , index its th child as . \begin{align} i_j &= \sigma\left(W^{(i)} x_j + \sum_{\ell = 1}^N U_{\ell}^{(i)} h_{j\ell} + b^{(i)}\right) \\ f_{jk} &= \sigma\left(W^{(f)} x_j + \sum_{\ell = 1}^N U_{k\ell}^{(f)} h_{j\ell} + b^{(f)}\right) \\ o_j &= \sigma\left(W^{(o)} x_j + \sum_{\ell = 1}^N U_{\ell}^{(o)} h_{j\ell} + b^{(o)}\right) \\ u_j &= \tanh\left(W^{(u)} x_j + \sum_{\ell = 1}^N U_{\ell}^{(u)} h_{j\ell} + b^{(u)}\right) \\ c_j &= i_j \odot u_j + \sum_{\ell = 1}^N f_{j\ell} \odot c_{j\ell} \\ h_j &= o_j \odot \tanh(c_j) \end{align}

experiments

didn’t look into too much detail but there were two experiments:

discussion

hypotheses:


references

  1. Tai, K. S., Socher, R., & Manning, C. D. (2015). Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Association for Computational Linguistics (ACL).
    @inproceedings{tai2015improved,
      title = {Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks},
      author = {Tai, Kai Sheng and Socher, Richard and Manning, Christopher D.},
      booktitle = {Association for Computational Linguistics (ACL)},
      year = {2015}
    }
    

[back]