Reading Assignment: Attention and Memory


For an overview on the topic, start with the blog post “Attention? Attention!” by Lilian Weng and the article “Attention and Augmented Recurrent Neural Networks” by Olah & Carter.

More on Transformers

More on neural networks with explicit memory (optional)

If you would like to dive more into this very exciting topic. have a look at the paper Graves et al. “Hybrid computing using a neural network with dynamic external memory” in Nature 538.7626 (2016): pp. 471-476.. There is also a video recording of Alex Graves’ talk at NIPS 2016.

Further Reading (Optional)

There are many exciting directions to explore further: