Reading Assignment: Autoencoders and Word Embeddings

This reading assignemt focusses on unsupervised and self-supervised learning tasks as a main driver for representation learning.

Main Reading 1: Autoencoders

The Deep Learning Book - Chapter 14: Autoencoders (optional reading!) covers the topic very well in depth. However, in favor of including the second topic, the blog post “Introduction to autoencoders” by Jeremy Jordan provides the most important details. Refer to the book chapter if you would like to know further details!

Main Reading 2: Word Embeddings

Besides the image processing domain that we already covered in reading 3 and 4, deep learning has also had a major impact on the field of Natural Language Processing (NLP). The following references provide a good overview. Feel free to dive deeper if you are interested!

• Sebastian Ruder’s blog post “A Review of the Neural History of Natural Language Processing” gives a nice introductory overview of the most important deep learning developments in NLP that nicely connects with the topics covered in the previous lectures.

• Next, read the two overview blog posts by Lilian Weng “Learning Word Embedding” and “Generalized Language Models”.

Further Reading (Optional)

• Sebastian Ruder describes the implications of models like BERT for NLP in his block post “NLP’s ImageNet moment has arrived”.
• The blog post “Unsupervised Cross-lingual Representation Learning” by Sebastian Ruder provides a glimpse at an exciting new NLP task that is tackled by deep learning.
• If you would like to dive deeper into self-supervised representation learning approaches (like BERT), Lilian Weng’s overview blog post “Self-Supervised Representation Learning” is highly recommended.
• For further in-depth reading, follow the links from the blog posts to the referenced articles.