Reading Assignment: Word Embeddings & Language Models

Besides the image processing domain that we already covered extensively (e.g. in reading 3), deep learning has also had a major impact on the field of Natural Language Processing (NLP) culminating in the recent boom of large languge models (LLMs). Here, we will look at important innovations that got us to this point. A further in-depth coverage of LLMs will be part of the “Learning Generative Models” course in the next term.

The following references provide a good overview. Feel free to dive deeper if you are interested!

• Sebastian Ruder’s blog post “A Review of the Neural History of Natural Language Processing” gives a nice introductory overview of the most important deep learning developments in NLP that nicely connects with the topics covered in the previous lectures.

• Next, read the two overview blog posts by Lilian Weng “Learning Word Embedding” and “Generalized Language Models”.

Further Reading (Optional)

• Sebastian Ruder describes the implications of models like BERT for NLP in his block post “NLP’s ImageNet moment has arrived”.
• For further in-depth reading, follow the links from the blog posts to the referenced articles.
• For those who cannot wait to dive deeper into LLMs, the paper “A Survey of Large Language Models” provides a quite comprehensive overview - on 97 pages!