Reading Assignment 3: Approximate Inference & Variational Autoencders

Preliminaries

You can start by reading chapter 19, intro and 19.1 for some context of what we understand under “inference”, why this is difficult as well as connecting to the previous topics (at least somewhat). You may want to check the definiton of the Kullback-Leibler Divergence first.

You can also read the remainder of the chapter if you want – however this is very hard to follow. Another, somewhat simpler explanation can be found in this short blog post. However, at the end of the day, variational inference is a highly mathematical topic, so formulas are hard to avoid.

The main point is for you to understand that we will be using a lower bound on the likelihood, namely the evidence lower bound (ELBO), as a training objective.

Finally, read section 20.9 in the Deep Learning Book on backpropagating through random operations. This is an important “trick” to successfully use VAEs. You can skip section 20.9.1 on discrete operations.

Variational Autoencoders

We offer you a variety of readings on VAEs. These explain the concept in different levels of detail and from different perspectives. See which ones work for you!

• Section 20.10.3 in the DL book is written in the familiar style and links back to many concepts of chapter 19 (which you may not have read).
• This blog post first introduces VAEs from the neural network perspective, which should be more familiar to you, and then introduces the graphical models perspective. If you only read one of these links, make it this one.
• This blog gives a rather simplified explanation. This can be good as a first piece of context, but you should aim for a deeper understanding in this class.
• A longer, more in-depth treatment can be found here.
• Finally, you can read the original paper, if you want.
  • This paper is often credited as essentially an independent “parallel discovery”.