Reading Assignment 6: Autoregressive Models

Chapter 22 from Probabilistic Machine Learning: Advanced Methods gives a good overview of autoregressive models. Since these models are conceptually very straightforward, there is not too much deep theory here.

Next, The unreasonable effectiveness of RNNs by Andrej Karpathy covers the basics of language modeling. Although outdated, state-of-the-art approaches like GPT work in essentially the same way.

Classic Success Stories

Consider these optional reading in case you want to get some more concrete examples. Some of these may be treated in more detail later in the class.

• PixelRNN models images directly on the pixel level. As the CNN variant was used a lot more, you can skip the details of the RNNs. There is also a follow-up, and another one.
• Wavenet was a revolutionary model that generated audio in an autoregressive manner. It’s basically PixelCNN in 1D.
• VQ-VAE encodes data in a lower-dimensional space and then uses an autoregressive model on that space. There is also a follow-up which uses a multi-level approach to generate very high-quality images.