Reading Assignment 5: Advanced VAE Techniques

In this reading assignment, you will develop a better understanding of the typical problems in VAE training and learn about techniques that aim to remedy them.

(V)AEs

• Start with the nice overview blog post “From Autoencoder to Beta-VAE” by Lilian Weng. (You can skim over the already familiar 1st half.)
• Continue with these blog posts on VAEs, their problems and variants: Part 1 Part 2
• Dive deeper with three recent papers on
  • Preventing posterior collapse with delta-VAEs - focus on the general idea
  • Don’t Blame the ELBO! A Linear VAE Perspective on Posterior Collapse - focus on the definition and the method to measure posterior collapse
  • NVAE: A Deep Hierarchical Variational Autoencoder paper code - try to gather as many good practices as possible
• Optionally, read more about Generating Diverse High-Fidelity Images with VQ-VAE-2 - Hi-Res Version (already covered in the blog above)

Further optional readings on VAEs

Read as many papers/posts as you can/want. Many are concerned with general architectural considerations and/or “tricks”, but some are about specific applications (bias towards music). Pick those that interest you. Also, please share any interesting sources you find on the topic!

• Learn about VAEs with spherical latent spaces in Hyperspherical Variational Auto-Encoders
• If you want to work with a VQ-VAE, this paper on Theory and Experiments on Vector Quantized Autoencoders will be helpful.
• Check out the hierarchical VQ-VAE “Jukebox” by OpenAI!
• Taming VAEs or a similar shorter paper
• InfoVAE
• Wasserstein AEs, along with a follow-up. Bonus question: How are InfoVAE and WAEs related?
• VAEs for learning timbre spaces

Further optional readings on VAE Hybrids (for later, when we have covered the other techniques)

• AE + GAN:
  • Adversarial Autoencoders
  • Adversarially Learned Inference
  • Autoencoding Beyond Pixels
  • Variational Approaches for AE GANs
• AE + Autoregressive
  • MusicVAE
  • PixelVAE
  • Autoregressive Discrete Autoencoders for music
  • NSynth with Wavenet autoencoders
• GAN + Autoregressive
  • ??? Find some :)

Feel free to share additional papers or other resources you come across!