Reading Assignment: Introspection

The blog about tf-explain gives a nice overview about different introspection techniques but is not very comprehensive.
The toolbox works with TF 2.0, but currently only works on image data.

This overview blog post and a more recent addition will give you a nice overview about feature visualization.

Now it’s time to look at some specific techniques. Dig as deep as you like! These websites (and papers) provide plenty of background information.

Beginnings of the Field

• Basic Saliency Maps and Feature Visualization in
  Simonyan et al. (2013) Deep Inside Convolutional Networks

Saliency Maps

• GradCAM Gradient-weighted Class Activation Mapping
• Heatmapping, especially Montavon et al. (2017) Methods for Interpreting and Understanding Deep Neural Networks
• Sanity Checks for Saliency Maps

Feature Visualization

• Deepvis (code and paper are linked)

Representation Analysis

• Linear Classifier Probes
• Visualizing NNs with the Grand Tour
• Visualizing Deep Neural Networks with Topographic Activation Maps
  This one is our work ;)

Optional

Some Architecture-Specific Approaches

• LSTMVis (code and paper are linked)
• exBert - Visual Analysis of Transformer Models (code and paper are linked)
• Abnar & Zuidema (2020) Quantify Attention Flow in Transformers

Overview Literature about Interpretable AI

• Christoph Molnar - Interpretable ML Book
• Nauta et al. (2023) From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
  published version, arXiv version including supplementary material