Model Interpretability and Understanding for PyTorch using Captum

Introduction

Methods of exemplary interpretability person gained increasing value successful caller years arsenic a nonstop consequence of the emergence successful exemplary complexity and the associated deficiency of transparency. Model knowing is simply a basking taxable of study and a focal area for applicable applications employing instrumentality learning crossed various sectors.

Captum supplies academics and developers pinch cutting-edge techniques, specified arsenic Integrated Gradients, that make it elemental to place the elements that lend to a model’s output. Captum makes it easier for ML researchers to usage PyTorch models to build interpretability methods.

By making it easier to place the galore elements that lend to a model’s output, Captum tin thief exemplary developers create amended models and hole models that supply unexpected results.

Algorithm Descriptions

Captum is simply a room that allows for the implementation of various interpretability approaches. It is imaginable to categorize Captum’s attribution algorithms into 3 wide categories:

• primary attribution: Determines the publication of each input characteristic to a model’s output.
• Layer Attribution: Each neuron successful a peculiar furniture is evaluated for its publication to the model’s output.
• Neuron Attribution: A hidden neuron’s activation is wished by evaluating the publication of each input feature.

The pursuing is simply a little overview of the various methods that are presently implemented wrong Captum for primary, layer, and neuron attribution. Also included is simply a explanation of the sound tunnel, which tin beryllium utilized to soft the results of immoderate attribution method. Captum provides metrics to estimate the reliability of exemplary explanations successful summation to its attribution algorithms. At this time, they supply infidelity and sensitivity metrics that assistance successful evaluating the accuracy of explanations.

Primary Attribution Techniques

Integrated Gradients

Let’s opportunity we person a general practice of a heavy network, F : Rn → [0, 1]. Let x ∈ Rn beryllium the existent input and x′ ∈ Rn beryllium the baseline input. The baseline successful image networks mightiness beryllium the achromatic image, whereas it mightiness beryllium the zero embedding vector successful matter models. From the baseline x′ to the input x, we compute gradients astatine each points on the straight-line way (in Rn). By cumulating these gradients, 1 tin make integrated gradients. Integrated gradients are defined arsenic the way integral of gradients on a nonstop way from the baseline x′ to the input x.

The 2 basal assumptions, sensitivity, and implementation invariance, shape the ground of this method. Please mention to the original paper to study much astir these axioms.

Gradient SHAP

Shapley values successful cooperative crippled theory are utilized to compute Gradient SHAP values, which are computed utilizing a gradient approach. Gradient SHAP adds Gaussian sound to each input sample aggregate times,  then picks a random constituent connected the way betwixt the baseline and the input to find the gradient of the outputs. As a result, the last SHAP values correspond the gradients’ expected value. * (inputs - baselines). SHAP values are approximated connected the premise that input features are independent and that the explanatory exemplary is linear betwixt the inputs and supplied baselines.

DeepLIFT

It is imaginable to usage DeepLIFT(a backmost propagation technique) to ascribe input changes based connected the differences betwixt inputs and their matching reference (or baseline). DeepLIFT attempts to explicate the disparity betwixt the output from reference utilizing the disparity betwixt the inputs from reference. DeepLIFT employs the thought of multipliers to “blame” individual neurons for the quality successful outputs. For a fixed input neuron x pinch difference-from-reference ∆x, and target neuron t pinch difference-from-reference ∆t that we wish to compute the publication to, we specify the multiplier m∆x∆t as:

DeepLIFT SHAP

DeepLIFT SHAP is simply a DeepLIFT hold based connected Shapley values established successful cooperative crippled theory. DeepLIFT SHAP computes the DeepLIFT attribution for each input-baseline brace and averages the resultant attributions per input illustration utilizing a distribution of baselines. DeepLIFT’s non-linearity rules thief to linearize the network’s non-linear functions, and the method’s approximation of SHAP values besides applies to the linearized network. Input features are likewise presumed to beryllium independent successful this method.

Saliency

Calculating input attribution via saliency is simply a straightforward process that yields the gradient of the output pinch respect to the input. A first-order Taylor network description is utilized astatine the input, and the gradients are the coefficients of each characteristic successful the model’s linear representation. The absolute worth of these coefficients tin beryllium utilized to bespeak the relevance of a feature. You tin find further accusation connected the saliency attack successful the original paper.

Input X Gradient

Input X Gradient is an hold of the saliency approach, taking the gradients of the output pinch respect to the input and multiplying by the input characteristic values. One intuition for this attack considers a linear model; the gradients are simply the coefficients of each input, and the merchandise of the input pinch a coefficient corresponds to the full publication of the characteristic to the linear model’s output.

Guided Backpropagation and Deconvolution

Gradient computation is performed via guided backpropagation and deconvolution, though backpropagation of ReLU functions is overridden specified that only non-negative gradients are backpropagated. While the ReLU usability is applied to the input gradients successful guided backpropagation, it is straight applied to the output gradients successful deconvolution. It is communal believe to employment these methods successful conjunction pinch convolutional networks, but they tin besides beryllium utilized successful different types of neural web architecture.

Guided GradCAM

Guided backpropagation attributions compute the element-wise merchandise of guided GradCAM attributions (guided GradCAM) pinch upsampled (layer) GradCAM attributions. Attribution computation is done for a fixed furniture and upsampled to fresh the input size. Convolutional neural networks are the attraction of this technique. However, immoderate furniture that tin beryllium spatially aligned pinch the input mightiness beryllium provided. Typically, the past convolutional furniture is  provided.

Feature Ablation

To compute attribution, a method known arsenic “feature ablation” employs a perturbation-based method that substitutes a known “baseline” aliases “reference value” (such arsenic 0) for each input characteristic earlier computing the output difference. Grouping and ablating input features is simply a amended replacement to doing truthful individually, and galore different applications tin use from this. By grouping and ablating segments of an image, we tin find the comparative value of the segment.

Feature Permutation

Feature permutation is simply a perturbation-based method successful which each characteristic is randomly permuted wrong a batch, and the alteration successful output (or loss) is computed arsenic a consequence of this modification. Features tin besides beryllium grouped together alternatively than individually successful the aforesaid measurement arsenic characteristic ablation. Note that successful opposition to the different algorithms disposable successful Captum, this algorithm is the only 1 that whitethorn supply due attributions erstwhile it is supplied pinch a batch of aggregate input examples. Other algorithms only request a azygous illustration arsenic input.

Occlusion

Occlusion is simply a perturbation-based attack to computing attribution, replacing each contiguous rectangular region pinch a fixed baseline/reference and computing the quality successful output. For features located successful aggregate areas (hyperrectangles), the corresponding output differences are averaged to compute the attribution for that feature. Occlusion is astir useful successful cases specified arsenic images, wherever pixels successful a contiguous rectangular region are apt to beryllium highly correlated.

Shapley Value Sampling

The attribution method Shapley value is based connected the cooperative crippled theory. This method takes each permutation of the input features and adds them 1 by 1 to a specified baseline. The quality successful output aft adding each characteristic corresponds to its contribution, and these differences are summed crossed each permutations to find attribution.

Lime

One of the astir wide utilized interpretability methods is Lime, which trains an interpretable surrogate exemplary by sampling information points astir an input illustration and utilizing exemplary evaluations astatine these points to train a simpler interpretable ‘surrogate’ model, specified arsenic a linear model.

KernelSHAP

Kernel SHAP is simply a method for calculating Shapley Values that uses the LIME framework. Shapley Values whitethorn beryllium obtained much efficiently successful the LIME model by mounting the nonaccomplishment function, weighting the kernel, and decently regularizing terms.

Layer Attribution Techniques

Layer Conductance

Layer Conductance is simply a method that builds a much broad image of a neuron’s value by combining the neuron’s activation pinch the partial derivatives of some the neuron pinch respect to the input and the output pinch respect to the neuron. Through the hidden neuron, conductance builds upon Integrated Gradients’ (IG’s) attribution flow. The full conductance of a hidden neuron y is defined arsenic follows successful the original paper:

Internal Influence

Using Internal Influence, 1 tin estimate the integral of gradients on the way from a baseline input to the provided input. This method is akin to applying integrated gradients, which involves integrating the gradient pinch respect to the furniture  (rather than the input).

Layer Gradient X Activation

Layer Gradient X Activation is the network’s balanced of the Input X Gradient method for hidden layers successful a network… It multiplies the activation of the furniture constituent by constituent pinch the gradients of the target output pinch respect to the specified layer.

GradCAM

GradCAM is simply a convolutional neural web furniture attribution method that is typically applied to the past convolutional layer. GradCAM computes the target output’s gradients pinch respect to the specified layer, averages each output transmission (output magnitude 2), and multiplies the mean gradient for each transmission by the furniture activations. A ReLU is applied to the output to guarantee that only non-negative attributions are returned from the sum of the results crossed each channels.

Neuron Attribution Techniques

Neuron Conductance

Conductance combines neuron activation pinch partial derivatives of some the neuron pinch respect to the input and the output pinch respect to the neuron to supply a much broad image of neuron relevance. To find the conductance of a circumstantial neuron, 1 examines the travel of IG attribution from each input that passes done that neuron. The pursuing is the original paper’s general meaning of conductance of neuron y fixed input attribution one :

According to this definition, It should beryllium noted that summing the conductance of a neuron (across each input features) is ever adjacent to the conductance of the furniture successful which that circumstantial neuron is located.

Neuron Gradient

The neuron gradient attack is the equivalence of the saliency method for a azygous neuron successful the network. It simply computes the gradient of neuron output comparative to exemplary input. This method, for illustration Saliency, whitethorn beryllium thought of arsenic doing a first-order Taylor description of the neuron’s output astatine the fixed input, pinch the gradients corresponding to the coefficients of each characteristic successful the model’s linear representation.

Neuron Integrated Gradients

It is imaginable to estimate the integral of input gradients pinch respect to a peculiar neuron passim the  path  from a baseline input to the input of liking utilizing a method called “Neuron Integrated Gradients.” Integral gradients are balanced to this method, assuming the output is conscionable that of the identified neuron. You tin find further accusation connected the integrated gradient attack successful the original insubstantial here.

Neuron GradientSHAP

Neuron GradientSHAP is the balanced of GradientSHAP for a circumstantial neuron. Neuron GradientSHAP adds Gaussian sound to each input sample aggregate times, chooses a random constituent on the way betwixt baseline and input, and computes the gradient of the target neuron pinch respect to each randomly picked point. The resultant SHAP values are adjacent to the predicted gradient values *. (inputs - baselines).

Neuron DeepLIFT SHAP

Neuron DeepLIFT SHAP is the balanced of DeepLIFT for a circumstantial neuron. Using distribution of baselines, the DeepLIFT SHAP algorithm computes the Neuron DeepLIFT attribution for each input-baseline brace and averages the resultant attributions per input example.

Noise Tunnel

Noise Tunnel is an attribution method that tin beryllium utilized successful conjunction pinch different methods. The sound passageway computes attribution aggregate times, adding Gaussian sound to the input each time, and past merges the resulting attributions depending connected the chosen type. The pursuing sound passageway types are supported:

• Smoothgrad: The sampled attributions’ mean is returned. Smoothing the specified attribution method utilizing a Gaussian Kernel is an approximation of this process.
• Smoothgrad Squared: The mean of the squared sample attributions is returned.
• Vargrad: The variance of the sample attributions is returned.

Metrics

Infidelity

Infidelity measures the mean squared correction betwixt exemplary explanations successful the magnitudes of input perturbations and predictor function’s changes to those input perturbtaions. Infidelity is defined arsenic follows:

From well-known attribution techniques specified arsenic the integrated gradient, this is simply a computationally much businesslike and extended conception of Sensitivy-n . The second analyzes the correlations betwixt the sum of the attributions and the differences of the predictor usability astatine its input and a predefined baseline.

Sensitivity

Sensitivity, which is defined arsenic the grade of mentation alteration to mini input perturbations utilizing Monte Carlo sampling-based approximation, is measured arsenic follows:

By default, we sample from a subspace of an L-Infinity shot pinch a default radius to approximate sensitivity. Users tin alteration the ball’s radius and the sample function.

Model Interpretation for Pretrained ResNet Model

This tutorial shows really to usage exemplary interpretability methods connected a pre-trained ResNet exemplary pinch a chosen image, and it visualizes the attributions for each pixel by overlaying them connected the image. In this tutorial, we will usage the mentation algorithms Integrated Gradients, GradientShape, Attribution pinch Layer GradCAM and Occlusion.

Before you start, you must person a Python situation that includes:

• Python type 3.6 aliases higher
• PyTorch type 1.2 aliases higher (the latest type is recommended)
• TorchVision type 0
• .6 aliases higher (the latest type is recommended)
• Captum (the latest type is recommended)

Depending connected whether you’re utilizing Anaconda aliases pip virtual environment, the pursuing commands will thief you group up Captum:

With conda:

conda instal pytorch torchvision captum -c pytorch

With pip:

pip instal torch torchvision captum

Let america import libraries.

import torch import torch.nn.functional arsenic F from PIL import Image import os import json import numpy arsenic np from matplotlib.colors import LinearSegmentedColormap import os, sys import json import numpy arsenic np from PIL import Image import matplotlib.pyplot arsenic plt from matplotlib.colors import LinearSegmentedColormap import torchvision from torchvision import models from torchvision import transforms from captum.attr import IntegratedGradients from captum.attr import GradientShap from captum.attr import Occlusion from captum.attr import LayerGradCam from captum.attr import NoiseTunnel from captum.attr import visualization arsenic viz from captum.attr import LayerAttribution

Loads pretrained Resnet exemplary and sets it to eval mode

model = models.resnet18(pretrained=True) model = model.eval()

The ResNet is trained connected the ImageNet data-set. Downloads and sounds the database of ImageNet dataset classes/labels successful memory.

wget -P $HOME/.torch/models https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json labels_path = os.getenv("HOME") + '/.torch/models/imagenet_class_index.json' with open(labels_path) arsenic json_data: idx_to_labels = json.load(json_data)

Now that we’ve completed the model, we tin download the image for analysis. In my case, I chose a feline image.

source

Your image files must incorporate the record cat.jpg. As we tin spot below, Image.open() opens and identifies the fixed image record and np.asarry() converts it to an array.

test_img = Image.open('path/cat.jpg') test_img_data = np.asarray(test_img) plt.imshow(test_img_data) plt.show()

In the codification below, we will specify transformers and normalizing functions for the image. To train our ResNet model, we utilized the ImageNet dataset, which requires images to beryllium of a peculiar size, pinch transmission information normalized to a specified scope of values. transforms.Compose() composes respective transforms together and transforms.Normalize() normalizes a tensor image pinch mean and modular deviation.

# exemplary anticipation is 224x224 3-color image transform = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), #crop the fixed tensor image astatine the center transforms.ToTensor() ]) # ImageNet normalization transform_normalize = transforms.Normalize( mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] ) img = Image.open('path/cat.jpg') transformed_img = transform(img) input = transform_normalize(transformed_img) #unsqueeze returns a caller tensor pinch a magnitude of size 1 inserted astatine the #specified position. input = input.unsqueeze(0)

Now, we will foretell the people of the input image. The mobility that tin beryllium asked is, “What does our exemplary consciousness this image represents?”

#call our model output = model(input) ## applied softmax() function output = F.softmax(output, dim=1) #torch.topk returns the k largest elements of the fixed input tensor on a fixed #dimension.K present is 1 prediction_score, pred_label_idx = torch.topk(output, 1) pred_label_idx.squeeze_() #convert into a dictionnary of keyvalues brace the foretell label, person it #into a drawstring to get the predicted label predicted_label = idx_to_labels[str(pred_label_idx.item())][1] print('Predicted:', predicted_label, '(', prediction_score.squeeze().item(), ')')

output:

Predicted: tabby ( 0.5530276298522949 )

The truth that ResNet thinks our image of a feline depicts an existent feline is verified. But what gives the exemplary the belief that this is an image of a cat? To get the solution to that question, we will consult Captum.

Feature Attribution pinch Integrated Gradients

One of the various characteristic attribution techniques successful Captum is Integrated Gradients. Integrated Gradients awards each input characteristic a relevance people by estimating the integral of the gradients of the model’s output pinch respect to the inputs.

For our case, we will return a peculiar constituent of the output vector - the 1 that indicates the model’s assurance successful its selected class - and usage integrated gradients to fig retired what aspects of the input image contributed to this output. It will let america to find which parts of the image were astir important successful producing this result. After we person obtained the value representation from Integrated Gradients, we will usage the visualization devices captured by Captum to supply a clear and understandable depiction of the value map.

Integrated gradients will determines the integral of the gradients of the output of the exemplary for the predicted people pred_label_idx pinch respect to the input image pixels on the way from the achromatic image to our input image.

print('Predicted:', predicted_label, '(', prediction_score.squeeze().item(), ')') #Create IntegratedGradients entity and get attributes integrated_gradients = IntegratedGradients(model) #Request the algorithm to delegate our output target to attributions_ig = integrated_gradients.attribute(input, target=pred_label_idx, n_steps=200)

Output:

Predicted: tabby ( 0.5530276298522949 )

Let’s spot the image and the attributions that spell on pinch it by overlaying the second connected apical of the image. The visualize_image_attr() method that Captum offers provides a group of possibilities for tailoring the position of the attribution information to your preferences. Here, we walk successful a civilization Matplotlib colour map(see LinearSegmentedColormap()).

#result visualization pinch civilization colormap default_cmap = LinearSegmentedColormap.from_list('custom blue', [(0, '#ffffff'), (0.25, '#000000'), (1, '#000000')], N=256) # usage visualize_image_attr helper method for visualization to show the #original image for comparison _ = viz.visualize_image_attr(np.transpose(attributions_ig.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)), method='heat_map', cmap=default_cmap, show_colorbar=True, sign='positive', outlier_perc=1)

Output:

You should beryllium capable to announcement successful the image that we person shown supra that the area surrounding the feline successful the image is wherever the Integrated Gradients algorithm gives america the strongest signal.

Let’s compute attributions by utilizing Integrated Gradients and past soft them retired complete respective images that person been produced by a noise tunnel. The second modifies the input by adding Gaussian sound pinch a modular deviation of one, 10 times (nt_samples=10). The smoothgrad_sq attack is utilized by sound passageway to make the attributions accordant crossed each nt_samples of noisy samples. The worth of smoothgrad_sq is the mean of the squared attributions crossed nt_samples samples. visualize_image_attr_multiple() visualizes attribution for a fixed image by normalizing the specified sign’s attribution values (positive, negative, absolute value, aliases all) and past displaying them successful a matplotlib fig utilizing the selected mode.

noise_tunnel = NoiseTunnel(integrated_gradients) attributions_ig_nt = noise_tunnel.attribute(input, nt_samples=10, nt_type='smoothgrad_sq', target=pred_label_idx) _ = viz.visualize_image_attr_multiple(np.transpose(attributions_ig_nt.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)), ["original_image", "heat_map"], ["all", "positive"], cmap=default_cmap, show_colorbar=True)

Output:

I tin spot successful the images supra that the exemplary concentrates connected the cat’s head.

Let’s decorativeness by utilizing GradientShap. GradientShap is simply a gradient attack that whitethorn beryllium utilized to compute SHAP values, and it is besides a awesome instrumentality for acquiring penetration into the world behavior. It is simply a linear mentation exemplary that explains the model’s predictions by utilizing a distribution of reference samples. It determines the expected gradients for an input picked randomly betwixt the input and a baseline. The baseline is picked astatine random from the supplied distribution of baselines.

torch.manual_seed(0) np.random.seed(0) gradient_shap = GradientShap(model) # Definition of baseline distribution of images rand_img_dist = torch.cat([input * 0, input * 1]) attributions_gs = gradient_shap.attribute(input, n_samples=50, stdevs=0.0001, baselines=rand_img_dist, target=pred_label_idx) _ = viz.visualize_image_attr_multiple(np.transpose(attributions_gs.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)), ["original_image", "heat_map"], ["all", "absolute_value"], cmap=default_cmap, show_colorbar=True)

Output:

Layer Attribution pinch Layer GradCAM

You tin subordinate the activity of hidden layers wrong your exemplary to features of your input pinch the thief of the Layer Attribution. We will use an algorithm for furniture attribution to analyse the activity of 1 of the convolutional layers included successful our model. GradCAM is responsible for computing the gradients of the target output pinch respect to the specified layer. These gradients are past averaged for each output transmission (dimension 2 of output), and the furniture activations are multiplied by the mean gradient for each channel. The results are summed crossed each channels. Since the activity of convolutional layers often maps spatially to the input, GradCAM attributions are often upsampled and utilized to disguise the input. It is worthy noting that GradCAM is explicitly developed for convolutional neural networks (convnets). Layer attribution is group up successful the aforesaid measurement arsenic input attribution, pinch the objection that successful summation to the model, you must supply a hidden furniture wrong the exemplary that you want to analyze. Similar to what was discussed before, erstwhile we telephone attribute(), we bespeak the target people of interest.

layer_gradcam = LayerGradCam(model, model.layer3[1].conv2) attributions_lgc = layer_gradcam.attribute(input, target=pred_label_idx) _ = viz.visualize_image_attr(attributions_lgc[0].cpu().permute(1,2,0).detach().numpy(), sign="all", title="Layer 3 Block 1 Conv 2")

To make a much meticulous comparison betwixt the input image and this attribution data, we will upsample it pinch the thief of the usability interpolate(), located successful the LayerAttribution guidelines class.

upsamp_attr_lgc = LayerAttribution.interpolate(attributions_lgc, input.shape[2:]) print(attributions_lgc.shape) print(upsamp_attr_lgc.shape) print(input.shape) _ = viz.visualize_image_attr_multiple(upsamp_attr_lgc[0].cpu().permute(1,2,0).detach().numpy(), transformed_img.permute(1,2,0).numpy(), ["original_image","blended_heat_map","masked_image"], ["all","positive","positive"], show_colorbar=True, titles=["Original", "Positive Attribution", "Masked"], fig_size=(18, 6))

Output:

Visualizations specified arsenic this 1 person the imaginable to supply you pinch unsocial insights into really your hidden layers respond to the input you provide.

Feature Attribution pinch Occlusion

Methods based connected gradients thief understand the exemplary successful position of straight computing the changes successful the output pinch respect to the input. The method known arsenic perturbation-based attribution takes a much nonstop attack to this problem by making modifications to the input to quantify the effect specified changes person connected the output. One specified strategy is called occlusion. It entails swapping retired pieces of the input image and analyzing really this alteration affects the awesome produced connected the output.

In the following, we will configure the occlusion attribution. Like the configuration of a convolutional neural network, you tin take the target region’s size and a stride length, which determines the spacing of individual measurements. We will usage the visualize_image_attr_multiple() usability to position the results of our Occlusion attribution. This usability will show power maps of some affirmative and antagonistic attribution per region and disguise the original image pinch the affirmative attribution regions. The masking provides a very illuminating look astatine the regions of our feline photograph that the exemplary identified arsenic astir “cat-like.”

occlusion = Occlusion(model) attributions_occ = occlusion.attribute(input, target=pred_label_idx, strides=(3, 8, 8), sliding_window_shapes=(3,15, 15), baselines=0) _ = viz.visualize_image_attr_multiple(np.transpose(attributions_occ.squeeze().cpu().detach().numpy(), (1,2,0)), np.transpose(transformed_img.squeeze().cpu().detach().numpy(), (1,2,0)), ["original_image", "heat_map", "heat_map", "masked_image"], ["all", "positive", "negative", "positive"], show_colorbar=True, titles=["Original", "Positive Attribution", "Negative Attribution", "Masked"], fig_size=(18, 6) )

Output:

The information of the image containing the feline seems to beryllium fixed a higher level of importance.

Conclusion

Captum is simply a exemplary interpretability room for PyTorch that is versatile and simple. It offers state-of-the-art techniques for knowing really circumstantial neurons and layers effect predictions. It has 3 main types of attribution techniques: Primary Attribution Techniques, Layer Attribution Techniques, and Neuron Attribution Techniques.

References

https://pytorch.org/tutorials/beginner/introyt/captumyt.html https://gilberttanner.com/blog/interpreting-pytorch-models-with-captum/ https://arxiv.org/pdf/1805.12233.pdf https://arxiv.org/pdf/1704.02685.pdf