Attentions
This notebook explains how to get various attention images with Saliency, SmoothGrad, GradCAM, GradCAM++ and ScoreCAM/Faster-ScoreCAM.
Preparation
Load libraries
[2]:
%reload_ext autoreload
%autoreload 2
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
import tensorflow as tf
from tf_keras_vis.utils import num_of_gpus
_, gpus = num_of_gpus()
print('Tensorflow recognized {} GPUs'.format(gpus))
Tensorflow recognized 1 GPUs
Load tf.keras.Model
In this notebook, we use VGG16 model, but if you want to use other tf.keras.Model, you can do so by modifying the section below.
[3]:
from tensorflow.keras.applications.vgg16 import VGG16 as Model
model = Model(weights='imagenet', include_top=True)
model.summary()
Model: "vgg16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
_________________________________________________________________
flatten (Flatten) (None, 25088) 0
_________________________________________________________________
fc1 (Dense) (None, 4096) 102764544
_________________________________________________________________
fc2 (Dense) (None, 4096) 16781312
_________________________________________________________________
predictions (Dense) (None, 1000) 4097000
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________
Load and preprocess images
tf-keras-vis supports batch-wise visualization. Here, we load and preprocess three pictures of goldfish, bear and assault-rifle as input data.
[4]:
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.applications.vgg16 import preprocess_input
# Image titles
image_titles = ['Goldfish', 'Bear', 'Assault rifle']
# Load images and Convert them to a Numpy array
img1 = load_img('images/goldfish.jpg', target_size=(224, 224))
img2 = load_img('images/bear.jpg', target_size=(224, 224))
img3 = load_img('images/soldiers.jpg', target_size=(224, 224))
images = np.asarray([np.array(img1), np.array(img2), np.array(img3)])
# Preparing input data for VGG16
X = preprocess_input(images)
# Rendering
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
ax[i].set_title(title, fontsize=16)
ax[i].imshow(images[i])
ax[i].axis('off')
plt.tight_layout()
plt.show()
Implement functions required to use attentions
Model modifier
When the softmax activation function is applied to the last layer of model, it may obstruct generating the attention images, so you should replace the function to a linear activation function. Although we create and use ReplaceToLinear instance here, we can also use the model modifier function defined by ourselves.
[5]:
from tf_keras_vis.utils.model_modifiers import ReplaceToLinear
replace2linear = ReplaceToLinear()
# Instead of using the ReplaceToLinear instance above,
# you can also define the function from scratch as follows:
def model_modifier_function(cloned_model):
cloned_model.layers[-1].activation = tf.keras.activations.linear
Score function
And then, you MUST create Score instance or define score function that returns target scores. Here, they return the score values corresponding Goldfish, Bear, Assault Rifle.
[6]:
from tf_keras_vis.utils.scores import CategoricalScore
# 1 is the imagenet index corresponding to Goldfish, 294 to Bear and 413 to Assault Rifle.
score = CategoricalScore([1, 294, 413])
# Instead of using CategoricalScore object,
# you can also define the function from scratch as follows:
def score_function(output):
# The `output` variable refers to the output of the model,
# so, in this case, `output` shape is `(3, 1000)` i.e., (samples, classes).
return (output[0][1], output[1][294], output[2][413])
Vanilla Saliency
Saliency generates a saliency map that appears the regions of the input image that contributes the most to the output value.
[7]:
%%time
from tensorflow.keras import backend as K
from tf_keras_vis.saliency import Saliency
# from tf_keras_vis.utils import normalize
# Create Saliency object.
saliency = Saliency(model,
model_modifier=replace2linear,
clone=True)
# Generate saliency map
saliency_map = saliency(score, X)
## Since v0.6.0, calling `normalize()` is NOT necessary.
# saliency_map = normalize(saliency_map)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
ax[i].set_title(title, fontsize=16)
ax[i].imshow(saliency_map[i], cmap='jet')
ax[i].axis('off')
plt.tight_layout()
plt.show()
CPU times: user 2.46 s, sys: 670 ms, total: 3.13 s
Wall time: 3.07 s
SmoothGrad
As you can see above, Vanilla Saliency map is too noisy, so let’s remove noise in the saliency map using SmoothGrad! SmoothGrad is a method that reduce the noise in saliency map by adding noise to input image.
Note: Because SmoothGrad calculates the gradient repeatedly, it might take much time around 2-3 minutes when using CPU.
[8]:
%%time
# Generate saliency map with smoothing that reduce noise by adding noise
saliency_map = saliency(score,
X,
smooth_samples=20, # The number of calculating gradients iterations.
smooth_noise=0.20) # noise spread level.
## Since v0.6.0, calling `normalize()` is NOT necessary.
# saliency_map = normalize(saliency_map)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
ax[i].set_title(title, fontsize=14)
ax[i].imshow(saliency_map[i], cmap='jet')
ax[i].axis('off')
plt.tight_layout()
plt.savefig('images/smoothgrad.png')
plt.show()
CPU times: user 1.33 s, sys: 122 ms, total: 1.45 s
Wall time: 1.43 s
GradCAM
Saliency is one of useful way of visualizing attention that appears the regions of the input image that contributes the most to the output value. GradCAM is another way of visualizing attention over input. Instead of using gradients of model outputs, it uses of penultimate layer output (that is the convolutional layer just before Dense layers).
[9]:
%%time
from matplotlib import cm
from tf_keras_vis.gradcam import Gradcam
# Create Gradcam object
gradcam = Gradcam(model,
model_modifier=replace2linear,
clone=True)
# Generate heatmap with GradCAM
cam = gradcam(score,
X,
penultimate_layer=-1)
## Since v0.6.0, calling `normalize()` is NOT necessary.
# cam = normalize(cam)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
heatmap = np.uint8(cm.jet(cam[i])[..., :3] * 255)
ax[i].set_title(title, fontsize=16)
ax[i].imshow(images[i])
ax[i].imshow(heatmap, cmap='jet', alpha=0.5) # overlay
ax[i].axis('off')
plt.tight_layout()
plt.show()
CPU times: user 951 ms, sys: 338 ms, total: 1.29 s
Wall time: 1.17 s
As you can see above, GradCAM is useful method for intuitively knowing where the attention is. However, when you take a look closely, you’ll see that the visualized attentions don’t completely cover the target (especially the head of Bear) in the picture.
Okay then, let’s move on to next method that is able to fix the problem above you looked.
GradCAM++
GradCAM++ can provide better visual explanations of CNN model predictions.
[10]:
%%time
from tf_keras_vis.gradcam_plus_plus import GradcamPlusPlus
# Create GradCAM++ object
gradcam = GradcamPlusPlus(model,
model_modifier=replace2linear,
clone=True)
# Generate heatmap with GradCAM++
cam = gradcam(score,
X,
penultimate_layer=-1)
## Since v0.6.0, calling `normalize()` is NOT necessary.
# cam = normalize(cam)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
heatmap = np.uint8(cm.jet(cam[i])[..., :3] * 255)
ax[i].set_title(title, fontsize=16)
ax[i].imshow(images[i])
ax[i].imshow(heatmap, cmap='jet', alpha=0.5)
ax[i].axis('off')
plt.tight_layout()
plt.savefig('images/gradcam_plus_plus.png')
plt.show()
CPU times: user 1.2 s, sys: 154 ms, total: 1.36 s
Wall time: 1.24 s
As you can see above, Now, the visualized attentions almost completely cover the target objects!
ScoreCAM
In the end, Here, we show you ScoreCAM. It is an another method that generates Class Activation Map. The characteristic of this method is that it’s the gradient-free method unlike GradCAM, GradCAM++ or Saliency.
[11]:
%%time
from tf_keras_vis.scorecam import Scorecam
from tf_keras_vis.utils import num_of_gpus
# Create ScoreCAM object
scorecam = Scorecam(model)
# Generate heatmap with ScoreCAM
cam = scorecam(score, X, penultimate_layer=-1)
## Since v0.6.0, calling `normalize()` is NOT necessary.
# cam = normalize(cam)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
heatmap = np.uint8(cm.jet(cam[i])[..., :3] * 255)
ax[i].set_title(title, fontsize=16)
ax[i].imshow(images[i])
ax[i].imshow(heatmap, cmap='jet', alpha=0.5)
ax[i].axis('off')
plt.tight_layout()
plt.show()
CPU times: user 19 s, sys: 699 ms, total: 19.6 s
Wall time: 19.1 s
As you can see above, The visualized images are more sophisticated!
Faster-ScoreCAM
ScoreCAM is a great method, however, it takes a bit more time to process it than other cam methods. There is a good news for us. Faster-ScorecAM that makes ScoreCAM to be more efficient was devised by @tabayashi0117.
https://github.com/tabayashi0117/Score-CAM/blob/master/README.md#faster-score-cam
We thought that several channels were dominant in generating the final heat map. Faster-Score-CAM adds the processing of “use only channels with large variances as mask images” to Score-CAM. (max_N = -1 is the original Score-CAM).
[12]:
%%time
from tf_keras_vis.scorecam import Scorecam
# Create ScoreCAM object
scorecam = Scorecam(model, model_modifier=replace2linear)
# Generate heatmap with Faster-ScoreCAM
cam = scorecam(score,
X,
penultimate_layer=-1,
max_N=10)
## Since v0.6.0, calling `normalize()` is NOT necessary.
# cam = normalize(cam)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, title in enumerate(image_titles):
heatmap = np.uint8(cm.jet(cam[i])[..., :3] * 255)
ax[i].set_title(title, fontsize=16)
ax[i].imshow(images[i])
ax[i].imshow(heatmap, cmap='jet', alpha=0.5)
ax[i].axis('off')
plt.tight_layout()
plt.show()
CPU times: user 3.58 s, sys: 191 ms, total: 3.77 s
Wall time: 3.64 s