Visualizing Conv filters using ActivationMaximization
Preparation
Load libraries
[2]:
%reload_ext autoreload
%autoreload 2
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
import tensorflow as tf
from tf_keras_vis.utils import num_of_gpus
_, gpus = num_of_gpus()
print('Tensorflow recognized {} GPUs'.format(gpus))
Tensorflow recognized 1 GPUs
Load tf.keras.Model
In this notebook, we use VGG16 model, however if you want to use other tf.keras.Model, you can do so by modifying the section below.
[3]:
from tensorflow.keras.applications.vgg16 import VGG16 as Model
# Load model
model = Model(weights='imagenet', include_top=True)
model.summary()
Model: "vgg16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
_________________________________________________________________
flatten (Flatten) (None, 25088) 0
_________________________________________________________________
fc1 (Dense) (None, 4096) 102764544
_________________________________________________________________
fc2 (Dense) (None, 4096) 16781312
_________________________________________________________________
predictions (Dense) (None, 1000) 4097000
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________
Implement functions required to use ActivationMaximization
Model modifier
You MUST define a model modifier to replace the model output to target layer’s output that has filters you want to visualize, and , in addion, replace the function of it to a linear activation function.
Although we create and use ExtractIntermediateLayer and ReplaceToLinear instance, we can also use the model modifier function defined by ourselves.
[4]:
from tf_keras_vis.utils.model_modifiers import ExtractIntermediateLayer, ReplaceToLinear
layer_name = 'block5_conv3' # The target layer that is the last layer of VGG16.
# This instance constructs new model whose output is replaced to `block5_conv3` layer's output.
extract_intermediate_layer = ExtractIntermediateLayer(index_or_name=layer_name)
# This instance modify the model's last activation function to linear one.
replace2linear = ReplaceToLinear()
# Instead of using ExtractIntermediateLayer and ReplaceToLinear instance,
# you can also define the function from scratch as follows:
def model_modifier_function(current_model):
target_layer = current_model.get_layer(name=layer_name)
target_layer.activation = tf.keras.activations.linear
new_model = tf.keras.Model(inputs=current_model.inputs,
outputs=target_layer.output)
return new_model
Score function
And then, you MUST create Score instance or define score function that returns target scores. Here, they return the value of 3rd filter in block5_conv3 layer.
[6]:
from tf_keras_vis.utils.scores import CategoricalScore
filter_number = 3
score = CategoricalScore(filter_number)
# Instead of using CategoricalScore object above,
# you can also define the function from scratch as follows:
def score_function(output):
return output[..., filter_number]
Visualizeing a conv filter
Create ActivationMaximization Instnace
When clone argument is True(default), the model will be cloned, so the model instance will be NOT modified, however the process may take a while.
Here, because the ExtractIntermediateLayer instance will construct new model, setting False.
[5]:
from tf_keras_vis.activation_maximization import ActivationMaximization
activation_maximization = ActivationMaximization(model,
# Please note that `extract_intermediate_layer` has to come before `replace2linear`.
model_modifier=[extract_intermediate_layer, replace2linear],
clone=False)
Visualize
ActivationMaximization will maximize the value computed by the score function. Here, we will visualize the 63rd convolutional filter.
[7]:
%%time
from tf_keras_vis.activation_maximization.callbacks import Progress
# Generate maximized activation
activations = activation_maximization(score,
callbacks=[Progress()])
## Since v0.6.0, calling `astype()` is NOT necessary.
# activations = activations[0].astype(np.uint8)
# Render
f, ax = plt.subplots(figsize=(4, 4))
ax.imshow(activations[0])
ax.set_title('filter[{:03d}]'.format(filter_number), fontsize=16)
ax.axis('off')
plt.tight_layout()
plt.show()
200/200 [==============================] - 8s 35ms/step - Score: 861.2347 - TotalVariation2D: 23.2731 - Norm: 39.9557
CPU times: user 8.09 s, sys: 422 ms, total: 8.52 s
Wall time: 8.49 s
Visualizing Conv filters
Then, let’s visualize multiple convolutional filters!
Modify Score function
Because change the target you want to visualize, you MUST create Score instance or define score function again. Here, our socre function returns the values corresponding to 63rd, 132nd and 320th filters of the layer.
[8]:
from tf_keras_vis.utils.scores import CategoricalScore
filter_numbers = [63, 132, 320]
scores = CategoricalScore(filter_numbers)
Create Seed-Input values
And then, you MUST create seed_input value. In default, when visualizing a specific filter, tf-keras-vis automatically generates seed_input to visualize a image. When visualizing multiple images, you MUST manually create seed_input.
[9]:
# Define seed inputs whose shape is (samples, height, width, channels).
seed_input = tf.random.uniform((3, 224, 224, 3), 0, 255)
Visualize
[10]:
%%time
from tf_keras_vis.activation_maximization.callbacks import Progress
# Generate maximized activation
activations = activation_maximization(scores,
seed_input=seed_input,
callbacks=[Progress()])
## Since v0.6.0, calling `astype()` is NOT necessary.
# activations = activations[0].astype(np.uint8)
# Render
f, ax = plt.subplots(nrows=1, ncols=3, figsize=(12, 4))
for i, filter_number in enumerate(filter_numbers):
ax[i].set_title('filter[{:03d}]'.format(filter_number), fontsize=16)
ax[i].imshow(activations[i])
ax[i].axis('off')
plt.tight_layout()
plt.show()
200/200 [==============================] - 14s 69ms/step - Score: 933.5200 - TotalVariation2D: 24.0820 - Norm: 39.8219
CPU times: user 14.2 s, sys: 325 ms, total: 14.5 s
Wall time: 14.7 s