Global Pooling in Convolutional Neural Networks

Introduction

Pooling operations person been a champion successful convolutional neural networks for immoderate time. While processes for illustration max pooling and mean pooling person often taken much of the halfway stage, their little known cousins world max pooling and world mean pooling person go arsenic as important. In this article, we will beryllium exploring what the world variants of the 2 communal pooling techniques entail and really they comparison to 1 another.

import torch import torch.nn as nn import torch.nn.functional as F import torchvision import torchvision.transforms as transforms import torchvision.datasets as Datasets from torch.utils.data import Dataset, DataLoader import numpy as np import matplotlib.pyplot as plt import cv2 from tqdm.notebook import tqdm import seaborn as sns from torchvision.utils import make_grid if torch.cuda.is_available(): instrumentality = torch.device('cuda:0') print('Running connected the GPU') else: instrumentality = torch.device('cpu') print('Running connected the CPU')

Prerequisites

• Basic Understanding of CNNs: Familiarity pinch the architecture of CNNs, including layers for illustration convolutional, pooling, and afloat connected layers.
• Pooling Concepts: Knowledge of communal pooling techniques (e.g., max pooling, mean pooling) utilized to trim spatial dimensions successful CNNs.
• Linear Algebra & Tensor Operations: Understanding of matrix operations and tensor manipulations, arsenic world pooling involves reducing a multi-dimensional tensor to a little dimension.
• Activation Functions: Basic knowledge of really activation functions (e.g., ReLU, sigmoid) effect the features extracted by CNN layers.
• Framework Proficiency: Experience pinch heavy learning frameworks for illustration TensorFlow aliases PyTorch, peculiarly successful implementing civilization pooling layers.

The Classical Convolutional Neural Network

Many beginners successful machine imagination often get introduced to convolutional neural networks arsenic the perfect neural web for image information arsenic it retains the spatial building of the input image while learning/extracting features from them. By doing truthful it is capable to study relationships betwixt neighboring pixels and the position of objects successful the image thereby making it a very powerful neural network.

A multi furniture perceptron would besides activity successful an image classification context, but its capacity will beryllium severely degraded compared to its convnet counterpart simply because it instantly destroys the spatial building of the image by flattening/vectorizing it, thereby removing astir of the narration betwixt neighboring pixels.

Many classical convolutional neural networks are really a operation of convnets and MLPs. Looking astatine the architectures of LeNet and AlexNet for instance, 1 tin distinctively spot that their architectures are conscionable a mates of convolution layers pinch linear layers attached astatine the end.

This configuration makes a batch of sense, it allowed the convolution layers to do what they do champion which is extracting features successful information pinch 2 spatial dimensions. Afterwards the extracted features are passed onto linear layers truthful they besides tin do what they are awesome at, uncovering relationships betwixt characteristic vectors and targets.

A Flaw successful the Design

The problem pinch this creation is that linear layers person a very precocious propensity to overfit to data. Dropout regularization was introduced to thief mitigate this problem but a problem it remained nonetheless. Furthermore, for a neural web which prides itself connected not destroying spatial structures, the classical convnet still did it anyway, albeit deeper into the web and to a lesser degree.

Modern Solutions to a Classical Problem

In bid to forestall this overfitting rumor successful convnets, the logical adjacent measurement aft trying dropout regularization was to wholly get free of the linear layers each together. If the linear layers are to beryllium excluded, an wholly caller measurement of down-sampling characteristic maps and producing a vector practice of adjacent size to the number of classes successful mobility is to beryllium sought. This precisely is wherever world pooling comes in.

Consider a 4 people classification task, while 1 x 1 convolution layers will thief to down-sample characteristic maps until they are 4 successful number, world pooling will thief to create a 4 constituent agelong vector practice which tin past beryllium utilized by the nonaccomplishment usability successful calculating gradients.

Global Average Pooling

Still connected the aforesaid classification task described above, ideate a script wherever we consciousness our convolution layers are astatine an capable extent but we person 8 characteristic maps of size (3, 3). We tin utilize a 1 x 1 convolution furniture successful bid to down-sample the 8 characteristic maps to 4. Now we person 4 matrices of size (3, 3) erstwhile what we really request is simply a vector of 4 elements.

One measurement to deduce a 4 constituent vector from these characteristic maps is to compute the mean of each pixels successful each characteristic representation and return that arsenic a azygous element. This is fundamentally what world mean pooling entails.

Global Max Pooling

Just for illustration the script supra wherever we would for illustration to nutrient a 4 constituent vector from 4 matrices, successful this lawsuit alternatively of taking the mean worth of each pixels successful each characteristic map, we return the maximum worth and return that arsenic an individual constituent successful the vector practice of interest.

Benchmarking Global Pooling Methods

The benchmarking nonsubjective present is to comparison some world pooling techniques based connected their capacity erstwhile utilized to make classification vector representations. The dataset to beryllium utilized for benchmarking is the FashionMNIST dataset which contains 28 pixel by 28 pixel images of communal manner items.

training_set = Datasets.FashionMNIST(root='./', download=True, transform=transforms.ToTensor()) validation_set = Datasets.FashionMNIST(root='./', download=True, train=False, transform=transforms.ToTensor())

Label

Description

T-Shirt

1

Trouser

2

Pullover

3

Dress

4

Coat

5

Sandal

6

Shirt

7

Sneaker

8

Bag

9

Ankle boot

Convnet pinch Global Average Pooling

The convnet defined beneath makes usage of a 1 x 1 convolution furniture successful tandem pinch world mean pooling alternatively of linear layers successful producing a 10 constituent vector practice without regularization. Concerning the implementation of world mean pooling successful PyTorch, each that needs to beryllium done is to utilize the regular mean pooling people but usage a kernel/filter adjacent successful size to the size of each individual characteristic map. To illustrate, the characteristic maps coming retired of furniture 6 are of size (3, 3) truthful successful bid to execute world mean pooling, a kernel of size 3 is used. Note: simply taking the mean worth of each characteristic representation will output the aforesaid result.

class ConvNet_1(nn.Module): def __init__(self): super().__init__() self.network = nn.Sequential( nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.Conv2d(8, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.Conv2d(16, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(32, 10, 1), nn.AvgPool2d(3) ) def forward(self, x): x = x.view(-1, 1, 28, 28) output = self.network(x) output = output.view(-1, 10) return torch.sigmoid(output)

Convnet pinch Global Max Pooling

ConvNet_2 beneath connected the different manus replaces linear layers pinch a 1 x 1 convolution furniture moving successful tandem pinch world max pooling successful bid to nutrient a 10 constituent vector without regularization. Similar to world mean pooling, to instrumentality world max pooling successful PyTorch, 1 needs to usage the regular max pooling people pinch a kernel size adjacent to the size of the characteristic representation astatine that point. Note: simply deriving the maximum pixel worth successful each characteristic representation would output the aforesaid results.

class ConvNet_2(nn.Module): def __init__(self): super().__init__() self.network = nn.Sequential( nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.Conv2d(8, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.Conv2d(16, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2), nn.Conv2d(32, 10, 1), nn.MaxPool2d(3) ) def forward(self, x): x = x.view(-1, 1, 28, 28) output = self.network(x) output = output.view(-1, 10) return torch.sigmoid(output)

Convolutional Neural Network Class

The beneath defined people contains the training and classification functions to beryllium utilized for the training and utilizing convnets.

class ConvolutionalNeuralNet(): def __init__(self, network): self.network = network.to(device) self.optimizer = torch.optim.Adam(self.network.parameters(), lr=3e-4) def train(self, loss_function, epochs, batch_size, training_set, validation_set): log_dict = { 'training_loss_per_batch': [], 'validation_loss_per_batch': [], 'training_accuracy_per_epoch': [], 'validation_accuracy_per_epoch': [] } def init_weights(module): if isinstance(module, nn.Conv2d): torch.nn.init.xavier_uniform_(module.weight) module.bias.data.fill_(0.01) def accuracy(network, dataloader): total_correct = 0 total_instances = 0 for images, labels in tqdm(dataloader): images, labels = images.to(device), labels.to(device) predictions = torch.argmax(network(images), dim=1) correct_predictions = sum(predictions==labels).item() total_correct+=correct_predictions total_instances+=len(images) return round(total_correct/total_instances, 3) self.network.apply(init_weights) train_loader = DataLoader(training_set, batch_size) val_loader = DataLoader(validation_set, batch_size) for epoch in range(epochs): print(f'Epoch {epoch+1}/{epochs}') train_losses = [] print('training...') for images, labels in tqdm(train_loader): images, labels = images.to(device), labels.to(device) self.optimizer.zero_grad() predictions = self.network(images) nonaccomplishment = loss_function(predictions, labels) log_dict['training_loss_per_batch'].append(loss.item()) train_losses.append(loss.item()) loss.backward() self.optimizer.step() with torch.no_grad(): print('deriving training accuracy...') train_accuracy = accuracy(self.network, train_loader) log_dict['training_accuracy_per_epoch'].append(train_accuracy) print('validating...') val_losses = [] with torch.no_grad(): for images, labels in tqdm(val_loader): images, labels = images.to(device), labels.to(device) predictions = self.network(images) val_loss = loss_function(predictions, labels) log_dict['validation_loss_per_batch'].append(val_loss.item()) val_losses.append(val_loss.item()) print('deriving validation accuracy...') val_accuracy = accuracy(self.network, val_loader) log_dict['validation_accuracy_per_epoch'].append(val_accuracy) train_losses = np.array(train_losses).mean() val_losses = np.array(val_losses).mean() print(f'training_loss: {round(train_losses, 4)} training_accuracy: '+ f'{train_accuracy} validation_loss: {round(val_losses, 4)} '+ f'validation_accuracy: {val_accuracy}\n') return log_dict def predict(self, x): return self.network(x)

ConvNet_1 (Global Average Pooling)

ConvNet_1 uses world mean pooling successful producing a classification vector. Setting parameters of liking and training for 60 epochs produces a metric log arsenic analyzed below.

model_1 = ConvolutionalNeuralNet(ConvNet_1()) log_dict_1 = model_1.train(nn.CrossEntropyLoss(), epochs=60, batch_size=64, training_set=training_set, validation_set=validation_set)

From the log obtained, some training and validation accuracy accrued complete the people of exemplary training. Validation accuracy starts disconnected astatine astir 66% earlier expanding steadily to a worth conscionable nether 80% by the 28th epoch. A crisp summation to a worth nether 85% is past observed by the 31st epoch earlier yet culminating astatine astir 87% by the 60th epoch.

sns.lineplot(y=log_dict_1['training_accuracy_per_epoch'], x=range(len(log_dict_1['training_accuracy_per_epoch'])), label='training') sns.lineplot(y=log_dict_1['validation_accuracy_per_epoch'], x=range(len(log_dict_1['validation_accuracy_per_epoch'])), label='validation') plt.xlabel('epoch') plt.ylabel('accuracy')

ConvNet_2 (Global Max Pooling)

ConvNet_2 utilizes world max pooling alternatively of world mean pooling successful producing a 10 constituent classification vector. Keeping each parameters the aforesaid and training for 60 epochs yields the metric log below.

model_2 = ConvolutionalNeuralNet(ConvNet_2()) log_dict_2 = model_2.train(nn.CrossEntropyLoss(), epochs=60, batch_size=64, training_set=training_set, validation_set=validation_set)

Overall, some training and validation accuracy accrued complete the people of 60 epochs. Validation accuracy starts disconnected astatine conscionable nether 70% earlier fluctuating whilst expanding steadily to a worth conscionable nether 85% by the 60th epoch.

sns.lineplot(y=log_dict_2['training_accuracy_per_epoch'], x=range(len(log_dict_2['training_accuracy_per_epoch'])), label='training') sns.lineplot(y=log_dict_2['validation_accuracy_per_epoch'], x=range(len(log_dict_2['validation_accuracy_per_epoch'])), label='validation') plt.xlabel('epoch') plt.ylabel('accuracy') plt.savefig('maxpool_benchmark.png', dpi=1000)

Comparing Performance

Comparing the capacity of some world pooling techniques, 1 tin easy infer that world mean pooling performs better, astatine slightest connected the dataset we chose to usage (FashionMNIST). It seems to beryllium rather logical really since world mean pooling produces a azygous worth which is typical of the wide quality of each pixels successful each characteristic representation arsenic opposed to world max pooling which produces a azygous worth successful isolation without regards to different pixels coming successful the characteristic map. However, to scope a much conclusive verdict, benchmarking should beryllium done crossed respective datasets.

Global Pooling Under the Hood

In bid to create an intuition for why world pooling really works, we request to constitute a usability which will alteration america to visualize the output of an intermediate furniture successful a convolutional neural network. Many times neural networks are thought to beryllium achromatic container models, but location are definite ways to astatine slightest effort to pry unfastened the achromatic container successful a bid to understand what goes connected inside. The usability beneath does conscionable that.

def visualize_layer(model, dataset, image_idx: int, layer_idx: int): """ This usability visulizes intermediate layers successful a convolutional neural web defined utilizing the PyTorch sequential people """ dataloader = DataLoader(dataset, 250) for images, labels in dataloader: images, labels = images.to(device), labels.to(device) break output = model.network.network[:layer_idx].forward(images[image_idx]) out_shape = output.shape predicted_class = model.predict(images[image_idx]) print(f'actual class: {labels[image_idx]}\npredicted class: {torch.argmax(predicted_class)}') plt.figure(dpi=150) plt.title(f'visualising output') plt.imshow(np.transpose(make_grid(output.cpu().view(out_shape[0], 1, out_shape[1], out_shape[2]), padding=2, normalize=True), (1,2,0))) plt.axis('off')

In bid to usage the function, the parameters should beryllium decently understood. The exemplary refers to a convolution neural web instantiated the aforesaid measurement we person done successful this article, different types will not activity pinch this function. Dataset successful this lawsuit could beryllium immoderate dataset, but preferably the validation set. Image_idx is the scale of an image successful the first batch of the dataset provided, the usability defines a batch arsenic 250 images truthful image_idx could scope from 0 - 249. Layer_idx connected the different manus does not precisely mention to convolution layers, it refers to layers arsenic defined by the PyTorch sequential people arsenic seen below.

model_1.network >>>> ConvNet_1( (network): Sequential( (0): Conv2d(1, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(8, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (5): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (6): ReLU() (7): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (8): ReLU() (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (10): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (11): ReLU() (12): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (13): ReLU() (14): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (15): Conv2d(32, 10, kernel_size=(1, 1), stride=(1, 1)) (16): AvgPool2d(kernel_size=3, stride=3, padding=0) ) )

Why Global Average Pooling Works

In bid to understand why world mean pooling works, we request to visualize the output of the output furniture correct earlier world mean pooling is done, this corresponds to furniture 15 truthful we request to grab/index layers up till furniture 15 which implies that layer_idx=16. Using model_1 (ConvNet_1), we nutrient the results below.

visualize_layer(model=model_1, dataset=validation_set, image_idx=2, layer_idx=16) >>>> existent class: 1 >>>> predicted class: 1

When we visualize the output of image 3 (index 2) conscionable earlier world mean pooling, we tin spot that the exemplary has predicted it’s people correctly arsenic people 1 (trouser) arsenic seen above. Looking astatine the visualization, we tin spot that the characteristic representation astatine scale 1 has the brightest pixels connected mean erstwhile compared to the different characteristic maps. In different words, the convnet has learnt to categorize images by ‘switching on’ much pixels successful the characteristic representation of liking conscionable earlier world mean pooling. When world mean pooling is past done, the highest weighted constituent will beryllium located astatine scale 1 hence why it is chosen arsenic the correct class.

Global mean pooling output.

Why Global Max Pooling Works

Keeping each parameters the aforesaid but utilizing model_2 (ConvNet_2) successful this instance, we get the results below. Again, the convnet correctly classifies this image arsenic belonging to people 1. Looking astatine the visualization produced, we tin spot that the characteristic representation astatine scale 1 contains the brightest pixel.

The convnet has successful this lawsuit learnt to categorize images by ‘switching on’ pixels the brightest successful the characteristic representation of liking conscionable earlier world max pooling.

visualize_layer(model=model_2, dataset=validation_set, image_idx=2, layer_idx=16) >>>> existent class: 1 >>>> predicted class: 1

Global max pooling output.

In this article, we explored what world mean and max pooling entail. We discussed why they person travel to beryllium utilized and really they measurement up against 1 another. We besides developed an intuition into why they activity by performing a biopsy of our convnets and visualizing intermediate layers.