Writing LeNet5 from Scratch in PyTorch

Introduction

In this article, we will beryllium building 1 of nan earliest convolutional neural networks ever introduced, LeNet5 (paper). We are building this CNN from scratch successful PyTorch, and will besides spot really it performs connected a real-world dataset.

We will commencement by exploring nan architecture of LeNet5. We will past load and analyse our dataset, MNIST, utilizing nan provided people from torchvision. Using PyTorch, we will build our LeNet5 from scratch and train it connected our data. Finally, we will spot really nan exemplary performs connected nan unseen trial data.

Prerequisites

Knowledge of neural networks will beryllium adjuvant successful knowing this article. This translates to being acquainted pinch nan different layers of neural networks (input layer, hidden layers, output layer), activation functions, optimization algorithms (variants of gradient descent), nonaccomplishment functions, etc. Additionally, familiarity pinch Python syntax and nan PyTorch room is basal for knowing nan codification snippets presented successful this article.

LeNet5

LeNet5 is 1 of nan earliest convolutional neural networks (CNNs). LeNet5 was projected by Yann LeCun and others successful 1998. LeCun had antecedently been progressive pinch publishing nan first study training CNNs pinch backpropagation successful 1989. You tin publication nan original LeNet5 insubstantial here: Gradient-Based Learning Applied to Document Recognition. In nan paper, nan LeNet5 was utilized for nan nickname of handwritten characters.

Let’s now understand nan architecture of LeNet5 arsenic shown successful nan fig below:

As nan sanction indicates, LeNet5 has 5 layers pinch 2 convolutional and 3 afloat connected layers. Let’s commencement pinch nan input. LeNet5 accepts arsenic input a greyscale image of 32x32, indicating that nan architecture is not suitable for RGB images (multiple channels). So nan input image should incorporate conscionable 1 channel. After this, we commencement pinch our convolutional layers

The first convolutional furniture has a select size of 5x5 pinch 6 specified filters. This will trim nan width and tallness of nan image while expanding nan extent (number of channels). The output would beryllium 28x28x6. After this, pooling is applied to alteration nan characteristic representation by half, i.e, 14x14x6. Same select size (5x5) pinch 16 filters is now applied to nan output followed by a pooling layer. This reduces nan output characteristic representation to 5x5x16.

After this, a convolutional furniture of size 5x5 pinch 120 filters is applied to flatten nan characteristic representation to 120 values. Then comes nan first afloat connected layer, pinch 84 neurons. Finally, we person nan output furniture which has 10 output neurons, since nan MNIST information person 10 classes for each of nan represented 10 numerical digits.

---

Data Loading

Let’s commencement by loading and analyzing nan data. We will beryllium utilizing nan MNIST dataset. The MNIST dataset contains images of handwritten numerical digits. The images are greyscale, each pinch a size of 28x28, and is composed of 60,000 training images and 10,000 testing images.

You tin spot immoderate of nan samples of images below:

Importing nan Libraries

Let’s commencement by importing nan required libraries and defining immoderate variables (hyperparameters and instrumentality are besides elaborate to thief nan package find whether to train connected GPU aliases CPU):

# Load successful applicable libraries, and othername wherever appropriate import torch import torch.nn arsenic nn import torchvision import torchvision.transforms arsenic transforms # Define applicable variables for nan ML task batch_size = 64 num_classes = 10 learning_rate = 0.001 num_epochs = 10 # Device will find whether to tally nan training connected GPU aliases CPU. device = torch.device('cuda' if torch.cuda.is_available() other 'cpu')

Loading and Transforming nan Data

Using torchvision , we will load nan dataset arsenic this will let america to execute immoderate pre-processing steps easily.

#Loading nan dataset and preprocessing train_dataset = torchvision.datasets.MNIST(root = './data', train = True, toggle shape = transforms.Compose([ transforms.Resize((32,32)), transforms.ToTensor(), transforms.Normalize(mean = (0.1307,), std = (0.3081,))]), download = True) test_dataset = torchvision.datasets.MNIST(root = './data', train = False, toggle shape = transforms.Compose([ transforms.Resize((32,32)), transforms.ToTensor(), transforms.Normalize(mean = (0.1325,), std = (0.3105,))]), download=True) train_loader = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle = True) test_loader = torch.utils.data.DataLoader(dataset = test_dataset, batch_size = batch_size, shuffle = True)

Let’s understand nan code:

• Firstly, nan MNIST information can’t beryllium utilized arsenic it is for nan LeNet5 architecture. The LeNet5 architecture accepts nan input to beryllium 32x32 and nan MNIST images are 28x28. We tin hole this by resizing nan images, normalizing them utilizing nan pre-calculated mean and modular deviation (available online), and yet storing them arsenic tensors.
• We group download=True incase nan information is not already downloaded.
• Next, we make usage of information loaders. This mightiness not impact nan capacity successful nan lawsuit of a mini dataset for illustration MNIST, but it tin really impede nan capacity successful lawsuit of ample datasets and is mostly considered a bully practice. Data loaders let america to iterate done nan information successful batches, and nan information is loaded while iterating and not astatine erstwhile successful start.
• We specify nan batch size and shuffle nan dataset erstwhile loading truthful that each batch has immoderate variance successful nan types of labels it has. This will summation nan efficacy of our eventual model.

---

LeNet5 from Scratch

Let’s first look into nan code:

#Defining nan convolutional neural network class LeNet5(nn.Module): def __init__(self, num_classes): super(ConvNeuralNet, self).__init__() self.layer1 = nn.Sequential( nn.Conv2d(1, 6, kernel_size=5, stride=1, padding=0), nn.BatchNorm2d(6), nn.ReLU(), nn.MaxPool2d(kernel_size = 2, stride = 2)) self.layer2 = nn.Sequential( nn.Conv2d(6, 16, kernel_size=5, stride=1, padding=0), nn.BatchNorm2d(16), nn.ReLU(), nn.MaxPool2d(kernel_size = 2, stride = 2)) self.fc = nn.Linear(400, 120) self.relu = nn.ReLU() self.fc1 = nn.Linear(120, 84) self.relu1 = nn.ReLU() self.fc2 = nn.Linear(84, num_classes) def forward(self, x): retired = self.layer1(x) retired = self.layer2(out) retired = out.reshape(out.size(0), -1) retired = self.fc(out) retired = self.relu(out) retired = self.fc1(out) retired = self.relu1(out) retired = self.fc2(out) return out

Defining nan LeNet5 Model

I’ll explicate nan codification linearly:

• In PyTorch,  we specify a neural web by creating a people that inherits from nn.Module arsenic it contains galore of nan methods that we will request to utilize.
• There are 2 main steps aft that. First is initializing nan layers that we are going to usage successful our CNN wrong __init__ , and nan different is to specify nan series successful which those layers will process nan image. This is defined wrong nan guardant function.
• For nan architecture itself, we first specify nan convolutional layers utilizing nan nn.Conv2D usability pinch nan due kernel size and nan input/output channels. We besides use max pooling utilizing nn.MaxPool2D function. The bully point astir PyTorch is that we tin harvester nan convolutional layer, activation function, and max pooling into 1 azygous furniture (they will beryllium separately applied, but it helps pinch organization) utilizing nan nn.Sequential function.
• Then we specify nan afloat connected layers. Note that we tin usage nn.Sequential present arsenic good and harvester nan activation functions and nan linear layers, but I wanted to show that either 1 is possible.
• Finally, our past furniture outputs 10 neurons which are our last predictions for nan digits.

---

Setting Hyperparameters

Before training, we request to group immoderate hyperparameters, specified arsenic nan nonaccomplishment usability and nan optimizer to beryllium used.

model = LeNet5(num_classes).to(device) #Setting nan nonaccomplishment function cost = nn.CrossEntropyLoss() #Setting nan optimizer pinch nan exemplary parameters and learning rate optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate) #this is defined to people really galore steps are remaining erstwhile training total_step = len(train_loader)

We commencement by initializing our exemplary utilizing nan number of classes arsenic an argument, which successful this lawsuit is 10. Then we specify our costs usability arsenic transverse entropy nonaccomplishment and optimizer arsenic Adam. There are a batch of choices for these, but these thin to springiness bully results pinch nan exemplary and nan fixed data. Finally, we specify total_step to support amended way of steps erstwhile training.

---

Model Training

Now, we tin train our model:

total_step = len(train_loader) for epoch successful range(num_epochs): for i, (images, labels) successful enumerate(train_loader): images = images.to(device) labels = labels.to(device) #Forward pass outputs = model(images) nonaccomplishment = cost(outputs, labels) #Backward and optimize optimizer.zero_grad() loss.backward() optimizer.step() if (i+1) % 400 == 0: people ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'

.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

Let’s spot what nan codification does:

• We commencement by iterating done nan number of epochs, and past nan batches successful our training data.
• We person nan images and nan labels according to nan instrumentality we are using, i.e., GPU aliases CPU.
• In nan guardant pass, we make predictions utilizing our exemplary and cipher nonaccomplishment based connected those predictions and our existent labels.
• Next, we do nan backward walk wherever we really update our weights to amended our model
• We past group nan gradients to zero earlier each update utilizing optimizer.zero_grad() function.
• Then, we cipher nan caller gradients utilizing nan loss.backward() function.
• And finally, we update nan weights pinch nan optimizer.step() function.

We tin spot nan output arsenic follows:

As we tin see, nan nonaccomplishment is decreasing pinch each epoch which shows that our exemplary is so learning. Note that this nonaccomplishment is connected nan training set, and if nan nonaccomplishment is measurement excessively mini (as is successful our case), it tin bespeak overfitting. There are aggregate ways to lick that problem specified arsenic regularization, information augmentation, and truthful connected but we won’t beryllium getting into that successful this article. Let’s now trial our exemplary to spot really it performs.

---

Model Testing

Let’s now trial our model:

# Test nan model # In trial phase, we don't request to compute gradients (for representation efficiency) with torch.no_grad(): correct = 0 total = 0 for images, labels successful test_loader: images = images.to(device) labels = labels.to(device) outputs = model(images) _, predicted = torch.max(outputs.data, 1) full += labels.size(0) correct += (predicted == labels).sum().item() print('Accuracy of nan web connected nan 10000 trial images: {} %'.format(100 * correct / total))

As you tin see, nan codification is not truthful different than nan 1 for training. The only quality is that we are not computing gradients (using pinch torch.no_grad()), and besides not computing nan nonaccomplishment because we don’t request to backpropagate here. To compute nan resulting accuracy of nan model, we tin simply cipher nan full number of correct predictions complete nan full number of images.

Using this model, we get astir 98.8% accuracy which is rather good:

Testing Accuracy

Note that MNIST dataset is rather basal and mini for today’s standards, and akin results are difficult to get for different datasets. Nonetheless, it’s a bully starting constituent erstwhile learning heavy learning and CNNs.

---

Conclusion

Let’s now reason what we did successful this article:

• We started by learning nan architecture of LeNet5 and nan different kinds of layers successful that.
• Next, we explored nan MNIST dataset and loaded nan information utilizing torchvision.
• Then, we built LeNet5 from scratch on pinch defining hyperparameters for nan model.
• Finally, we trained and tested our exemplary connected nan MNIST dataset, and nan exemplary seemed to execute good connected nan trial dataset.

---

Future Work

Although this seems a really bully preamble to heavy learning successful PyTorch, you tin widen this activity to study much arsenic well:

• You tin effort utilizing different datasets but for this exemplary you will request grey standard datasets. One specified dataset is FashionMNIST.
• You tin research pinch different hyperparameters and spot nan champion operation of them for nan model.
• Finally, you tin effort adding aliases removing layers from nan dataset to spot their effect connected nan capacity of nan model.