My learnings from building Deep Learning models from scratch

By Utkarsh Malaiya

Building deep learning architectures from scratch isn’t just a cool project you build to brush up your resume, it is also a great way to test and extend your understanding of the underlying details of these architectures, improve your math and coding skills and this proves to be a very good tool to have when you’re going to be building more complex versions of these models. As it also turns out, this tests your debugging skills a lot more than you’d imagine.

This past week, I set out to build a simple Feedforward Neural Network and a Convolutional Neural Network from scratch, without using GPUs and tensors, i.e., only in NumPy, and I’d finished the FastAI Course Part 1 only 3 days before I decided on this, so suffice it to say, I didn’t have a vast amount of experience before I started this.

My goal behind this was to test my knowledge from that course and to test my programming and math skills. I’ll admit, at the time, I didn’t think this would be much difficult.

The result: I succeeded! I built a slow Feedforward Neural Network to work on the Titanic dataset (view here), which I’d say worked pretty well, and then built a base CNN on the MNIST dataset (view here) that also did averagely well. I got an accuracy on both models of around 70% on the test set after extensive training, so I can’t exactly call them good. More than the performance of the models, this was a phenomenal learning experience as I learnt a whole lot of things that both, FastAI part 1 and 3 courses of Deep Learning Specialization did not, about building deep learning models (They’re both great courses, believe me but these were things you can’t learn in a course/class). In the following portion of the article, I’ll tell you all those things, so if you’re thinking of building deep learning models from scratch, you won’t have to struggle with them.

1. Optimize for a good learning experience, not a fast and accurate model

Chances are, if you’re reading till here, you’re thinking of building deep learning models from scratch, or you’ve built one already. If the former, I would very much encourage you to not prioritize building a State-of-the-art model or something similar but rather to prioritize what you’re learning.

If you want to build a SOTA model, you can very easily use TensorFlow or PyTorch, and train a very large, deep Neural Network for 8–9 hours on a GPU can get a SOTA model. But when you’re trying to build these from scratch, chances are they are going to be very scrappy models, that can just barely get good accuracy after training for 1000’s of epochs.

This is, of course, assuming that you’re only using NumPy like I did, but still, you don’t want to spend a lot of computing power for a from-scratch model that you or anyone else may not use later on.

2. You can un-generalize the model if you want

Although I don’t recommend doing this, this isn’t a crime either.
If, like me, you’re building an MNIST-classifying CNN, then the MaxPooling layer you code or the Convolutional layer you code doesn’t have to generalize to images that have 3 colour channels. I admit, I didn’t create my Convolutional layer to use an RGB image, no, I just used it such that it would work for an image in the MNIST dataset because I was trying to make the model fit just for the MNIST dataset, I don’t plan on using it later.

If you want to build a general model, please do so. I encourage you to. Since you’re building these models to learn the nitty-gritty details of these models, you might as well go all the way. I didn’t, because even though I wanted to, it added a layer of complexity that I wasn’t prepared for, and I do plan on generalizing my model to 3 channel images, but for now, I decided not to do so. This isn’t to say you should always do this. You can do this in these kinds of models because they’re your personal projects, but not with models you’re building for your company or ones you plan on productionising.

However, for this project, even if you don’t build generalizable models, that’s okay. This is a learning opportunity, even if you learn 90% from this opportunity, that’s not a wasted opportunity. Maybe you’ll come back to it later.

3. If you can’t code it yourself, you probably don’t understand it well enough.

It’s very easy to go to a GitHub repository or an article someone wrote about something and copy that code without understanding what it’s doing, but you don’t want to do that in this scenario.
If you’re building from scratch models to understand what’s going on under the hood, that’s probably the worst thing you can do.

So if there’s a concept that you can’t code by yourself, there’s something in there to suggest that you don’t understand that concept well enough yourself. And you’re only admitting that to yourself, so there’s no shame in saying that. It took me over 3 days to code the Convolutional layer in my model, because that’s how long it took for me to understand that layer. It’s fine.

My suggestion is to use Feynman’s technique and write on a paper about the concept you don’t understand as if you were explaining it to a child. I did that with the Convolutional layer (see my explanation here and my supporting diagram here) and not only did it increase my understanding of the layer, but it also helped me find the gaps in my knowledge so I could very specifically improve those areas.

If you succeed in explaining the concept successfully, then your knowledge is thorough, and if you can’t, then you can watch videos about the things you’re unclear on, and improve your understanding.
This will allow you to easily code everything from scratch while maximizing your understanding of the concepts.

4. The math and the code are only 50% of the entire process, debugging is the rest

This is true for all of Machine Learning, so the faster you internalize this, the better.

When you are building these models, it’s easy enough to learn the math and write the code from pseudocode, but the real challenge I encountered was to debug my code. This is one of the reasons why I thought of building a Feedforward Neural Network despite having done it in the Deeplearning.ai specialization. In the assignments there, you don’t get the chance to debug your code because mostly, what you have to do is translate pseudocode in those assignments.

Debugging your code is an important skill as a programmer, but it becomes even more important as a Machine Learning developer. That is because in ML/AI, you don’t just have to debug the errors, you also have to make sure the dimensions match, the gradients are being updated properly, the loss is going down and the accuracy goes up. Most of this happens under the hood, so it becomes much more difficult to debug these things.

The biggest headache I had when building both my models was to make sure my loss went down, because it was either stagnant or it was increasing. Same with accuracy. If it’s increasing, that’s easy to debug. You’re putting a negative sign somewhere it’s not supposed to be, or you’re putting it where it is supposed to be. When it is stagnant, that’s a whole different headache.

For this reason alone, I can say that I spent half the time debugging my model, and only half the time actually writing the code for the layers and the pipeline.

My tip to make debugging easier is to print out everything, starting from the gradients to the loss to the weights of different layers to make sure they’re decreasing. Sometimes, it’s also a matter of your model’s complexity or the amount of data you are training your model with also causes bugs so be sure to check for that too.

The next two points also deal with specific examples that’ll help you catch bugs easily, ones I had to learn the harsh way.

5. If you’re stuck somewhere, start by working out the matrix shapes with a pen and paper

The only math you need in Deep Learning, as per what I’ve encountered, is Matrix Multiplication and the knowledge of derivatives.

If you know how to calculate derivatives, you’re golden. If you use a framework like TensorFlow or PyTorch, you don’t even need to calculate the gradients because the framework does that for you.
Even when you’re building deep learning models from scratch, you only need basic knowledge of derivatives for backpropagation, and you’re good.

Matrix multiplication, however, is the foundation that a deep learning model lives on. Your entire processes are carried out in n-dimensional matrices, everything. So whenever you face any problem, your loss is stagnant, or the accuracy is coming out to be NaN or anything else, I suggest first working out your matrices’ shapes across all your layers with a pen and paper. You can also use NumPy to tell you all the matrices’ shapes, but I find working them out on paper to much easier for me.

I cannot tell you how many times over the last week I faced an error that I wracked my head over for hours, all because the matrix dimensions didn’t match, and it was leading to some inconsistency in my model and felt embarrassed afterwards, because it was a silly mistake.
I found that, in my case, the only mistake you can make, relating to the math is to not check the matrix shapes’ consistency across layers. Don’t be like me, don’t wait 2–3 hours to work your matrix shapes.

In things like this, it’s better to be proactive. So I suggest, if you’re thinking of building these models, work out your matrix shapes as you decide on the input and output of each layer. Saves you a lot of hassle in the building phase. And one more thing,

6. Always double-check the variables. ALWAYS.

I made this mistake one too many times, more than I care to admit, but it’s possible to do that. When there are many lines of code on your 15.6-inch screen, and your variables are so descriptive and similar as mine were, it’s a possibility for this to happen.

I would’ve spent 3–6 fewer hours on both projects if I’d just double-checked if I was using the correct variables everywhere. I don’t think this is a mistake many of you will make, but I did, and it was very dumb of me to make this mistake (But I’m learning, so I don’t plan on staying this dumb).

To avoid this, don’t make the mistake of keeping your variables descriptive. Have a key whenever necessary for your variables, but keep the variables distinctive and short. Reduces the chances of you making the same mistakes I did.

7. Test everything out as you code it, not all at once.

This is a general tip for whenever you’re doing any kind of programming, but I had to learn the hard way just how important this is when you’re building out Neural Networks from scratch.

When you’re building these kinds of projects, you don’t have anything laid out in front of you. You don’t know what the outputs will look like, or the backpropagated weights will look like, or even the gradients. This makes the entire process daunting.

As soon as you start to code, you need to have a clear plan of how you’re going to be testing different parts of your models, how you’ll test the forward propagation for each layer, how you’ll test the backpropagation or the gradient updation functions of your model. And if you don’t test them as soon as they’re ready, if you don’t have a unit test ready for them, they will cause a whole heap of problems when you’re testing out the entire pipeline/model.

This is something I didn’t do (because I was an idiot not to do this), but I highly recommend you do it, but always have a parallel script that uses TensorFlow or PyTorch to test your functions/layers’ working.
If you’re building a Fully Connected (or Dense) layer and you’ve just written the forward propagation code for it, have a script that has a set seed for both, NumPy and the framework, with both of them having the same settings, and compare the results of both the models. They may not be identical, but you’ll know if your implementation is working correctly or not.

8. Don’t be afraid to ask for help. Just, give it your best before you do.

Again, a pretty general tip but it should be known.

I didn’t build these models all on my own. I was stuck many times when building these out, and I did ask for help on public forums, and private communities because there were times I couldn’t do it on my own no matter how much I tried. Sometimes, an external eye can spot errors in your code much more easily than you.
(Remember how I mentioned earlier that I would’ve spent 3–6 hours less on the project if I’d double-checked that I was using the right variables, well I didn’t catch that mistake by myself, someone told me on an online forum about this mistake because I was unable to find it even after a long time.)

When I didn’t understand the math, I watched YouTube videos and watched tutorials to see how to do something if I couldn’t do it myself first. That’s all to say, you should know when you’re lacking in some help and ask for others’ help rather than try and beat yourself up over not being do something.

However, all I ask if you’re planning to embark on this journey, or are already on it, is to first try everything yourself. You want to learn from this experience as much as you can, so make sure you’ve exhausted all your possibilities and your knowledge before you ask for help.