Interactive demonstration of Convolutional Neural Network processing MNIST digits
MNIST handwritten digit
Grayscale: 0-255 pixel values
Click 'Start CNN' to begin
Convolutional Neural Networks (CNNs) are a class of deep learning models primarily used for analyzing visual imagery. They are inspired by the organization of the animal visual cortex and are designed to automatically and adaptively learn spatial hierarchies of features from input data.
The core building block of a CNN is the convolutional layer. It performs a convolution operation on the input, passing a "filter" (or "kernel") over the input to produce a "feature map." This process helps in extracting features like edges, textures, and patterns.
For a 2D input image $I$ and a 2D kernel $K$, the convolution $(I * K)$ at position $(i, j)$ is given by:
$$(I * K)(i, j) = \sum_m \sum_n I(i-m, j-n) K(m, n)$$
Nomenclature:
$I$: Input image
$K$: Kernel/Filter
$(i, j)$: Output pixel coordinates
$(m, n)$: Kernel coordinates
A kernel is a small matrix that slides over the input data, performing element-wise multiplication with the input values it covers and summing the results. Each kernel is designed to detect a specific feature (e.g., horizontal edges, vertical edges).
Padding involves adding extra rows and columns of zeros (or other values) around the input image. This is done for two main reasons:
Stride defines the step size at which the kernel slides over the input image.
Output size calculation for a 2D convolution:
$$O_h = \lfloor \frac{I_h - K_h + 2P}{S} \rfloor + 1$$ $$O_w = \lfloor \frac{I_w - K_w + 2P}{S} \rfloor + 1$$
Nomenclature:
$O_h, O_w$: Output height and width
$I_h, I_w$: Input height and width
$K_h, K_w$: Kernel height and width
$P$: Padding
$S$: Stride
$\lfloor . \rfloor$: Floor function
After the convolution operation, an activation function is applied element-wise to the feature map. The Rectified Linear Unit (ReLU) is a popular choice:
$$ReLU(x) = \max(0, x)$$
ReLU introduces non-linearity into the model, allowing it to learn more complex patterns. It converts negative inputs to zero, effectively acting as a threshold.
Pooling layers are used to reduce the spatial dimensions (width and height) of the feature maps, thereby reducing the number of parameters and computational cost. Max pooling is the most common type: it takes the maximum value from a window (e.g., 2x2) of the feature map. This helps in making the model more robust to small translations and distortions in the input.
After several convolutional and pooling layers, the 2D feature maps are "flattened" into a single, long 1D vector. This transformation is necessary to connect the convolutional part of the CNN to the fully connected (dense) layers for classification.
These are traditional neural network layers where every input neuron is connected to every output neuron. In a CNN, dense layers typically come after the flattening layer and are responsible for performing the final classification based on the features extracted by the convolutional layers.
The final layer in a classification CNN is usually a dense layer with a Softmax activation function. Softmax converts the raw output scores into probabilities, where each probability corresponds to the likelihood of the input belonging to a particular class. The sum of all probabilities is 1.
For a vector $z$ of $K$ real numbers, the Softmax function is:
$$\sigma(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$$
Nomenclature:
$z_i$: Input score for class $i$
$K$: Number of classes
$e$: Euler's number (base of natural logarithm)