Welcome!

This site contains lecture notes, tutorials, and assignment instructions for COMP 4630: Machine Learning. Use the sidebar on the left to view the material, and if you notice a problem, please report an issue at the GitHub repo.

Sometimes I’ll write stuff on the board. You can find a version of my scribbles here, which I’ll try to keep more or less up to date.

Lecture 1: Data Exploration

HTML Slides | PDF Slides

Welcome to Machine Learning!

COMP 4630 | Winter 2026 Charlotte Curtis

What is this course about?

• Continuing the supervised/unsupervised learning algorithms from COMP 3652, with a focus on Neural Networks
• First half: the history, theory, and math behind neural networks
• Second half: applications of NNs in computer vision, natural language processing, and more

This is not (just) a course on building models using libraries like TensorFlow or PyTorch, it is a course on understanding the theory

How did I get involved with ML?

What do you want to learn about ML?

Grade Assessment

Course materials repo: https://github.com/mru-comp4630/w26 Rendered at: https://mru-comp4630.github.io/w26/

Textbooks and other readings

Primary Textbook:

• Hands on Machine Learning with Scikit-Learn and [Tensorflow/PyTorch]
• Associated GitHub repo (Tensorflow)
• Associated GitHub repo (PyTorch)

More mathy details:

• Deep Learning

Journal club list: on D2L under “Course Info” (requires MRU library login)

Generative AI policy

• Yes, AI can do a lot of what I’m asking for in this course
• No, I do not want to read about what AI “thinks”
• ❓ What do you think is an appropriate use?

Machine Learning Project Checklist

1. Look at the big picture

Example Dataset: California housing prices (1990)

❓ Discussion questions:

• How does the company expect to use and benefit from this model?
• What is the current solution?
• What kind of ML task is this?
• What kind of performance measure should we use?

Where we left off on Wednesday, January 7

First, some stuff about assessments

• Assignment 1
• Journal club guidelines
• Example of a math-heavy paper
• Additional references for papers:
  • Google Scholar
  • ArXiv

2. Get the data

For this class, we’ll use readily available datasets. Some sources are:

• UCI Machine Learning Repository
• Kaggle
• Google Dataset Search
• Various Government open data portals (e.g. Calgary, Alberta, Canada)

After fetching the data, set aside a test set and don’t look at it.

2a. Set aside a test set

❓ Discussion questions:

• Why do we need an independent test set?
  • Avoid data snooping bias
  • Relevant XKCD
• Why would we use a random seed?
• What is naive about simply selecting a random sample?
• What else could we do?
• What is stratified sampling?

Side tangent: Sampling bias

• Simple example: assume 80% of population likes cilantro
• Goal: ensure our sample is representative of the population, $\pm 5\%$

The binomial distribution can be used to model the probability of choosing $k$ people who like cilantro from $n$ total participants:

$$P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k},\mathrm{where}(\genfrac{}{}{0ex}{}{n}{k})=\frac{n!}{k!(n-k)!}$$

Side tangent: Sampling bias continued

$P(X=k)$ is the probability mass function, and the corresponding cumulative distribution function is just the sum up to $k$:

$$P(X\le k)=\sum _{i=0}^k\left(\genfrac{}{}{0ex}{}{n}{i}\right)p^i(1-p{)}^{n-i}$$

Suppose we randomly sample 100 people. What is the probability of fewer than 75 or more than 85 cilantro lovers?

This is also my excuse to review some probability theory and notation

3. Explore the data

❓ Discussion questions:

• What do you notice about the data?
• Do the values make sense for the labels?
• Is the scale of the features comparable? Does this matter?
• What possible biases might be present in the data?

3a. Look for correlations

The Pearson correlation coefficient is a measure of the linear correlation between two variables $X$ and $Y$ (commonly denoted as $r$):

$$r=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}\sqrt{{\sum }_{i=1}^n(y_i-\bar{y}{)}^2}}$$

where $\bar{x}$ and $\bar{y}$ are the sample means of $X$ and $Y$, respectively.

• What do correlations of 0, 1, and -1 mean?
• What are some limitations of Pearson correlation?

Where we left off on Monday, January 12

4. Prepare the data

General goals:

• Handle missing data, and maybe outliers
• Drop irrelevant features
• Combine features using domain knowledge
• Apply various transformations (e.g. scaling, encoding)
• Apply scaling when necessary

4a. Handling missing data

In the book 3 options are listed to handle the NaN values:

housing.dropna(subset=["total_bedrooms"], inplace=True) ## option 1
housing.drop("total_bedrooms", axis=1)                  ## option 2
median = housing["total_bedrooms"].median()             ## option 3
housing["total_bedrooms"].fillna(median, inplace=True)

❓ Discussion questions:

• What is each option doing?
• What are the pros and cons of each option?
• Which one should we choose?

4b. Handling non-numeric data

Most of the math in ML algorithms is based on numbers, so we need to convert text and categorical attributes to numbers. This is called encoding.

❓ Discussion questions:

• Which columns of our data are categorical?
• What methods could we use to convert them to numbers?
• What are the assumptions about the various encoding methods?

4c. Scaling the data

Many ML algorithms don’t like features with vastly different scales. Common scaling methods are min-max scaling and standardization.

Important: scaling is computed on the training set and applied to the validation and test sets - they are not scaled independently!

❓ Discussions questions:

• What are the bounds of each method?
• Which method is more affected by outliers?
• How would you decide which method to use?

4e. Standardization details

A general Gaussian distribution is given by:

$$f(x)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}$$

where $\mu$ is the mean and $\sigma$ is the standard deviation. The standard normal distribution is a special case where $\mu =0$ and $\sigma =1$, reducing the equation to:

$$f(x)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{x}^2}$$

4f. Other transformations

• Log transformation: useful for data that is heavily skewed
• Also square root, squaring, etc.: try to remove heavy tails
• Feature engineering: combining features to create new ones
• Binning: turning continuous data into discrete categories
  • Possibly using K-means clustering
  • Relies on domain knowledge
• Best to create a transformation pipeline and apply it to the data rather than saving the transformed data

Coming up next

• Math review:
  • Linear algebra
  • Differential calculus
  • Statistics
• A brief introduction to vector calculus

Lecture 2: Math Review

HTML Slides | PDF Slides

Math review

COMP 4630 | Winter 2026 Charlotte Curtis

Math review

• MATH 1203: Linear algebra
• MATH 1200: Differential calculus
• MATH 2234: Statistics

Further reading:

• Linear algebra: notebook, deep learning book
• Calculus: notebook

Linear algebra

Vector operations

• Addition: ${\mathbf{v}}_1+{\mathbf{v}}_2=\left[\begin{array}{c}{v}_{11}+v_{21}\\ v_{12}+v_{22}\end{array}\right]$
• Scalar multiplication: $c\mathbf{v}=\left[\begin{array}{c}c{v}_1\\ c{v}_2\end{array}\right]$
• Dot product: ${\mathbf{v}}_1\cdot {\mathbf{v}}_2=v_{11}{v}_{21}+v_{12}{v}_{22}$ (yields a scalar)
  • Can be thought of as the projection of one vector onto another, or how much two vectors are aligned in the same direction

Vector norms

• The norm of a vector is a measure of its length
• Most common is the Euclidean norm (or $L^2$ norm): $$\Vert \mathbf{v}{\Vert }_2=\Vert \mathbf{v}\Vert =\sqrt{\left(\sum _{i=1}^n{v}_i^2\right)}$$
• You might also see the $L^1$ norm, particularly as a regularization term: $$\Vert \mathbf{v}{\Vert }_1=\sum _{i=1}^n|v_i|$$

Useful vectors

• Unit vector: A vector with a norm of 1, e.g. $\mathbf{x}=\left[\begin{array}{c}1\\ 0\end{array}\right]$, $\mathbf{y}=\left[\begin{array}{c}0\\ 1\end{array}\right]$
• Normalized vector: A vector divided by its norm, e.g. $\mathbf{v}=\hat{\mathbf{v}}=\frac{\mathbf{v}}{\Vert \mathbf{v}\Vert }$
• Dot product can also be written as ${\mathbf{v}}_1\cdot {\mathbf{v}}_2=\Vert {\mathbf{v}}_1\Vert \Vert {\mathbf{v}}_2\Vert \cos (\theta )$

Matrices

A matrix is a 2D array of numbers:

$$A=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right]$$

Notation: Element $a_{ij}$ is in row $i$, column $j$, also written as $A_{ij}$.

Rows then columns! $M\times N$ matrix has $M$ rows and $N$ columns

Matrix operations

• Addition: element-wise if dimensions match. $A+B=B+A$
• Scalar multiplication: just like vectors
• Matrix multiplication: $C=AB$ where the elements of $C$ are: $$c_{ij}=\sum _{k=1}^n{a}_{ik}{b}_{kj}$$
  • Multiply and sum rows of $A$ with columns of $B$
  • Usually, $AB\ne BA$

Matrix multiplication examples

Matrix times a matrix: $$A=\left[\begin{array}{cc}2& 0\\ 1& 3\\ -4& 5\end{array}\right],B=\left[\begin{array}{ccc}-1& 0& 1\\ 1& 3& 7\end{array}\right]$$

Matrix times a vector: $$A=\left[\begin{array}{cc}0& -1\\ 1& 0\end{array}\right],\mathbf{v}=\left[\begin{array}{c}1\\ 0\end{array}\right]$$

Where we left off on January 14

Matrix transpose

• Transpose: $A^T$ swaps rows and columns $$A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right],A^T=\left[\begin{array}{cc}1& 3\\ 2& 4\end{array}\right]$$

• Inverse: just as $\frac{1}{x}\cdot x=1$, $A^{-1}A=I$, where $I$ is the identity matrix $$A=\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right],A^{-1}=\left[\begin{array}{cc}-2& 1\\ 1.5& -0.5\end{array}\right],A^{-1}A=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$$

Calculus: Notation

The derivative of a function $y=f(x)$ is represented as:

$$f^{\prime }(x)=\frac{dy}{dx}=\underset{h\to 0}{\lim }\frac{f(x+h)-f(x)}{h}$$

The second derivative is denoted:

$$f^{\prime \prime }(x)=\frac{d^{2}y}{d{x}^2}=\frac{d}{dx}\left(\frac{dy}{dx}\right)$$

and so on.

Differentiability

For a function to be differentiable at a point $x_A$, it must be:

• Defined at $x_A$
• Continuous at $x_A$
• Smooth at $x_A$
• Non-vertical at $x_A$

Select rules of differentiation

Chain rule example

1. Find $\frac{df}{dx}$ for $f(x)=\sigma (x)=\frac{1}{1+e^{-x}}$

2. Now, let, $y=\sigma (x_1)$, where $x_1=wx$. What is $\frac{dy}{dx}$?

Partial derivatives

For a scalar valued function $y=f(x_1,x_2)$, there are two partial derivatives:

$$\frac{\partial y}{\partial x_1},\frac{\partial y}{\partial x_2}$$

These are computed by holding the “other” variable(s) constant. For example, if $y=2x_1+x_2+x_1{x}_2$, then:

$$\frac{\partial y}{\partial x_1}=2+x_2,\frac{\partial y}{\partial x_2}=1+x_1$$

A brief introduction to vector calculus

Putting together partial derivatives with vectors and matrices we get:

Most of the time we’ll just be working with the gradient

Statistics: Notation

• A random variable $\mathrm{x}\sim P$ is a variable that can take on random variables according to some probability distribution $P$
• $\mathrm{x}$ may take on discrete (e.g. dice rolls) or continuous (e.g. age) values
• $X$ or $\mathrm{x}$ for the random variable and $x$ or $x_i$ for a specific value
• $P(\mathrm{x})$ for a a discrete distribution and $p(\mathrm{x})$ for continuous
• ${\mathrm{x}}_P\equiv \mathrm{x}\sim P$ and ${\mathrm{x}}_p\equiv \mathrm{x}\sim p$

Discrete random variables

• A discrete probability mass function describes the probability of $\mathrm{x}$ taking on a specific value
• Example: for a balanced 6-sided die, $P(\mathrm{x}=1)=\frac{1}{6}$
• You can add together probabilities, e.g. $P(\mathrm{x}\le 3)=\sum _{i=1}^{3}P(\mathrm{x}=i)$
• $\sum _{x}P(\mathrm{x})=1$ and $P(x_i)\ge 0$ for any valid distribution

Continuous random variables

• A continuous probability density function gives the probability of being in some tiny interval $\delta x$ given by $p(x)\delta x$
• Example: the uniform distribution, $p(\mathrm{x})=\frac{1}{b-a}$ for $a\le x\le b$
• $p(\mathrm{x}=x_i)=0$ for any specific value $x_i$
• Need to integrate to get a concrete value, e.g. $p(\mathrm{x}\le a)={\int }_{-\infty }^{a}p(x)dx$
• ${\int }_{-\infty }^{\infty }p(x)dx=1$ and ${\int }_a^{b}p(x)dx\ge 0$ for any valid distribution

Expectation and variance

• The expectation or expected value is its average value $\mathbb{E}[\mathrm{x}]$
• $\mathbb{E}[{\mathrm{x}}_P]=\sum _{x}xP(\mathrm{x})$ and $\mathbb{E}[{\mathrm{x}}_p]={\int }_{-\infty }^{\infty }xp(x)dx$
• More generally, for any function $f(\mathrm{x})$: $$\mathbb{E}[f(\mathrm{x})]=\sum _{x}f(x)P(\mathrm{x})\mathrm{and}{\int }_{-\infty }^{\infty }f(x)p(x)dx$$
• The variance describes how much the values vary from their mean: $$\mathrm{Var}[\mathrm{x}]=\mathbb{E}[(\mathrm{x}-\mathbb{E}[\mathrm{x}]{)}^2]$$

Multiple random variables

• Joint probability $P(\mathrm{x},\mathrm{y})$ is the probability of $\mathrm{x}$ and $\mathrm{y}$ occurring together
• Conditional probability $P(\mathrm{x}=x\mid \mathrm{y}=y)$ is the probability that $\mathrm{x}$ takes on value $x$ given that $\mathrm{y}=y$ has already happened
• In general, $P(\mathrm{x}=x\mid \mathrm{y}=y)=\frac{P(\mathrm{x}=x,\mathrm{y}=y)}{P(\mathrm{y}=y)}$
• For independent variables, $P(\mathrm{x}=x\mid \mathrm{y}=y)=P(\mathrm{x}=x)$

Covariance

• The covariance between $f(\mathrm{x})$ and $g(\mathrm{y})$ gives a sense of how linearly related they are and how much they vary together: $$\mathrm{Cov}(f(\mathrm{x}),g(\mathrm{y}))=\mathbb{E}[(f(\mathrm{x})-\mathbb{E}[f(\mathrm{x})])(g(\mathrm{y})-\mathbb{E}[g(\mathrm{y})])]$$
• Related to correlation as $\mathrm{Corr}(f(\mathrm{x}),g(\mathrm{y}))=\frac{\mathrm{Cov}(f(\mathrm{x}),g(\mathrm{y}))}{\sqrt{\mathrm{Var}(f(\mathrm{x}))\mathrm{Var}(g(\mathrm{y}))}}$
• The covariance matrix of a random vector $\mathbf{x}$ is a square matrix where the $(i,j)$ element is the covariance between $x_i$ and $x_j$
• The diagonal of the covariance matrix gives $\mathrm{Var}(x_i)$

The Normal distribution

$$N(x;u,{\sigma }^2)=\sqrt{\frac{1}{2\pi {\sigma }^2}}{\exp }^{\left(-\frac{1}{2{\sigma }^2}(x-\mu {)}^2\right)}$$

Good “default choice” for two reasons:

• The central limit theorem shows that the sum of many ($>30$ ish) independent random variables is normally distributed
• Has the most uncertainty of any distribution with the same variance

Coming up next

• Training (regression) models
  • Linear regression
  • Gradient descent
• References and suggested reading:
  • Scikit-learn book:
    • Chapter 4: Training Models
  • Deep Learning Book
    • Section 5.1.4: Linear Regression

Lecture 3: Training models

HTML Slides | PDF Slides

Training Models with Regression and Gradient Descent

COMP 4630 | Winter 2026 Charlotte Curtis

Overview

• Linear Regression and the Normal Equation
• Gradient Descent and its various flavours
• References and suggested reading:
  • Scikit-learn book:
    • Chapter 4: Training Models
  • Deep Learning Book
    • Section 5.1.4: Linear Regression

Linear Regression

Unlike most models, linear regression has a closed-form solution called the Normal Equation:

$$\hat{\theta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$$

where

• $\hat{\theta }$ are the weights of the model minimizing the cost function
• $\mathbf{y}$ is the vector of target values
• $\mathbf{X}$ is the design matrix of feature values

Consider the 1-d case:

$$\hat{y}={\theta }_0+{\theta }_{1}x$$

we want the values of ${\theta }_0$ and ${\theta }_1$ that minimize the Mean Square Error between the actual and predicted $y$ values:

$$\begin{array}{rl}MSE& =\frac{1}{m}\sum _{i=1}^m(\hat{y}-y_i{)}^2\\ MSE& =\frac{1}{m}\sum _{i=1}^m({\theta }_0+{\theta }_1{x}_i-y_i{)}^2\end{array}$$

Solving for ${\theta }_0$ and ${\theta }_1$

Math time!

Solving for ${\theta }_0$ and ${\theta }_1$

After some algebraic gymnastics, we get:

$$\begin{array}{rl}{\theta }_1& =\frac{{\mu }_y{\sum }_m{x}_i-{\sum }_m{x}_i{y}_i}{{\mu }_x{\sum }_m{x}_i-{\sum }_m{x}_i^2}\\ {\theta }_0& ={\mu }_y-{\theta }_1{\mu }_x\end{array}$$

where ${\mu }_x$ and ${\mu }_y$ are the means of the $x$ and $y$ values, respectively.

Expanding to matrix form

Instead of the scalar $x$ or even vector $\mathbf{x}$, it’s common to use a design matrix $\mathbf{X}$ to represent the feature values:

$$\mathbf{X}=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1n}\\ x_{21}& x_{22}& \cdots & x_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ x_{m1}& x_{m2}& \cdots & x_{mn}\end{array}\right]$$

where each row is an instance (sample) and each column is a feature.

The first column is all ones, representing the bias term

Back to the linear regression problem…

• We can rewrite the estimate in matrix notation: $$\hat{\mathbf{y}}=\mathbf{X}\theta$$

• The MSE can be written as:

  $$MSE=\frac{1}{m}\sum _{i=1}^m({\hat{y}}_i-y_i{)}^2=\frac{1}{m}(\mathbf{X}\theta -\mathbf{y}{)}^T(\mathbf{X}\theta -\mathbf{y})$$

  where we’ve used the trick of substituting ${\mathbf{a}}^T\mathbf{a}={\sum }_i{a}_i^2$

:abacus: Find the gradient of the MSE w.r.t $\theta$, set it to zero, and solve for $\theta$

Properties of matrices and their transpose

The following properties are useful for solving linear algebra problems:

• ${(\mathbf{AB})}^T={\mathbf{B}}^T{\mathbf{A}}^T$

• ${(\mathbf{A}+\mathbf{B})}^T={\mathbf{A}}^T+{\mathbf{B}}^T$

• ${({\mathbf{A}}^{-1})}^T=({\mathbf{A}}^T{)}^{-1}$

• ${({\mathbf{A}}^{\mathbf{T}})}^T=\mathbf{A}$

Additionally, any matrix or vector multiplied by $\mathbf{I}$ is unchanged.

The Normal Equation

We made it! The Normal Equation is again:

$$\hat{\theta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$$

• No optimization is required to find the optimal $\theta$
• Limitations:
  • ${\mathbf{X}}^T\mathbf{X}$ must be invertible and small enough to fit in memory
  • The computational complexity is (at least) $O(n^3)$
• Even in linear regression problems, it is common to use gradient descent instead due to these limitations

Gradient Descent

The goal of gradient descent is still to minimize the cost function, but it follows an iterative process:

1. Start with a random $\theta$

2. Calculate the gradient ${\nabla }_{\theta }$ for the current $\theta$

3. Update $\theta$ as $\theta =\theta -\eta {\nabla }_{\theta }$

4. Repeat 2-3 until some stopping criterion is met

   where $\eta$ is the learning rate, or the size of step to take in the direction opposite the gradient.

Stochastic Gradient Descent

• Standard or batch gradient descent uses the entire training set to calculate the gradient for each instance at every step
• Stochastic Gradient Descent uses a single random instance at each step:
  1. Start with a random $\theta$
  2. Pick a random instance ${\mathbf{x}}_i$ (row in the design matrix)
  3. Calculate the gradient ${\nabla }_{\theta }$ for the current $\theta$ and ${\mathbf{x}}_i$
  4. Update $\theta$ as $\theta =\theta -\eta {\nabla }_{\theta }$
  5. Repeat 2-4 until some stopping criterion is met

Mini-batch Gradient Descent

• Mini-batch gradient descent uses a random subset of the training set
• Less chaotic than stochastic, but faster than batch
• Most common type of gradient descent used in practice

Gradient Descent Hyperparameters

• The learning rate $\eta$ - size of step taken
• No rule that it needs to be constant! A simple learning schedule is to decrease $\eta$ over time, e.g.: $$\eta =\frac{t_0}{t+t_1}$$ where $t$ is the current iteration and $t_0$ and $t_1$ are hyper-parameters
• For mini-batch, the batch size is another hyper-parameter
• The number of epochs, or times to process the entire training set

Stopping Criteria

• The simplest stopping criterion is to set a maximum number of epochs
• Early stopping is another option:
  • Evaluate on a validation set at regular intervals
  • Stop when the validation error starts to increase
• The comparison between training and validation performance can also help prevent overfitting

Loss functions

• The loss function is the function being minimized by gradient descent

• MSE is convex and guaranteed to have a single global minimum, but many other loss functions have multiple local minima

• The relative scale of the features can affect the convergence:

  

Higher-order Polynomials

• Higher order polynomials can be solved with the Normal Equation as well: $$y={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+\cdots +{\theta }_n{x}^n$$
• Just include the higher order terms in $\mathbf{X}$
• This is still a linear regression problem because the coefficients are linear!
• Risk of overfitting the data
• Easy way to regularize: drop one or more of the higher order terms

Regularization

• If the model fits the training data too well, but doesn’t generalize to new data, it is overfitting
• Regularization imposes additional constraints on the weights
• Example: Ridge Regression adds a term to the loss function: $$J(\theta )=MSE(\theta )+\alpha \frac{1}{2}\sum _{i=1}^n{\theta }_i^2$$ where $\alpha$ is the regularization parameter
• The regularization term is only added during training, not evaluation

Logistic regression and beyond

Logistic regression is a binary classifier that uses the logistic function (aka sigmoid function) to map the output to a range of 0 to 1:

$$\sigma (t)=\frac{1}{1+e^{-t}}$$

We can then minimize the log loss or cross-entropy loss function:

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y_i\log ({\hat{p}}_i)+(1-y_i)\log (1-{\hat{p}}_i)\right]$$

where ${\hat{p}}_i=\sigma ({\theta }^T{\mathbf{x}}_i)$ is the probability that instance $i$ is positive.

The gradient of the log loss ends up being:

$${\nabla }_{\theta }J(\theta )=\frac{1}{m}\sum _{i=1}^m\left(\sigma ({\theta }^T{\mathbf{x}}_i)-y_i\right){\mathbf{x}}_i$$

• There is no (known) analytical solution this time, but we can still use gradient descent!
• In this case it’s still convex, so we don’t have to worry about local minima
• In general, for a loss function to work with gradient descent, it must be:
  • Continuous and
  • Differentiable
  • … at the locations where you evaluate it

Next up: Backpropagation!

Lecture 4: Backpropagation

HTML Slides | PDF Slides

Backpropagation

COMP 4630 | Winter 2026 Charlotte Curtis

Overview

• A brief review of the history of neural networks
• Neurons, perceptrons, and multilayer perceptrons
• Backpropagation
• References and suggested reading:
  • Scikit-learn book: Chapter 10, introduction to artificial neural networks
  • Deep Learning Book: Chapter 6, deep feedforward networks

The rise and fall of neural networks

In between each era of excitement and advancement there was an “AI winter”

Model of a neuron

Threshold Linear Units (TLUs)

Training a perceptron

Multilayer perceptrons (MLPs)

• If a perceptron can’t even solve XOR, how can it do higher order logic?

• Consider that XOR can be rewritten as:

  A xor B = (A and !B) or (!A and B)
  

• A perceptron can solve and and or and not… so what if the input to the or perceptron is the output of two and perceptrons?

A solution to XOR

Backpropagation

• I just gave you the weights to solve XOR, but how do we actually find them?
• Applying the perceptron learning rule no longer works, need to know how much to adjust each weight relative to the overall output error
• Solution presented in 1986 by Rumelhart, Hinton, and Williams
• Key insight: Good old chain rule! Plus some recursive efficiencies

Training MLPs with backpropagation

1. Initialize the weights, through some random-ish strategy

2. Perform a forward pass to compute the output of each neuron

3. Compute the loss of the output layer (e.g. MSE)

4. Calculate the gradient of the loss with respect to each weight

5. Update the weights using gradient descent (minibatch, stochastic, etc)

6. Repeat steps 2-5 until stopping criteria met

   Step 4 is the “backpropagation” part

Example: forward pass

Example: calculate error and gradient

• We never picked a loss function! Let’s assume we’re using MSE
• For a single sample: $$\mathcal{L}({\mathbf{w}}^{(1)},{\mathbf{w}}^{(2)})=\frac{1}{2}(\hat{y}-y{)}^2=\frac{1}{2}{\left(\sum _{j=1}^2{w}_j^{(2)}\sum _{i=1}^2{x}_i{w}_{ij}^{(1)}-y\right)}^2$$ with the $1/2$ added for convenience
• The goal is to update each weight by a small amount to minimize the loss
• Fortunately, we know how to find a small change in a function with respect to one of the variables: the partial derivative!

Recursively applying the chain rule

• Weights in the second layer (connecting hidden and output): $$\frac{\partial \mathcal{L}}{\partial w_j^{(2)}}=\frac{\partial \mathcal{L}}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w_j^{(2)}}=(\hat{y}-y)\frac{\partial \hat{y}}{\partial w_j^{(2)}}=(\hat{y}-y)\sum _i{x}_i{w}_{ij}^{(1)}$$

• For the first layer (connecting inputs to hidden): $$\frac{\partial \mathcal{L}}{\partial w_{ij}^{(1)}}=\frac{\partial \mathcal{L}}{\partial \hat{y}}\frac{\partial \hat{y}}{h_j}\frac{\partial h_j}{\partial w_{ij}^{(1)}}=(\hat{y}-y)w_j^{(2)}{x}_i$$ where $h_j=x_i{w}_{ij}^{(1)}$ is the output of the hidden layer

Bias terms

• The toy example did not include bias terms, but these are very important (as seen in the perceptron examples)
• With a single layer we can add a column of 1s to $\mathbf{X}$, but with multiple layers we need to add bias at every layer
• The forward pass becomes: $$\hat{\mathbf{y}}=(\mathbf{X}{\mathbf{W}}^{(1)}+{\mathbf{b}}^{(1)}){\mathbf{W}}^{(2)}+{\mathbf{b}}^{(2)}$$
• Or in summation form: $$\hat{y}=\sum _{j=1}^2{w}_j^{(2)}\left(\sum _{i=1}^2{x}_i{w}_{ij}^{(1)}+b^{(1)}\right)+b^{(2)}$$

Gradient with respect to the bias terms

• For layer 2 (the output layer):

$$\frac{\partial \mathcal{L}}{\partial b_j^{(2)}}=\frac{\partial \mathcal{L}}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial b^{(2)}}=(\hat{y}-y)(1)$$

• For layer 1: $$\frac{\partial \mathcal{L}}{\partial b^{(1)}}=\frac{\partial \mathcal{L}}{\partial \hat{y}}\frac{\partial \hat{y}}{h}\frac{\partial h}{\partial b^{(1)}}=(\hat{y}-y)\sum _i{w}_{ij}^{(2)}$$

  where $h={\sum }_i{x}_i{w}_{ij}^{(1)}+b^{(1)}$ is the input to the hidden layer

Summary in matrix form

Computational considerations

• Many of the terms computed in the forward pass are reused in the backward pass (such as the inputs to each layer)

• Similarly, gradients computed in layer $l+1$ are reused in layer $l$

• Typically each intermediate value is stored, but modern networks are big

  Model	Parameters
  Our example	6
  AlexNet (2012)	60 million
  GPT-3 (2020)	175 billion

Choices in neural network design

Activation functions

• The simple example used a linear activation function (identity)

• To include other activation functions, the forward pass becomes: $$\hat{\mathbf{y}}={\mathbf{f}}_2({\mathbf{f}}_1(\mathbf{X}{\mathbf{W}}^{(1)}+{\mathbf{b}}^{(1)}){\mathbf{W}}^{(2)}+{\mathbf{b}}^{(2)})$$

• The gradient in the output layer becomes: $$\frac{\partial \mathcal{L}}{\partial {\mathbf{W}}^{(2)}}=\frac{\partial \mathcal{L}}{\partial {\mathbf{f}}_2}\frac{\partial {\mathbf{f}}_2}{\partial {\mathbf{z}}^{(2)}}\frac{\partial {\mathbf{z}}^{(2)}}{\partial {\mathbf{W}}^{(2)}}$$ where ${\mathbf{z}}^{(2)}={\mathbf{f}}_1(\mathbf{X}{\mathbf{W}}^{(1)}+{\mathbf{b}}^{(1)}){\mathbf{W}}^{(2)}+{\mathbf{b}}^{(2)}$, or the summation the second layer before applying the activation function

• Problem! That step function in the original perceptron is not differentiable

Activation functions

• A common early choice was the sigmoid function: $$\sigma (z)=\frac{1}{1+e^{-z}},\frac{d\sigma }{dz}=\sigma (z)(1-\sigma (z))$$
• A more computationally efficient choice common today is the “ReLU” (Rectified Linear Unit) function: $$\text{ReLU}(z)=\max (0,z),\frac{d\text{ReLU}}{dz}=\{\begin{array}{ll}0& z<0\\ 1& z>0\end{array}$$

Activation functions in hidden layers

The design of hidden units is an extremely active area of research and does not yet have many definitive guiding theoretical principles. – Deep Learning Book, Section 6.3

• Activation functions in hidden layers serve to introduce nonlinearity
• Common for multiple hidden layers to use the same activation function
• Sigmoid, ReLU, and tanh (hyperbolic tangent) are common choices
• Also “leaky” ReLU, Parameterized ReLU, absolute value, etc
• Can be considered a hyperparameter of the network

Loss functions

• The choice of loss function is very important!
• Depends on the task at hand, e.g.:
  • Regression: MSE, MAE, etc
  • Classification: Usually some kind of cross-entropy (log likelihood)
• May or may not include regularization terms
• Must be differentiable, just like the activation functions

Activation functions in the output layer

• Activation functions in the output layer should be chosen based on the loss function (and thus the task)
  • Regression: linear
  • Binary classification: sigmoid
  • Multiclass classification: softmax (generalization of sigmoid)
• Again, must be differentiable

A complete fully connected network

Next up: Classification loss functions and metrics

Lecture 5: Classification

HTML Slides | PDF Slides

Classification loss functions and metrics

COMP 4630 | Winter 2025 Charlotte Curtis

Overview

• All the derivation thus far has been for mean squared error
• Cross-entropy loss is more appropriate for classification problems
• References and suggested reading:
  • Scikit-learn book: Chapter 4, training models
  • Scikit-learn docs: Log loss
  • Deep Learning Book: Sections 3.1, 3.8, and 6.2

Revisiting the expected value

The expected value of some function $f(x)$ when $x$ is distributed as $P(x)$ is given in discrete form as:

$$\mathbb{E}[f(x)]=\sum _{x}P(x)f(x)$$

where the sum is over all possible values of $x$.

In continuous form, this is an integral:

$$\mathbb{E}[f(x)]=\int p(x)f(x)dx$$

Binary case: Bernoulli distribution

• If a random variable $x$ has a $p$ probability of being 1 and a $1-p$ probability of being 0, then $x$ is distributed as a Bernoulli distribution: $$P(x)=p^x(1-p{)}^{1-x}=\{\begin{array}{ll}p& \text{for }x=1\\ 1-p& \text{for }x=0\end{array}$$
• The expected value of $x$ is then: $$\mathbb{E}[x]=\sum _{x}P(x)x=0\cdot (1-p)+1\cdot p=p$$

Information theory

Originally developed for message communication, with the intuition that less likely events carry more information, defined for a single event as:

$$I(x)=-\log P(x)$$

Entropy

• We can measure the expected information of a distribution $P(x)$ as: $$H(X)=\mathbb{E}[I(x)]=-{\mathbb{E}}_{x\sim P}[\log P(x)]$$
• This is called the Shannon entropy
• Measured in bits (base 2) or nats (base $e$)
• :abacus: Find the entropy of a bernoulli distribution

Cross-entropy

• The KL divergence is a measure of the extra information needed to encode a message from a true distribution $P(x)$ using an approximate distribution $Q(x)$: $$D_{KL}(P||Q)={\mathbb{E}}_{x\sim P}\left[\log \frac{P(x)}{Q(x)}\right]={\mathbb{E}}_{x\sim P}[\log P(x)-\log Q(x)]$$
• The cross-entropy is a simplification that drops the term $\log P(x)$: $$H(P,Q)=-{\mathbb{E}}_{x\sim P}[\log Q(x)]$$
• Minimizing the cross-entropy is equivalent to minimizing the KL divergence
• If $P(x)=Q(x)$, then $D_{KL}(P||Q)=0$ and $H(P,Q)=H(P)$

Cross-entropy loss

For a true label $y\in \{0,1\}$ and predicted $\hat{p}\in [0,1]$, the cross-entropy loss is:

$$\begin{array}{rl}\mathcal{L}(y,\hat{p})=& -{\mathbb{E}}_y[\log P(x)]\\ =& -y\log \hat{p}-(1-y)\log (1-\hat{p})\end{array}$$

where $\hat{p}=\sigma ({\mathbf{w}}^T\mathbf{h}+b)$ is the output of the final layer of a neural network (thresholded to obtain the prediction $\hat{y}$)

This is also called log loss or binary cross-entropy

Terminology for evaluation

• True positive: predicted positive, label was positive ($TP$) ✔️

• True negative: predicted negative, label was negative ($TN$) ✔️

• False positive: predicted positive, label was negative ($FP$) ❌ (type I)

• False negative: predicted negative, label was positive ($FN$) ❌ (type II)

• Accuracy is the fraction of correct predictions, given as:

  $$\mathrm{accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$

Precision and recall

• Precision: Out of all the positive predictions, how many were correct? $$\mathrm{precision}=\frac{TP}{TP+FP}$$

• Recall: Out of all the positive labels, how many were correct? $$\mathrm{recall}=\frac{TP}{TP+FN}$$

• Specificity: Out of all the negative labels, how many were correct? $$\mathrm{specificity}=\frac{TN}{TN+FP}$$

Confusion matrix

• The axes might be reversed, but a good predictor will have strong diagonals
• There’s also the F1 score, or harmonic mean of precision and recall: $$F1=2\cdot \frac{\mathrm{precision}\cdot \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}$$

ROC Curves

• The receiver operating characteristic curve is a plot of the true positive rate (recall or sensitivity) vs. false positive rate (1 - specificity) as the detection threshold changes

• The diagonal is the same as random guessing

• A perfect classifier would hug the top left corner

Which classifier is better?

Multiclass case

• For $K$ classes, the output is a vector $\hat{\mathbf{p}}$ with ${\hat{p}}_i=P(y=i|\mathbf{x})$
• The cross-entropy loss is then: $$\mathcal{L}(\mathbf{y},\hat{\mathbf{p}})=-\sum _{i=1}^K{y}_i\log {\hat{p}}_i$$
• For a one-hot encoded vector $\mathbf{y}$, this simplifies to: $$\mathcal{L}(\mathbf{y},\hat{\mathbf{p}})=-\log {\hat{p}}_k$$ where $k$ is the index of the true class

The softmax function

• For binary classification, the sigmoid function $\sigma (z)=\frac{1}{1+e^{-z}}$ is used to predict the probability of the positive class

• For multiclass classification, the softmax function is used:

  $${\hat{p}}_i=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$$

  where $z_i={\mathbf{w}}_i^T\mathbf{h}+b_i$ is the output of neuron $i$ in the final layer before the activation function is applied

• This means that $K$ neurons are needed in the final layer, one for each class

Next up: Convolution and NN frameworks

Lecture 6: Modern Neural Networks

HTML Slides | PDF Slides

Intro to modern neural networks

COMP 4630 | Winter 2024 Charlotte Curtis

Overview

• More decisions when making a neural network
  • Weight initialization
  • Number of neurons and layers
  • Optimization algorithms
• References and suggested reading:
  • Scikit-learn book: Chapters 10-11
  • Deep Learning Book: Chapter 8
  • Understanding Deep Learning: Chapter 7

Revisiting Backpropagation

• For a network with $l$ layers, the gradients of the loss function with respect to the weights in the last layer are given by: $$\frac{\partial L}{\partial W^{(l)}}=\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial f^{(l)}}\frac{\partial f^{(l)}}{\partial z^{(l)}}\frac{\partial z^{(l)}}{\partial W^{(l)}}$$

  assuming that the output $\hat{y}=f^{(l)}(z^{(l)})$ is a function of layer $l$’s input $z^{(l)}=W^{(l)}{f}^{(l-1)}(z^{(l-1)})+b^{(l)}$.

• At layer $l-1$, the gradients are computed as: $$\frac{\partial L}{\partial W^{(l-1)}}=\left(\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial f^{(l)}}\frac{\partial f^{(l)}}{\partial z^{(l)}}\right)\frac{\partial z^{(l)}}{\partial f^{(l-1)}}\frac{\partial f^{(l-1)}}{\partial z^{(l-1)}}\frac{\partial z^{(l-1)}}{\partial W^{(l-1)}}$$

• At layer $l-2$, this becomes: $$\frac{\partial L}{\partial W^{(l-2)}}=\left(\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial f^{(l)}}\frac{\partial f^{(l)}}{\partial z^{(l)}}\frac{\partial z^{(l)}}{\partial f^{(l-1)}}\frac{\partial f^{(l-1)}}{\partial z^{(l-1)}}\right)\frac{\partial z^{(l-1)}}{\partial f^{(l-2)}}\frac{\partial f^{(l-2)}}{\partial z^{(l-2)}}\frac{\partial z^{(l-2)}}{\partial W^{(l-2)}}$$

• And so on, until we reach the first layer.

• We are recursively applying the chain rule and re-using the gradients computed at the previous layer

• This is great for computational efficiency, but it can also lead to vanishing or exploding gradients

Vanishing and Exploding Gradients

• Vanishing/exploding gradients are where the gradients become near zero or near infinity as they are propagated back through the network
• Particularly problematic for recurrent neural networks, where the same weights are multiplied by themselves repeatedly
• Also a problem for very deep networks, and part of the reason that deep learning was not popular until the 2010s
• ❓ What changed?

Consider the variance

• At the input layer, $Z^{(0)}=W^{(0)}X+b^{(0)}$, and $X$ has some variance ${\sigma }^2$
• Assume $W^{(0)}$ and $b^{(0)}$ are initialized to 0
• :abacus: What is the variance of $Z^{(0)}$?
• What about after the activation function $f^{(0)}(Z^{(0)})$?

Initialization strategies

• In 2010, Glorot and Bengio proposed the Xavier initialization for a layer with $m$ inputs and $n$ outputs: $$W_{i,j}\sim U\left(-\sqrt{\frac{6}{m+n}},\sqrt{\frac{6}{m+n}}\right)$$
• Goal is to preserve the variance of the input and output in both directions
• Similar to LeCun initialization, and apparently an overlooked feature of networks from the 1990s

Initialization for ReLU

• Glorot initialization was derived under the assumption of linear activation functions (even though they knew this wasn’t the case)
• In 2015, He et al. proposed the He initialization specifically for ReLU activations: $$W_{i,j}\sim N\left(0,\sqrt{\frac{2}{m}}\right)$$
• The choice of normal vs uniform is apparently not very important
• Default in PyTorch is $U(-\sqrt{1/k},\sqrt{1/k})$

Batch normalization

• Also in 2015, Ioffe and Szegedy proposed batch normalization as a way to mitigate vanishing/exploding gradients
• This is simply a normalization at each layer, shifting and scaling the inputs to have a mean of 0 and a variance of 1 (across the batch)
• A moving average of the mean and variance is maintained during training, and used for normalization during inference
• It also ends up acting as regularization, magic!
• ❓ Why wouldn’t you want to use batch normalization?

RELU and its variants

• In early works, the sigmoid or tanh functions were popular
• Both have a small range of non-zero gradients
• ReLU has a stable gradient for positive inputs, but can lead to the dying ReLU problem whereby certain neurons are “turned off”
• ❓ How can we prevent dying ReLUs?

Number of neurons and layers

• Number of neurons in the input layer is defined by number of features
• Number of neurons in the output layer is defined by prediction task
• In between is a design choice
• Common early choice was a pyramid shape, but it turns out that a stack of layers with the same number of neurons works well too
• Deeper networks can solve more complex problems with the same number of total parameters, but are also prone to vanishing/exploding gradients

Ultimately, yet another a hyperparameter to be tuned

Optimization algorithms: variations on gradient descent

• Gradient descent takes small regular steps, constant or otherwise
• Many variations exist! For example, momentum keeps track of the previously computed gradient and uses it to inform the new step: $$\begin{array}{rl}\mathbf{m}& =\beta \mathbf{m}-\eta {\nabla }_{\mathbf{W}}J(\mathbf{W})\\ \mathbf{W}& =\mathbf{W}+\mathbf{m}\end{array}$$ where $\beta$ is a hyperparameter between 0 and 1
• Adaptive moment estimation (Adam) is a popular choice that adds on an exponentially decaying average of the squared gradients

The Adam optimizer

• Keeps track of first ($\mathbf{s}$) and second ($\mathbf{r}$) moments of the gradient, with two exponential decay terms ${\rho }_1$ and ${\rho }_2$
• At each time step, the update is now: $$\begin{array}{rl}\mathbf{s}& ={\rho }_1\mathbf{s}+(1-{\rho }_1){\nabla }_{\mathbf{W}}J(\mathbf{W})\\ \mathbf{r}& ={\rho }_2\mathbf{r}+(1-{\rho }_2){\nabla }_{\mathbf{W}}J(\mathbf{W}{)}^2\\ \mathbf{W}& =\mathbf{W}-\eta \frac{\mathbf{s}}{\sqrt{\mathbf{r}}+ϵ}\end{array}$$
• ${\rho }_1$ and ${\rho }_2$ are typically 0.9 and 0.999, respectively

Regularization via dropout

• Dropout is a regularization technique that randomly sets a fraction of the neurons to zero during training
• During each training pass, a neuron has a $p$ probability of being dropped
• Similar to training an ensemble of models then bagging
• Helps to prevent overfitting, but can slow down training
• Typical values: 0.5 for hidden layers, 0.2 for input layers

Choices for starting

From the Deep Learning Book, section 11.2:

• “Depending on the complexity of your problem, you may even want to begin without using deep learning.”
• Tabular data: fully connected, images: convolutional, sequences: recurrent*
• ReLU or variants, with He initialization
• SGD or Adam, add batch normalization if unstable
• Use some kind of regularization, such as dropout

Implementation time

Lecture 7: Convolutional Neural Networks

HTML Slides | PDF Slides

Convolution and Convolutional Neural Networks

COMP 4630 | Winter 2026 Charlotte Curtis

Overview

• Convolutional neural networks (CNNs) are a type of neural network that is particularly well-suited to image data
• Before we can understand CNNs, we need to understand convolution
• References and suggested reading:
  • Scikit-learn book: Chapter 14
  • Deep Learning Book: Chapter 9
  • 3blue1brown video: What is convolution?

Convolution

• Convolution is defined as: $$(f\ast g)(t)={\int }_{-\infty }^{\infty }f(\tau )g(t-\tau )d\tau$$

• Or in the discrete case: $$(f\ast g)[n]=\sum _{m=-\infty }^{\infty }f[m]g[n-m]$$

• Can be though of as “flipping” one function and sliding it over the other, multiplying and summing at each point

Example

We’re in a hospital dealing with an outbreak. For the first 5 days we have 1 patient on Monday, 2 on Tuesday, etc:

$$\mathrm{patients}(x)=\left[\begin{array}{c}1,2,3,4,5\end{array}\right]$$

Fortunately, we know how to treat them: 3 doses on day 1, then 2, then 1:

$$\mathrm{doses}(x)=\left[\begin{array}{c}3,2,1\end{array}\right]$$

And after 3 days they’re cured.

How many doses do we need on each day?

Convolution in 2D

• Extending to 2D basically adds another summation/integration: $$\begin{array}{rl}(f\ast g)[n,m]=& \sum _{i=0}^n\sum _{j=0}^{m}f[i,j]g[n-i,m-j]\\ (f\ast g)(x,y)=& {\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }f(u,v)g(x-u,y-v)dudv\end{array}$$
• This can also be extended to higher dimensions
• Caution: a colour image is a 3D array, not a 2D array
• For typical image processing applications, the colour channels are convolved independently such that the output is still a 3D array

Convolution kernels

Some common kernels

A side tangent on frequency representation

• Any signal can be represented as a weighted summation of sinusoids
• For a discrete signal $x[n]$, you can think of this as: $$x[n]=\sum _{k=0}^{N-1}\left[a_k\cos \left(\frac{2\pi kn}{N}\right)+b_k\sin \left(\frac{2\pi kn}{N}\right)\right]$$
• Or, using Euler’s formula $e^{j\theta }=\cos \theta +j\sin \theta$: $$x[n]=\sum _{k=0}^{N-1}{c}_k{e}^{j\frac{2\pi kn}{N}}$$ where the complex coefficients $c_k=a_k+j{b}_k$

Fourier Transform

• To figure out what the coefficients $c_k$ are, we can use the Discrete Fourier Transform (DFT): $$X[k]=\sum _{n=0}^{N-1}x[n]e^{-j\frac{2\pi kn}{N}}$$ where each element of $X[k]$ is the coefficient $c_k$ for frequency $k$
• The Fast Fourier Transform (FFT) computes the DFT in $O(n\log n)$ time
• Convolution is $O(n^2)$

Convolution is multiplication in frequency

• Sometimes it is useful to use the Fourier Transform to obtain a frequency representation of an image (or signal)
• In the frequency domain, convolution is simply element-wise multiplication
• This allows for some efficient operations such as blurring/sharpening an image, as well as some fancy stuff like deconvolution
• Sharp edges in space become ringing in frequency, and vice versa
• ❓ why do CNNs operate in the spatial domain?

Convolutional neural networks

• 1958: Hubel and Wiesel experiment on cats and reveal the structure of the visual cortex
• Determine that specific neurons react to specific features and receptive fields
• Modelled in the “neocognitron” by Kunihiko Fukushima in 1980
• LeCun’s work in the 1990s led to modern CNNs

Why CNNs?

• A fully connected network has a 1:1 mapping of weights to inputs
• Fine for MNIST (28x28) pixels, but quickly grows out of control
• ❓ If you train a (fully connected) network on 100x100 images, how would you infer on 200x200 images?
• ❓ What if an object is shifted, rotated, or flipped within the image?

Convolutional layers

• A convolution layer is a set of kernels whose weights are learned
• Instead of the straight up weighted sum of inputs, the input image is convolved with the learned kernel(s)
• The output is often referred to as a feature map
• The dimensionality of the feature map is determined by the:
  • Size of the input image
  • Number of kernels
  • Padding (usually “same” or “valid”)
  • “Stride”, or shift of the kernel at each step

Dimensionality examples