D-DL4CV-Lec04a-Gradient

Introduction

Intuition

A gradient is a vector that points to the steepest increase of the function (here $L(w)$) at the given point

The additive inverse of the gradient vector points to the steepest decrease of the function at the given point

Math Definition

If $f$ is a function of two variables $x$ and $y$, then the gradient of $f$ is the vector function $\nabla f$ defined by

$$\nabla f(x,y)=⟨f_x(x,y),f_y(x,y)⟩=\frac{\partial f}{\partial x}\hat{i}+\frac{\partial f}{\partial y}\hat{j}$$

---

Numeric Gradient

Strategy

Numeric gradient rely on the formula below

$$\frac{\partial f}{\partial x}=\underset{h\to 0}{\lim }\frac{f(x+h)-f(x)}{h}$$

This strategy observes the loss change in small change of weight. Then, by dividing the small change in weight by the loss change, we can approximate $\frac{\partial f}{\partial x}$

By doing the same thing $n$ times, which is the dimension of the weight, we can get the gradient $\frac{dL}{dW}$

Advantages & Disadvantages

Advantages:

1. Easy to implement

Disadvantages:

1. Slow: $O(n)$ ($n$ is dimension of weight)
2. This method “approximate” the gradient using numerical approximation rather than getting the actual value

---

Analytic Gradient

Strategy

We apply calculus rules to the loss function and receive $\partial L/\partial W={\nabla }_{W}L(W)$. After that, we can put in the coordinates (pixel value of image) and get the accurate gradient

Sketch of Implementation

$$L(W)=\frac{1}{N}\sum _{i=1}^N{L}_i(x_i,y_i,W)+\lambda R(W)$$ $$\text{Gradient}={\nabla }_{W}L(W)=\frac{1}{N}\sum _{i=1}^N{\nabla }_W{L}_i(x_i,y_i,W)+{\nabla }_{W}R(W)$$

Advantages & Disadvantages

Advantages:

1. Fast: By the method backpropagation
2. Get the exact gradient value rather than approximate it

Disadvantage:

1. Error-Prone: Hard to implement, often has bugs

In real world, we'll almost always use analytic gradient rather than numeric gradient

---

Gradient Check

We’ve mentioned that analytic gradient is error-prone and numeric gradient is easy to implement. Hence, we can use numeric gradient to check the correctness of analytic gradient. We call this step “gradient check”

To overcome the inefficiency of numeric gradient, in gradient check, we’ll instead use lower dimensional samples. This way, we can check analytic gradient’s correctness without wasting so much time

PyTorch offers torch.autograd.gradcheck function for gradient check