Multiclass Support Vector Machine (SVM) Loss

Introduction

Core Concept

The score of the correct class should be higher than all the other incorrect classes

Explanation

Let $s$ be the score

$$s=f(x_i,W,b)$$

then the loss is defined by

$$L_i=\sum _{j\ne y_i}\max (0,s_j-s_{y_i}+1)$$

note: $s_j$ mean the score of training sample $i$ for label $j$

---

Problems & Answers to SVM

Q1: What happen when the scores for an image change a bit? A1: If the score of the correct label is already much larger than incorrect labels’ scores, then the loss will stay 0

Q2: If all the scores were random, what do we expect? A2: $s_j-s_{y_i}=0$ thus $L_i=1$ and $L=K$ where $K$ is number of labels

Q3: What if we make $L_i={\sum }_j\max (0,s_j-s_{y_i}+1)$ A3: $L_i$ will add a constant $1$

Q4: What if we use mean instead of sum when calculating the loss A4: The loss will be lower and the update to hyperparameters in linear classifier will be slighter