Cross-Entropy Loss

Introduction

Core Concept

After we computed the scores, we transform them into probability to give us better intuition about the classifier’s confidence. Next, we

Explanation

Let the $s$ be the output of score function, i.e.,

$$s=f(x_i,W,b)$$

We convert the score into probability using “Softmax Function” The conditional probability represent the score image $x_i$ get on label $k$

$$P(Y=k|X=x_i)=\frac{\exp (s_k)}{{\sum }_j\exp (s_j)}$$

Then, the loss of image $x_i$ is defined as

$$L_i=-\log P(Y=y_i|x=x_i)$$

Putting it together we get the equation below:

$$L_i=-\log \left(\frac{\exp (s_{y_i})}{{\sum }_j\exp (s_j)}\right)$$

---

Problems & Solutions for Cross-Entropy Loss

Q1: What is min/max possible loss $L_i$ A1: min:$0$ , max: $\infty$

Q2: If all scores are small random values, what is the loss for any input? A2: $\log K$, where $K$ is number of labels

At the start of the training, the scores will be small random value, we can then compare the loss value with $-\log \left(\frac{1}{K}\right)$ to check if there's bug in our code