D-DL4CV-Lec19b-Explicit_Density_Estimation

Explicit Density Estimation

Goal

The goal of explicit density estimation is to write down a function $f$ such that for any given input $x$, we can give its probability $p(x)=f(x,W)$

Explanation

For each input $x$, we split it into parts: $x^{(1)}$, $x^{(2)}$… These can be pixels in the context of images or words in context of texts

Then, we want to find parameters $W^{\star }$ such that

$$\begin{array}{rl}{W}^{\star }& =\arg \underset{W}{\max }\prod _{i=1}^n\prod _{j}p(x_i^{(j)})\\ & =\arg \underset{W}{\max }\sum _{i=1}^n\ln \left(\prod _{j}f(x_i^{(j)},W)\right)\\ & =\arg \underset{W}{\max }\sum _{i=1}^n\sum _j\ln \left(f(x_i^{(j)},W)\right)\end{array}$$

Why do we multiply $x^{(i)}$ together?

We suppose each pixel is independent, so multiplying them together can be written as:

$$p(x^{(1)}\cap x^{(2)}\cap x^{(3)}\cap \dots )$$

That is, the probability of the model generating the exact image $x$ we input

In context of neural network, $W^{\ast }$ refers to all the learnable parameters appear in your network. Hence, the above equation is just telling us we want to optimize our network

Intuition

We want our learning parameters $W^{\star }$ to learn the share pattern from our training example. For example, if we input lots of cats’ images, then we want our model to be excelled at predicting pixel so that the pixels form a cat image