Likelihood

Main Idea: measures how well a statistical model explains the observed data.

$$ \text{for } D = {x_1, ..., x_n} \text{ given parameter } \theta\\

\text{likelihood: } L(\theta) = P(D | \theta) $$

aka the probability of observing the data D given the parameter

MLE & Log Likelihood

Main Idea: finds the parameter that maximizes the probability of observing the data

$$ L(\theta) = \prod_i P(x_i|\theta) $$

• in practice log-likelihood is used

$$ log(L(\theta)) = \sum_i P(x_i|\theta) $$

• using log allows for products to be converted into sums
• this is often easier in optimization problems
• MLE w/ Log likelihood:

$$ \text{arg max }{\theta} \text{ } log(L(\theta))\\ = \text{arg max }{\theta}\sum_i P(x_i|\theta) $$

Maximum A Pasteriori

Main Idea: incorporate prior belief in to estimation, recall Bayes Theorem:

$$ P(\theta | D) = \frac{P(D | \theta) P(\theta)}{P(D)} $$

Maximizing the prior:

$$ \hat{\theta}{\text{MAP}} = \arg\max\theta P(\theta | D) $$

$$ \text{Bayes Theorem (P(D) constant due to maximizing w/ respect to } \theta \text{)}\\

\hat{\theta}{\text{MAP}} = \arg\max\theta P(D | \theta) P(\theta) $$