Entropy is a way to measure uncertainty. If an outcome is predictable, entropy is low. If many outcomes are plausible, entropy is high.
For a discrete distribution p, Shannon entropy is:
$$ H(p) = -\sum_x p(x)\log p(x) $$
The negative sign is there because log p(x) is negative when probabilities are between 0 and 1.
Intuition
Imagine a classifier looking at an image and producing probabilities:
| Class | Probability |
|---|---|
| cat | 0.92 |
| dog | 0.05 |
| fox | 0.03 |
This distribution has low entropy because the model is confident. Compare it to:
| Class | Probability |
|---|---|
| cat | 0.34 |
| dog | 0.33 |
| fox | 0.33 |
This distribution has higher entropy because the model is uncertain.
Cross entropy
In supervised learning, we often compare a true distribution q with a predicted distribution p.
$$ H(q, p) = -\sum_x q(x)\log p(x) $$
For one-hot labels, this simplifies to:
$$ \operatorname{loss} = -\log p(y_{\text{correct}}) $$
So if the correct label is cat:
import math
def cross_entropy_for_label(probabilities, label):
return -math.log(probabilities[label])
loss = cross_entropy_for_label(
{"cat": 0.92, "dog": 0.05, "fox": 0.03},
"cat",
)
print(loss)
Assigning high probability to the correct class gives a small loss. Assigning low probability gives a large loss.
Why it works well
Cross entropy does not just ask whether the model is right. It asks whether the model assigned calibrated probability to the right answer.
That makes it useful for classification because the loss has a strong gradient when the model is confidently wrong.
Entropy in different places
| Concept | What it measures |
|---|---|
| Entropy | Uncertainty inside one distribution |
| Cross entropy | How surprised predictions are by true labels |
| KL divergence | Extra surprise from using one distribution instead of another |
| Perplexity | Exponentiated average cross entropy, common in language models |
Practical note
Entropy is not automatically good or bad. High entropy can mean healthy uncertainty. Low entropy can mean confidence, but it can also mean overconfidence.
A good model should be confident when the evidence is strong and uncertain when the evidence is ambiguous.
That is why entropy shows up in calibration, active learning, reinforcement learning exploration, and language modeling.