Physics · Primitives
MutualInformation
Information-theoretic dependence between two variables
Definition
Mechanism
The substrate information-theoretic measure of dependence between two random variables, defined generally as the Kullback-Leibler divergence between their joint distribution and the product of their marginals. When the relevant Entropy terms are finite, I(X; Y) = H(X) + H(Y) − H(X, Y), equivalently the reduction in one variable's entropy from knowing the other. Zero mutual information indicates statistical independence; a positive value indicates dependence, but its unnormalized magnitude has no universal scale. The property exists whether or not anyone estimates it. Distinct from normalized correlation coefficients, whose sensitivity depends on the selected method, and from mere co-occurrence.
Invariants
- Non-negativity: I(X; Y) ≥ 0 for all random variables X, Y.
- Symmetry: I(X; Y) = I(Y; X).
- Zero iff independent: I(X; Y) = 0 if and only if X and Y are statistically independent.
- Discrete entropy bound: for discrete X and Y with finite entropies, I(X; Y) ≤ min(H(X), H(Y)).
Risks
Failure modes
Estimator conflation
reading a finite-sample estimate of zero as proof of independence. Zero-iff-independent is a property of the population quantity, not of an estimate.
Relationships
Builds on
References
Measure of disorder
Nothing in this library builds on this pattern yet.