Materials

Mathematics for Machine Learning

CVPR 2021 Tutorial: Normalizing Flows and Invertible Neural Networks in Computer Vision

Normalizing Flows Architecture

Change-of-variables trick: Build complex distributions from simple distributions via a flow of successive (invertible) transformations

Change of Variables Trick

Key Idea: Transform random variable $X$ into random variable $Z$ using an invertible transformation $\phi$, while keeping track of the change in distribution

• $p_X=\phi(p_Z)$
• $p_Z=\phi^{-1}(p_X)$

Jacobian determinant

$$ \left|\det(\frac{dz}{dx})\right|=\left|\det(\frac{d\phi(x)}{dx})\right| $$

Determinant of Jacobian tells us how much the domain $dx$ is stretched to $dz$

Volume preservation

Rescale $p_Z$ by the inverse of Jacobian determinant

$$ p_Z(\mathbf z)=p_X(\mathbf x)=p_X(\mathbf x)\left|\det\frac{d\phi(\mathbf x)}{d\mathbf x})\right|^{-1} $$

• Express target distribution $p_Z$ in terms of known distribution $p_X$ and the Jacobian determinant of an invertible mapping $\phi$