The Problem with Jensen-Shannon Divergence:
When two distributions have non-overlapping supports (which happens early in GAN training), the JS divergence returns a constant value:

$$ \text{JSD}(P_r || P_g) = \log 2 $$
This leads to uninformative gradients for the generator, causing training difficulties.

The Wasserstein-1 Distance (Earth Mover's Distance):
$$ W_1(P_r, P_g) = \inf_{\gamma \in \Pi(P_r, P_g)} \mathbb{E}_{(x,y) \sim \gamma} \left[ \|x - y\| \right] $$
Where:

• $\Pi(P_r, P_g)$ is the set of all joint distributions $\gamma(x,y)$ whose marginals are $P_r$ and $P_g$
• $\|x - y\|$ represents the cost of moving a unit of mass from $x$ to $y$

This can be interpreted as finding the optimal way to "move earth" from one distribution to another with minimal effort.

The Dual Form of the Wasserstein Distance:
The Wasserstein distance can be rewritten using Kantorovich-Rubinstein duality as: $$ W_1(P_r, P_g) = \sup_{f \in \text{Lip}_1} \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] $$
Where $\text{Lip}_1$ is the set of all 1-Lipschitz functions (functions where $|f(x) - f(y)| \leq |x - y|$ for all $x, y$).

Proof of Kantorovich-Rubinstein Duality
Let's show the equivalence between the primal form: $$ W_1(P_r, P_g) = \inf_{\gamma \in \Pi(P_r, P_g)} \mathbb{E}_{(x,y) \sim \gamma} \left[ \|x - y\| \right] $$ And the dual form: $$ W_1(P_r, P_g) = \sup_{f \in \text{Lip}_1} \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] $$ The proof relies on linear programming duality. The primal problem can be formulated as a linear program over coupling measures, and the dual involves optimizing over the space of 1-Lipschitz functions. For any coupling $\gamma \in \Pi(P_r, P_g)$ and any 1-Lipschitz function $f$: $$ \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)] = \int f(x)dP_r(x) - \int f(y)dP_g(y) $$ Since $\gamma$ has marginals $P_r$ and $P_g$: $$ \int f(x)dP_r(x) - \int f(y)dP_g(y) = \iint f(x)d\gamma(x,y) - \iint f(y)d\gamma(x,y) $$ $$ = \iint [f(x) - f(y)]d\gamma(x,y) $$ Since $f$ is 1-Lipschitz, $f(x) - f(y) \leq \|x - y\|$, so: $$ \iint [f(x) - f(y)]d\gamma(x,y) \leq \iint \|x - y\|d\gamma(x,y) = \mathbb{E}_{(x,y) \sim \gamma}[\|x - y\|] $$ This shows that the dual objective is bounded by the primal objective. The equality can be proved by carefully constructing optimal solutions.

The WGAN Objective Function:
$$ \min_G \max_{D \in \mathcal{D}} \mathbb{E}_{x \sim P_r}[D(x)] - \mathbb{E}_{z \sim p(z)}[D(G(z))] $$
Where:

• $D$ is the critic (renamed from discriminator) constrained to be 1-Lipschitz
• $G$ is the generator
• $z$ is random noise
• $\mathcal{D}$ is the set of 1-Lipschitz functions

WGAN-GP Objective:
$$ L = \mathbb{E}_{\tilde{x} \sim P_g}[D(\tilde{x})] - \mathbb{E}_{x \sim P_r}[D(x)] + \lambda \mathbb{E}_{\hat{x} \sim P_{\hat{x}}}[(||\nabla_{\hat{x}} D(\hat{x})||_2 - 1)^2] $$
Where:

• $\hat{x}$ are points sampled uniformly along lines between pairs of points from $P_r$ and $P_g$
• $\lambda$ is the gradient penalty coefficient (usually 10)

Detailed Explanation of the Gradient Penalty
The 1-Lipschitz constraint requires that: $$ |D(x) - D(y)| \leq |x - y| \quad \text{for all } x, y $$ This is equivalent to requiring the gradient norm to be at most 1 everywhere: $$ ||\nabla_x D(x)||_2 \leq 1 \quad \text{for all } x $$ The WGAN-GP enforces this by penalizing the model when the gradient norm deviates from 1 at randomly sampled points. The samples are taken along straight lines between real and generated data points: $$ \hat{x} = t \cdot x + (1-t) \cdot \tilde{x} $$ where $t \sim U[0,1]$, $x \sim P_r$, and $\tilde{x} \sim P_g$. The penalty term is: $$ \lambda \cdot \mathbb{E}_{\hat{x}}[(||\nabla_{\hat{x}} D(\hat{x})||_2 - 1)^2] $$ This penalty forces the gradient norm to be close to 1 along important regions of the data space, effectively enforcing the Lipschitz constraint without explicitly clipping weights.

Comparing Different Probability Divergences/Distances:

• KL Divergence: $D_{KL}(P || Q) = \int P(x) \log \frac{P(x)}{Q(x)} dx$
• JS Divergence: $D_{JS}(P || Q) = \frac{1}{2} D_{KL}(P || M) + \frac{1}{2} D_{KL}(Q || M)$ where $M = \frac{1}{2}(P+Q)$
• Wasserstein Distance: $W_1(P, Q) = \inf_{\gamma \in \Pi(P, Q)} \mathbb{E}_{(x,y) \sim \gamma} \left[ \|x - y\| \right]$

Key Differences:

• KL and JS divergences are not well-defined when distributions don't overlap
• Wasserstein distance gives meaningful values and gradients even for non-overlapping distributions
• Wasserstein distance respects the underlying geometry of the space

Visual Intuition: Why Wasserstein Works Better
Consider two 1D Gaussian distributions $P$ and $Q$ moving away from each other: When they have significant overlap: - KL: Defined, provides useful gradients - JS: Defined, provides useful gradients - W1: Defined, provides useful gradients As they move further apart with minimal overlap: - KL: Approaches infinity (extremely high gradients) - JS: Approaches $\log 2$ (vanishing gradients) - W1: Equals the distance between distributions (stable gradients) When they have no overlap: - KL: Infinite (undefined) - JS: Constant $\log 2$ (zero gradients) - W1: Exactly the distance between means (meaningful, useful gradients) This is why Wasserstein distance provides a more stable learning signal throughout the entire training process.