Jacobian in Deep Learning

🧮 The Jacobian in Deep Learning

A detailed, math‑friendly guide with design & implementation notes

In deep learning, the Jacobian is the silent workhorse behind backpropagation, sensitivity analysis, and adversarial robustness. While we often talk about gradients, the Jacobian generalises the derivative to vector‑valued functions — exactly what we need when a neural network maps an input vector $\mathbf{x} \in \mathbb{R}^n$ to an output $\mathbf{y} \in \mathbb{R}^m$.

This post walks through the Jacobian formula, its role in deep learning, and gives a concrete implementation outline. The layout is Blogger‑ready: just paste the HTML (with MathJax) into your post editor.

📐 What is the Jacobian?

For a function $\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m$, the Jacobian matrix $\mathbf{J} \in \mathbb{R}^{m \times n}$ collects all first‑order partial derivatives:

$$ \mathbf{J} = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \cdots & \dfrac{\partial f_1}{\partial x_n} \\[1em] \vdots & \ddots & \vdots \\[0.8em] \dfrac{\partial f_m}{\partial x_1} & \cdots & \dfrac{\partial f_m}{\partial x_n} \end{bmatrix} $$

In deep learning, $\mathbf{f}$ is a layer or the entire network, $\mathbf{x}$ is the input (or an activation), and $\mathbf{y} = \mathbf{f}(\mathbf{x})$ is the output. The Jacobian tells us how each output dimension reacts to a tiny change in each input dimension.

💡 Key insight: For a single layer $\mathbf{y} = \sigma(\mathbf{W}\mathbf{x} + \mathbf{b})$, the Jacobian w.r.t. $\mathbf{x}$ is $\mathbf{J} = \operatorname{diag}(\sigma'(\mathbf{z})) \, \mathbf{W}$, where $\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}$ and $\sigma'$ is the derivative of the activation.

⚙️ Jacobian in backpropagation

Backpropagation computes gradients using the chain rule. For a composition of functions $\mathbf{f} = \mathbf{f}^{(L)} \circ \cdots \circ \mathbf{f}^{(1)}$, the Jacobian of the entire network is the product of per‑layer Jacobians:

$$ \mathbf{J}_{\text{total}} = \mathbf{J}^{(L)} \cdot \mathbf{J}^{(L-1)} \cdots \mathbf{J}^{(1)} $$

In practice we never form the full Jacobian (it’s huge), but we use vector‑Jacobian products (VJPs) — that’s exactly what autograd does. The gradient of a loss $L$ w.r.t. an input $\mathbf{x}$ is:

$$ \nabla_{\mathbf{x}} L = \mathbf{J}_{\text{total}}^\top \, \nabla_{\mathbf{y}} L $$

where $\nabla_{\mathbf{y}} L$ is the gradient of the loss w.r.t. the network output. This is the essence of reverse‑mode autodiff.

🧠 Detailed design: Jacobian‑vector product (JVP) & VJP

Two key operations appear in deep learning frameworks:

• JVP (forward‑mode): $\mathbf{J} \mathbf{v}$ — efficient when $n \ll m$.
• VJP (reverse‑mode): $\mathbf{v}^\top \mathbf{J}$ — efficient when $m \ll n$.

In neural networks, $m$ (output dimension) is often the loss (scalar), so reverse‑mode is the default. But for tasks like style transfer or Jacobian regularisation, we need full or partial Jacobians.

📌 Design tip: To compute a Jacobian in PyTorch, use torch.autograd.functional.jacobian. For large models, prefer VJPs via torch.autograd.grad with create_graph=True to build higher‑order derivatives.

🐍 PyTorch implementation (sketch)

Below is a compact example of computing the Jacobian of a simple layer and using it for a sensitivity analysis.

import torch # Define a small network: 3 → 5 → 2 model = torch.nn.Sequential( torch.nn.Linear(3, 5), torch.nn.Tanh(), torch.nn.Linear(5, 2) ) x = torch.randn(1, 3, requires_grad=True) y = model(x) # shape (1, 2) # Compute full Jacobian dy/dx (2 x 3) from torch.autograd.functional import jacobian def f(x_flat): return model(x_flat.unsqueeze(0)).squeeze(0) x_flat = x.squeeze(0) J = jacobian(f, x_flat) # shape (2, 3) print("Jacobian:\n", J) # VJP: gradient of a scalar loss w.r.t. input loss = y.sum() loss.backward() print("∇_x loss:", x.grad) # equals J^T @ ones

Notice how the VJP ($\mathbf{J}^\top \mathbf{1}$) matches the gradient from backward(). This is the chain rule in action.

📊 When do we need the full Jacobian?

Full Jacobians are expensive ($O(mn)$ memory), but they appear in:

• Adversarial attacks (e.g., FGSM uses the gradient, but Jacobian‑based attacks use full sensitivity).
• Jacobian regularisation — penalising $\|\mathbf{J}\|_F^2$ to improve robustness.
• Style transfer — Gram matrices are built from Jacobian‑like features.
• Neural ODEs — the Jacobian of the dynamics is needed for the adjoint method.

For these cases, efficient Jacobian computation via batched VJPs or forward‑mode AD is essential.

⚡ Pro tip: Use torch.func (formerly functorch) for jacrev / jacfwd — they handle batching and are more memory‑efficient than naive loops.

📝 Summary

The Jacobian generalises gradients to vector‑valued functions and is the mathematical backbone of backpropagation. While we rarely materialise it, understanding it clarifies why VJPs are efficient and how higher‑order derivatives work. Whether you're implementing a custom layer, analysing sensitivity, or building robust models, the Jacobian formula is a fundamental tool in the deep learning toolbox.

Deep Learning Mathematics PyTorch 📅 Published · 2026