Adversarial attacks

Threat model first#

A robustness claim without a threat model is not a claim. Specify:

• Perturbation set $\Delta$. Usually an $\ell_p$ ball ${\delta : |\delta|_p \le \varepsilon}$, with $p \in {0, 1, 2, \infty}$. Also: spatial transformations, colour shifts, physically realisable patches.
• Adversary knowledge. White-box (weights and gradients), black-box score-based (probabilities), black-box decision-based (labels only), or transfer-only.
• Adversary capability. Query budget, ability to modify the physical scene, ability to poison training data.

$\ell_p$ balls are a tractable proxy for "perceptually similar", not a definition of it. Robustness inside an $\ell_\infty$ ball says nothing about robustness to a small rotation.

The objective#

Madry et al. (2018) frame robust training as a saddle-point problem:

$$ \min_{\theta} \; \mathbb{E}{(x,y) \sim \mathcal{D}} \Big[ \max{\delta \in \Delta} \; \mathcal{L}\big(f_\theta(x + \delta),\, y\big) \Big] $$

$\theta$ are the model parameters, $\mathcal{D}$ the data distribution, $\mathcal{L}$ the loss, $\Delta$ the perturbation set. The inner maximisation is the attack; the outer minimisation is adversarial training. The inner problem is non-concave, so it is solved approximately, and the quality of that approximation determines whether the outer solution is meaningfully robust.

Attacks#

FGSM (Goodfellow et al., 2014): $x' = x + \varepsilon\,\text{sign}(\nabla_x \mathcal{L})$. One step. Exact for linear models — the demonstration in the developer block — and a weak approximation otherwise.

PGD (Madry et al., 2018): iterate $x^{t+1} = \Pi_{\mathcal{B}(x,\varepsilon)}\big(x^t + \alpha\,\text{sign}(\nabla_x \mathcal{L})\big)$, with $\Pi$ the projection back onto the ball and random restarts. Treated as the standard first-order attack.

C&W (Carlini and Wagner, 2017): minimise $|\delta|_2 + c \cdot g(x + \delta)$ with a margin-based $g$ optimised in a change of variables that removes the box constraint. Finds much smaller perturbations than PGD and broke defensive distillation.

AutoAttack (Croce and Hein, 2020): an ensemble of APGD with two losses, FAB and Square Attack, parameter-free. The current default for reporting, and RobustBench standardises on it.

Black-box. Square Attack (Andriushchenko et al., 2020) is score-based and random-search driven. Boundary Attack (Brendel et al., 2018) needs only decision labels. Transfer attacks exploit the shared decision boundaries documented by Papernot et al. (2016).

Why adversarial examples exist#

Two accounts, both supported.

Linearity (Goodfellow et al., 2014). For a linear model the logit shift under an $\ell_\infty$ perturbation is exactly $\varepsilon |w|_1$, which grows with input dimension. Modern networks are locally close enough to linear for the argument to bite.

Non-robust features (Ilyas et al., 2019). Adversarial vulnerability arises from genuinely predictive but brittle features present in the data. Their key experiment: a dataset relabelled purely by adversarial perturbations still trains a model that generalises to the clean test set, showing the perturbations carry real signal. This reframes robustness as a choice about which features to use, not as a defect to patch.

Defences, and the evaluation problem#

Adversarial training is the only broadly surviving defence. Madry et al. (2018) train on PGD examples. Costs: roughly $k\times$ training compute for $k$ inner steps, and a consistent clean-accuracy drop. Zhang et al. (2019), TRADES, make the accuracy-robustness trade explicit through a decomposed loss.

Certified defences give provable guarantees on a radius. Randomised smoothing (Cohen et al., 2019) constructs $g(x) = \arg\max_c \Pr_{\eta \sim \mathcal{N}(0,\sigma^2 I)}[f(x + \eta) = c]$, certified in $\ell_2$ with radius $\frac{\sigma}{2}(\Phi^{-1}(p_A) - \Phi^{-1}(p_B))$ where $p_A, p_B$ are the top two class probabilities under noise and $\Phi^{-1}$ the inverse Gaussian CDF. Scales to ImageNet; certified radii remain small.

The methodological warning. Athalye, Carlini and Wagner (2018), Obfuscated Gradients Give a False Sense of Security, broke seven of nine defences accepted at ICLR 2018 within weeks. Carlini et al. (2019) publish an evaluation checklist. The diagnostic signals of gradient masking are concrete: unbounded attacks fail to reach 100% success, black-box attacks beat white-box attacks, iterative attacks do not beat single-step attacks, and increasing $\varepsilon$ does not monotonically increase success.

RobustBench (Croce et al., 2021) maintains standardised leaderboards under AutoAttack. Current CIFAR-10 $\ell_\infty$ robustness at $\varepsilon = 8/255$ sits well below clean accuracy, and progress has been incremental for years. Treat any claim far above the leaderboard as an evaluation artefact until independently reproduced.

Physical and non-vision attacks#

Eykholt et al. (2018) achieved high misclassification rates on physical stop signs with printed stickers under varied distances and angles. Athalye et al. (2018) produced 3D-printed objects misclassified from most viewpoints. Sharif et al. (2016) used printed eyeglass frames to impersonate identities in face recognition.

Beyond vision, the same framework covers audio (Carlini and Wagner, 2018), malware detection, and — for language models — prompt injection, where the perturbation set is natural-language text and no $\ell_p$ ball applies.

Papers#

• Szegedy et al., Intriguing Properties of Neural Networks, 2013 — arxiv.org/abs/1312.6199
• Goodfellow, Shlens and Szegedy, Explaining and Harnessing Adversarial Examples, 2014 — arxiv.org/abs/1412.6572
• Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018 — arxiv.org/abs/1706.06083
• Athalye, Carlini and Wagner, Obfuscated Gradients Give a False Sense of Security, ICML 2018 — arxiv.org/abs/1802.00420
• Ilyas et al., Adversarial Examples Are Not Bugs, They Are Features, NeurIPS 2019 — arxiv.org/abs/1905.02175
• Cohen, Rosenfeld and Kolter, Certified Adversarial Robustness via Randomized Smoothing, ICML 2019 — arxiv.org/abs/1902.02918
• Croce and Hein, Reliable Evaluation of Adversarial Robustness (AutoAttack), ICML 2020 — arxiv.org/abs/2003.01690

What to learn next#

• Prompt injection — the same idea, aimed at language models.
• Deploying responsibly — the system-level defences that do not need a robust model.
• Image classification — the task these attacks were first built against.