Are Adversarial Examples also Features in NLP?

Adversarial perturbations in vision models encode discriminative features, not random noise. We ask whether the same holds for language models, and find that it does, along with a troubling consequence: those features encode gender and religious bias.

With Javier Maroto & Apostolos Modas · EPFL

Project Overview

It's well established in computer vision that adversarial perturbations aren't just noise: they span a low-dimensional, highly discriminative subspace that captures the features a model relies on most. Remove that subspace from the data and accuracy collapses; keep only that subspace and accuracy is nearly preserved.

This project asks: does the same geometric structure exist in NLP? Using BERT fine-tuned on IMDB sentiment classification, we compute L∞ PGD adversarial perturbations in the continuous word embedding space, then analyse them with Singular Value Decomposition to identify the dominant eigen-directions of those perturbations.

Core finding: Yes. Adversarial perturbations in NLP span a compact subspace that encodes most of BERT's discriminative power. And alarmingly, that same subspace is strongly aligned with gender and religious bias features.

Methodology

We use two BERT models, one trained normally (94% test accuracy) and one adversarially trained with TextAttack (95% test accuracy), on the IMDB 25k-review sentiment dataset.

  1. Adversarial perturbation generation: For each word embedding in the dataset, apply a 10-step L∞ PGD attack (ε = 1) to produce adversarial perturbations δ ∈ ℝW×D, where W = 512 tokens and D = 1024 dimensions.
  2. Singular Value Decomposition: Stack all perturbations into a tensor T ∈ ℝ(N×W)×D (N=25,000 reviews) and run SVD to obtain the eigen basis V with singular values S ranked by importance.
  3. Discriminative power test: Project clean embeddings onto the top-k eigenvectors and measure classification accuracy: how much of the model's knowledge lives in each subspace?
  4. Bias alignment test: Use WEAT bias word pairs (e.g. he/she, church/mosque, art/science) and measure how much of their energy concentrates in the top eigen components.

Results: A Compact, Powerful Subspace

The singular value spectrum drops sharply after the first few hundred dimensions, indicating that the adversarial perturbation space is highly structured, not uniform noise. The discriminative power test confirms this:

400
eigenvectors needed to recover 96% of discriminative power (out of 1024)
53%
accuracy after removing just the top 50 eigenvectors (near random)
80%
accuracy using only the top 200 eigenvectors

The singular value profiles of both the standard and the adversarially trained BERT drop sharply, which is the signature of a structured subspace. Projecting onto the top-k eigenvectors recovers near-full accuracy; removing those same k directions makes accuracy collapse. Together these confirm that the subspace is where the discriminative information lives.

Crucially, this pattern holds for both the standard and adversarially trained model; adversarial training compresses the subspace slightly (300 eigenvectors needed instead of 400) but does not remove its discriminative structure.

Bias Analysis: What Does the Subspace Encode?

Knowing the subspace is discriminative, the next question is: what does the model consider discriminative? We probe this using the Word Embedding Association Test (WEAT), measuring how much energy bias word-pair differences (e.g. embeddings of "brother" minus "sister") project onto the top adversarial eigen-components.

A high-energy concentration in the top components means the model treats that difference as an important feature for classification. A flat distribution means the model is effectively invariant.

Decomposing energy on the adversarial eigen-basis separates the three bias types cleanly. Gender and religion concentrate their energy in the top components: the model is using them as discriminative features. Subject bias (art versus science) spreads out far more evenly, meaning the model is largely invariant to it.

Across synonym substitution energy profiles, the robust model spreads energy more evenly than the standard one, which is consistent with the synonym invariance adversarial training is supposed to buy. Gender and religious bias, however, stay concentrated in the top components of both models.

Input Manipulation: Gender Pronoun Swaps

We also test bias directly: swapping all male pronouns to female (and vice versa) across the 25k test reviews, then measuring whether predictions change. A fair model should be perfectly consistent.

Swap % Positive (Standard) % Positive (Robust) % Consistency (Standard) % Consistency (Robust)
Male → Female 49.91 48.00 97.91 97.10
Female → Male 49.30 48.30

The robust model is more affected by the male→female swap (larger drop in positive rate), suggesting adversarial training can amplify certain gender biases rather than removing them.

Key insight: The robust model associates male pronouns more strongly with positive sentiment. When male → female pronouns are swapped, its positive prediction rate drops more sharply than the standard model's. Adversarial training did not make it fairer; it may have made it less fair.

Takeaways

Future directions include semantically constrained adversarial attacks (ensuring grammatical fluency), full gender-swap datasets with name and occupation replacement, and applying this framework to decoder-based LLMs where the embedding space has richer structure.