IEEE Transactions on Information Forensics and Security (TIFS), 2024
Hangcheng Liu · Yuan Zhou · Ying Yang · Qingchuan Zhao · Tianwei Zhang · Tao Xiang
How stealthy is an adversarial perturbation to the human eye?
Widely used metrics such as (L_p) norms, PSNR, and SSIM were not designed specifically for subtle adversarial perturbations and can disagree with human visual perception. This work provides a systematic study of pixel-level adversarial stealthiness assessment from both subjective and objective perspectives.
We introduce:
- A new subjective assessment criterion that covers both hardly perceptible and clearly perceptible perturbations.
- APSD, a large-scale Adversarial Perturbation Stealthiness Assessment Dataset with human opinion scores.
- A2SM, an Attention-based Adversarial Stealthiness Assessment Model that mimics human scoring habits for objective stealthiness prediction.
| 👁️ Human-centric assessment | A two-step protocol provides fine-grained subjective scores across the full stealthiness spectrum. |
| 🗂️ 10,586 adversarial examples | APSD contains 10,586 clean–adversarial pairs generated from 400 reference images. |
| ⚔️ 12 attacks / 5 attack families | Gradient-, optimization-, generative model-, pixel-, and block-based attacks are covered. |
| 👥 60 human observers | Every adversarial example is evaluated by 15 observers; Mean Opinion Score (MOS) is used as the label. |
| 🧠 Human-inspired objective model | A2SM combines global/local features with self-, cross-, and spatial-attention mechanisms. |
| 📈 High HVS consistency | A2SM achieves 0.984 PLCC / 0.978 SROCC on APSD. |
We propose a two-step subjective assessment protocol.
Observers first inspect a clean/adversarial image pair for a fixed period and classify the perturbation as:
- Strong stealthiness — the difference is hardly perceptible.
- Weak stealthiness — the difference is readily perceptible.
For strong stealthiness, observers use progressively amplified Scaled Grayscale Difference Maps (SGDM) to distinguish subtle pixel-level differences.
For weak stealthiness, observers assess the degree of visible degradation / annoyance.
The resulting scores are aggregated into a Mean Opinion Score (MOS) in the range ([0,1]), where a larger value means better stealthiness.
Rating software.
We construct the Adversarial Perturbation Stealthiness Assessment Dataset (APSD), a dedicated benchmark for human-aligned adversarial stealthiness evaluation.
| Property | APSD |
|---|---|
| Reference images | 400 |
| Clean–adversarial pairs | 10,586 |
| Image categories | 200+ |
| Attack methods | 12 |
| Attack families | 5 |
| Human observers | 60 |
| Ratings per adversarial example | 15 |
| Label | MOS ∈ [0, 1] |
| Image size used for subjective study | 512 × 512 |
- Gradient-based: FGSM, MIFGSM, PGD, NES
- Optimization-based: CW
- Generative model-based: CDP, AdvGAN, GAP, GUAP
- Pixel-based: SimBA, Pixel
- Block-based: Square
APSD contains perturbations spanning a broad range of perceptual stealthiness, allowing models to learn both subtle and visually obvious adversarial distortions.
Distribution of adversarial examples generated by different attacks in APSD. The suffix of each attack indicates the corresponding parameter.
The APSD dataset is publicly available through the project repository:
The released dataset should contain the clean images, adversarial images, and their corresponding human subjective scores.
A recommended local organization is:
APSD/
├── REF-299 (Reference)/
│ ├── xxx.png
│ └── ...
└── AE_targetmodel0 (Adversarial example)/
├── 0 FGSM/
├── 1 MIFGSM/
├── 2 PGD/
├── 3 NES/
├── 4 CW/
├── 5C DP/
├── 6 AdvGAN/
├── 7 GAP/
├── 8 GUAP/
├── 9 SimBA/
├── 10 Pixel/
└── 11 Square/
└── score.csv
Note: the exact released directory/file names should follow the actual dataset package. Update the tree above if your final release uses a different structure.
If you find APSD or A2SM useful in your research, please cite:
@article{liu2024stealthiness,
title = {Stealthiness Assessment of Adversarial Perturbation: From a Visual Perspective},
author = {Liu, Hangcheng and Zhou, Yuan and Yang, Ying and Zhao, Qingchuan and Zhang, Tianwei and Xiang, Tao},
journal = {IEEE Transactions on Information Forensics and Security},
volume = {20},
pages = {898--913},
year = {2024},
doi = {10.1109/TIFS.2024.3520016}
}For questions about the paper, dataset, or code, please open a GitHub issue or contact the authors.
Please add the license that applies to the released code and dataset before public release.
If the dataset contains images inherited from ImageNet, COCO, or VOC2012, their original licenses/terms should also be respected.