# Scaleable input gradient regularization for adversarial robustness

@article{Finlay2019ScaleableIG, title={Scaleable input gradient regularization for adversarial robustness}, author={Chris Finlay and Adam M. Oberman}, journal={ArXiv}, year={2019}, volume={abs/1905.11468} }

In this work we revisit gradient regularization for adversarial robustness with some new ingredients. First, we derive new per-image theoretical robustness bounds based on local gradient information. These bounds strongly motivate input gradient regularization. Second, we implement a scaleable version of input gradient regularization which avoids double backpropagation: adversarially robust ImageNet models are trained in 33 hours on four consumer grade GPUs. Finally, we show experimentally and… Expand

#### Figures, Tables, and Topics from this paper

#### 21 Citations

Input Hessian Regularization of Neural Networks

- Computer Science, Mathematics
- ArXiv
- 2020

It is proved that the Hessian operator norm relates to the ability of a neural network to withstand an adversarial attack and, furthermore, that it increases the robustness of neural networks over input gradient regularization. Expand

Improving Gradient Regularization using Complex-Valued Neural Networks

- Computer Science
- ICML
- 2021

Experimental results show that the performance of gradient regularized CVNN surpasses that of real-valued neural networks with comparable storage and computational complexity and that the properties of the CVNN parameter derivatives resist decrease of performance on the standard objective that is caused by competition with the gradient regularization objective. Expand

Adversarial Robustness Through Local Lipschitzness

- Computer Science, Mathematics
- ArXiv
- 2020

The results show that having a small Lipschitz constant correlates with achieving high clean and robust accuracy, and therefore, the smoothness of the classifier is an important property to consider in the context of adversarial examples. Expand

Adversarial Boot Camp: label free certified robustness in one epoch

- Computer Science, Mathematics
- ArXiv
- 2020

This work presents a deterministic certification approach which results in a certifiably robust model based on an equivalence between training with a particular regularized loss, and the expected values of Gaussian averages. Expand

On the human-recognizability phenomenon of adversarially trained deep image classifiers

- Computer Science
- ArXiv
- 2021

This work demonstrates that state-of-theart methods for adversarial training incorporate two terms – one that orients the decision boundary via minimizing the expected loss, and another that induces smoothness of the classifier’s decision surface by penalizing the local Lipschitz constant. Expand

Deterministic Gaussian Averaged Neural Networks

- Computer Science, Mathematics
- ArXiv
- 2020

A deterministic method to compute the Gaussian average of neural networks used in regression and classification is presented, comparable to known stochastic methods such as randomized smoothing, but requires only a single model evaluation during inference. Expand

A principled approach for generating adversarial images under non-smooth dissimilarity metrics

- Computer Science, Mathematics
- AISTATS
- 2020

This work proposes an attack methodology not only for cases where the perturbations are measured by $\ell_p$ norms, but in fact any adversarial dissimilarity metric with a closed proximal form, and eliminates the differentiability requirement of the metric. Expand

Are Adversarial Examples Created Equal? A Learnable Weighted Minimax Risk for Robustness under Non-uniform Attacks

- Computer Science
- AAAI
- 2021

A weighted minimax risk optimization that defends against non-uniform attacks, achieving robustness against adversarial examples under perturbed test data distributions and significantly improves state-of-the-art adversarial accuracy under non- uniform attacks. Expand

Min-Max Optimization without Gradients: Convergence and Applications to Black-Box Evasion and Poisoning Attacks

- Computer Science
- ICML
- 2020

A principled optimization framework, integrating a zeroth-order (ZO) gradient estimator with an alternating projected stochastic gradient descent-ascent method, where the former only requires a small number of function queries and the later needs just one-step descent/ascent update. Expand

Defending Adversarial Attacks without Adversarial Attacks in Deep Reinforcement Learning

- Computer Science, Mathematics
- ArXiv
- 2020

A new policy distillation loss that consists of a prescription gap maximization loss aiming at simultaneously maximizing the likelihood of the action selected by the teacher policy and the entropy over the remaining actions and a Jacobian regularization loss that minimizes the magnitude of Jacobian with respect to the input state is proposed. Expand

#### References

SHOWING 1-10 OF 71 REFERENCES

Adversarially Robust Training through Structured Gradient Regularization

- Computer Science, Mathematics
- ArXiv
- 2018

We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled… Expand

Unifying Adversarial Training Algorithms with Data Gradient Regularization

- Mathematics, Computer Science
- Neural Computation
- 2017

The proposed DataGrad framework, which can be viewed as a deep extension of the layerwise contractive autoencoder penalty, cleanly simplifies prior work and easily allows extensions such as adversarial training with multitask cues. Expand

Adversarial Vulnerability of Neural Networks Increases With Input Dimension

- Mathematics, Computer Science
- ArXiv
- 2018

This work shows that adversarial vulnerability increases with the gradients of the training objective when seen as a function of the inputs, and rediscover and generalize double-backpropagation, a technique that penalizes large gradients in the loss surface to reduce adversarialulnerability and increase generalization performance. Expand

Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers

- Computer Science, Mathematics
- NeurIPS
- 2019

It is demonstrated through extensive experimentation that this method consistently outperforms all existing provably $\ell-2$-robust classifiers by a significant margin on ImageNet and CIFAR-10, establishing the state-of-the-art for provable $\ell_ 2$-defenses. Expand

Scaling provable adversarial defenses

- Computer Science, Mathematics
- NeurIPS
- 2018

This paper presents a technique for extending these training procedures to much more general networks, with skip connections and general nonlinearities, and shows how to further improve robust error through cascade models. Expand

Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

- Computer Science, Mathematics
- NeurIPS
- 2018

From the relationship between the Lipschitz constants and prediction margins, a computationally efficient calculation technique is presented to lower-bound the size of adversarial perturbations that can deceive networks, and that is widely applicable to various complicated networks. Expand

Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

- Computer Science, Mathematics
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- 2019

A new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input that achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10. Expand

Improving DNN Robustness to Adversarial Attacks using Jacobian Regularization

- Computer Science, Mathematics
- ECCV
- 2018

This work suggests a theoretically inspired novel approach to improve the networks' robustness using the Frobenius norm of the Jacobian of the network, which is applied as post-processing, after regular training has finished and demonstrates empirically that it leads to enhanced robustness results with a minimal change in the original network's accuracy. Expand

Stabilizing Training of Generative Adversarial Networks through Regularization

- Computer Science, Mathematics
- NIPS
- 2017

This work proposes a new regularization approach with low computational cost that yields a stable GAN training procedure and demonstrates the effectiveness of this regularizer accross several architectures trained on common benchmark image generation tasks. Expand

Robustness via Curvature Regularization, and Vice Versa

- Computer Science, Mathematics
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2019

It is shown in particular that adversarial training leads to a significant decrease in the curvature of the loss surface with respect to inputs, leading to a drastically more "linear" behaviour of the network. Expand