logo

SCIENTIA SINICA Informationis, Volume 51 , Issue 4 : 602(2021) https://doi.org/10.1360/SSI-2019-0205

Instability analysis for generative adversarial networks and its solving techniques

More info
  • ReceivedNov 14, 2019
  • AcceptedMar 1, 2020
  • PublishedNov 20, 2020

Abstract


Funded by

国家自然科学基金(61732019)


References

[1] Goodfellow I, Pougetabadie J, Mirza M, et al. Generative adversarial nets. In: Proceedings of the 28th Conference on Neural Information Processing Systems (NeurIPS), Montréal, 2014. 2672--2680. Google Scholar

[2] Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, 2016. Google Scholar

[3] Mao X D, Li Q, Xie H R, et al. Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, 2017. Google Scholar

[4] Karras T, Aila T, Laine S, et al. Progressive growing of GANs for improved quality, stability, and variation. In: Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, 2018. Google Scholar

[5] Brock A, Donahua J, Simonyan K. Large scale GANs training for high fidelity natural image synthesis. In: Proceedings of the 7th International Conference on Learning Representations (ICLR), Louisiana, 2019. Google Scholar

[6] Luc P, Couprie C, Chintala S, et al. Semantic segmentation using adversarial networks. In: Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS), Barcelona, 2016. Google Scholar

[7] Zhu W T, Xiang X, Tran T D, et al. Adversarial deep structural networks for mammographic mass segmentation. In: Proceedings of the 15th IEEE International Symposium on Biomedical Imaging, Washington, 2018. Google Scholar

[8] Arjovsky M, Bottou L. Towards principled methods for training generative adversarial networks. In: Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, 2017. Google Scholar

[9] Wiatrak M, Albrecht S V. Stabilizing generative adversarial network training: a survey. 2019,. arXiv Google Scholar

[10] Durall R, Pfreundt F, Keuper J. Stabilizing GANs with Octave Convolutions. 2019,. arXiv Google Scholar

[11] Grey R. Entropy and Information Theory. Berlin: Springer, 2013. Google Scholar

[12] Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), Sydney, 2017. Google Scholar

[13] Nagarajan V, Kolter J Z. Gradient descent GAN optimization is locally stable. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), California, 2017. Google Scholar

[14] Mescheder L, Geiger A, Nowozin S. Which training methods for GANs do actually converge? In: Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, 2018. Google Scholar

[15] Sharma M, Makwana M, Upadhyay A, et al. Robust image colorization using self attention based progressive generative adversarial network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Califonia, 2019. Google Scholar

[16] Salimans T, Goodfellow I, Zaremba W, et al. Improved techniques for training GANs. In: Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS), Barcelona, 2016. Google Scholar

[17] Heusel M, Ramsauer H, Unterthiner T, et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), California, 2017. Google Scholar

[18] Metz L, Poole B, Pfau D, et al. Unrolled generative adversarial networks. In: Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, 2017. Google Scholar

[19] Su Y X, Zhao S L, Chen X X, et al. Parallel wasserstein generative adversarial nets with multiple discriminators. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), Macao, 2019. Google Scholar

[20] Dam N, Hoang Q, Le T, et al. Three-player wasserstein GAN via amortised duality. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), Macao, 2019. Google Scholar

[21] Liu H D, Gu X F, Samaras D. Wasserstein GAN with quadratic transport cost. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), seoul, 2019. Google Scholar

[22] Miyato T, Kataoka T, Koyama M, et al. Spectral normalization for generative adversarial networks. In: Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), Montréal, 2018. Google Scholar

[23] Zhang H, Zhang Z Z, Odena A, et al. Consistency regularization for generative adversarial networks. 2019,. arXiv Google Scholar

[24] Kurach K, Mario L, Zhai X H, et al. A large-scale study on regularization and normalization in GANs. In: Proceedings of the 36th International Conference on Machine Learning (ICML), Long Beach, 2019. Google Scholar

[25] Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of Wasserstein GANs. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), California, 2017. Google Scholar

[26] Thanh-Tung H, Tran T, Venkatesh S. Improving generalization and stability of generative adversarial networks. In: Proceedings of the 7th International Conference on Learning Representations (ICLR), Louisiana, 2019. Google Scholar

[27] Petzka H, Fischer A, Lukovnicov D. On the regularization of wasserstein GANs. In: Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, 2018. Google Scholar

[28] Roth K, Lucchi A, Nowozin S, et al. Stabilizing training of generative adversarial networks through regularization. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), California, 2017. Google Scholar

[29] Edraki M, Qi G J. Generalized loss-sensitive adversarial learning with manifold margins. In: Proceedings of the 15th European Conference on Computer Vision (ECCV), Munich, 2018. Google Scholar

[30] Dudley R M. Real Analysis and Probability. Cambridge: Cambridge University Press, 2004. Google Scholar

[31] Fedus W, Rosca M, Lakshminarayanan B, et al. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. In: Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, 2018. Google Scholar

[32] Biau G, Cadre B, Sangnier M, et al. Some theoretical properties of GANs. 2018,. arXiv Google Scholar

[33] Qin Y P, Mitra N, Wonka P. How does Lipschitz regularization influence GANs training? 2019,. arXiv Google Scholar

[34] Handel R V. Probability in high dimension. 2016. https://web.math.princeton.edu/~rvan/APC550.pdf. Google Scholar

[35] Paszke, Gross A, Chintala S, et al. Automatic differentiation in PyTorch. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS) Workshop, California, 2017. Google Scholar

[36] Krizhevsky A. Learning multiple layers of features from tiny images. Dissertation for M.S. Degree. Toronto: University of Toronto, 2009. Google Scholar

[37] Netzer Y, Wang T, Coates A, et al. Reading digits in natural images with unsupervised feature learning. In: Proceedings of the 25th Conference on Neural Information Processing Systems (NeurIPS), Granada, 2011. Google Scholar

[38] Coates A, Lee H, Andrew Y. An analysis of single layer networks in unsupervised feature learning. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), Florida, 2011. Google Scholar

[39] Russakovsky O, Deng J, Su H. ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis, 2015, 115: 211-252 CrossRef Google Scholar

[40] Liu Z W, Luo P, Wang X G, et al. Deep learning face attributes in the wild. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), Santiago, 2015. Google Scholar

[41] Yu F, Seff A, Zhang Y D, et al. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop. 2015,. arXiv Google Scholar

[42] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, 2015. Google Scholar

[43] Saxe A, Mcclelland L, Ganguli S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In: Proceedings of the 2nd International Conference on Learning Representations (ICLR), Banff, 2014. Google Scholar

[44] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), Sardinia, 2010. Google Scholar

[45] Daskalakis C, Ilyas A, Syrgkanis V, et al. Training GANs with optimism. In: Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, 2018. Google Scholar

  • Figure 1

    (Color online) Different $L$ value and its corresponding loss function variation when $\lambda=10$ (CIFAR-10).protectłinebreak (a) Discriminator loss $D$_loss; (b) generator loss $G$_loss

  • Figure 2

    (Color online) Comparison of the loss function variation and the general variation trend of gradient between two penalty techniques. (a) The variation of loss function; (b) the general variation trend of gradient

  • Figure 3

    (Color online) The variation of loss function in three algorithms. (a) The variation of loss function in discriminator (critic function); (b) the variation of loss function in generator

  • Figure 4

    (Color online) The general variation trend of gradient in three algorithms. (a) The general variation trend of gradient in discriminator (critic function); (b) the general variation trend of gradient in generator

  • Table 1   Different $\lambda$ and its corresponding $\rm~FID$ value when $L=0$ (CIFAR-10)
    $\lambda$ 5 10 20 50 100
    $\rm~FID$ 15.41 12.91 16.28 34.61 32.82
  • Table 2   Different $L$ and its corresponding $\rm~FID$ value when $\lambda=10$ (CIFAR-10)
    $L$ 10 5 1 0.5 0
    $\rm~FID$ 18.59 18.56 16.47 15.40 12.91
  • Table 3   Algorithms performance comparison (The smaller the FID value, the better)
    CIFAR-10 SVHN STL-10 CelebA LSUN (bedroom)
    LSGAN (2017) 22.20 3.84 20.17 5.10 5.23
    SNGAN (2018) 20.70 4.53 18.11 5.56 12.05
    WGAN-GP (2017) 21.89 4.09 18.19 5.01 14.61
    WGAN-LP (2018) 21.01 3.62 17.40 5.12 15.21
    GAN-0GP (2019) 18.91 6.10 14.49 4.53 7.14
    Ours 12.91 2.78 12.85 3.91 6.85
    Real datasets 0.46 0.24 0.84 0.34 0.55
qqqq

Contact and support