logo

SCIENTIA SINICA Informationis, Volume 51 , Issue 6 : 959(2021) https://doi.org/10.1360/SSI-2020-0100

Super-resolution reconstruction of MR image with self-attention based generate adversarial network algorithm

More info
  • ReceivedApr 21, 2020
  • AcceptedJun 24, 2020
  • PublishedMay 11, 2021

Abstract


Funded by

国家自然科学基金(61672466,620115300130)

浙江省自然科学基金–数理医学学会联合基金重点项目(LSZ19F010001)

浙江省科技厅重点研发项目(2020C03060)

浙江省科技厅公益项目(2015C31075)

浙江省自然科学基金项目(LY18D060009)


Author information




References

[1] Wang X Y, Li D. Application of generative countermeasure network in super-resolution image reconstruction. Journal of Frontiers of Computer Science and Technology, 2020, 14(04): 680-687. Google Scholar

[2] Wang Y, Zhao R, Hu S. The algorithm of image super-resolution reconstruction via separable dictionaries. Sci Sin-Inf, 2020, 50: 275-288 CrossRef Google Scholar

[3] Gholipour A, Estroff J A, Warfield S K. Robust Super-Resolution Volume Reconstruction From Slice Acquisitions: Application to Fetal Brain MRI. IEEE Trans Med Imag, 2010, 29: 1739-1758 CrossRef Google Scholar

[4] Zeng W, Peng J, Wang S, et al. A Comparative Study of CNN-Based Super-Resolution Methods in MRI Reconstruction. In: Proceedings of the 16th International Symposium on Biomedical Imaging, Venezia, 2019. 1678--1682. Google Scholar

[5] Wang S, Cheng H, Ying L. DeepcomplexMRI: Exploiting deep residual network for fast parallel MR imaging with complex convolution. Magn Reson Imag, 2020, 68: 136-147 CrossRef Google Scholar

[6] Wang S, Ke Z, Cheng H, et al. DIMENSION: Dynamic MR imaging with both k-space and spatial prior knowledge obtained via multi-supervised network training. NMR in Biomedicine, 2019, e4131,. arXiv Google Scholar

[7] Wang S, Tan S, Gao Y. Learning Joint-Sparse Codes for Calibration-Free Parallel MR Imaging. IEEE Trans Med Imag, 2018, 37: 251-261 CrossRef Google Scholar

[8] Manjón J V, Coupé P, Buades A. Non-local MRI upsampling. Med Image Anal, 2010, 14: 784-792 CrossRef Google Scholar

[9] Jafari-Khouzani K. MRI upsampling using feature-based nonlocal means approach. IEEE Trans Med Imaging, 2014, 33: 1969-1985. Google Scholar

[10] Rueda A, Malpica N, Romero E. Single-image super-resolution of brain MR images using overcomplete dictionaries. Med Image Anal, 2013, 17: 113-132 CrossRef Google Scholar

[11] Tourbier S, Bresson X, Hagmann P. An efficient total variation algorithm for super-resolution in fetal brain MRI with adaptive regularization. NeuroImage, 2015, 118: 584-597 CrossRef Google Scholar

[12] Dong C, Loy C C, He K. Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans Pattern Anal Mach Intell, 2016, 38: 295-307 CrossRef Google Scholar

[13] Shi J, Li Z, Ying S, et al. Image super-resolution via wide residual networks with fixed skip connection. IEEE Biomed Health Informat, 2018, 23: 1129-1140. Google Scholar

[14] Mahmoudzadeh A P, Kashou N H. Interpolation-based super-resolution reconstruction: effects of slice thickness. J Med Imag, 2014, 1: 034007 CrossRef Google Scholar

[15] Luo J, Mou Z, Qin B, et al. Fast single image super-resolution using estimated low-frequency k-space data in MRI. Magn Reson Imaging, 2017, 40: 1-11. Google Scholar

[16] Shi J, Liu Q, Wang C. Super-resolution reconstruction of MR image with a novel residual learning network algorithm. Phys Med Biol, 2018, 63: 085011 CrossRef ADS Google Scholar

[17] Jog A, Carass A, Prince JL. Improving magnetic resonance resolution with supervised learning. In: Proceedings of the 11th International Symposium on Biomedical Imaging, Beijing, 2014. 987--990. Google Scholar

[18] Shi W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, 2016. 1874--1883. Google Scholar

[19] Seeliger K, Güclü U, Ambrogioni L, et al. Generative adversarial networks for reconstructing natural images from brain activity. Neuro Image, 2018, 181: 775-785. Google Scholar

[20] Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, 2017. 4700--4708. Google Scholar

[21] Armanious K, Gatidis S, Nikolaou K, et al. Retrospective correction of Rigid and Non-Rigid MR motion artifacts using GANs. In: Proceedings of the IEEE 16th International Symposium on Biomedical Imaging, Venezia, 2019. 1550--1554. Google Scholar

[22] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets. In: Proceedings of the Neural Information Processing Systems, Montreal, 2014. 2672--2680. Google Scholar

[23] Jin W, Yang P, Tang P. Double discriminator generative adversarial networks and their application in detecting nests built in catenary and semisupervized learning. Sci Sin-Inf, 2018, 48: 888-902 CrossRef Google Scholar

[24] Quan T M, Nguyen-Duc T, Jeong W K. Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic Loss. IEEE Trans Med Imag, 2018, 37: 1488-1497 CrossRef Google Scholar

[25] Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. 4681--4690. Google Scholar

[26] Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, 2017. 4700--4708. Google Scholar

[27] Liu H, Xu J, Wu Y. Learning deconvolutional deep neural network for high resolution medical image reconstruction. Inf Sci, 2018, 468: 142-154 CrossRef Google Scholar

[28] Latif S, Asim M, Usman M, et al. Automating motion correction in multishot MRI using generative adversarial networks. 2018,. arXiv Google Scholar

[29] Dong C, Loy C C, He K, et al. Learning a deep convolutional network for image super-resolution. In: Proceedings of the European Conference on Computer Vision, Switzerland, 2014. 184--199. Google Scholar

[30] Zhang H, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks. 2018,. arXiv Google Scholar

[31] Ramachandran P, Zoph B, Le Q V. Searching for Activation Functions. 2017,. arXiv Google Scholar

[32] Miyato T, Kataoka T, Koyama M, et al. Spectral normalization for generative adversarial networks. 2018,. arXiv Google Scholar

[33] Iandola F N, Moskewicz M W, Ashraf K, et al. Firecaffe: near-linear acceleration of deep neural network training on compute clusters. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. 2592--2600. Google Scholar

  • Figure 1

    (Color online) Generator model (a) and discriminator model (b)

  • Figure 1

    (Color online) Generator model (a) and discriminator model (b)

  • Figure 4

    (Color online) The reconstructed super resolution cardiac MR images by using different GAN based methods. (a) The real high resolution MR image, and the reconstructed super-resolution MRI using (b) SR-GAN, PSNR = 31.27, SSIM = 0.9947, (c) SR-WGAN, PSNR = 33.58, SSIM = 0.9953, (d) SA-SR-GAN, PSNR = 35.46, SSIM = 0.9962, protectłinebreak (e) SA-SR-GAN(SN), PSNR = 36.16, SSIM = 0.9977

  • Figure 4

    (Color online) The reconstructed super resolution cardiac MR images by using different GAN based methods. (a) The real high resolution MR image, and the reconstructed super-resolution MRI using (b) SR-GAN, PSNR = 31.27, SSIM = 0.9947, (c) SR-WGAN, PSNR = 33.58, SSIM = 0.9953, (d) SA-SR-GAN, PSNR = 35.46, SSIM = 0.9962, protectłinebreak (e) SA-SR-GAN(SN), PSNR = 36.16, SSIM = 0.9977

  • Figure 6

    (Color online) The reconstructed super resolution clinical brain MR images by using different GAN based methods. (a) The real high resolution MR image, and the reconstructed super-resolution MRI using (b) SR-GAN, PSNR = 29.52, SSIM = 0.9872, (c) SR-WGAN, PSNR = 29.87, SSIM = 0.9905, (d) SA-SR-GAN, PSNR = 30.17, SSIM = 0.9912, (e) SA-SR-GAN(SN), PSNR = 30.96, SSIM = 0.9931

  • Figure 6

    (Color online) The reconstructed super resolution clinical brain MR images by using different GAN based methods. (a) The real high resolution MR image, and the reconstructed super-resolution MRI using (b) SR-GAN, PSNR = 29.52, SSIM = 0.9872, (c) SR-WGAN, PSNR = 29.87, SSIM = 0.9905, (d) SA-SR-GAN, PSNR = 30.17, SSIM = 0.9912, (e) SA-SR-GAN(SN), PSNR = 30.96, SSIM = 0.9931

  • Table 1   Average PSNR, SSIM under different methods (cardiac)
    rrrr
    SR-GANSR-WGANSA-SR-GANSA-SR-GAN(SN)
    PSNR SSIM PSNRSSIM PSNR SSIMPSNR SSIM
    $X2~$ 33.73 0.9957 33.98 0.9968 35.66 0.9962 34.70 0.9965
    $X4~$ 32.30 0.9946 33.22 0.9951 34.09 0.9959 37.58 0.9979
    $X8~$ 31.35 0.9951 32.52 0.9943 33.45 0.9956 35.60 0.9962
  • Table 2   Average PSNR, SSIM under different methods (brain)
    rrrr
    SR-GANSR-WGANSA-SR-GANSA-SR-GAN(SN)
    PSNR SSIM PSNRSSIM PSNR SSIMPSNR SSIM
    $X2~$ 43.35 0.9989 43.76 0.9990 43.97 0.9991 44.23 0.9993
    $X4~$ 39.88 0.9955 40.37 0.9961 41.76 0.9970 42.76 0.9976
    $X8~$ 30.83 0.8891 30.97 0.8902 31.02 0.8907 31.59 0.8922
  • Table 3   Average PSNR, SSIM under different methods (clinical data)
    rrrr
    SR-GANSR-WGANSA-SR-GANSA-SR-GAN(SN)
    PSNR SSIM PSNRSSIM PSNR SSIMPSNR SSIM
    $X2~$ 34.21 0.9963 35.43 0.9964 36.28 0.9966 37.72 0.9968
    $X4~$ 28.89 0.9903 29.34 0.9908 30.13 0.9913 30.85 0.9920
    $X8~$ 26.31 0.9722 27.67 0.9824 28.22 0.9842 28.71 0.9869
  • Table 4   Parameters by using different methods
    Method Parameters (M)
    SR-GAN 1.55
    SR-WGAN 1.57
    SA-SR-GAN 1.58
    SA-SR-GAN(SN) 1.60
qqqq

Contact and support