SCIENCE CHINA Information Sciences, Volume 64 , Issue 2 : 122401(2021) https://doi.org/10.1007/s11432-020-2998-1

Optoelectronic convolutional neural networks based on time-stretch method

More info
  • ReceivedMay 14, 2020
  • AcceptedJul 17, 2020
  • PublishedJan 20, 2021



This work was supported by National Key Research and Development Program of China (Grant No. 2019YFB1803501), National Natural Science Foundation of China (Grant No. 61771284), and Beijing Natural Science Foundation (Grant No. L182043).


Appendix A for lists of important variables and notations used in the article, Appendix B for the detailed generation of the the signals for modulation in each layer, Appendix C for the lists of feature maps extracted by electronical implemented CNN and TS-CNN under different relative noise levels, and Appendix D for the estimation of time consumption in TS-CNN.


[1] Garg R, Bg V K, Carneiro G, et al. Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Proceedings of European Conference on Computer Vision, 2016. 740--756. Google Scholar

[2] Sercu T, Puhrsch C, Kingsbury B, et al. Very deep multilingual convolutional neural networks for LVCSR. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016. Google Scholar

[3] Prabhakar G, Kailath B, Natarajan S, et al. Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. In: Proceedings of IEEE Region 10 Symposium (TENSYMP), 2017. Google Scholar

[4] Al-Qizwini M, Barjasteh I, Al-Qassab H, et al. Deep learning algorithm for autonomous driving using GoogLeNet. In: Proceedings of IEEE Intelligent Vehicles Symposium (IV), 2017. Google Scholar

[5] Stefaniga S A, Gaianu M. Face detection and recognition methods using deep learning in autonomous driving. In: Proceedings of the 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), 2018. Google Scholar

[6] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60: 84-90 CrossRef Google Scholar

[7] Schaller R R. Moore's law: past, present and future. IEEE Spectr, 1997, 34: 52-59 CrossRef Google Scholar

[8] Shen Y, Harris N C, Skirlo S. Deep learning with coherent nanophotonic circuits. Nat Photon, 2017, 11: 441-446 CrossRef ADS arXiv Google Scholar

[9] Lin X, Rivenson Y, Yardimci N T. All-optical machine learning using diffractive deep neural networks. Science, 2018, 361: 1004-1008 CrossRef ADS arXiv Google Scholar

[10] Paquot Y, Duport F, Smerieri A. Optoelectronic Reservoir Computing. Sci Rep, 2012, 2: 287 CrossRef ADS arXiv Google Scholar

[11] Larger L, Soriano M C, Brunner D. Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. Opt Express, 2012, 20: 3241-3249 CrossRef ADS Google Scholar

[12] Feldmann J, Youngblood N, Wright C D. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature, 2019, 569: 208-214 CrossRef ADS Google Scholar

[13] Bueno J, Maktoobi S, Froehly L. Reinforcement learning in a large-scale photonic recurrent neural network. Optica, 2018, 5: 756-760 CrossRef ADS arXiv Google Scholar

[14] Qian C, Lin X, Lin X. Performing optical logic operations by a diffractive neural network. Light Sci Appl, 2020, 9: 59 CrossRef ADS Google Scholar

[15] Hughes T W, Minkov M, Shi Y. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica, 2018, 5: 864-871 CrossRef ADS arXiv Google Scholar

[16] Hengameh B, Scott S, Yichen S, et al. On-chip optical convolutional neural networks. 2018,. arXiv Google Scholar

[17] Mahjoubfar A, Churkin D V, Barland S. Time stretch and its applications. Nat Photon, 2017, 11: 341-351 CrossRef ADS Google Scholar

[18] Ng W, Rockwood T D, Sefler G A. Demonstration of a Large Stretch-Ratio (M=41) Photonic Analog-to-Digital Converter With 8 ENOB for an Input Signal Bandwidth of 10 GHz. IEEE Photon Technol Lett, 2012, 24: 1185-1187 CrossRef ADS Google Scholar

[19] Wong J H, Lam H Q, Li R M. Photonic Time-Stretched Analog-to-Digital Converter Amenable to Continuous-Time Operation Based on Polarization Modulation With Balanced Detection Scheme. J Lightwave Technol, 2011, 29: 3099-3106 CrossRef ADS Google Scholar

[20] Valley G C. Photonic analog-to-digital converters. Opt Express, 2007, 15: 1955-1982 CrossRef ADS Google Scholar

[21] Chen C L, Mahjoubfar A, and Jalali B. Optical Data Compression in Time Stretch Imaging. PLoS One, 2015, 10(4): 1-11, doi: 10.1007/978-3-319-51448-2_9. Google Scholar

[22] Goda K, Tsia K K, Jalali B. Serial time-encoded amplified imaging for real-time observation of fast dynamic phenomena. Nature, 2009, 458: 1145-1149 CrossRef ADS Google Scholar

[23] Goda K, Ayazi A, Gossett D R. High-throughput single-microparticle imaging flow analyzer. Proc Natl Acad Sci USA, 2012, 109: 11630-11635 CrossRef ADS Google Scholar

[24] Xing F, Chen H, Lei C. A 2-GHz discrete-spectrum waveband-division microscopic imaging system. Optics Commun, 2015, 338: 22-26 CrossRef ADS Google Scholar

[25] Zang Y, Chen M, Yang S. Electro-Optical Neural Networks Based on Time-Stretch Method. IEEE J Sel Top Quantum Electron, 2020, 26: 1-10 CrossRef ADS arXiv Google Scholar

[26] Goda K, Solli D R, Tsia K K. Theory of amplified dispersive Fourier transformation. Phys Rev A, 2009, 80: 043821 CrossRef ADS Google Scholar

[27] Plancherel M, Leffler M. Contribution à ?étude de la représentation D'une fonction arbitraire par des intégrales définies. Rend Circ Matem Palermo, 1910, 30: 289-335 CrossRef Google Scholar

[28] Palm R B. MNIST_recognization done by BP networks. 2019. https://github.com/rasmusbergpalm/DeepLearnToolbox. Google Scholar

[29] Palm R B. Prediction as a candidate for learning deep hierarchical models of data. Dissertation for M.Sc. Degree. Copenhagen: Technical University of Denmark, 2012. Google Scholar

[30] Lecun Y, Cortes C, Burges C J C. The mnist database of handwritting digits. 2019. http://yann.lecun.com/exdb/mnist/. Google Scholar

[31] Kato K, Tohmori Y. PLC hybrid integration technology and its application to photonic components. IEEE J Sel Top Quantum Electron, 2000, 6: 4-13 CrossRef ADS Google Scholar

[32] Davey R P, Smith K, McGuire A. High-speed, mode-locked, tunable, integrated erbium fibre laser. Electron Lett, 1992, 28: 482 CrossRef Google Scholar

[33] Zhang D, Feng X, Huang Y. Tunable and Reconfigurable Bandpass Microwave Photonic Filters Utilizing Integrated Optical Processor on Silicon-on-Insulator Substrate. IEEE Photon Technol Lett, 2012, 24: 1502-1505 CrossRef ADS Google Scholar

[34] Kensuke O. High-speed silicon-based integrated optical modulators for optical-fiber telecommunications. In: Proceedings of the International Society for Optical Engineering, 2014. Google Scholar

[35] Miscuglio M, Mehrabian A, Hu Z. All-optical nonlinear activation function for photonic neural networks [Invited]. Opt Mater Express, 2018, 8: 3851 CrossRef ADS arXiv Google Scholar

[36] Selden A C. Pulse transmission through a saturable absorber. Br J Appl Phys, 1967, 18: 743-748 CrossRef ADS Google Scholar

[37] Bao Q, Zhang H, Ni Z. Monolayer graphene as a saturable absorber in a mode-locked laser. Nano Res, 2011, 4: 297-307 CrossRef Google Scholar

[38] Lim G K, Chen Z L, Clark J. Giant broadband nonlinear optical absorption response in dispersed graphene single sheets. Nat Photon, 2011, 5: 554-560 CrossRef ADS Google Scholar

[39] Hu X, Wang A, Zeng M, et al. Graphene-assisted multiple-input high-base optical computing. Sci Rep, 2016, 6:32911, doi: 10.1109/ICOCN.2016.7875616. Google Scholar

[40] Yadav R K, Aneesh J, Sharma R. Designing Hybrids of Graphene Oxide and Gold Nanoparticles for Nonlinear Optical Response. Phys Rev Appl, 2018, 9: 044043 CrossRef ADS Google Scholar

  • Figure 1

    (Color online) Structure of TS-CNN system. Light pulses propagate through the loop-shaped structure to implement linear computations in CNN. Index in circles represents important nodes whose signals will be shown in Subsection 3.2. Note that EDFA(s)1 represents one kind of untunable EDFA(s).

  • Figure 2

    (Color online) Structure of theoretical CNN model. It includes two convolution layers, two mean pooling layers and one full-connected layer. The sample `0' from the training set is used as an example to show all procedures.

  • Figure 3

    (Color online) Illustration of convolution operations implemented by broadened pulses in TS-CNN. All numbers and alphabets only represent pixel order.

  • Figure 4

    (Color online) Important signals of the last layer of TS-CNN system. (a) Pulses generated by mode-locked laser [\textcircled{1}]; (b) broadened pulses after pulse broaden component [\textcircled{2}]; (c) pulses after two intensity modulators [\textcircled{3}]; (d) zoomed-in picture of modulated pulse [\textcircled{3}]; (e) compressed pulses after compression component [\textcircled{4}]; (f) accumulated signals after PD [\textcircled{5}]. Note that index of circles in the square brackets represent the corresponding nodes in Figure 1. This picture depicts the convolution operations between the first kernel with input picture in convolution layer1 in TS-CNN system.

  • Figure 5

    (Color online) Confusion matrices of testing of TS-CNN. (a) The trend of accuracy of handwriting digit recognition with regard to the gain of adjustable EDFA2 in the optoelectronic TS-CNN system. Detailed information of confusion matrix of each data point can be seen in Figure 5(d)–(i). The numbers in circle mark the data points which are shown in Figure 5(a) with their corresponding confusion matrices. (b) Test of theoretical CNN by all pictures from MNIST test set. (c) Test of theoretical CNN by 100 pictures from MNIST test set. (d) Test of noise free TS-CNN [{\textcircled{1}}]. (e) Test of TS-CNN with the gain of EDFA2 equals 20 dB [{\textcircled{2}}]. (f) Test of TS-CNN with the gain of EDFA2 equals 17 dB [{\textcircled{3}}]. (g) Test of TS-CNN with the gain of EDFA2 equals 15 dB [{\textcircled{4}}]. (h) Test of TS-CNN with the gain of EDFA2 equals 12 dB [{\textcircled{5}}]. (i) Test of TS-CNN with the gain of EDFA2 equals 10 dB [{\textcircled{6}}].