logo

SCIENCE CHINA Physics, Mechanics & Astronomy, Volume 64 , Issue 11 : 114211(2021) https://doi.org/10.1007/s11433-021-1730-x

High-throughput fast full-color digital pathology based on Fourier ptychographic microscopy via color transfer

More info
  • ReceivedMar 10, 2021
  • AcceptedJun 3, 2021
  • PublishedJul 27, 2021

PACS numbers

Abstract


Funded by

the National Natural Science Foundation of China(Grant,No.,81427802)


Acknowledgment

This work was supported by the National Natural Science Foundation of China (Grant No. 81427802). The authors thank Dr. Jila Rafighdoost from Iran for helpful modifications to this manuscript.


Supplement

Appendix Discussion of the sizes of neighborhood

In experiments, we tested different neighborhood sizes with respect to the CFPM approach, and two representative datasets are shown in Figure a1 with single and double dyes, respectively. We found serious problems when using the 1 × 1 neighborhood. Choosing a pixel of donor image to match a certain pixel of the acceptor image is hard because many pixels have the same gray value. That is, for the acceptor image pixel j, multiple pixels in the donor image satisfy the dyeing condition. The gray value of acceptor images may be wrong because of the noise and ringing effect. Therefore, when using the 1 × 1 neighborhood, we chose the nearest pixel of donor image for a certain pixel of acceptor image, and obvious chromatic aberration could be observed in Figure a1(a1) and (b1). The performances, RMSE, and IHCS results of these two samples for neighborhoods with different sizes are provided in Figure a1(a1)-(a9) and (b1)-(b9), respectively. The RMSE and the IHCS continuously decline before the size of 7 × 7 and the performances gradually improve. An obvious leap can be observed from 1 × 1 to 3 × 3 in Figure a1(a9) and (b9). However, too large a neighborhood selection may cause the images to be too smooth so that the RMSE and the IHCS may rise instead (Figure a1(b9)), which also requires more computation. Therefore, we balanced the pros and cons of the situation and chose the 5 × 5 neighborhood in our CFPM algorithm to resist the noise and ringing effect of coherent images. In addition, we also explored whether or not the neighborhood size is related to the objective lenses. Two objective lenses, 4×/0.1NA and 10×/0.25NA, were tested. According to the performances and the curves in Figure a1(a9) and (b9), the objective lens does not exert a significant effect. Given the nonparametric requirement of the algorithm, we suggest using the 5 × 5 neighborhood for different objectives andthe readers can also choose the neighborhood size they prefer.


References

[1] Burger P. C., Fuller G. N.. Neurol. Clin., 1991, 9: 249 CrossRef Google Scholar

[2] Lin H. J., Chen J. Y., Lin C. F., Kao S. T., Cheng J. C., Chen H. L., Chen C. M.. J. EthnoPharmacol., 2011, 134: 953 CrossRef Google Scholar

[3] Nakano T., Cheng Y. F., Lai C. Y., Hsu L. W., Chang Y. C., Deng J. Y., Huang Y. Z., Honda H., Chen K. D., Wang C. C., Chiu K. W., Jawan B., Eng H. L., Goto S., Chen C. L.. J. Hepatology, 2011, 55: 415 CrossRef Google Scholar

[4] Titford M.. Biotech. HistoChem., 2005, 80: 73 CrossRef Google Scholar

[5] Xu K. Z. Y., Zhu C., Kim M. S., Yamahara J., Li Y.. J. EthnoPharmacol., 2009, 123: 280 CrossRef Google Scholar

[6] Zheng G., Horstmeyer R., Yang C.. Nat. Photon., 2013, 7: 739 CrossRef ADS arXiv Google Scholar

[7] Ou X., Horstmeyer R., Yang C., Zheng G.. Opt. Lett., 2013, 38: 4845 CrossRef ADS arXiv Google Scholar

[8] Pan A., Zuo C., Yao B.. Rep. Prog. Phys., 2020, 83: 096101 CrossRef ADS Google Scholar

[9] Ou X., Zheng G., Yang C.. Opt. Express, 2014, 22: 4960 CrossRef ADS Google Scholar

[10] Pan A., Zhang X. F., Wang B., Zhao Q., Shi Y. S.. Acta Phys. Sin., 2016, 65: 014204 CrossRef Google Scholar

[11] Pan A., Zhou M., Zhang Y., Min J., Lei M., Yao B.. Opt. Commun., 2019, 430: 73 CrossRef ADS Google Scholar

[12] J. García, P. Garcíamartínez, V. Mico, and Z. Zalevsky, J. Opt. Soc. Am. A 23, 3162 (2006). Google Scholar

[13] Horstmeyer R., Ou X., Zheng G., Willems P., Yang C.. Comput.ized Med. Imag. Graph., 2015, 42: 38 CrossRef Google Scholar

[14] Guo K., Liao J., Bian Z., Heng X., Zheng G.. Biomed. Opt. Express, 2015, 6: 3210 CrossRef Google Scholar

[15] Guo C., Bian Z., Jiang S., Murphy M., Zhu J., Wang R., Song P., Shao X., Zhang Y., Zheng G.. Opt. Lett., 2020, 45: 260 CrossRef ADS arXiv Google Scholar

[16] Kim J., Henley B. M., Kim C. H., Lester H. A., Yang C.. Biomed. Opt. Express, 2016, 7: 3097 CrossRef Google Scholar

[17] Pan A., Zhang Y., Wen K., Zhou M., Min J., Lei M., Yao B.. Opt. Express, 2018, 26: 23119 CrossRef ADS arXiv Google Scholar

[18] Chan A. C. S., Kim J., Pan A., Xu H., Nojima D., Hale C., Wang S., Yang C.. Sci. Rep., 2019, 9: 11114 CrossRef ADS Google Scholar

[19] Tian L., Li X., Ramchandran K., Waller L.. Biomed. Opt. Express, 2014, 5: 2376 CrossRef Google Scholar

[20] Tian L., Liu Z., Yeh L. H., Chen M., Zhong J., Waller L.. Optica, 2015, 2: 904 CrossRef ADS arXiv Google Scholar

[21] A. Pan, C. Shen, B. Yao, and C. Yang, in Single-shot Fourier ptychographic microscopy via annular monochrome LED array: OSA Technical Digest (Optical Society of America, Washington D C, 2019), FTh3F.4. Google Scholar

[22] Pan A., Yao B.. Opt. Express, 2019, 27: 5433 CrossRef ADS Google Scholar

[23] Horstmeyer R., Chung J., Ou X., Zheng G., Yang C.. Optica, 2016, 3: 827 CrossRef ADS Google Scholar

[24] Tian L., Waller L.. Optica, 2015, 2: 104 CrossRef ADS Google Scholar

[25] Dong S., Shiradkar R., Nanda P., Zheng G.. Biomed. Opt. Express, 2014, 5: 1757 CrossRef Google Scholar

[26] Dong S., Guo K., Jiang S., Zheng G.. Opt. Express, 2015, 23: 30393 CrossRef ADS Google Scholar

[27] Williams A., Chung J., Ou X., Zheng G., Rawal S., Ao Z., Datar R., Yang C., Cote R.. J. Biomed. Opt, 2014, 19: 066007 CrossRef ADS Google Scholar

[28] Chung J., Martinez G. W., Lencioni K. C., Sadda S. R., Yang C.. Optica, 2019, 6: 647 CrossRef ADS arXiv Google Scholar

[29] Pan A., Wen K., Yao B.. Opt. Lett., 2019, 44: 2032 CrossRef ADS Google Scholar

[30] Xiang M., Pan A., Zhao Y., Fan X., Zhao H., Li C., Yao B.. Opt. Lett., 2021, 46: 29 CrossRef ADS Google Scholar

[31] Zhou Y., Wu J., Bian Z., Suo J., Zheng G., Dai Q.. J. Biomed. Opt, 2017, 22: 066006 CrossRef ADS Google Scholar

[32] Pan A, Wang D, Shi Y. S., Yao B. L., Ma Z., Han Y.. Acta Phys. Sin., 2016, 65: 124201 CrossRef Google Scholar

[33] Sun J., Chen Q., Zhang Y., Zuo C.. Opt. Express, 2016, 24: 15765 CrossRef ADS Google Scholar

[34] Wang M., Zhang Y., Chen Q., Sun J., Fan Y., Zuo C.. Opt. Commun., 2017, 405: 406 CrossRef ADS Google Scholar

[35] Rivenson Y., Wang H., Wei Z., de Haan K., Zhang Y., Wu Y., Günaydın H., Zuckerman J. E., Chong T., Sisk A. E., Westbrook L. M., Wallace W. D., Ozcan A.. Nat. Biomed. Eng., 2019, 3: 466 CrossRef Google Scholar

[36] Zhang Y., de Haan K., Rivenson Y., Li J., Delis A., Ozcan A.. Light. Sci. Appl., 2020, 9: 78 CrossRef ADS arXiv Google Scholar

[37] Nguyen T., Xue Y., Li Y., Tian L., Nehmetallah G.. Opt. Express, 2018, 26: 26470 CrossRef ADS arXiv Google Scholar

[38] Xue Y., Cheng S., Li Y., Tian L.. Optica, 2019, 6: 618 CrossRef ADS arXiv Google Scholar

[39] Ruderman D. L., Cronin T. W., Chiao C. C.. J. Opt. Soc. Am. A, 1998, 15: 2036 CrossRef ADS Google Scholar

[40] T. Wlesh, M. Ashikhmin, and K. Mueller, ACM Trans. Graphics 21, 277 (2002). Google Scholar

[41] K. Tan, and J. Oakley, in Enhancement of color images in poor visibility conditions: Proceedings of the IEEE 2000 International Conference on Image Processing (ICIP), Vancouver, 2000, pp. 788-791. Google Scholar

[42] Dan D., Lei M., Yao B., Wang W., Winterhalder M., Zumbusch A., Qi Y., Xia L., Yan S., Yang Y., Gao P., Ye T., Zhao W.. Sci. Rep., 2013, 3: 1116 CrossRef ADS Google Scholar

[43] Y. Zhang, A. Pan, M. Lei, and B. Yao, Opt. Eng. 56, 123107 (2017). Google Scholar

[44] Pan A., Zhang Y., Zhao T., Wang Z., Dan D., Lei M., Yao B.. J. Biomed. Opt., 2017, 22: 1 CrossRef ADS arXiv Google Scholar

[45] Pan A., Zuo C., Xie Y., Lei M., Yao B.. Opt. Lasers Eng., 2019, 120: 40 CrossRef ADS Google Scholar

[46] R. C. Gonzalez, and R. E. Woods, Digital Image Processing, 3rd ed. (Prentice Hall, Upper Saddle River, 2007). Google Scholar

  • Figure a1

    (Color online) Discussion of the sizes of neighborhood. (a), (b) Ground truth of two samples with two and single dyes. (a1)-(a8), (b1)-(b8) Results with different neighborhood sizes and different objectives, respectively. (a9), (b9) Curves of RMSE and the IHCS with respect to the neighborhood size under different objective lens.

  • Figure 1

    (Color online) Importance of true colorization of grayscale images. (a) Grayscale image of a cell section. (b) Pseudo-color image. (c) True-color image of the cell section. The types of cells in (b) are difficult to distinguish according to our visual habits compared with (c).

  • Figure 2

    (Color online) Inspiration of color matching and principle of the CFPM approach. (a) Match color: transfer the hue of the donor image to the acceptor image; (b) color transfer: transfer the color texture of the donor image to the acceptor image.

  • Figure 3

    (Color online) Schematic of CFPM. (a) Donor image: LR full-color image with the same FOV of FPM captured by the same objective; (b) acceptor image: FPM recovery with a single wavelength; (c) CFPM final recovery.

  • Figure 4

    (Color online) CFPM setup with the illumination of LED board. Schematic (a) and its experimental photograph (b). (a1) A 32 × 32 programmable R/G/B LED matrix. (a2) Enlargement of a compact inverted microscope with a light path diagram. It can be simply modified to realize multimodal imaging (a3). MO: microscope objective; TL: tube lens; M1 and M2: mirrors; BS: beam splitter; DM1 and DM2: dichroic mirror. (c) Standard color gamut chart of 1931 CIE-XYZ and our color space of FPM (blue triangle). (c1)-(c4) Wavelength and intensity calibration. He-Ne laser: 5.6 nm deviation tested by the spectrometer of Ocean Optics. Red LED: 630.1 nm of center wavelength, 20.8 nm of FWHM. Green LED: 515.0 nm of center wavelength, 38.0 nm of FWHM. Blue LED: 462.6 nm of center wavelength, 34.6 nm of FWHM.

  • Figure 5

    (Color online) Results of stained resting sporangia. (a) LR donor image with the entire FOV of a 4×/0.1NA objective. (b) FPM recovery image under green channel (515.0 nm). (a1) Tile of (a). (b1)-(b3) Corresponding close-ups of FPM recovery images under red, green, and blue channels, respectively. (c) Ground truth captured by a 10×/0.3NA objective. (c1)-(c3) Staining results with different acceptor images via CFPM. (d1)-(d3) Different maps between the ground truth and CFPM results, respectively. The RMSE and the IHCS are given below the results of CFPM, respectively. Among them, the best performance for stained resting sporangia is under the green channel with an RMSE of 5.37% and an IHCS of 11.5737°.

  • Figure 6

    (Color online) Results of stained lily bud cells. (a) LR donor image with the entire FOV of a 4×/0.1NA objective. (b) FPM recovery image under the red channels (630.1 nm). (a1) A tile of (a). (b1)-(b3) Corresponding close-ups of FPM recovery images under red, green, and blue channels, respectively. (c) Ground truth captured by a 10×/0.3NA objective. (c1)-(c3) Staining results with different acceptor images via CFPM. (d1)-(d3) Different maps between the ground truth and CFPM results. The RMSE and the IHCS are given below the results of CFPM. Among them, the best performance for lily bud cells is under the red channel with an RMSE of 4.83% and an IHCS of 17.9055°. Comparison of Figures 5 and 6 reveal that the CFPM approach has wavelength selectivity, which depends on dye absorption and can be easily obtained as a priori knowledge according to its chemical property.

  • Figure 7

    (Color online) Comparisons between three types of FPM colorization results. (a) RMSE curves of conventional R+G+B, our CFPM, and WMFPM with 30 pathologic or biological samples at a statistical level. The difference between the results of the conventional R+G+B method and CFPM is less than 1%, whereas WMFPM is worse than both methods with an average level of 11.85%, roughly twice that of RMSE of the former two schemes. (b1)-(b3) Ground truth of the No. 24 (two dyes), No. 7 (single dye), and No. 12 (single dye) stained biological samples, respectively. (c1)-(e1), (c2)-(e2), (c3)-(e3) Recovery result was obtained by CFPM, conventional R+G+B, and WMFPM, respectively. RMSEs are given below the corresponding results. (f1)-(f3) Color histograms with different methods, where the numbers in the brackets of legend are the IHCS results. Among the results of 30 images, unexpectedly, 43% of CFPM results are better than the results of the conventional method. The CFPM approach is less affected by chromatic aberration because of the reconstruction with a single wavelength only (red box in (c1)-(e1)). If the proportion of background is much larger than the sample, then the CFPM approach would be much better than the conventional method ((b2)-(f2) and (b3)-(f3)). Different from the sample Nos. 7 and 12 stained with a single dye, sample No. 24 is stained with two dyes for the specific recognition of different parts. The CFPM may not be able to obtain the right color (blue box in (c1)-(e1)) because different dyes may have different absorption levels. Therefore, currently, the CFPM method is better for stained samples with a single dye.

  • Figure 8

    (Color online) Color transfer of different acceptor images using an HR donor image. (a) An HR donor image captured by a higher NA and magnification objective. Therefore, its FOV is much smaller. (b)-(d) Three random tiles as the acceptor images were captured by the green channels. (b1)-(d1) show the corresponding results of this type of color transfer method. (b2)-(d2) represent the corresponding ground truth. (b3)-(d3) show the difference maps between the ground truth and this kind of color transfer results. RMSE and IHCS are given below the results. The results are not stable and worse than our CFPM method. It also has a double dye problem. Therefore, the double dye problem in our CFPM method is not caused by the poor resolution of donor images.

qqqq

Contact and support