logo

SCIENCE CHINA Life Sciences, https://doi.org/10.1007/s11427-021-1946-0

Facing small and biased data dilemma in drug discovery with enhanced federated learning approaches

More info
  • ReceivedApr 8, 2021
  • AcceptedMay 16, 2021
  • PublishedJul 26, 2021

Abstract


Funded by

the Shanghai Municipal Science and Technology Major Project

the National Natural Science Foundation of China(81773634)

the National Science and Technology Major Project of the Ministry of Science and Technology of China(2018ZX09711002)

and the Strategic Priority Research Program of the Chinese Academy of Sciences(XDA12050201,XDA12020368)


Acknowledgment

This work was supported by the Shanghai Municipal Science and Technology Major Project, the National Natural Science Foundation of China (81773634), the National Science and Technology Major Project of the Ministry of Science and Technology of China (2018ZX09711002), and the Strategic Priority Research Program of the Chinese Academy of Sciences (XDA12050201 and XDA12020368).


Interest statement

The author(s) declare that they have no conflict of interest.


Supplement

SUPPORTING INFORMATION

The supporting information is available online at https://doi.org/10.1007/s11427-021-1946-0. The supporting materials are published as submitted, without typesetting or editing. The responsibility for scientific accuracy and content remains entirely with the authors.


References

[1] Ai X., Sun Y., Wang H., Lu S.. A systematic profile of clinical inhibitors responsive to EGFR somatic amino acid mutations in lung cancer: implication for the molecular mechanism of drug resistance and sensitivity. Amino Acids, 2014, 46: 1635-1648 CrossRef PubMed Google Scholar

[2] Anastassiadis T., Deacon S.W., Devarajan K., Ma H., Peterson J.R.. Comprehensive assay of kinase catalytic activity reveals features of kinase inhibitor selectivity. Nat Biotechnol, 2011, 29: 1039-1045 CrossRef PubMed Google Scholar

[3] Anwar-Mohamed A., Barakat K.H., Bhat R., Noskov S.Y., Tyrrell D.L., Tuszynski J.A., Houghton M.. A human ether-á-go-go-related (hERG) ion channel atomistic model generated by long supercomputer molecular dynamics simulations and its use in predicting drug cardiotoxicity. Toxicol Lett, 2014, 230: 382-392 CrossRef PubMed Google Scholar

[4] Aronov A.M., Goldman B.B.. A model for identifying HERG K+ channel blockers. Bioorg Med Chem, 2004, 12: 2307-2315 CrossRef PubMed Google Scholar

[5] Aronov A.. Predictive in silico modeling for hERG channel blockers. Drug Discov Today, 2005, 10: 149-155 CrossRef Google Scholar

[6] Beaugrand M., Arnold A.A., Bourgault S., Williamson P.T.F., Marcotte I.. Comparative study of the structure and interaction of the pore helices of the hERG and Kv1.5 potassium channels in model membranes. Eur Biophys J, 2017, 46: 549-559 CrossRef PubMed Google Scholar

[7] Benson A.P., Al-Owais M., Holden A.V.. Quantitative prediction of the arrhythmogenic effects of de novo hERG mutations in computational models of human ventricular tissues. Eur Biophys J, 2011, 40: 627-639 CrossRef PubMed Google Scholar

[8] Bento A.P., Gaulton A., Hersey A., Bellis L.J., Chambers J., Davies M., Krüger F.A., Light Y., Mak L., McGlinchey S., et al. The ChEMBL bioactivity database: an update. Nucl Acids Res, 2014, 42: D1083-D1090 CrossRef PubMed Google Scholar

[9] Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H.B., Patel, S., Ramage, D., Segal, A., and Seth, K. (2017). Practical Secure Aggregation for privacy-preserving machine learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. New York: Association for Computing Machinery. 1175–1191. Google Scholar

[10] Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, H.B., et al. (2019). Towards federated learning at scale: system design. arXiv, 1902.01046. Google Scholar

[11] Braga R.C., Alves V.M., Silva M.F.B., Muratov E., Fourches D., Lião L.M., Tropsha A., Andrade C.H.. Pred-hERG: a novel web-accessible computational tool for predicting cardiac toxicity. Mol Inf, 2015, 34: 698-701 CrossRef PubMed Google Scholar

[12] Cai C., Guo P., Zhou Y., Zhou J., Wang Q., Zhang F., Fang J., Cheng F.. Deep learning-based prediction of drug-induced cardiotoxicity. J Chem Inf Model, 2019, 59: 1073-1084 CrossRef PubMed Google Scholar

[13] Chen B., Garmire L., Calvisi D.F., Chua M.S., Kelley R.K., Chen X.. Harnessing big ‘omics’ data and AI for drug discovery in hepatocellular carcinoma. Nat Rev Gastroenterol Hepatol, 2020, 17: 238-251 CrossRef PubMed Google Scholar

[14] Chen S., Xue D., Chuai G., Yang Q., Liu Q.. FL-QSAR: a federated learning-based QSAR prototype for collaborative drug discovery. Bioinformatics, 2021, 36: 5492-5498 CrossRef PubMed Google Scholar

[15] Christmann-Franck S., van Westen G.J.P., Papadatos G., Beltran Escudie F., Roberts A., Overington J.P., Domine D.. Unprecedently large-scale kinase inhibitor set enabling the accurate prediction of compound-kinase activities: a way toward selective promiscuity by design?. J Chem Inf Model, 2016, 56: 1654-1675 CrossRef PubMed Google Scholar

[16] Daub H., Specht K., Ullrich A.. Strategies to overcome resistance to targeted protein kinase inhibitors. Nat Rev Drug Discov, 2004, 3: 1001-1010 CrossRef PubMed Google Scholar

[17] Davis M.I., Hunt J.P., Herrgard S., Ciceri P., Wodicka L.M., Pallares G., Hocker M., Treiber D.K., Zarrinkar P.P.. Comprehensive analysis of kinase inhibitor selectivity. Nat Biotechnol, 2011, 29: 1046-1051 CrossRef PubMed Google Scholar

[18] Delaney J.S.. ESOL:  estimating aqueous solubility directly from molecular structure. J Chem Inf Comput Sci, 2004, 44: 1000-1005 CrossRef PubMed Google Scholar

[19] Doddareddy M.R., Klaasse E.C., Shagufta E., IJzerman A.P., Bender A.. Prospective validation of a comprehensive in silico hERG model and its applications to commercial compound and drug databases. Chemmedchem, 2010, 5: 716-729 CrossRef PubMed Google Scholar

[20] Dranchak P., MacArthur R., Guha R., Zuercher W.J., Drewry D.H., Auld D.S., Inglese J.. Profile of the GSK published protein kinase inhibitor set across ATP-dependent and-independent luciferases: implications for reporter-gene assays. PLoS ONE, 2013, 8: e57888 CrossRef PubMed ADS Google Scholar

[21] Elkins J.M., Fedele V., Szklarz M., Abdul Azeez K.R., Salah E., Mikolajczyk J., Romanov S., Sepetov N., Huang X.P., Roth B.L., et al. Comprehensive characterization of the Published Kinase Inhibitor Set. Nat Biotechnol, 2015, 34: 95-103 CrossRef PubMed Google Scholar

[22] Haddadpour, F., Kamani, M.M., Mahdavi, M., and Cadambe, V.R. (2019). Local SGD with periodic averaging: tighter analysis and adaptive synchronization. arXiv, 1910.13598. Google Scholar

[23] Huang, Y., Chu, L., Zhou, Z., Wang, L., Liu, J., Pei, J., and Zhang, Y. (2020). Personalized federated learning: an attentive collaboration approach. arXiv, 2007.03797. Google Scholar

[24] Hunter A.J., Lee W.H., Bountra C.. Open innovation in neuroscience research and drug discovery. Brain Neurosci Adv, 2018, 2: 239821281879927 CrossRef PubMed Google Scholar

[25] Huuskonen J.. Estimation of aqueous solubility for a diverse set of organic compounds based on molecular topology. J Chem Inf Comput Sci, 2000, 40: 773-777 CrossRef PubMed Google Scholar

[26] Jiang, Y., Konečný, J., Rush, K., and Kannan, S. (2019). Improving federated learning personalization via model agnostic meta learning. arXiv, 1909.12488. Google Scholar

[27] Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. (2019). Advances and open problems in federated learning. arXiv, 1912.04977. Google Scholar

[28] Kaissis G.A., Makowski M.R., Rückert D., Braren R.F.. Secure, privacy-preserving and federated machine learning in medical imaging. Nat Mach Intell, 2020, 2: 305-311 CrossRef Google Scholar

[29] Keserü G.M.. Prediction of hERG potassium channel affinity by traditional and hologram QSAR methods. Bioorg Med Chem Lett, 2003, 13: 2773-2775 CrossRef Google Scholar

[30] Knapp S., Arruda P., Blagg J., Burley S., Drewry D.H., Edwards A., Fabbro D., Gillespie P., Gray N.S., Kuster B., et al. A public-private partnership to unlock the untargeted kinome. Nat Chem Biol, 2013, 9: 3-6 CrossRef PubMed Google Scholar

[31] Li, W., Milletarì, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M.J., et al. (2019). Privacy-preserving federated brain tumour segmentation. In: Suk, H.I., Liu, M., Yan, P., and Lian, C., eds. Machine Learning in Medical Imaging. MLMI 2019. Cham: Springer. 133–141. Google Scholar

[32] Liu L., Lu J., Lu Y., Zheng M., Luo X., Zhu W., Jiang H., Chen K.. Novel Bayesian classification models for predicting compounds blocking hERG potassium channels. Acta Pharmacol Sin, 2014, 35: 1093-1102 CrossRef PubMed Google Scholar

[33] Liu, D., Xu, C., He, W., Xu, Z., Fu, W., Zhang, L., Yang, J., Peng, G., Han, D., Bai, X., et al. (2019). AutoGenome: an autoML tool for genomic research. bioRxiv, 10.1101/842526. Google Scholar

[34] Ma R., Li Y., Li C., Wan F., Hu H., Xu W., Zeng J.. Secure multiparty computation for privacy-preserving drug discovery. Bioinformatics, 2020, 36: 2872-2880 CrossRef PubMed Google Scholar

[35] McMahan, B., Moore, E., Ramage, D., Hampson, S. and Arcas, B.A.y. (2017). Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Fort Lauderdale: PMLR. 1273–1282. Google Scholar

[36] Merget B., Turk S., Eid S., Rippmann F., Fulle S.. Profiling prediction of kinase inhibitors: toward the virtual assay. J Med Chem, 2017, 60: 474-485 CrossRef PubMed Google Scholar

[37] Metz J.T., Johnson E.F., Soni N.B., Merta P.J., Kifle L., Hajduk P.J.. Navigating the kinome. Nat Chem Biol, 2011, 7: 200-202 CrossRef PubMed Google Scholar

[38] Raevsky O.A., Grigor’ev V.Y., Polianczyk D.E., Raevskaja O.E., Dearden J.C.. Calculation of aqueous solubility of crystalline un-ionized organic chemicals and drugs based on structural similarity and physicochemical descriptors. J Chem Inf Model, 2014, 54: 683-691 CrossRef PubMed Google Scholar

[39] Riley P.. Three pitfalls to avoid in machine learning. Nature, 2019, 572: 27-29 CrossRef PubMed ADS Google Scholar

[40] Rogers D., Hahn M.. Extended-connectivity fingerprints. J Chem Inf Model, 2010, 50: 742-754 CrossRef PubMed Google Scholar

[41] Schneider P., Walters W.P., Plowright A.T., Sieroka N., Listgarten J., Goodnow Jr. R.A., Fisher J., Jansen J.M., Duca J.S., Rush T.S., et al. Rethinking drug design in the artificial intelligence era. Nat Rev Drug Discov, 2020, 19: 353-364 CrossRef PubMed Google Scholar

[42] Siramshetty V.B., Nguyen D.T., Martinez N.J., Southall N.T., Simeonov A., Zakharov A.V.. Critical assessment of artificial intelligence methods for prediction of hERG channel inhibition in the “big data” era. J Chem Inf Model, 2020, 60: 6007-6019 CrossRef PubMed Google Scholar

[43] Smalley E.. AI-powered drug discovery captures pharma interest. Nat Biotechnol, 2017, 35: 604-605 CrossRef PubMed Google Scholar

[44] Smirnov E.A., Timoshenko D.M., Andrianov S.N.. Comparison of regularization methods for ImageNet classification with deep convolutional neural networks. AASRI Procedia, 2014, 6: 89-94 CrossRef Google Scholar

[45] Sorkun M.C., Khetan A., Er S.. AqSolDB, a curated reference set of aqueous solubility and 2D descriptors for a diverse set of compounds. Sci Data, 2019, 6: 143 CrossRef PubMed ADS Google Scholar

[46] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. J Mach Learn Res 15, 1929–1958. Google Scholar

[47] Sun X., Xu B., Xue Y., Li H., Zhang H., Zhang Y., Kang L., Zhang X., Zhang J., Jia Z., et al. Characterization and structure-activity relationship of natural flavonoids as hERG K+ channel modulators. Int Immunopharmacol, 2017, 45: 187-193 CrossRef PubMed Google Scholar

[48] Tang J., Szwajda A., Shakyawar S., Xu T., Hintsanen P., Wennerberg K., Aittokallio T.. Making sense of large-scale kinase inhibitor bioactivity data sets: a comparative and integrative analysis. J Chem Inf Model, 2014, 54: 735-743 CrossRef PubMed Google Scholar

[49] Volkamer A., Eid S., Turk S., Jaeger S., Rippmann F., Fulle S.. Pocketome of human kinases: prioritizing the ATP binding sites of (yet) untapped protein kinases for drug discovery. J Chem Inf Model, 2015, 55: 538-549 CrossRef PubMed Google Scholar

[50] Wang J., Hou T., Xu X.. Aqueous solubility prediction based on weighted atom type counts and solvent accessible surface areas. J Chem Inf Model, 2009, 49: 571-581 CrossRef PubMed Google Scholar

[51] Wang, K., Mathews, R., Kiddon, C., Eichner, H., Beaufays, F., and Ramage, D. (2019). Federated evaluation of on-device personalization. arXiv, 1910.10252. Google Scholar

[52] Yang, Q., Liu, Y., Chen, T., and Tong, Y. (2019). Federated machine learning: concept and applications. arXiv, 1902.04885. Google Scholar

[53] Yang, T., Andrew, G., Eichner, H., Sun, H., Li, W., Kong, N., Ramage, D., and Beaufays, F. (2018). Applied federated learning: improving Google keyboard query suggestions. arXiv, 1812.02903. Google Scholar

[54] Zhang S., Zhou Z., Gong Q., Makielski J.C., January C.T.. Mechanism of block and identification of the verapamil binding domain to HERG potassium channels. Circ Res, 1999, 84: 989-998 CrossRef PubMed Google Scholar

[55] Zhang W., Roederer M.W., Chen W.Q., Fan L., Zhou H.H.. Pharmacogenetics of drugs withdrawn from the market. Pharmacogenomics, 2012, 13: 223-231 CrossRef PubMed Google Scholar

  • Figure 1

    The life cycle of a federated learning system for drug discovery. In federated training: (i) the coordinator server broadcast the latest shared global model to each client; (ii) the client locally computes the model updates, (iii) encrypts and uploads the model updates; (iv) finally, the coordinator server aggregates all the encrypted model updates securely and uses them to update the shared global model for the next round of training. After the training is done, the best model is selected for rollout and might be personalized for the users who have their own labeled data.

  • Figure 2

    Performance comparisons between federated modeling and centralized modeling. A, Centralized models are trained on individual clients (Only F1, F2, F3 and F4) or on the union/pooling of datasets F1/F2/F3/F4 (Union), while federated learning models are trained cross clients F1/F2/F3/F4. The mean absolute error (MAE) of Log10S on the test set of each client are reported. B, The deep neural network architecture of the model, which is shared by federated modeling and centralized modeling. FC is short for fully connected layer with relu activation. BN is short for batch normalization layer. C, When the weights of FC1–4 are initialized with normal distribution. D, The weight distributions of the Union model do not vary tangibly after centralized training. E, The weight distributions of the federated model vary tangibly with more weights concentrated on 0.

  • Table 1   Data statistics and the mean absolute deviation of the values for the compounds shared among different sourcesa)

    F1

    F2

    F3

    F4

    C1

    C2

    C3

    F1

    6,110 (0.00)

     

    F2

    763 (0.27)

    4,650 (0.00)

     

    F3

    72 (0.35)

    215 (0.12)

    2,603 (0.00)

     

    F4

    520 (0.24)

    952 (0.04)

    136 (0.15)

    2,115 (0.00)

     

    C1

    52 (0.19)

    194 (0.06)

    20 (0.22)

    185 (0.06)

    1,291 (0.00)

     

    C2

    4 (1.52)

    29 (0.09)

    59 (0.11)

    12 (0.12)

    7 (0.22)

    1,210 (0.00)

     

    C3

    195 (0.31)

    545 (0.12)

    32 (0.28)

    486 (0.09)

    174 (0.02)

    8 (0.19)

    1,144 (0.00)

    The numbers outside the parentheses are shared molecules between two datasets, and the numbers inside are the MAD of Log10S of these shared molecules. The values in bold indicate the count of molecules in individual client.

  • Table 2   Performance of using different Federated Averaging epochsa)

    Client/Test size

    Federated Averaging epochs

    Every 1

    Every 5

    Every 10

    Every 15

    Every 20

    F1/611

    0.878±0.022

    0.864±0.023

    0.872±0.017

    0.888±0.013

    0.870±0.012

    F2/465

    0.543±0.018

    0.517±0.011

    0.522±0.007

    0.519±0.009

    0.531±0.011

    F3/260

    0.534±0.007

    0.503±0.011

    0.511±0.002

    0.515±0.020

    0.530±0.006

    F4/212

    0.324±0.037

    0.296±0.007

    0.311±0.011

    0.304±0.009

    0.310±0.008

    C1/258

    0.299±0.023

    0.277±0.003

    0.280±0.004

    0.283±0.004

    0.287±0.006

    C2/242

    0.801±0.024

    0.775±0.004

    0.789±0.007

    0.779±0.005

    0.786±0.005

    C3/229

    0.328±0.021

    0.313±0.005

    0.315±0.006

    0.319±0.010

    0.322±0.004

    The reported performance is the mean absolute error (MAE) of Log10S on test sets in 5 independent runs. Clients F1–4 are the federated training participants, and have 1/10 of the dataset as the test set (8/10 for federated training and 1/10 for validation), while clients C1–3 participate in personalization only, and has 1/5 of the dataset as test set (3/5 for personalization training and 1/5 for validation). The best performance on each client is highlighted in bold.

  • Table 3   Performances (MAE) of using different network architectures and federated learning algorithmsa)

    Client

    Test size

    Union+MLP

    Union+RFCN

    FedAvg+MLP

    FedAvg+RFCN

    FedAMP+MLP

    FedAMP+RFCN

    F1

    509

    0.901

    0.872

    0.871

    0.889

    0.872

    0.887

    (0.021)

    (0.017)

    (0.022)

    (0.024)

    (0.018)

    (0.021)

    F2

    388

    0.610

    0.511

    0.531

    0.545

    0.514

    0.510

    (0.020)

    (0.023)

    (0.022)

    (0.023)

    (0.023)

    (0.021)

    F3

    217

    0.644

    0.503

    0.527

    0.546

    0.492

    0.479

    (0.025)

    (0.024)

    (0.021)

    (0.025)

    (0.024)

    (0.024)

    F4

    177

    0.391

    0.287

    0.307

    0.283

    0.282

    0.260

    (0.026)

    (0.018)

    (0.019)

    (0.026)

    (0.018)

    (0.020)

    C1

    1,291

    0.365

    0.260

    0.285

    0.268

    0.287

    0.291

    (0.011)

    (0.003)

    (0.008)

    (0.013)

    (0.006)

    (0.008)

    C2

    1,210

    0.697

    0.748

    0.787

    0.807

    0.793

    0.774

    (0.020)

    (0.015)

    (0.014)

    (0.023)

    (0.019)

    (0.014)

    C3

    1,144

    0.408

    0.304

    0.319

    0.308

    0.323

    0.332

    (0.005)

    (0.006)

    (0.013)

    (0.009)

    (0.009)

    (0.013)

    Weighted mean

    0.544

    0.484

    0.507

    0.508

    0.506

    0.504

    Unweighted mean

    0.574

    0.498

    0.518

    0.521

    0.509

    0.505

    The best performance on each client is highlighted in bold. The standard deviation of 5 independent runs are shown in parentheses.

  • Table 4   The kinase inhibition predictive performance comparison (MAE)a)

    Client

    Test size

    Individual MLP model

    Union+RFCN

    FedAVG+RFCN

    FedAMP+RFCN

    BioMedX

    Christmann

    PKIS

    Tang

    BioMedX

    142

    0.642

    2.531

    2.630

    2.190

    0.483

    0.762

    0.609

    (0.030)

    (0.065)

    (0.023)

    (0.067)

    (0.019)

    (0.016)

    (0.023)

    Christmann

    93

    0.678

    0.676

    0.638

    0.210

    0.261

    1.178

    0.272

    (0.027)

    (0.033)

    (0.032)

    (0.029)

    (0.006)

    (0.007)

    (0.029)

    PKIS

    31

    0.411

    1.034

    0.584

    0.628

    0.080

    0.387

    0.120

    (0.038)

    (0.055)

    (0.029)

    (0.048)

    (0.101)

    (0.108)

    (0.031)

    Tang

    75

    0.503

    0.850

    1.062

    0.710

    0.295

    0.478

    0.235

    (0.025)

    (0.021)

    (0.034)

    (0.024)

    (0.046)

    (0.045)

    (0.026)

    Weighted mean

    0.600

    1.520

    1.557

    1.183

    0.345

    0.779

    0.390

    Unweighted mean

    0.559

    1.273

    1.229

    0.935

    0.280

    0.701

    0.309

    The best performance on each client is highlighted in bold. The standard deviation of 5 independent runs are shown in parentheses.

  • Table 5   The hERG inhibition predictive performance comparison (F1 Score)a)

    Client

    Test size

    Individual MLP model

    Union+MLP

    FedAVG+MLP

    FedAMP+MLP

    ChEMBL

    NCATS

    JHICC

    Cai

    ChEMBL

    748

    0.848

    0.643

    0.081

    0.839

    0.894

    0.872

    0.886

    (0.004)

    (0.004)

    (0.003)

    (0.002)

    (0.006)

    (0.002)

    (0.009)

    NCATS

    1,652

    0.457

    0.702

    0.020

    0.551

    0.758

    0.685

    0.637

    (0.066)

    (0.019)

    (0.062)

    (0.014)

    (0.059)

    (0.068)

    (0.067)

    JHICC

    423

    0.162

    0.108

    0.424

    0.125

    0.402

    0.261

    0.428

    (0.028)

    (0.003)

    (0.041)

    (0.008)

    (0.026)

    (0.042)

    (0.035)

    Cai

    786

    0.840

    0.629

    0.030

    0.810

    0.884

    0.879

    0.888

    (0.014)

    (0.007)

    (0.007)

    (0.000)

    (0.005)

    (0.001)

    (0.007)

    FDA-Drugs

    177

    0.521

    0.459

    0.029

    0.512

    0.441

    0.402

    0.432

    (0.036)

    (0.037)

    (0.047)

    (0.038)

    (0.037)

    (0.039)

    (0.040)

    Keserü

    66

    0.866

    0.866

    0.047

    0.924

    0.926

    0.914

    0.804

    (0.017)

    (0.009)

    (0.020)

    (0.010)

    (0.013)

    (0.025)

    (0.024)

    Doddareddy

    1,636

    0.908

    0.757

    0.018

    0.913

    0.921

    0.910

    0.829

    (0.005)

    (0.005)

    (0.007)

    (0.011)

    (0.002)

    (0.013)

    (0.013)

    Siramshetty

    5,804

    0.969

    0.657

    0.026

    0.897

    0.975

    0.958

    0.871

    (0.005)

    (0.004)

    (0.009)

    (0.003)

    (0.003)

    (0.010)

    (0.011)

    Zhang

    1,565

    0.821

    0.699

    0.024

    0.864

    0.884

    0.866

    0.756

    (0.007)

    (0.002)

    (0.010)

    (0.001)

    (0.006)

    (0.014)

    (0.011)

    Sun

    3,024

    0.494

    0.914

    0.023

    0.586

    0.868

    0.820

    0.672

    (0.006)

    (0.006)

    (0.047)

    (0.011)

    (0.011)

    (0.047)

    (0.046)

    Weighted mean

    0.743

    0.684

    0.034

    0.745

    0.859

    0.829

    0.750

    Unweighted mean

    0.689

    0.643

    0.072

    0.702

    0.795

    0.757

    0.720

    The best performance on each client is highlighted in bold. The standard deviation of 5 independent runs are shown in parentheses.

  •   

    Algorithm 1 Federated Averaging with Secure Aggregation

    Coordinator Server executes:

    initialize w0

    for each round t=1,2,... do

    for each client k in parallel do

    (Δt+1kλnk) ← ClientUpdate(k, wt)

    Δt+1k=1kΔt+1k // sum of weighted updates

    λnallk=1kλnk  // sum of weights

    Δt+1Δt+1/ λnall // averaged updates

    w t + 1 w t + Δ t + 1

     

    ClientUpdate(k, w): // executed on clientk

    w shared w

    for each local epoch i from 1 to E do

    batches ← (data Pk split into batches of size B)

    for batch b in batches do

    w w η l ( w ; b )

    Δλnk·(wwshared) // weighted updates

    // note is more amenable to compression than w

    // then, encrypt it with random variables s

    // specified between client k and all other clients u

    // by Secure Aggregation protocol

    Δ Δ + k < u s k , u k > u s u , k

    returnΔ,λnk to server

qqqq

Contact and support