Distributed Collaborative Learning with Representative Knowledge Sharing

Joseph Casey; Qianjiao Chen; Mengchen Fan; Baocheng Geng; Roman Shterenberg

doi:10.3390/math13061004

journal article Open Access Mar 19, 2025

Distributed Collaborative Learning with Representative Knowledge Sharing

Joseph Casey

Qianjiao Chen

Mengchen Fan Baocheng Geng

Roman Shterenberg

Mathematics Vol. 13 No. 6 pp. 1004 · MDPI AG

View at Publisher Save 10.3390/math13061004

Abstract

Distributed Collaborative Learning (DCL) addresses critical challenges in privacy-aware machine learning by enabling indirect knowledge transfer across nodes with heterogeneous feature distributions. Unlike conventional federated learning approaches, DCL assumes non-IID data and prediction task distributions that span beyond local training data, requiring selective collaboration to achieve generalization. In this work, we propose a novel collaborative transfer learning (CTL) framework that utilizes representative datasets and adaptive distillation weights to facilitate efficient and privacy-preserving collaboration. By leveraging Energy Coefficients to quantify node similarity, CTL dynamically selects optimal collaborators and refines local models through knowledge distillation on shared representative datasets. Simulations demonstrate the efficacy of CTL in improving prediction accuracy across diverse tasks while balancing trade-offs between local and global performance. Furthermore, we explore the impact of data spread and dispersion on collaboration, highlighting the importance of tailored node alignment. This framework provides a scalable foundation for cross-domain generalization in distributed machine learning.

Topics

No keywords indexed for this article. Browse by subject →

References

32

[1]

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., and Wang, P. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.

[2]

Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.

[3]

McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B.A. (2017, January 20–22). Communication-efficient learning of deep networks from decentralized data. Proceedings of the Artificial Intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA.

[4]

Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H.B. (2021). Adaptive Federated Optimization. arXiv.

[5]

Li "Federated optimization in heterogeneous networks" Proc. Mach. Learn. Syst. (2020)

[6]

Tan "AdaFed: Optimizing participation-aware federated learning with adaptive aggregation weights" IEEE Trans. Netw. Sci. Eng. (2022) 10.1109/tnse.2022.3168969

[7]

Xie, C., Koyejo, S., and Gupta, I. (2020). Asynchronous Federated Optimization. arXiv.

[8]

Chen, Y., Huang, W., and Ye, M. (2024, January 16–22). Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA. 10.1109/cvpr52733.2024.01148

[9]

Acar, D.A.E., Zhao, Y., Navarro, R.M., Mattina, M., Whatmough, P.N., and Saligrama, V. (2021). Federated Learning Based on Dynamic Regularization. arXiv.

[10]

Seyedmohammadi, S.J., Atapour, S.K., Abouei, J., and Mohammadi, A. (2024). KnFu: Effective Knowledge Fusion. arXiv. 10.23919/fusion59988.2024.10706495

[11]

Zhang "Parameterized knowledge transfer for personalized federated learning" Adv. Neural Inf. Process. Syst. (2021)

[12]

Towards Personalized Federated Learning

Alysa Ziying Tan, Han Yu, Lizhen Cui et al.

IEEE Transactions on Neural Networks and Learning... 2022 10.1109/tnnls.2022.3160699

[13]

Jeong, W., Yoon, J., Yang, E., and Hwang, S.J. (2021). Federated Semi-Supervised Learning with Inter Client Consistency & Disjoint Learning. arXiv.

[14]

Malinin, A., Mlodozeniec, B., and Gales, M. (2019). Ensemble distribution distillation. arXiv.

[15]

Cho, J.H., and Hariharan, B. (November, January 27). On the efficacy of knowledge distillation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.

[16]

Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardt, K., and Schölkopf, B. (2009). Covariate shift and local learning by distribution matching. Dataset Shift in Machine Learning, MIT Press.

[17]

A Survey on Transfer Learning

Sinno Jialin Pan, Qiang Yang

IEEE Transactions on Knowledge and Data Engineerin... 2010 10.1109/tkde.2009.191

[18]

Tian, Y., Krishnan, D., and Isola, P. (2022). Contrastive Representation Distillation. arXiv.

[19]

Li, T., Hu, S., Beirami, A., and Smith, V. (2021, January 18–24). Ditto: Fair and robust federated learning through personalization. Proceedings of the International Conference on Machine Learning. PMLR, Online.

[20]

Fallah, A., Mokhtari, A., and Ozdaglar, A. (2020). Personalized federated learning: A meta-learning approach. arXiv.

[21]

Sui, D., Chen, Y., Zhao, J., Jia, Y., Xie, Y., and Sun, W. (2020, January 16–20). Feded: Federated learning via ensemble distillation for medical relation extraction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. 10.18653/v1/2020.emnlp-main.165

[22]

Passalis, N., and Tefas, A. (2018, January 8–14). Learning deep representations with probabilistic knowledge transfer. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany. 10.1007/978-3-030-01252-6_17

[23]

Huang, Z., and Wang, N. (2017). Like What You Like: Knowledge Distill via Neuron Selectivity Transfer. arXiv.

[24]

Score-matching representative approach for big data analysis with generalized linear models

Jie Yang

Electronic Journal of Statistics 2022 10.1214/21-ejs1965

[25]

Communication-efficient federated learning via knowledge distillation

Chuhan Wu, Fangzhao Wu, Lingjuan Lyu et al.

Nature Communications 2022 10.1038/s41467-022-29763-x

[26]

FedProc: Prototypical contrastive federated learning on non-IID data

Xutong Mu, Yulong Shen, Ke Cheng et al.

Future Generation Computer Systems 2023 10.1016/j.future.2023.01.019

[27]

Székely, G.J. (2003). E-Statistics: The Energy of Statistical Samples, Bowling Green State University.

[28]

Rizzo "A new test for multivariate normality" J. Multivar. Anal. (2005) 10.1016/j.jmva.2003.12.002

[29]

Fan, M., Geng, B., Shterenberg, R., Casey, J.A., Chen, Z., and Li, K. (2025). Measuring Heterogeneity in Machine Learning with Distributed Energy Distance. arXiv.

[30]

Bennett "Robust linear programming discrimination of two linearly inseparable sets" Optim. Methods Softw. (1992) 10.1080/10556789208805504

[31]

Li, D., and Wang, J. (2019). Fedmd: Heterogenous federated learning via model distillation. arXiv.

[32]

Chen, H.Y., and Chao, W.L. (2020). Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv.

Metrics

3

Citations

32

References

Details

Published: Mar 19, 2025
Vol/Issue: 13(6)
Pages: 1004
License: View

Authors

J

Joseph Casey

Department of Mathematics, University of Alabama at Birmingham, Birmingham, AL 35294, USA

Q

Qianjiao Chen

Department of Mathematics, University of Alabama at Birmingham, Birmingham, AL 35294, USA

M

Mengchen Fan

Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL 35294, USA

B

Baocheng Geng

Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL 35294, USA

R

Roman Shterenberg

Department of Mathematics, University of Alabama at Birmingham, Birmingham, AL 35294, USA

Cite This Article

Joseph Casey, Qianjiao Chen, Mengchen Fan, et al. (2025). Distributed Collaborative Learning with Representative Knowledge Sharing. Mathematics, 13(6), 1004. https://doi.org/10.3390/math13061004

Distributed Collaborative Learning with Representative Knowledge Sharing

You May Also Like