Communication-efficient federated learning : a second order Newton-type method with analog over-the-air aggregation
Krouka, Mounssif; Elgabli, Anis; Issaid, Chaouki Ben; Bennis, Mehdi (2022-05-09)
M. Krouka, A. Elgabli, C. B. Issaid and M. Bennis, "Communication-Efficient Federated Learning: A Second Order Newton-Type Method With Analog Over-the-Air Aggregation," in IEEE Transactions on Green Communications and Networking, vol. 6, no. 3, pp. 1862-1874, Sept. 2022, doi: 10.1109/TGCN.2022.3173420
© 2022 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
https://creativecommons.org/licenses/by/4.0/
https://urn.fi/URN:NBN:fi-fe2019080123354
Tiivistelmä
Abstract
Owing to their fast convergence, second-order Newton-type learning methods have recently received attention in the federated learning (FL) setting. However, current solutions are based on communicating the Hessian matrices from the devices to the parameter server, at every iteration, incurring a large number of communication rounds; calling for novel communication-efficient Newton-type learning methods. In this article, we propose a novel second-order Newton-type method that, similarly to its first-order counterpart, requires every device to share only a model-sized vector at each iteration while hiding the gradient and Hessian information. In doing so, the proposed approach is significantly more communication-efficient and privacy-preserving. Furthermore, by leveraging the over-the-air aggregation principle, our method inherits privacy guarantees and obtains much higher communication efficiency gains. In particular, we formulate the problem of learning the inverse Hessian-gradient product as a quadratic problem that is solved in a distributed way. The framework alternates between updating the inverse Hessian-gradient product using a few alternating direction method of multipliers (ADMM) steps, and updating the global model using Newton’s method. Numerical results show that our proposed approach is more communication-efficient and scalable under noisy channels for different scenarios and across multiple datasets.
Kokoelmat
- Avoin saatavuus [32026]