FlexReduce: Flexible All-reduce for Distributed Deep Learning on Asymmetric Network Topology
- Title
- FlexReduce: Flexible All-reduce for Distributed Deep Learning on Asymmetric Network Topology
- Authors
- Lee, Jinho; HWANG, INSEOK; Shah, Soham; Cho, Minsik
- Date Issued
- 2020-07-21
- Publisher
- ACM/IEEE
- Abstract
- We propose FlexReduce, an efficient and flexible all-reduce algorithm for distributed deep learning under irregular network hierarchies. With ever-growing deep neural networks, distributed learning over multiple nodes is becoming imperative for expedited training. There are several approaches leveraging the symmetric network structure to optimize the performance over different hierarchy levels of the network. However, the assumption of symmetric network does not always hold, especially in shared cloud environments. By allocating an uneven portion of gradients to each learner (GPU), FlexReduce outperforms conventional algorithms on asymmetric network structures, and still performs even or better on symmetric networks.
- URI
- https://oasis.postech.ac.kr/handle/2014.oak/104285
- Article Type
- Conference
- Citation
- DAC 2020 (57th ACM/IEEE Design Automation Conference), 2020-07-21
- Files in This Item:
- There are no files associated with this item.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.