MALT: Distributed Data-Parallelism for Existing ML Applications

摘要

Systems and methods are disclosed for parallel machine learning with a cluster of N parallel machine network nodes by determining k network nodes as a subset of the N network nodes to update learning parameters, wherein k is selected to disseminate the updates across all nodes directly or indirectly and to optimize predetermined goals including freshness, balanced communication and computation ratio in the cluster; sending learning unit updates to fewer nodes to reduce communication costs with learning convergence; and sending reduced learning updates and ensuring that the nodes send/receive learning updates in a uniform fashion.