Efficient semantic-aware coflow scheduling for data-parallel jobs

摘要

This paper studies the communication pattern of data-parallel applications from the perspective of job execution, and discovers multiple inter-coflow dependencies. These inter-coflow dependencies, collectively named as semantic flow (seflow), can expose job-level semantics. It is observed that most distributed computing frameworks describe their job execution as directed acyclic graphs (DAG). So a seflow comprises not only all the coflows of a job but also the DAG-based relationship between them. Seflow, coflow and flow can be viewed as the top-down abstractions for communication of jobs.