Compile a graph-based program specification with automated clustering of graph components based on identifying specific data port connections

摘要

A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports, including scalar data ports for sending or receiving a single data element and collection data ports for sending or receiving a collection of multiple data elements; and one or more links, each connecting an output port of an upstream component to an input port of a downstream component. Prepared code is generated representing subsets of the plurality of components, including: identifying one or more subset boundaries, including identifying one or more links connecting a collection data port of a component to a scalar data port of a component; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.