Parallel processing of large data files on distributed file systems with dynamic workload balancing

摘要

Parallel processing files on a distributed file system, responds to a request to process the file, by partitioning the file into a set of predetermined sized blocks by a single coordinator and assigns blocks to a reader in a set of readers. Block assignments are recorded to a control table accessible to all readers. The set of readers search the control table for a row assigned to a respective reader. Responsive to locating the row, the respective reader changes a state of the row to in progress updating an associated timestamp. The respective reader scans the control table for an available set of the blocks, when all blocks currently assigned are exhausted and dynamically re-assigns an available block from another owning reader to itself and changes ownership to itself. When no rows are available, the respective reader completes processing and returns an end of file indication to the single coordinator.