Using Pilot-Jobs for Developing eThread, a Meta-threading Pipeline.

摘要

The genome revolution has produced vast amount of sequence information, but the functional annotation of most of the gene products are yet to be explored in depth. Functional inference of low sequence identity is brought about by the structure based template methods. To model and understand these proteome scale functions, state of-the-art algorithms like eThread is used. They are compute intensive and demand effe cient and optimal use of the underlying resources. Combination of large scale data and complex workload raises the need for pilot based approaches. eThread is a metathreading protein structure modeling algorithm which is supported by ten independent single-threading algorithms whose computational complexity also depends on the number and size of the input sequences. In this thesis, eThread pipeline is developed on an extensible, scalable and interoperable pilot-job based framework and it supports concurrent tasks execution and data-parallelization on heterogeneous resources deployed on Amazon EC2 with S3 as data repository. This study aims to understand the dominant factors which influence the performance of eThread on EC2. This analysis suggests an optimized solution based on execution time and cost of implementation. It primarily achieves better utilization of resources by scaling workload on multiple resources. Further ideas on increasing resource capacity and discussions on the importance of dynamic execution of tasks are also laid out.