Research Output

An optimized Speculative Execution Strategy Based on Local Data Prediction in Heterogeneous Hadoop Environment

  Hadoop is a famous parallel computing framework that is applied to process large-scale data, but there exists such a task in hadoop framework, which is called “Straggling task” and has a serious impact on Hadoop. Speculative execution (SE) is an effective way to deal with the “Straggling task” by monitoring the real-time rate of running tasks and back up the “Straggler” on another node to increase the opportunity of completing backup task ahead of original. There are many problems in the proposed SE strategies, such as “Straggling task” misjudgment, improper selection of backup nodes, which will result in inefficient implementation of SE. In this paper, we propose an optimized SE strategy based on local data prediction, it collects task execution information in real time and uses Local regression to predict remaining time of the current task, and selects the appropriate backup task node according to the actual requirements, at the same time, it uses the consumption and benefit model to maximizes the effectiveness of SE. Finally, the strategy is implemented in Hadoop-2.6.0, the experiment proves that the optimized strategy not only enhances the accuracy of selecting the “Straggler” task candidates, but also shows better performance in heterogeneous Hadoop environment.

  • Type:

    Article

  • Date:

    31 December 2018

  • Publication Status:

    Published

  • DOI:

    10.3966/199115992019063003010

  • ISSN:

    1991-1599

  • Library of Congress:

    QA76 Computer software

  • Dewey Decimal Classification:

    006 Special Computer Methods

  • Funders:

    Edinburgh Napier Funded

Citation

Jin, D., Liu, Q., Liu, X., & Linge, N. (2019). An optimized Speculative Execution Strategy Based on Local Data Prediction in Heterogeneous Hadoop Environment. Journal of Computers, 30(3), 130-142. https://doi.org/10.3966/199115992019063003010

Authors

Keywords

Hadoop, Speculative execution, Straggling task, Local Regression, Prediction accuracy

Monthly Views:

Available Documents