Research Output

An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment

  MapReduce (MRV1), a popular programming model, proposed by Google, has been well used to process large datasets in Hadoop, an open source cloud platform. Its new version MapReduce 2.0 (MRV2) developed along with the emerging of Yarn has achieved obvious improvement over MRV1. However, MRV2 suffers from long finishing time on certain types of jobs. Speculative Execution (SE) has been presented as an approach to the problem above by backing up those delayed jobs from low-performance machines to higher ones. In this paper, an adaptive SE strategy (ASE) is presented in Hadoop-2.6.0. Experiment results have depicted that the ASE duplicates tasks according to real-time resources usage among work nodes in a cloud. In addition, the performance of MRV2 is largely improved using the ASE strategy on job execution time and resource consumption, whether in a multi-job environment.

  • Type:

    Article

  • Date:

    28 February 2017

  • Publication Status:

    Published

  • Publisher

    Korean Society for Internet Information (KSII)

  • DOI:

    10.3837/tiis.2017.02.004

  • ISSN:

    1976-7277

  • Library of Congress:

    QA75 Electronic computers. Computer science

  • Dewey Decimal Classification:

    005.4 Systems programming and programs

  • Funders:

    European Commission

Citation

Liu, Q., Cai, W., Liu, Q., Shen, J., Fu, Z., Liu, X., & Linge, N. (2017). An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment. KSII transactions on internet and information systems, 11(2), https://doi.org/10.3837/tiis.2017.02.004

Authors

Keywords

Computer Networks and Communications; Information Systems

Monthly Views:

Available Documents