Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters

X. Xu, L. Cao, X. Wang

Research output: Contribution to journalArticle

25 Citations (Scopus)

Abstract

The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. According to the dynamic change of load of each task node and the difference of node performance of different tasks in the heterogeneous Hadoop cluster, a novel adaptive task scheduling strategy based on dynamic workload adjustment (ATSDWA) is presented. With ATSDWA, tasktrackers can adapt to the change of load at runtime, obtain tasks in accordance with the computing ability of their own, and realize the self-regulation, while avoiding the complexity of algorithm, which is the prime reason to make jobtracker the system performance bottleneck. Experimental results show that ATSDWA is a highly efficient and reliable algorithm, which can make heterogeneous Hadoop clusters stable, scalable, efficient, and load balancing. Furthermore, its performance is superior to the original and improved task scheduling strategy of Hadoop, from the aspects of the execution time of tasks, the resource utilization, and the speed-up ratio.
Original languageEnglish
Pages (from-to)1-12
Number of pages12
JournalIEEE Systems Journal
Volume10
Issue number2
DOIs
Publication statusPublished - 12 Jun 2014

    Fingerprint

Keywords

  • Cloud computing
  • Dynamic scheduling
  • Heart beat
  • Real-time systems
  • Scheduling algorithms
  • Adaptive scheduling
  • clustering methods
  • computational efficiency
  • distributed computing
  • dynamic scheduling

Cite this