Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters

X. Xu, L. Cao, X. Wang

Research output: Contribution to journalArticle

Abstract

The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. According to the dynamic change of load of each task node and the difference of node performance of different tasks in the heterogeneous Hadoop cluster, a novel adaptive task scheduling strategy based on dynamic workload adjustment (ATSDWA) is presented. With ATSDWA, tasktrackers can adapt to the change of load at runtime, obtain tasks in accordance with the computing ability of their own, and realize the self-regulation, while avoiding the complexity of algorithm, which is the prime reason to make jobtracker the system performance bottleneck. Experimental results show that ATSDWA is a highly efficient and reliable algorithm, which can make heterogeneous Hadoop clusters stable, scalable, efficient, and load balancing. Furthermore, its performance is superior to the original and improved task scheduling strategy of Hadoop, from the aspects of the execution time of tasks, the resource utilization, and the speed-up ratio.
Original languageEnglish
Pages (from-to)1-12
Number of pages12
JournalIEEE Systems Journal
Volume10
Issue number2
DOIs
Publication statusPublished - 12 Jun 2014

Fingerprint

Scheduling
Scheduling algorithms
Resource allocation

Keywords

  • Cloud computing
  • Dynamic scheduling
  • Heart beat
  • Real-time systems
  • Scheduling algorithms
  • Adaptive scheduling
  • clustering methods
  • computational efficiency
  • distributed computing
  • dynamic scheduling

Cite this

@article{ff81fc72315a42a5b8f18d2aeb5462f8,
title = "Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters",
abstract = "The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. According to the dynamic change of load of each task node and the difference of node performance of different tasks in the heterogeneous Hadoop cluster, a novel adaptive task scheduling strategy based on dynamic workload adjustment (ATSDWA) is presented. With ATSDWA, tasktrackers can adapt to the change of load at runtime, obtain tasks in accordance with the computing ability of their own, and realize the self-regulation, while avoiding the complexity of algorithm, which is the prime reason to make jobtracker the system performance bottleneck. Experimental results show that ATSDWA is a highly efficient and reliable algorithm, which can make heterogeneous Hadoop clusters stable, scalable, efficient, and load balancing. Furthermore, its performance is superior to the original and improved task scheduling strategy of Hadoop, from the aspects of the execution time of tasks, the resource utilization, and the speed-up ratio.",
keywords = "Cloud computing, Dynamic scheduling, Heart beat, Real-time systems, Scheduling algorithms, Adaptive scheduling, clustering methods, computational efficiency, distributed computing, dynamic scheduling",
author = "X. Xu and L. Cao and X. Wang",
year = "2014",
month = "6",
day = "12",
doi = "10.1109/JSYST.2014.2323112",
language = "English",
volume = "10",
pages = "1--12",
journal = "IEEE Systems Journal",
issn = "1932-8184",
publisher = "IEEE",
number = "2",

}

Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters. / Xu, X.; Cao, L.; Wang, X.

In: IEEE Systems Journal, Vol. 10, No. 2, 12.06.2014, p. 1-12.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Adaptive Task Scheduling Strategy Based on Dynamic Workload Adjustment for Heterogeneous Hadoop Clusters

AU - Xu, X.

AU - Cao, L.

AU - Wang, X.

PY - 2014/6/12

Y1 - 2014/6/12

N2 - The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. According to the dynamic change of load of each task node and the difference of node performance of different tasks in the heterogeneous Hadoop cluster, a novel adaptive task scheduling strategy based on dynamic workload adjustment (ATSDWA) is presented. With ATSDWA, tasktrackers can adapt to the change of load at runtime, obtain tasks in accordance with the computing ability of their own, and realize the self-regulation, while avoiding the complexity of algorithm, which is the prime reason to make jobtracker the system performance bottleneck. Experimental results show that ATSDWA is a highly efficient and reliable algorithm, which can make heterogeneous Hadoop clusters stable, scalable, efficient, and load balancing. Furthermore, its performance is superior to the original and improved task scheduling strategy of Hadoop, from the aspects of the execution time of tasks, the resource utilization, and the speed-up ratio.

AB - The original task scheduling algorithm of Hadoop cannot meet the performance requirements of heterogeneous clusters. According to the dynamic change of load of each task node and the difference of node performance of different tasks in the heterogeneous Hadoop cluster, a novel adaptive task scheduling strategy based on dynamic workload adjustment (ATSDWA) is presented. With ATSDWA, tasktrackers can adapt to the change of load at runtime, obtain tasks in accordance with the computing ability of their own, and realize the self-regulation, while avoiding the complexity of algorithm, which is the prime reason to make jobtracker the system performance bottleneck. Experimental results show that ATSDWA is a highly efficient and reliable algorithm, which can make heterogeneous Hadoop clusters stable, scalable, efficient, and load balancing. Furthermore, its performance is superior to the original and improved task scheduling strategy of Hadoop, from the aspects of the execution time of tasks, the resource utilization, and the speed-up ratio.

KW - Cloud computing

KW - Dynamic scheduling

KW - Heart beat

KW - Real-time systems

KW - Scheduling algorithms

KW - Adaptive scheduling

KW - clustering methods

KW - computational efficiency

KW - distributed computing

KW - dynamic scheduling

U2 - 10.1109/JSYST.2014.2323112

DO - 10.1109/JSYST.2014.2323112

M3 - Article

VL - 10

SP - 1

EP - 12

JO - IEEE Systems Journal

JF - IEEE Systems Journal

SN - 1932-8184

IS - 2

ER -