Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters

Tomohiro Matsuno; Bijoy Chand Chatterjee; Eiji Oki; Malathi Veeraraghavan; Satoru Okamoto; Naoaki Yamanaka

doi:10.1109/HPSR.2016.7525667

Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters

Tomohiro Matsuno, Bijoy Chand Chatterjee, Eiji Oki, Malathi Veeraraghavan, Satoru Okamoto, Naoaki Yamanaka

Department of Information and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

This paper aims to design a Hadoop system and evaluates the performance of a task allocation scheme. The task allocation scheme splits each job into tasks using an appropriate splitting ratio, and assigns tasks to slave servers based on server processing performance and network resource availability. We experimentally evaluate the performance of the scale out of the task allocation scheme with five machines. We focus on the configuration of jobtracker and tasktracker in Hadoop. In cases with heterogeneous Hadoop clusters, we distribute task blocks to high-capability slaves with proportionally larger-sized tasks than to low-capability slaves. We create an environment in which high-capability slaves perform more work than low-capability slaves. The experimental testbed results indicate that the task allocation scheme is effective.

Original language	English
Title of host publication	2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016
Publisher	IEEE Computer Society
Pages	200-205
Number of pages	6
Volume	2016-July
ISBN (Electronic)	9781479989508
DOIs	https://doi.org/10.1109/HPSR.2016.7525667
Publication status	Published - 2016 Jul 28
Event	17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016 - Yokohama, Japan Duration: 2016 Jun 14 → 2016 Jun 17

Other

Other	17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016
Country/Territory	Japan
City	Yokohama
Period	16/6/14 → 16/6/17

Keywords

Hadoop
heterogeneous clusters
implementation
jobtracker

ASJC Scopus subject areas

Hardware and Architecture
Electrical and Electronic Engineering

Access to Document

10.1109/HPSR.2016.7525667

Cite this

Matsuno, T., Chatterjee, B. C., Oki, E., Veeraraghavan, M., Okamoto, S., & Yamanaka, N. (2016). Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters. In 2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016 (Vol. 2016-July, pp. 200-205). Article 7525667 IEEE Computer Society. https://doi.org/10.1109/HPSR.2016.7525667

Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters. / Matsuno, Tomohiro; Chatterjee, Bijoy Chand; Oki, Eiji et al.
2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016. Vol. 2016-July IEEE Computer Society, 2016. p. 200-205 7525667.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Matsuno, T, Chatterjee, BC, Oki, E, Veeraraghavan, M, Okamoto, S & Yamanaka, N 2016, Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters. in 2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016. vol. 2016-July, 7525667, IEEE Computer Society, pp. 200-205, 17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016, Yokohama, Japan, 16/6/14. https://doi.org/10.1109/HPSR.2016.7525667

@inproceedings{34bace73330e4bf586ef375fe816c8de,

title = "Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters",

abstract = "This paper aims to design a Hadoop system and evaluates the performance of a task allocation scheme. The task allocation scheme splits each job into tasks using an appropriate splitting ratio, and assigns tasks to slave servers based on server processing performance and network resource availability. We experimentally evaluate the performance of the scale out of the task allocation scheme with five machines. We focus on the configuration of jobtracker and tasktracker in Hadoop. In cases with heterogeneous Hadoop clusters, we distribute task blocks to high-capability slaves with proportionally larger-sized tasks than to low-capability slaves. We create an environment in which high-capability slaves perform more work than low-capability slaves. The experimental testbed results indicate that the task allocation scheme is effective.",

keywords = "Hadoop, heterogeneous clusters, implementation, jobtracker",

author = "Tomohiro Matsuno and Chatterjee, {Bijoy Chand} and Eiji Oki and Malathi Veeraraghavan and Satoru Okamoto and Naoaki Yamanaka",

year = "2016",

month = jul,

day = "28",

doi = "10.1109/HPSR.2016.7525667",

language = "English",

volume = "2016-July",

pages = "200--205",

booktitle = "2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016",

publisher = "IEEE Computer Society",

note = "17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016 ; Conference date: 14-06-2016 Through 17-06-2016",

}

TY - GEN

T1 - Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters

AU - Matsuno, Tomohiro

AU - Chatterjee, Bijoy Chand

AU - Oki, Eiji

AU - Veeraraghavan, Malathi

AU - Okamoto, Satoru

AU - Yamanaka, Naoaki

PY - 2016/7/28

Y1 - 2016/7/28

N2 - This paper aims to design a Hadoop system and evaluates the performance of a task allocation scheme. The task allocation scheme splits each job into tasks using an appropriate splitting ratio, and assigns tasks to slave servers based on server processing performance and network resource availability. We experimentally evaluate the performance of the scale out of the task allocation scheme with five machines. We focus on the configuration of jobtracker and tasktracker in Hadoop. In cases with heterogeneous Hadoop clusters, we distribute task blocks to high-capability slaves with proportionally larger-sized tasks than to low-capability slaves. We create an environment in which high-capability slaves perform more work than low-capability slaves. The experimental testbed results indicate that the task allocation scheme is effective.

AB - This paper aims to design a Hadoop system and evaluates the performance of a task allocation scheme. The task allocation scheme splits each job into tasks using an appropriate splitting ratio, and assigns tasks to slave servers based on server processing performance and network resource availability. We experimentally evaluate the performance of the scale out of the task allocation scheme with five machines. We focus on the configuration of jobtracker and tasktracker in Hadoop. In cases with heterogeneous Hadoop clusters, we distribute task blocks to high-capability slaves with proportionally larger-sized tasks than to low-capability slaves. We create an environment in which high-capability slaves perform more work than low-capability slaves. The experimental testbed results indicate that the task allocation scheme is effective.

KW - Hadoop

KW - heterogeneous clusters

KW - implementation

KW - jobtracker

UR - http://www.scopus.com/inward/record.url?scp=84991735306&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84991735306&partnerID=8YFLogxK

U2 - 10.1109/HPSR.2016.7525667

DO - 10.1109/HPSR.2016.7525667

M3 - Conference contribution

AN - SCOPUS:84991735306

VL - 2016-July

SP - 200

EP - 205

BT - 2016 IEEE 17th International Conference on High Performance Switching and Routing, HPSR 2016

PB - IEEE Computer Society

T2 - 17th IEEE International Conference on High Performance Switching and Routing, HPSR 2016

Y2 - 14 June 2016 through 17 June 2016

ER -

Task allocation scheme based on computational and network resources for heterogeneous Hadoop clusters

Abstract

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this