TY - GEN
T1 - Hadoop triggered opt/electrical data-center orchestration architecture for reducing power consumption
AU - Yamashita, Akira
AU - Muro, Wataru
AU - Hirono, Masayuki
AU - Sato, Takehiro
AU - Okamoto, Satoru
AU - Yamanaka, Naoaki
AU - Veeraraghavan, Malathi
N1 - Funding Information:
This work is supported by ”HOLST (High-speed Optical Layer 1 Switch system for Time slot switching based optical data center networks) Project” funded by New Energy and Industrial Technology Development Organization (NEDO) of Japan.
Publisher Copyright:
© 2017 IEEE.
PY - 2017/9/1
Y1 - 2017/9/1
N2 - In this paper, a data-center network (DCN) system that distinguishes Hadoop job types and allocates optical/electrical circuits to data flows depending on the types automatically is proposed. The proposed system calculates the predicted shuffle value (PSV) of the Hadoop job and determines which type of flow is allocated by comparing the PSV and the threshold value. The PSV can be calculated from a part of the input data based on the shuffle ratio, which is known to have a relationship to the heaviness of the shuffle phase of the Hadoop job. The threshold of PSV can be dynamically changed according to the network load condition to exploit the optical circuit efficiently. By orchestrating the DCN and the Hadoop system, the proposed system achieves the reduction of power consumption. In this study, the orchestration part of the proposed DCN system is implemented and the feasibility of switching data flows between optical and electrical circuits is verified.
AB - In this paper, a data-center network (DCN) system that distinguishes Hadoop job types and allocates optical/electrical circuits to data flows depending on the types automatically is proposed. The proposed system calculates the predicted shuffle value (PSV) of the Hadoop job and determines which type of flow is allocated by comparing the PSV and the threshold value. The PSV can be calculated from a part of the input data based on the shuffle ratio, which is known to have a relationship to the heaviness of the shuffle phase of the Hadoop job. The threshold of PSV can be dynamically changed according to the network load condition to exploit the optical circuit efficiently. By orchestrating the DCN and the Hadoop system, the proposed system achieves the reduction of power consumption. In this study, the orchestration part of the proposed DCN system is implemented and the feasibility of switching data flows between optical and electrical circuits is verified.
KW - Hadoop
KW - data-center network
KW - optical switch
KW - orchestration
KW - shuffle-heavy
UR - http://www.scopus.com/inward/record.url?scp=85031027866&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85031027866&partnerID=8YFLogxK
U2 - 10.1109/ICTON.2017.8025150
DO - 10.1109/ICTON.2017.8025150
M3 - Conference contribution
AN - SCOPUS:85031027866
T3 - International Conference on Transparent Optical Networks
BT - ICTON 2017 - 19th International Conference on Transparent Optical Networks
PB - IEEE Computer Society
T2 - 19th International Conference on Transparent Optical Networks, ICTON 2017
Y2 - 2 July 2017 through 6 July 2017
ER -