Effect of counters in performance of hadoop
Keywords:
Big Data, Map Reduce, hadoop, Hadoop Distributed File System (HDFS), CPU time spent, Size of data, number of blocks, Heap Size, HDFS Bytes Read, Spilled RecordsAbstract
Recent technological advancements have led to an overflow of data from distinctive domains (e.g., health care and scientific sensors, user-generated data, Internet and financial companies, and supply chain systems) over the past two decades [1]. Big data is commonly unstructured, huge in volume and require more real-time analysis. This paper discusses a Big Data problem from NCDC for huge volume of weather data collected from various parts of world. We had generated map () and reduce () function for solving this problem and experimental results of these applications on a Hadoop cluster are being discussed. In this paper, performance of above application has been shown with respect to some counters available.
References
Maurya and Mahajan, S., “Performance analysis of MapReduce programs on Hadoop cluster”, Information and Communication Technologies (WICT), 2012 World Congress, ISBN:978-1-4673-4806-5, Oct. 30 2012.
A. Alexandrov, R. Bergmann, S. Ewen, J.C. Freytag, F. Hueske, A. Heise, O. Kao, M. Leich, U. Leser, V. M. Felix Naumann, M. Peters, A. Rheinländer, M. J. Sax and S. Schelter, “The Stratosphere Plateform for Big Data Analytics”, The VLDB Journal, Springer-Verlag Berlin Heidelberg, July 2013.
Shilpa and Manjit Kaur, “Big Data Visualization tool with Advancement of Chalange” International Journal of Advanced Research in Computer and Software Engineering”, ISSN: 2277 128X, Volume 4, Issue 3, March 2014.
D. Borthakur, “The Hadoop Distributed File System: Architecture and Design”.
Han Hu, Yonggang Wen , Tat-Seng Chua and Xuelong Li, “Toward Scalable Systems for Big Data Analytics: A Technology Tutorial”, ISSN :2169-3536, INSPEC Accession Number: 14429402
P. Kumar and Dr. V. S. Rathore, “Efficient Capabilities of Processing of Big Data using Hadoop Map Reduce”, International Journal of Advanced Research in Computer and Communication Engineering, Vol. 3, Issue 6, June 2014.
Shilpa and Manjit Kaur, “Big Data Visualization tool with Advancement of Chalange” International Journal of Advanced Research in Computer and Software Engineering”, ISSN: 2277 128X, Volume 4, Issue 3, March 2014.
X. Zhang, “Simple example to demonstrate how does the map reduce work”, June 2013.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2015 COMPUSOFT: An International Journal of Advanced Computer Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
©2023. COMPUSOFT: AN INTERNATIONAL OF ADVANCED COMPUTER TECHNOLOGY by COMPUSOFT PUBLICATION is licensed under a Creative Commons Attribution 4.0 International License. Based on a work at COMPUSOFT: AN INTERNATIONAL OF ADVANCED COMPUTER TECHNOLOGY. Permissions beyond the scope of this license may be available at Creative Commons Attribution 4.0 International Public License.
