Weblog Analysis with Map-Reduce and Performance Comparison of Single v/s Multinode Hadoop Cluster

Abstract

In this internet era websites are useful source of many information. Because of growing popularity of World Wide Web a website receives thousands to millions requests per day. Thus, the log files of such websites are growing in size day by day. These log files are useful source of information to identify user’s behavior. This paper is an attempt to analyze the weblogs using Hadoop Map-Reduce algorithm. Hadoop is an open source framework that provides parallel storage and processing of large datasets. This paper makes use of Hadoop’s this feature to analyze the large, Semi structured dataset of websites log. The performance of the algorithm is compared on pseudo distributed and fully distributed mode Hadoop cluster.