A Novel Approach for Generic log analyser

Abstract

To capture the meaning of this emerging trend the term big data was formulated. In addition to its sheer volume, big data also shows other unique characteristics as compared with traditional data. For instance, big data requires more real-time analysis and is commonly unstructured. For data acquisition, transmission, storage, and large-scale data processing components, this improvement calls for new system architectures. In all databases there are log ?les that keep records of changes in database. This can include tracking distinct user events. For log processing Apache Hadoop is used. A standard part of large applications are the log files and are essential in operating systems, computer networks and distributed systems. The only ways to identify and locate an error in software log ?les are used, because log ?le analysis is not affected by anytime-based issues known as probe effect. This is opposite to analysis of a running program, when the investigative process can obstruct with time-critical or resource-critical conditions within the analyzed program. The global goal of this project is to design a generic log analyzer using hadoop map-reduce framework. Different kinds of log ?les such as- Email logs, Web logs; Firewall logs Server logs, Call data logs are analyzed using generic log analyzer.