Analysis and Implementation of a Data Pre-processing System

Abstract

Today, we generate vast amounts of data each day, most of which is unstructured, incomplete, and more importantly inconsistent. In order to overcome the shortcomings of manually analyzing data, we have designed a data pre-processing system that cleans, integrates and transforms a data set. User specified files are integrated and stored in an automatically generated output file. A data table is also generated and the values are updated into their corresponding locations. In order to clean the missing values, we perform mean, median and mode on the complete data tuples in order to replace the missing data with these values. Transformation of our data is done by normalizing from wide ranges to narrow ranges [-1, 1] by implementing decimal scaling normalization, min-max normalization and z-score normalization .The processed data is stored in a .ARFF file which can be used for business requirements in a productive way.