Building a Platform for Data Mining in a Big Data Environment Combining R-language with Hadoop

Abstract

With the exponential growth of data in various industries, the need for effective data mining platforms has become crucial. This research focuses on constructing a data mining platform suitable for processing large-scale and diverse data sets. The platform leverages the power of the R language and utilizes the scalability and processing capabilities of Hadoop. The system architecture encompasses a physical layer, virtualization layer, service layer, and application layer. Heterogeneous hardware resources are deployed at the physical layer, while virtual machines are created and managed using CloudStack at the virtualization layer. The service layer integrates the R language, enabling the implementation of various data mining functions. Finally, the application layer provides users with a user-friendly interface to customize flow paths and configure parameters. The proposed method effectively processes big data, facilitates comprehensive data analysis, and achieves high processing efficiency.