Adaptive Data Quality Scoring Using Machine Learning for Large-Scale Enterprise Data Pipelines: A Systematic Review

Abstract

Enterprise data pipelines increasingly ingest heterogeneous, high-velocity data, and the quality of that data directly determines the reliability of downstream analytics and machine learning systems. This review synthesises the literature on data quality assessment and on the use of machine learning to automate and scale that assessment. It examines foundational multidimensional quality frameworks, data profiling and rule-based error detection, supervised and unsupervised approaches to anomaly and duplicate detection, the theory of probabilistic record linkage, and production validation systems for continuously evolving pipelines. The synthesis traces a transition from static, rule-based checking towards learned scoring functions that adapt to schema evolution, concept drift, and distributional shift. Holistic, inference-based repair methods and constraint-learning validators substantially reduce the manual specification burden of earlier rule-based approaches, while ensemble methods such as random forests and gradient-boosted trees provide effective classifiers for error and duplicate detection over mixed-type data. Three problems persist across the literature: the scarcity of labelled ground truth for supervised scoring, the entanglement of data quality with the broader technical debt of production machine learning systems, and the difficulty of keeping scores stable under non-stationary data distributions. The review argues that adaptive quality scoring is best structured as three coordinated layers—dimension-specific statistical profiling, learned anomaly and duplicate detection models, and drift-aware recalibration—and offers guidance on integrating quality scoring with automated pipeline orchestration.