Quality of data has a vital role in predictive analytics that shapes the accuracy, reliability and utility of its outcomes. Data quality dimensions encompass accuracy, completeness, consistency, timeliness, reliability, and relevance. Common challenges in ensuring high data quality include human errors in manual data handling, interpretation, and processing, as well as issues related to inadequate and inconsistent data accuracy. The establishment of a robust data governance framework, the formulation of clear data standards, and the regular monitoring of data quality metrics are essential for effective data quality management. Standards such as ISO 8000 and ISO/IEC 25012 address key data quality issues, including accuracy, completeness, credibility, accessibility, compliance, confidentiality, efficiency, precision, and traceability. Furthermore, the General Data Protection Regulation (GDPR) underscores the rights of individuals to data accuracy, while the Basel Accords mandate that banks maintain precise and comprehensive records of their lending activities, risk exposures, and financial reserves.
Completness refers to the sufficient information about a product that enables decision-makers to generate a clear insight about the product. Similarly, uniqueness of the data is vital to assure the avoidance of overlaps in data and analysis.
Efficiency of the data is affected by inconsistencies and incompleteness of the data, often resulting low accuracy. Quality of raw data influences the results of analysis whether it is financial modelling or spatial modelling which depends on spatial resources such as satellite images, aerial photography and digital copies. Outcomes of the modelling could be affected by fluctuations in data, leading to overfitting. Overfitting of the data can be corrected by using cross-validation, regularisation, pruning and ensemble methods. Whether it’s a linear regression or classification model, accuracy determines the model performance. Error is a measure of difference between the actual values and predicted values. Errors can be standard or relative; standard error calculates the scale of the errors whereas relative errors such as Mean Percentage Error (MPE) and Mean Absolute Percentage Error (MAPE) compute average errors as a percentage. Mean Absolute Deviation(MAD) is a standard measure to calculate the scale of the errors. MSE calculates the error by squaring the errors that increases its sensitivity to outliers, more applicable for the computations where large errors should be avoided.
Handling of missing data is essential to achieve desired outcomes. Missing data can be handled with data imputation techniques such as mean, median or regression imputation. For time series data, forward fill or imputing missing values by carrying forward the last observed value and backward fill or filling the next available value, maintains the temporal order of the data.
Errors can be systematic or random errors. Systematic errors are predictable that occurs as a result of bias in the data collection process. For example; inaccuracies in the rainfall data by 0.5 mm less than normal value due to faulty measurement device. Random errors can occur as a result of unexpected error during data collection such as errors of pollution level measurement. Random errors don’t have predictable pattern and the variations are not consistent. Errors in data can be prevented during the pre-processing steps whether it is machine learning research or spatial modelling. Data cleansing techniques such as standardisation, missing data handling, validation and outlier detection can be applied to improve reliability and accuracy of the data.

Leave a Reply