benmerveille commented on issue #9397: URL: https://github.com/apache/hop/issues/9397#issuecomment-6727161554
My only concern with my method is the need to load a full dataset, and it can be millions of records, to perform data quality checks on a per column based (Single column DQ checks). This would exhaust the memory usage in the running JVM I believe. -> For each dimensional column of the dataset I need to perform a distinct selection of values -> All distinct values of dimensional columns are evaluated on per Data Quality criteria (Similar as for Data Validation Transform) -> If the column is a metric/fact then some DQ rules can be evaluated as well. Another approach would be to store the dataset as Hive external table, where physical file format would be optimized for column store like parquet or ORC. Then I can build a pipeline who loops across all single dimensional column and retrieve distinct values with a Select. Afterwards another pipeline can perform DQ check at column level. But I want to definitively try to involve "Data Validation" transform as well. I'll definitively try both approaches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
