benmerveille commented on issue #9397:
URL: https://github.com/apache/hop/issues/9397#issuecomment-6727161554

   My only concern with my method is the need to load a full dataset, and it 
can be millions of records, to perform data quality checks on a per column 
based (Single column DQ checks).  This would exhaust the memory usage in the 
running JVM I believe.
   -> For each dimensional column of the dataset I need to perform a distinct 
selection of values
   -> All distinct values of dimensional columns are evaluated on per Data 
Quality criteria (Similar as for Data Validation Transform)
   -> If the column is a metric/fact then some DQ rules can be evaluated as 
well.
   
   Another approach would be to store the dataset as Hive external table, where 
physical file format would be optimized for column store like parquet or ORC.  
Then I can build a pipeline who loops across all single dimensional column and 
retrieve distinct values with a Select.  Afterwards another pipeline can 
perform DQ check at column level.  But I want to definitively try to involve 
"Data Validation" transform as well.  
   
   I'll definitively try both approaches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to