nsivabalan commented on code in PR #17931:
URL: https://github.com/apache/hudi/pull/17931#discussion_r2729679889
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/validator/SqlQueryEqualityPreCommitValidator.java:
##########
@@ -52,23 +52,34 @@ protected String getQueryConfigName() {
@Override
protected void validateUsingQuery(String query, String prevTableSnapshot,
String newTableSnapshot, SQLContext sqlContext) {
- Dataset<Row> prevRows = executeSqlQuery(
- sqlContext, query, prevTableSnapshot, "previous state").cache();
- log.info("Total rows in prevRows " + prevRows.count());
- Dataset<Row> newRows = executeSqlQuery(
- sqlContext, query, newTableSnapshot, "new state").cache();
- log.info("Total rows in newRows " + newRows.count());
- printAllRowsIfDebugEnabled(prevRows);
- printAllRowsIfDebugEnabled(newRows);
- boolean areDatasetsEqual = prevRows.intersect(newRows).count() ==
prevRows.count();
- log.info("Completed Equality Validation, datasets equal? " +
areDatasetsEqual);
- if (!areDatasetsEqual) {
- log.error("query validation failed. See stdout for sample query results.
Query: " + query);
- System.out.println("Expected result (sample records only):");
- prevRows.show();
- System.out.println("Actual result (sample records only):");
- newRows.show();
- throw new HoodieValidationException("Query validation failed for '" +
query + "'. See stdout for expected vs actual records");
+ Dataset<Row> prevRows = null;
+ Dataset<Row> newRows = null;
+ try {
+ prevRows = executeSqlQuery(
+ sqlContext, query, prevTableSnapshot, "previous state").cache();
+ log.info("Total rows in prevRows " + prevRows.count());
+ newRows = executeSqlQuery(
+ sqlContext, query, newTableSnapshot, "new state").cache();
+ log.info("Total rows in newRows " + newRows.count());
+ printAllRowsIfDebugEnabled(prevRows);
+ printAllRowsIfDebugEnabled(newRows);
+ boolean areDatasetsEqual = prevRows.intersect(newRows).count() ==
prevRows.count();
+ log.info("Completed Equality Validation, datasets equal? " +
areDatasetsEqual);
+ if (!areDatasetsEqual) {
+ log.error("query validation failed. See stdout for sample query
results. Query: " + query);
+ System.out.println("Expected result (sample records only):");
Review Comment:
can we remove the `System.out.println` statements.
either do log.info or log.error, but not SOPs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
