wgtmac commented on code in PR #1026:
URL: https://github.com/apache/parquet-mr/pull/1026#discussion_r1105193381
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/rewrite/ParquetRewriter.java:
##########
@@ -183,12 +186,61 @@ public ParquetRewriter(TransParquetFileReader reader,
}
}
+ // Open all input files to validate their schemas are compatible to merge
+ private void openInputFiles(List<Path> inputFiles, Configuration conf) {
+ Preconditions.checkArgument(inputFiles != null && !inputFiles.isEmpty(),
"No input files");
+
+ for (Path inputFile : inputFiles) {
Review Comment:
It is tricky to set a max size, whether in terms of number of files or total
file sizes. Based on my knowledge, the performance varies when input files have
different number of rows, number of columns, number of blocks, block sizes,
etc. So it would be better for the caller to do the planning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@parquet.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
