yihua commented on code in PR #19914:
URL: https://github.com/apache/hudi/pull/19914#discussion_r4203311468
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -262,8 +273,11 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
// For overly large single files, we can use multiple concurrent tasks to
read them, thereby reducing the overall job reading time consumption
val superSplitable = super.isSplitable(sparkSession, options, path)
val isLance = hoodieFileFormat == HoodieFileFormat.LANCE
- val splitable = !isMOR && !isIncremental && !isBootstrap && !isLance &&
superSplitable
- logDebug(s"isSplitable: $splitable, super.isSplitable: $superSplitable,
isMOR: $isMOR, isIncremental: $isIncremental, isBootstrap: $isBootstrap")
+ // COW incremental reads have no log files to merge, so file splitting is
safe.
+ // MOR, CDC and bootstrap reads need to disable splitting.
+ val splitable = !isMOR && !isCDC && !isBootstrap && !isLance &&
superSplitable
Review Comment:
non-blocking: Now that COW incremental reads can be split, could we add a
case that actually produces more than one split per base file (small parquet
block size plus a low `spark.sql.files.maxPartitionBytes`) and asserts the
incremental range result against the unsplit read? The existing ranges test
only has single-split files, so the splitting change itself is currently
unguarded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
