wombatu-kun commented on code in PR #19005:
URL: https://github.com/apache/hudi/pull/19005#discussion_r4573591777
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -88,7 +90,7 @@ case class MergeOnReadIncrementalRelationV2(override val
sqlContext: SQLContext,
mergeType = mergeType,
fileSplits = fileSplits,
includedInstantTimeSet =
Option(includedCommits.map(_.requestedTime).toSet),
- instantRangeOpt = getInstantRange,
+ instantRangeOpt = if (fullTableScan) getInstantRange else
windowEndInstantRange,
Review Comment:
Every new test reads through the file-group-reader format, so the legacy
`buildScan` path (the streaming source with
`hoodie.file.group.reader.enabled=false`) and its new `windowEndInstantRange`
bound are never exercised. Could the later-commits test also read through
`spark.baseRelationToDataFrame(MergeOnReadIncrementalRelationV2(...))`, as
`TestLegacyParquetReadPath` does?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderFunction.scala:
##########
@@ -150,11 +154,38 @@ private[parquet] class
HoodieFileGroupReaderFunction(baseFileReader: Broadcast[S
fileSliceMapping.getSlice(fileGroupName) match {
case Some(fileSlice) if !state.isCount &&
(state.requiredSchema.nonEmpty || fileSlice.getLogFiles.findAny().isPresent) =>
val tableConfig = state.tableState.getTableConfig
+ // For an incremental read of a MOR file group that needs merging,
the commit-time filter in
+ // requiredFilters must not be pushed into the file reads: it
would drop the base-file row of a
+ // record whose latest version is outside the window before the
merge, which breaks partial
+ // updates (unchanged columns come from the base file) and
event-time ordering. The full file
+ // group is merged with inputs bounded to the window end, and the
merged rows are then filtered
+ // by commit time.
+ val windowCommitTimes =
+ if (state.isIncremental &&
fileSlice.getLogFiles.findAny().isPresent) {
+ incrementalWindowCommitTimes(state.requiredFilters)
+ } else {
+ Set.empty[String]
+ }
+ val postMergeCommitTimeFilter =
+ if (windowCommitTimes.isEmpty) {
+ None
+ } else {
+
state.projectionInputSchema.getFieldIndex(HoodieRecord.COMMIT_TIME_METADATA_FIELD)
+ .map(idx => commitTimeRowFilter(idx, windowCommitTimes))
+ }
+ val (readFilters, instantRangeOpt) =
+ if (postMergeCommitTimeFilter.isDefined) {
+ (Seq.empty[Filter], HOption.of(InstantRange.builder()
+
.rangeType(InstantRange.RangeType.CLOSED_CLOSED).nullableBoundary(true)
+ .endInstant(windowCommitTimes.max).build()))
Review Comment:
With NBCC writers, a delta commit requested before the window's last instant
but completed after the window end passes this requested-time bound (the listed
slice keeps its log file), so under EVENT_TIME_ORDERING its version can win the
merge and drop the in-window record. Should the bound also exclude instants
that completed after the window end, as the exact-match `state.instantRangeOpt`
this branch replaces did for the full-scan fallback?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -171,6 +163,79 @@ case class MergeOnReadIncrementalRelationV2(override val
sqlContext: SQLContext,
override def shouldIncludeLogFiles(): Boolean = fullTableScan
+ /**
+ * Collects the incremental file slices for the given modified partitions.
+ *
+ * The slices start from the files recorded in the window's commits
(`affectedFilesInCommits`). On a
+ * MOR table, a slice whose base file was written before the window holds
only the window's log files,
+ * yet resolving the current value of a record changed in the window needs
the full file slice
+ * (HUDI #18943):
+ * - EVENT_TIME_ORDERING: a record written in the window may carry a lower
ordering value than the
+ * version already in the base/earlier-log, so the existing version wins.
+ * - Partial updates: a window log block holds only the changed columns, so
the unchanged columns
+ * must be filled in from the base file.
+ * Such slices are replaced by the full slice of the same file group from a
listing of their
+ * partitions (metadata-table-aware), so partitions whose touched file
groups all have a base file in
+ * the window are not listed. The listing timeline is bounded to
`latestCommit` so files written by
+ * later commits are not visible; the commit-time filter applied after the
merge still restricts the
+ * returned records to the window.
+ *
+ * A slice is replaced when the listing holds every log file the window
recorded for it, or when
+ * those log files still exist but a compaction scheduled up to
`latestCommit` has already folded them
+ * into a newer base file. When some were removed (e.g. by cleaning) and the
full-table-scan fallback
+ * is disabled, the recorded slice is kept, so the scan fails early with a
file-not-found error
+ * pointing the user at
`hoodie.datasource.read.incr.fallback.fulltablescan.enable` instead of
+ * returning a partial result.
+ */
+ private def collectIncrementalFileSlices(partitionPaths: Seq[String],
latestCommit: String): Seq[FileSlice] = {
+ val affectedSlices = latestMergedFileSlices(new
HoodieTableFileSystemView(metaClient, timeline, affectedFilesInCommits),
+ partitionPaths, latestCommit, _ => true)
+ val slicesWithoutBaseFile = affectedSlices.filter(slice =>
!slice.getBaseFile.isPresent)
+ if (metaClient.getTableType == HoodieTableType.COPY_ON_WRITE ||
slicesWithoutBaseFile.isEmpty) {
+ affectedSlices
+ } else {
+ val fileIdsWithoutBaseFile = slicesWithoutBaseFile.map(_.getFileId).toSet
+ val engineContext = new HoodieSparkEngineContext(new
JavaSparkContext(sqlContext.sparkContext))
+ val fsView =
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(
+ engineContext, metaClient, fileIndex.getMetadataConfig,
timeline.findInstantsBeforeOrEquals(latestCommit))
Review Comment:
With `hoodie.datasource.read.incr.skip_compact=true`, `timeline` has no
completed compactions, so for a file group compacted before the window's writes
this listing returns the pre-compaction slice, and the `logFilesExist` branch
swaps it in for the window's log files, which drops that file group's in-window
updates from the result. Could this view use the unfiltered timeline instead,
e.g.
`metaClient.getCommitsAndCompactionTimeline.filterCompletedAndCompactionInstants.findInstantsBeforeOrEquals(latestCommit)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]