wombatu-kun commented on code in PR #19005:
URL: https://github.com/apache/hudi/pull/19005#discussion_r4573591777


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -88,7 +90,7 @@ case class MergeOnReadIncrementalRelationV2(override val 
sqlContext: SQLContext,
       mergeType = mergeType,
       fileSplits = fileSplits,
       includedInstantTimeSet = 
Option(includedCommits.map(_.requestedTime).toSet),
-      instantRangeOpt = getInstantRange,
+      instantRangeOpt = if (fullTableScan) getInstantRange else 
windowEndInstantRange,

Review Comment:
   Every new test reads through the file-group-reader format, so the legacy 
`buildScan` path (the streaming source with 
`hoodie.file.group.reader.enabled=false`) and its new `windowEndInstantRange` 
bound are never exercised. Could the later-commits test also read through 
`spark.baseRelationToDataFrame(MergeOnReadIncrementalRelationV2(...))`, as 
`TestLegacyParquetReadPath` does?



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderFunction.scala:
##########
@@ -150,11 +154,38 @@ private[parquet] class 
HoodieFileGroupReaderFunction(baseFileReader: Broadcast[S
         fileSliceMapping.getSlice(fileGroupName) match {
           case Some(fileSlice) if !state.isCount && 
(state.requiredSchema.nonEmpty || fileSlice.getLogFiles.findAny().isPresent) =>
             val tableConfig = state.tableState.getTableConfig
+            // For an incremental read of a MOR file group that needs merging, 
the commit-time filter in
+            // requiredFilters must not be pushed into the file reads: it 
would drop the base-file row of a
+            // record whose latest version is outside the window before the 
merge, which breaks partial
+            // updates (unchanged columns come from the base file) and 
event-time ordering. The full file
+            // group is merged with inputs bounded to the window end, and the 
merged rows are then filtered
+            // by commit time.
+            val windowCommitTimes =
+              if (state.isIncremental && 
fileSlice.getLogFiles.findAny().isPresent) {
+                incrementalWindowCommitTimes(state.requiredFilters)
+              } else {
+                Set.empty[String]
+              }
+            val postMergeCommitTimeFilter =
+              if (windowCommitTimes.isEmpty) {
+                None
+              } else {
+                
state.projectionInputSchema.getFieldIndex(HoodieRecord.COMMIT_TIME_METADATA_FIELD)
+                  .map(idx => commitTimeRowFilter(idx, windowCommitTimes))
+              }
+            val (readFilters, instantRangeOpt) =
+              if (postMergeCommitTimeFilter.isDefined) {
+                (Seq.empty[Filter], HOption.of(InstantRange.builder()
+                  
.rangeType(InstantRange.RangeType.CLOSED_CLOSED).nullableBoundary(true)
+                  .endInstant(windowCommitTimes.max).build()))

Review Comment:
   With NBCC writers, a delta commit requested before the window's last instant 
but completed after the window end passes this requested-time bound (the listed 
slice keeps its log file), so under EVENT_TIME_ORDERING its version can win the 
merge and drop the in-window record. Should the bound also exclude instants 
that completed after the window end, as the exact-match `state.instantRangeOpt` 
this branch replaces did for the full-scan fallback?



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelationV2.scala:
##########
@@ -171,6 +163,79 @@ case class MergeOnReadIncrementalRelationV2(override val 
sqlContext: SQLContext,
 
   override def shouldIncludeLogFiles(): Boolean = fullTableScan
 
+  /**
+   * Collects the incremental file slices for the given modified partitions.
+   *
+   * The slices start from the files recorded in the window's commits 
(`affectedFilesInCommits`). On a
+   * MOR table, a slice whose base file was written before the window holds 
only the window's log files,
+   * yet resolving the current value of a record changed in the window needs 
the full file slice
+   * (HUDI #18943):
+   *  - EVENT_TIME_ORDERING: a record written in the window may carry a lower 
ordering value than the
+   *    version already in the base/earlier-log, so the existing version wins.
+   *  - Partial updates: a window log block holds only the changed columns, so 
the unchanged columns
+   *    must be filled in from the base file.
+   * Such slices are replaced by the full slice of the same file group from a 
listing of their
+   * partitions (metadata-table-aware), so partitions whose touched file 
groups all have a base file in
+   * the window are not listed. The listing timeline is bounded to 
`latestCommit` so files written by
+   * later commits are not visible; the commit-time filter applied after the 
merge still restricts the
+   * returned records to the window.
+   *
+   * A slice is replaced when the listing holds every log file the window 
recorded for it, or when
+   * those log files still exist but a compaction scheduled up to 
`latestCommit` has already folded them
+   * into a newer base file. When some were removed (e.g. by cleaning) and the 
full-table-scan fallback
+   * is disabled, the recorded slice is kept, so the scan fails early with a 
file-not-found error
+   * pointing the user at 
`hoodie.datasource.read.incr.fallback.fulltablescan.enable` instead of
+   * returning a partial result.
+   */
+  private def collectIncrementalFileSlices(partitionPaths: Seq[String], 
latestCommit: String): Seq[FileSlice] = {
+    val affectedSlices = latestMergedFileSlices(new 
HoodieTableFileSystemView(metaClient, timeline, affectedFilesInCommits),
+      partitionPaths, latestCommit, _ => true)
+    val slicesWithoutBaseFile = affectedSlices.filter(slice => 
!slice.getBaseFile.isPresent)
+    if (metaClient.getTableType == HoodieTableType.COPY_ON_WRITE || 
slicesWithoutBaseFile.isEmpty) {
+      affectedSlices
+    } else {
+      val fileIdsWithoutBaseFile = slicesWithoutBaseFile.map(_.getFileId).toSet
+      val engineContext = new HoodieSparkEngineContext(new 
JavaSparkContext(sqlContext.sparkContext))
+      val fsView = 
FileSystemViewManager.createInMemoryFileSystemViewWithTimeline(
+        engineContext, metaClient, fileIndex.getMetadataConfig, 
timeline.findInstantsBeforeOrEquals(latestCommit))

Review Comment:
   With `hoodie.datasource.read.incr.skip_compact=true`, `timeline` has no 
completed compactions, so for a file group compacted before the window's writes 
this listing returns the pre-compaction slice, and the `logFilesExist` branch 
swaps it in for the window's log files, which drops that file group's in-window 
updates from the result. Could this view use the unfiltered timeline instead, 
e.g. 
`metaClient.getCommitsAndCompactionTimeline.filterCompletedAndCompactionInstants.findInstantsBeforeOrEquals(latestCommit)`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to