danny0405 commented on code in PR #20679:
URL: https://github.com/apache/hudi/pull/20679#discussion_r4625054896
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSourceV1.scala:
##########
@@ -97,6 +99,18 @@ class HoodieStreamSourceV1(sqlContext: SQLContext,
}
}
+ override protected def admissionParameters: Map[String, String] = parameters
+
+ override protected def admissionInitialOffset: HoodieSourceOffset =
initialOffsets
+
+ override protected def completedWriteOffsets: Seq[String] = {
+ metaClient.reloadActiveTimeline()
+ val timeline = handleHollowCommitIfNeeded(
+ metaClient.getActiveTimeline.getWriteTimeline.filterCompletedInstants(),
metaClient, hollowCommitHandlingMode)
+ timeline.getInstants.asScala.map(instant =>
+ if (hollowCommitHandlingMode == USE_TRANSITION_TIME)
instant.getCompletionTime else instant.requestedTime()).toSeq
Review Comment:
Fixed in de440b7004e. V1 no longer translates getBatch boundaries based on
the target write version: BLOCK/FAIL keep requested times, and
USE_TRANSITION_TIME keeps completion times throughout the source and V1
relation.
Added cap=1 ProcessingTime regressions on a v6 MOR table with a compaction
completing after subsequent delta commits, in both BLOCK and
USE_TRANSITION_TIME modes. Each key in the requested range is consumed exactly
once, with one data commit per batch. Added capped CDC coverage and validation
of invalid options at readStream.load(); the config is now marked advanced.
Validated on JDK 17 / Spark 3.5: 4 admission-control tests, 21
streaming-source tests, and the separate out-of-order source test passed (2
expected Spark-4.1-only skips). Style/license checks passed. This revision has
not been rerun on Spark 4.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]