kevinjqliu opened a new pull request, #16311: URL: https://github.com/apache/iceberg/pull/16311
Backport of #15992 to `spark/v3.4`. **Depends on #16307** (backport of #15683 `SerializableFileIOWithSize`). This branch is stacked on top of that PR; please merge #16307 first. Once #16307 lands the diff here will be limited to the async micro-batch changes. ## Adaptations from the source PR - `SparkMicroBatchStream.java` was replaced wholesale with the v3.5 post-#15992 version because v3.4 had structural drift relative to v3.5. The refactor extracts the planning logic into the new planner classes (`BaseSparkMicroBatchPlanner`, `SyncSparkMicroBatchPlanner`, `AsyncSparkMicroBatchPlanner`, `MicroBatchUtils`) and there are no v3.4-only features in this file, so a verbatim copy keeps the implementations aligned. - `TestStructuredStreamingRead3.java` was likewise replaced with the v3.5 version (which adds parameterized sync/async coverage). The only non-mechanical change was using `SparkCatalogConfig.SPARK` instead of `SparkCatalogConfig.SPARK_SESSION` because v3.4 still uses the older enum name. - All other files in #15992 applied cleanly via `git apply --3way`. ## Validation - `./gradlew -DsparkVersions=3.4 :iceberg-spark:iceberg-spark-3.4_2.12:classes :iceberg-spark:iceberg-spark-3.4_2.12:testClasses` - `./gradlew -DsparkVersions=3.4 :iceberg-spark:iceberg-spark-3.4_2.12:test --tests "*TestAsyncSparkMicroBatchPlanner*" --tests "*TestMicroBatchPlanningUtils*"` - `./gradlew -DsparkVersions=3.4 :iceberg-spark:iceberg-spark-3.4_2.12:test --tests "org.apache.iceberg.spark.source.TestStructuredStreamingRead3.testReadStreamOnIcebergTableWithMultipleSnapshots"` - `./gradlew :iceberg-spark:iceberg-spark-3.4_2.12:spotlessApply :iceberg-spark:iceberg-spark-extensions-3.4_2.12:spotlessApply` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
