tkaymak opened a new pull request, #39906: URL: https://github.com/apache/beam/pull/39906
First slice of the Spark 4 Structured Streaming work, split out of the POC #39576 as announced there. Addresses #36841. This prepares the structured streaming runner for a streaming translator without changing batch behavior on either Spark version. - Adds PipelineTranslatorFactory, the seam the Spark 4 module will shadow to dispatch streaming pipelines. The shared base rejects streaming with a clear message instead of the previous generic checkArgument. - Opens up EvaluationContext (non final, protected constructor, leaves()) and adds a no-op stop() so a streaming context can override evaluation later. - Adds a createEvaluationContext hook to PipelineTranslator and skips the persist and lineage breaking optimizations for streaming datasets. - Plumbs the EvaluationContext into SparkStructuredStreamingPipelineResult so cancel() can stop a running streaming query. - Defaults the state store provider to RocksDB, required by Spark 4 transformWithState and inert for batch. - Adds the watermarkDelayMillis, maxRecordsPerMicroBatch, maxBatchDurationMillis and streamingStopAfterIdleBatches options. - Narrows the Spark 4 test source override exclude to the legacy DStream package. The previous glob also matched structuredstreaming and would have silently dropped structured streaming tests. No behavior change for Spark 3, proven by the full :runners:spark:3:test suite (220 tests, 0 failures) and :runners:spark:4:test (196 tests, 0 failures) locally on JDK 17, plus spotless, checkstyle and a live ErrorProne compile. The end to end evidence that this seam carries a working streaming runner is in draft #39576. Remaining slices, in order: Kryo registrations, the DataSourceV2 unbounded source, the state and timer bridge on transformWithState, and the translators with the end to end tests. R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
