tkaymak opened a new pull request, #39939: URL: https://github.com/apache/beam/pull/39939
Second slice of the Spark 4 Structured Streaming work split out of #39576, following the dispatch seam in #39906. Addresses #36841. A Structured Streaming query serializes two Spark internals behind the runner's back, so streaming pipelines fail under spark.kryo.registrationRequired=true without these registrations: - StateSchemaMetadata is broadcast for every transformWithState query, hit on the very first micro-batch of any pipeline using Beam state or timers. - MemoryWriterCommitMessage is the memory sink's commit message, nested inside the already registered DataWritingSparkTaskResult. Design notes, mirroring the discussion in the POC: - Both are registered by name because the shared base also compiles against Spark 3, where neither class exists. ClassNotFound is caught and logged, so Spark 3 behavior is unchanged. - Both use a JavaSerializer so their whole Scala object graph (immutable.Map, StructType, avro Schema, Row) is covered without tracking Spark's internal field layout across versions. Neither is on a hot path, one broadcast per query and one message per task commit. - The registration call sits at the end of the registrator on purpose. Kryo auto assigns registration ids sequentially, so appending these conditional by-name registrations keeps the auto assigned ids of every registration above identical on Spark 3 and Spark 4 classpaths. SparkKryoRegistratorStreamingTest locks down both registrations and the id parity. No behavior change for Spark 3, full :runners:spark:3:test and :runners:spark:4:test suites green locally on JDK 17 plus spotless, checkstyle and a live ErrorProne compile. Next slices in order: the DataSourceV2 unbounded source, the state and timer bridge on transformWithState, and the translators with the end to end tests. End to end evidence remains in draft #39576. R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
