tkaymak opened a new pull request, #40766: URL: https://github.com/apache/beam/pull/40766
Part of #36841. In streaming mode the Spark 4 runner now accepts bounded Read and Impulse. A streaming branch in `SparkTransformOverrides` replaces `PrimitiveBoundedRead` and `Impulse` with a `PrimitiveUnboundedRead` over `BoundedToUnboundedSourceAdapter` while keeping the output bounded, mirroring the Dataflow `StreamingBoundedRead`. This is what `Create` and PAssert need. To let global window results complete, the unbounded source emits one end of stream row with an empty payload at `TIMESTAMP_MAX_VALUE`. It is emitted only once every split has reported end of stream through a done marker under the source checkpoint, and a marker is written only after the batch that drained its split is committed. So the Spark watermark, which is the maximum over partitions, passes the end of the global window only after the slowest split is drained, also after a restart. The row is dropped at decode. This needs `watermarkDelayMillis` under one day, noted in the option description. The `Never` trigger that PAssert applies is now allowed on GroupByKey. A failing streaming query surfaces as `Pipeline.PipelineExecutionException` with the user exception as cause, as in batch. `PAssertStreamingTest` covers `containsInAnyOrder` in the global window and in fixed windows, uneven split progress, `empty`, a failing assertion that fails the pipeline, and `Create` with one and three elements. With the end of stream row disabled all cases fail, and without the all splits gate the uneven case loses the slow split's last element. Full `:runners:spark:4:test` and `:runners:spark:3:test` pass locally. The streaming ValidatesRunner suite with its workflow and the CHANGES.md entry follow in the next PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
