tkaymak commented on issue #36841: URL: https://github.com/apache/beam/issues/36841#issuecomment-5831832049
Status update on slice 5b. Stateful ParDo, meaning state plus event time timers, now runs on the Spark 4 Structured Streaming runner through `transformWithState`. It is coded and tested locally. The PR is not opened yet. There is one known blocker for 5c. Spark has no watermark hold. So output from an event time timer is emitted after the watermark has already passed the timer timestamp. That is harmless when only stateless ParDos follow. A downstream GroupByKey would drop that output as late. Before 5c supports a GroupByKey after a stateful ParDo, it needs a way to hold back output timestamps, or it has to reject that pipeline shape. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
