tkaymak opened a new pull request, #41307: URL: https://github.com/apache/beam/pull/41307
Fixes #41297, part of #36841. In streaming, when one shuffle feeds two stateful transforms, for example a `Reshuffle` followed by two `GroupByKey`s, Spark reuses the shuffle for the second consumer. Spark 4.0.2 then gives the stateful operator above the reused shuffle no input watermark, so its timers never fire and its output is lost. This is [SPARK-60862](https://issues.apache.org/jira/browse/SPARK-60862), fixed only in Spark 4.4.0, which is not released yet. The runner now sets `spark.sql.exchange.reuse` to false on the session before it starts its streaming queries. It is not restored afterwards, since pipelines can share a session and a restore could reach a query that is still starting. Reuse off only costs recomputation. The TODO points at #41297 to drop it once the runner uses Spark 4.4.0 or later. `PAssertStreamingTest.testReshuffleFeedingTwoGroupByKeys` runs a `Reshuffle` into two `GroupByKey`s. It fails on master (one output missing) and passes with this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
