tkaymak opened a new issue, #41297:
URL: https://github.com/apache/beam/issues/41297
### What happened?
In streaming mode the Spark 4 Structured Streaming runner can silently drop
the output of a stateful transform. When one shuffle feeds two stateful
transforms, for example a `Reshuffle` followed by two `GroupByKey`s, Spark
reuses the shuffle for the second consumer. Spark 4.0.2 then gives the stateful
operator above the reused shuffle no input watermark, so its event time timers
never fire and its windows never close.
Reproduced on master: `Create.of(1, 2, 3)`, `WithKeys.of("k")`,
`Reshuffle.of()`, two `GroupByKey`s with a sum each, `Flatten`, `PAssert`
expecting both sums. Only the first sum arrives, 2 of 2 runs. It also
reproduces in plain Spark 4.0.2 with two `transformWithState` operators behind
one reused exchange: the second fires 0 timers with
`spark.sql.exchange.reuse=true` and all of them with `false`.
This is [SPARK-60862](https://issues.apache.org/jira/browse/SPARK-60862),
fixed in Spark 4.4.0 only. No released Spark version has the fix and the runner
pins 4.0.2. Until the runner moves to a fixed Spark, it should disable exchange
reuse for its streaming queries.
Part of #36841.
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [x] Component: Spark Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
