tkaymak opened a new issue, #41199:
URL: https://github.com/apache/beam/issues/41199
### What needs to happen?
`Combine.perKey().withHotKeyFanout(...)` computes a wrong result on the
Spark 4 Structured Streaming runner in streaming mode.
`CombineTest$BasicTests.testHotKeyCombining` expects the mean `KV{a, 2.0}` and
gets `KV{a, 2.064516129032258}`. It was observed with #40949 applied, which
stops rejecting the stateless multi output ParDo that splits hot and cold keys.
The split is deterministic per element, so the cause is in the hot path
combine, not yet found. In batch mode the same test is excluded for a different
failure, `INTERNAL_ERROR Couldn't find method fromByteArray`.
Until this is fixed the streaming translator rejects
`Combine.PerKeyWithHotKeyFanout`, and `testHotKeyCombining` and
`testHotKeyCombiningWithAccumulationMode` stay excluded from the streaming
ValidatesRunner suite.
Part of #36841.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Spark Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
