tkaymak commented on issue #41199: URL: https://github.com/apache/beam/issues/41199#issuecomment-6836470931
Root cause found, this is #41198. `CombineTest` uses `SPLIT_HOT_KEY_FANOUT = input -> Math.random() < 0.5 ? 3 : 0`, so every element is routed hot or cold at random. In streaming the hot and the cold branch each rerun the splitting DoFn with fresh random values, so an element can land in both branches or in neither. Measured with the hot key rejection lifted locally, `Combine.perKey(Mean.of()).withHotKeyFanout(...)` on the test input, expected `a 2.0, b 7.0`: - deterministic fanout (`a` hot, `b` cold): correct in 3 of 3 runs - random fanout as in the test: wrong in 3 of 3 runs, `a` 1.75, 2.147, 2.0645, `b` 5.8, 6.33, 6.5 A constant fanout gives correct results, but the runner cannot tell a deterministic fanout function from a random one, so the streaming translator keeps rejecting hot key fanout until #41198 is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
