cshuo opened a new issue, #19818: URL: https://github.com/apache/hudi/issues/19818
### Describe the problem HoodieFlinkEngineContext.mapGroupsByKey submits only a lazy Stream.map transformation to executeParallelStream. The transform returns a Stream before any terminal operation runs. executeParallelStream then shuts down its dedicated ForkJoinPool, and the caller performs flatMap and collect afterward. As a result, processFunc and consumption of its result iterator do not run in the newly created ForkJoinPool. Because the source stream remains parallel, execution normally falls back to ForkJoinPool.commonPool when invoked from a regular Flink thread. The requested dedicated parallelism is therefore not honored and the work can contend with unrelated common-pool work. ### Current execution order 1. Create groupedValues.parallelStream. 2. Submit stream.map to the dedicated pool. 3. Return the unevaluated Stream from the submitted task. 4. Shut down the dedicated pool. 5. Execute flatMap and collect outside the pool, which finally invokes processFunc. Exceptions thrown by processFunc also occur outside executeParallelStream and bypass its HoodieException wrapping. There is an additional empty-input issue: groupedValues.size is zero, causing new ForkJoinPool(0) to throw IllegalArgumentException. ### Expected behavior The complete parallel stream pipeline, including the terminal collect operation, should execute inside the dedicated ForkJoinPool. Empty input should return empty HoodieData without constructing a zero-parallelism pool. ### Suggested fix Move flatMap and collect into the transform passed to executeParallelStream, handle empty groupedValues explicitly, and add tests that verify processFunc runs in a non-common ForkJoinPool and that empty input succeeds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
