mxm commented on issue #10386: [BEAM-8972] Add Jenkins job with Combine test for portable Java URL: https://github.com/apache/beam/pull/10386#issuecomment-601606381 >@mxm I ran into a problem with TaskManagers timing out in a streaming case, the latest failed test is an example of this. Do you have any insight into why this might have happened? @mwalenia Not from the top of my head. I would need a stack trace or log to see what happened. >Also there's an issue of intermittent OutOfDirectMemoryError getting thrown by gRPC on batch as well (eg. here) > Also there's an issue of intermittent `OutOfDirectMemoryError` getting thrown by gRPC on batch as well (eg. [here](https://builds.apache.org/job/beam_LoadTests_Java_Combine_Portable_Flink_Batch_PR/30/console) ) Flink 1.10 configures has changed the memory configuration. It looks like GRPC's direct memory allocation is competing with Flink's managed memory. Flink allocates a fix amount of direct memory upfront leaving only a small amount left. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#memory-configuration, especially `taskmanager.memory.framework.off-heap.size` and `taskmanager.memory.jvm-overhead.fraction`.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
