Just in case, make sure that you are not using Kafka SSL port without setting security protocol, see [1]
[1] https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4090 ________________________________ From: bat man <tintin0...@gmail.com> Sent: Wednesday, July 14, 2021 10:55:54 AM To: Timo Walther <twal...@apache.org> Cc: user <user@flink.apache.org> Subject: Re: High DirectByteBuffer Usage Hi Timo, I am looking at these options. However, I had a couple of questions - 1. The off-heap usage grows overtime. My job does not do any off-heap operations so I don't think there is a leak there. Even after GC it keeps adding a few MBs after hours of running. 2. Secondly, I am seeing as the incoming record volume increases the off-heap usage grows. What's the reason for this? I am using 1.9. Is there any known bug which is causing this issue? Thanks, Hemant On Wed, Jul 14, 2021 at 7:30 PM Timo Walther <twal...@apache.org<mailto:twal...@apache.org>> wrote: Hi Hemant, did you checkout the dedicated page for memory configuration and troubleshooting: https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-direct-buffer-memory https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#container-memory-exceeded It is likely that the high number of output streams could cause your issues. Regards, Timo On 14.07.21 08:46, bat man wrote: > Hi, > I have a job which reads different streams from 5 kafka topics. It > filters data and then data is streamed to different operators for > processing. This step involves data shuffling. > > Also, once data is enriched in 4 joins(KeyedCoProcessFunction) > operators. After joining the data is written to different kafka topics. > There are a total of 16 different output streams which are written to 4 > topics. > > I have been facing some issues with yarn killing containers. I took the > heap dump and ran it through JXray [1]. Heap usage is not high. One > thing which stands out is off-heap usage which is very high. My guess is > this is what is killing the containers as the data inflow increases. > > Screenshot 2021-07-14 at 11.52.41 AM.png > > > From the stack above is this usage high because of many output streams > being written to kafka topics. As the stack shows RecordWriter holding > off this DirectByteBuffer. I have assigned Network Memory as 1GB, and > --MaxDirectMemorySize also shows ~1GB for task managers. > > From here[2] I found that setting -Djdk.nio.maxCachedBufferSize=262144 > limits the temp buffer cache. Will it help in this case? > jvm version used is - JVM: OpenJDK 64-Bit Server VM - Red Hat, Inc. - > 1.8/25.282-b08 > > [1] - https://jxray.com <https://jxray.com> > [2] - > https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo > <https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo> > > Thanks, > Hemant
