Hi Pranjul, First of all, you adopted on-heap state backend: HashMapStateBackend, which would not use native off-heap memory. Moreover, JobManager would not initialize any keyed state backend instance. And if not enable high availability, JobManagerCheckpointStorage would also not use direct memory to write checkpoint stream out. Did you use some third-party native library?
You could use native memory tracking [1] to see whether JVM used too much overhead native memory. And use memory allocator analysis tool such as JeMalloc [2] to see whether existed unexpected native memory usage. [1] https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/ [2] https://gist.github.com/thomasdarimont/79b3cef01e5786210309 Best Yun Tang ________________________________ From: Pranjul Ahuja <ahuja0...@gmail.com> Sent: Monday, August 16, 2021 13:10 To: user@flink.apache.org <user@flink.apache.org> Subject: JobManager Resident memory Always Increasing Hi, We are running the JobManager container with 1024GB out of which 512MB is allocated to the heap. We observe that the JobManager container's resident memory is always increasing and it never comes down. Heap usage remains to be constant and not rising abruptly. Can anyone help here where else this memory is going? Statebackend Used - HashMapStateBackend Checkpoint Storage - JobManagerCheckpointStorage
