Re: Memory issue in Flink 1.10

2020-05-27 Thread Andrey Zagrebin
Hi Steve, RocksDB does not contribute to the JVM direct memory. RocksDB off-heap memory consumption is part of managed memory [1]. You got `OutOfMemoryError: Direct buffer memory` which is related to the JVM direct memory, also off-heap but managed by JVM. The JVM direct memory limit depends on t

Memory issue in Flink 1.10

2020-05-27 Thread Steven Nelson
We recently migrated to Flink 1.10, but are experiencing some issues with memory. Our cluster is: 1) Running inside of Kubernetes 2) Running in HA mode 3) Checkpointing/Savepointing to an HDFS cluster inside of Kubernetes 4) Using RocksDB for checkpointing 5) Running on m5d.4xlarge EC2 instances w