Hello, I'm having a bit of trouble understanding the memory configuration on flink. I'm using flink10.0.0 to read some datasets of edges and extract features. I run this on a cluster consisting of 4 nodes , with 32cores and 252GB Ram each, and hopefully I could expand this as long as I can add extra nodes to the cluster.
So regarding the configuration file (flink-conf.yaml). a) I can't understand when should I use process.size and when .flink.size. b) From the detailed memory model I understand that Direct memory is included in both of flink and process size, however if I don't specify off-heap.task.size I get " OutOfMemoryError: Direct buffer memory " . Also should I change off-heap.fraction as well? c)When I fix this, I get network buffers error, which if I understand correctly, flink.size * network fraction , should be between min and max. I can't find the 'perfect' configuration regarding my setup. What is the optimal way to use the system I have currently? Thank you for your time.
