Hi,
You have to enable CPU scheduling in YARN, otherwise it always shows
that only 1 CPU is allocated for each container,
regardless of how many Flink try to allocated.
TaskManager memory is 1400MB, but Flink reserves some amount for for
off-heap memory, so the actual heap size is smaller.
This is controlled by 2 settings:
containerized.heap-cutoff-min: default 600MB
containerized.heap-cutoff-ratio: default 15% of TM's memory
That's why your TM's heap size is limitted to ~800MB (1400 - 600)
Regards,
Kien
On 4/29/2018 12:36 PM, Soheil Pourbafrani wrote:
Hi, I have an Flink .jar file and I submit it on yarn cluster using
the command:
flink run -m yarn-cluster -yn 5 -yjm 768 -ytm 1400 -ys 2 -yqu streamQ
my_program.jar
According to the submitting command I expect:
// It will create 5 containers ----> satisfied
// each container should use 2 core --> not satisfied according to the
YARN webui but in jobmanager webui it shows each taskmanager has
2 slot
// jobmanager should use 768m ram --> satisfied
// each taskmanager should use 1400m ram --> satisfied according to
the YARN webui but in jobmanager webui it shows for each
taskmanager: 767m heap memory and 478m used by Flink
Why the number of used cores and memory is not as desired?
