Hi,
I have a question regarding configuration of task manager heap size when
running YARN session on EMR.
I am running 2 task managers on m4.4xlarge (64GB RAM). I would like to use
as much as possible of that memory for the task manager heap.
However when requesting 56000 MB when staring YARN actually only around
42GB is assigned to TM. Do you know how I can increase that?
This is how I start YARN session:
/usr/lib/flink/bin/yarn-session.sh --container 2 --taskManagerMemory 56000
--slots 16 --detached -Dparallelism.default=32
-Dtaskmanager.network.numberOfBuffers=20480 ...
This is the output of *ps aux *on TM box
yarn 42843 1030 67.7 46394740 44688084 ? Sl 15:27 175:56
/usr/lib/jvm/java-openjdk/bin/java -Xms42000m -Xmx42000m ....
yarn 42837 0.0 0.0 113104 2684 ? Ss 15:27 0:00 /bin/bash
-c /usr/lib/jvm/java-openjdk/bin/java -Xms42000m -Xmx42000m ....
I would expect around 56GB set as max heap size for TM.
some settings from yarn-site.xml that might be of interest:
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>57344</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>57344</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>5</value>
</property>
Cheers,
Pawel
