When we run Flink jobs in EMR (typically batch, though) we disable the pmem
(permanent memory) and vmem (virtual memory) checks.
This was initially done for much older versions of Flink (1.6???), where the
memory model wasn’t so well documented or understood by us.
But I think the pmem check might still have an issue, due to Flink’s use of
off-heap.
So something like:
[
{
"classification": "yarn-site",
"properties": {
"yarn.nodemanager.pmem-check-enabled": "false",
"yarn.nodemanager.vmem-check-enabled": "false"
}
}
]
…might help.
— Ken
> On Jul 2, 2021, at 8:36 AM, bat man <tintin0...@gmail.com> wrote:
>
> Hi,
>
> I am running a streaming job (Flink 1.9) on EMR on yarn. Flink web UI or
> metrics reported from prometheus shows total memory usage within specified
> task manager memory - 3GB.
>
> Metrics shows below numbers(in MB) -
> Heap - 577
> Non Heap - 241
> DirectMemoryUsed - 852
>
> Non-heap does rise gradually, starting around 210MB and reaching 241 when
> yarn kills the container. Heap fluctuates between 1.x - .6GB,
> DirectMemoryUsed is constant at 852.
>
> Based on configurations these are the tm params from yarn logs -
> -Xms1957m -Xmx1957m -XX:MaxDirectMemorySize=1115m
>
> These are other params as configuration in flink-conf
> yarn-cutoff - 270MB
> Managed memory - 28MB
> Network memory - 819MB
>
> Above memory values are from around the same time the container is killed by
> yarn for - <container-xxx> is running beyond physical memory limits.
>
> Is there anything else which is not reported by flink in metrics or I have
> been misinterpreting as seen from above total memory consumed is below - 3GB.
>
> Same behavior is reported when I have run the job with 2GB, 2.7GB and now 3GB
> task mem. My job does have shuffles as data from one operator is sent to 4
> other operators after filtering.
>
> One more thing is I am running this with 3 yarn containers(2 tasks in each
> container), total parallelism as 6. As soon as one container fails with this
> error, the job re-starts. However, within minutes other 2 containers also
> fail with the same error one by one.
>
> Thanks,
> Hemant
--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch
