Wangda Tan created YARN-7739: -------------------------------- Summary: Revisit scheduler resource normalization behavior for max allocation Key: YARN-7739 URL: https://issues.apache.org/jira/browse/YARN-7739 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Priority: Critical
Currently, YARN Scheduler normalizes requested resource based on the maximum allocation derived from configured maximum allocation and maximum registered node resources. Basically, the scheduler will silently cap asked resource by maximum allocation. This could cause issues for applications, for example, a Spark job which needs 12 GB memory to run, however in the cluster, registered NMs have at most 8 GB mem on each node. So scheduler allocates 8GB memory container to the requested application. Once app receives containers from RM, if it doesn't double check allocated resources, it will lead to OOM and hard to debug because scheduler silently caps maximum allocation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org