ruiliang created SPARK-52242:
--------------------------------
Summary: spark support yarn containerType opportunistic
Key: SPARK-52242
URL: https://issues.apache.org/jira/browse/SPARK-52242
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 3.5.4
Reporter: ruiliang
Attachments: spark-support-yarn-OPPORTUNISTIC.patch
I checked the relevant spark source code and found that since there was no
relevant configuration supporting yarn containerType, Please check the
yarn:[https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html.]
If yarn elastic memory management is enabled, this type is relatively important
and can be killed first. Therefore, spark needs to support yarn opportunistic
containerType.
The relevant changes are as follows:
1)Support containerType opportunistic
2) Add the container exit code-108 times and upgrade it to GUARANTEED to
prevent the job from being killed all the time
3) container exit code 137 is that after yarn enables elastic memory, it will
continuously kill the container. This is a normal phenomenon and should not be
counted in the limit of spark.task.maxFailures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
