[
https://issues.apache.org/jira/browse/SPARK-52242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ruiliang updated SPARK-52242:
-----------------------------
Attachment: spark-support-yarn-OPPORTUNISTIC.patch
> spark support yarn containerType opportunistic
> ----------------------------------------------
>
> Key: SPARK-52242
> URL: https://issues.apache.org/jira/browse/SPARK-52242
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.5.4
> Reporter: ruiliang
> Priority: Minor
> Attachments: spark-support-yarn-OPPORTUNISTIC.patch
>
>
> I checked the relevant spark source code and found that since there was no
> relevant configuration supporting yarn containerType, Please check the
> yarn:[https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html.]
> If yarn elastic memory management is enabled, this type is relatively
> important and can be killed first. Therefore, spark needs to support yarn
> opportunistic containerType.
> The relevant changes are as follows:
> 1)Support containerType opportunistic
> 2) Add the container exit code-108 times and upgrade it to GUARANTEED to
> prevent the job from being killed all the time
> 3) container exit code 137 is that after yarn enables elastic memory, it will
> continuously kill the container. This is a normal phenomenon and should not
> be counted in the limit of spark.task.maxFailures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
