[
https://issues.apache.org/jira/browse/SPARK-28022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Henry Yu updated SPARK-28022:
-----------------------------
Description:
Hi, in order to achieve cloud native friendly autoscaling , I propose to add a
pod affinity feature.
Traditionally, when we use spark in fix size yarn cluster, it make sense to
spread container evenly to every node.
But come to cloud native resource manage, we want to release node when we don't
need any more.
pod affinity feature can help when we try to place all pod of certain job to
some node instead of all node.
we can not use pod template to achieve this, for we must add application id to
affinity .
was:
Hi, in order to achieve friendly auto scaling , I propose to add a pod affinity
feature.
Traditionally, when we use spark in fix size yarn cluster, it make sense to
spread container evenly to every node.
But come to cloud native resource manage, we want to release node when we don't
need any more.
pod affinity feature can help when we try to place all pod of certain job to
some node instead of all node.
we can not use pod template to achieve this, for we must add application id to
affinity .
> k8s pod affinity achieve cloud native friendly auto scaling
> ------------------------------------------------------------
>
> Key: SPARK-28022
> URL: https://issues.apache.org/jira/browse/SPARK-28022
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 2.4.3
> Reporter: Henry Yu
> Priority: Major
>
> Hi, in order to achieve cloud native friendly autoscaling , I propose to add
> a pod affinity feature.
> Traditionally, when we use spark in fix size yarn cluster, it make sense to
> spread container evenly to every node.
> But come to cloud native resource manage, we want to release node when we
> don't need any more.
> pod affinity feature can help when we try to place all pod of certain job to
> some node instead of all node.
> we can not use pod template to achieve this, for we must add application id
> to affinity .
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
