[
https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478907#comment-17478907
]
Dongjoon Hyun commented on SPARK-24432:
---------------------------------------
[~pralabhkumar]. What you are looking at is not a single PR. For the
configuration, please check
{code}
spark.dynamicAllocation.* (including spark.dynamicAllocation.shuffleTracking.*)
spark.decommission.*
spark.storage.decommission.*
{code}
In addition, `master` branch is already for Apache Spark 3.3.0. It seems that
you are using outdated Spark versions.
bq. The K8s dynamic allocation with storage migration between executors is
already in `master` branch for Apache Spark 3.1.0.
If you didn't try to use the latest Apache Spark 3.2, please try Apache Spark
3.2.1 RC2. Although it's not Apache Spark 3.3.0-SNAPSHOT, it has most available
features you need.
- https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-bin/
- https://dist.apache.org/repos/dist/dev/spark/v3.2.1-rc2-docs/
> Add support for dynamic resource allocation
> -------------------------------------------
>
> Key: SPARK-24432
> URL: https://issues.apache.org/jira/browse/SPARK-24432
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Spark Core
> Affects Versions: 3.1.0
> Reporter: Yinan Li
> Priority: Major
>
> This is an umbrella ticket for work on adding support for dynamic resource
> allocation into the Kubernetes mode. This requires a Kubernetes-specific
> external shuffle service. The feature is available in our fork at
> github.com/apache-spark-on-k8s/spark.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
