[
https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861707#comment-16861707
]
Junjie Chen commented on SPARK-27499:
-------------------------------------
Yes, In KubernetesExecutorBuilder.scala, the LocalDrisFeatureStep is built
before MountVolumesFeatureStep which means we cannot use any volumes mount
later. I think we should build localDirsFeature at last, so that we can check
if directories in SPARK_LOCAL_DIRS are set to volumes mounted either hostPath,
PV, or others may support later and use that as local storage.
With that way, we can utilize specified media to improve the local storage
performance instead of just emptyDir which is just a ephemeral directory on
node.
> Support mapping spark.local.dir to hostPath volume
> --------------------------------------------------
>
> Key: SPARK-27499
> URL: https://issues.apache.org/jira/browse/SPARK-27499
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Junjie Chen
> Priority: Minor
>
> Currently, the k8s executor builder mount spark.local.dir as emptyDir or
> memory, it should satisfy some small workload, while in some heavily workload
> like TPCDS, both of them can have some problem, such as pods are evicted due
> to disk pressure when using emptyDir, and OOM when using tmpfs.
> In particular on cloud environment, users may allocate cluster with minimum
> configuration and add cloud storage when running workload. In this case, we
> can specify multiple elastic storage as spark.local.dir to accelerate the
> spilling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
