[
https://issues.apache.org/jira/browse/SPARK-23529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397662#comment-16397662
]
Suman Somasundar commented on SPARK-23529:
------------------------------------------
I have a couple of use-cases for this as explained below:
# There are multiple Spark jobs running in the cluster and I would want to be
able to extract the log files from these jobs. In this case, I can mount a
hostpath volume in all the driver and executor pods and have them write logs to
that path. I can login to the nodes and copy all the log files from all jobs.
# My jobs have a dependency on large data files and fat jars. I dont want to
copy them over (from HDFS) to the driver and executor pods every time I submit
the job. Packaging these data files and jars in the image is not an option as
the image size will be large and I dont want to have images specific to a job.
In this case, I can download the dependencies to a local volume and mount it as
hostpath on the driver and executor pods and use the dependencies as local
files.
> Specify hostpath volume and mount the volume in Spark driver and executor
> pods in Kubernetes
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-23529
> URL: https://issues.apache.org/jira/browse/SPARK-23529
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.3.0
> Reporter: Suman Somasundar
> Assignee: Anirudh Ramanathan
> Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
