[
https://issues.apache.org/jira/browse/SPARK-29474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035372#comment-17035372
]
Adam Antal commented on SPARK-29474:
------------------------------------
AFAIK YARN obtains the Docker image from the repository provided with the name
of the Docker image like (I guess dockerhub by default) or other either public
or private repository. If using offline image has important use cases, one can
file an improvement jira for YARN.
> CLI support for Spark-on-Docker-on-Yarn
> ---------------------------------------
>
> Key: SPARK-29474
> URL: https://issues.apache.org/jira/browse/SPARK-29474
> Project: Spark
> Issue Type: Improvement
> Components: Spark Shell, YARN
> Affects Versions: 3.0.0
> Reporter: Adam Antal
> Priority: Major
>
> The Docker-on-Yarn feature is stable for a while now in Hadoop.
> One can run Spark on Docker using the Docker-on-Yarn feature by providing
> runtime environments to the Spark AM and Executor containers similar to this:
> {noformat}
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=repo/image:tag
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro"
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=repo/image:tag
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro"
> {noformat}
> This is not very user friendly. I suggest to add CLI options to specify:
> - whether docker image should be used ({{--docker}})
> - which docker image should be used ({{--docker-image}})
> - what docker mounts should be used ({{--docker-mounts}})
> for the AM and executor containers separately.
> Let's discuss!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
