Github user foxish commented on a diff in the pull request:
https://github.com/apache/spark/pull/20059#discussion_r158568498
--- Diff: docs/running-on-kubernetes.md ---
@@ -120,6 +120,23 @@ by their appropriate remote URIs. Also, application
dependencies can be pre-moun
Those dependencies can be added to the classpath by referencing them with
`local://` URIs and/or setting the
`SPARK_EXTRA_CLASSPATH` environment variable in your Dockerfiles.
+### Using Remote Dependencies
+When there are application dependencies hosted in remote locations like
HDFS or HTTP servers, the driver and executor pods need a Kubernetes
[init-container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
for downloading the dependencies so the driver and executor containers can use
them locally. This requires users to specify the container image for the
init-container using the configuration property
`spark.kubernetes.initContainer.image`. For example, users simply add the
following option to the `spark-submit` command to specify the init-container
image:
--- End diff --
HDFS and HTTP sound good. We can cover GCS elsewhere. Line breaks were for
ease of reviewing by others (being able to comment on individual lines) and for
consistency with the rest of the docs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
