GitHub user madanadit opened a pull request: https://github.com/apache/spark/pull/20989
[SPARK-23529][K8s] Support mounting hostPath volumes for executors ## What changes were proposed in this pull request? This PR introduces a new config `spark.kubernetes.executor.volumes` taking a values of the form `hostPath:containerPath[:ro|rw][,...]`; where `hostPath` is the path for the executor pod volume, `containerPath` is the mount path and `ro` is read-only mode. The use case is to enable short-circuit writes to distributed storage on k8s. The Alluxio File System uses domain sockets to enable short-circuit writes from the client to worker memory when co-located on the same host machine. A directory, lets say /tmp/domain on the host, is mounted on the Alluxio worker container as well as the Alluxio client ( = Spark executor) container. The worker creates a domain socket /tmp/domain/d and if the client container mounts the same directory, it can write directly to the Alluxio worker w/o passing through network stack. The end result is faster data access when data is local. ## How was this patch tested? Manual testing on a k8s v1.8 cluster. You can merge this pull request into a Git repository by running: $ git pull https://github.com/madanadit/spark k8s-volumes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20989.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20989 ---- commit addb8fec7c7fcd1535cfdc29594d74b04981ae19 Author: madanadit <adit@...> Date: 2018-04-05T01:02:25Z Spark executor volumes configuration commit f33eaf53b914c3414a4d4a2bca9c5bc3d84aa268 Author: madanadit <adit@...> Date: 2018-04-05T20:05:48Z Fix style commit 4016d6d33c296823e338f28deb7815c4be6607a1 Author: madanadit <adit@...> Date: 2018-04-05T20:12:28Z Undo unintended style change commit c5b6c0bdc0ac46e87d4ca41ab27b81be62beee6f Author: madanadit <adit@...> Date: 2018-04-05T20:57:45Z Read mode for mounted volumes commit 17cfdf3afede72efd970c1d3ac92105257c1f63c Author: madanadit <adit@...> Date: 2018-04-05T22:05:07Z Refactor ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org