GitHub user madanadit opened a pull request:
https://github.com/apache/spark/pull/21095
[SPARK-23529][K8s] Support mounting hostPath volumes
## What changes were proposed in this pull request?
This PR introduces a new config `spark.kubernetes.driver/executor.volumes`
taking a values of the format documented
[here](https://docs.google.com/document/d/15-mk7UnOYNTXoF6EKaVlelWYc9DTrTXrYoodwDuAwY4/edit?usp=sharing)
The use case is to enable short-circuit writes to distributed storage on
k8s. The Alluxio File System uses domain sockets to enable short-circuit writes
from the client to worker memory when co-located on the same host machine. A
directory, lets say /tmp/domain on the host, is mounted on the Alluxio worker
container as well as the Alluxio client ( = Spark executor) container. The
worker creates a domain socket /tmp/domain/d and if the client container mounts
the same directory, it can write directly to the Alluxio worker w/o passing
through network stack. The end result is faster data access when data is local.
## How was this patch tested?
Manual testing on a k8s v1.8 cluster. Unit tests added to
`Driver/ExecutorPodFactorySuite`.
This PR replaces https://github.com/apache/spark/pull/21032.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/madanadit/spark k8s-vol
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21095.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21095
----
commit 51a877a7d86b6c0f18d381ba9300de18ccef9189
Author: madanadit <adit@...>
Date: 2018-04-05T01:02:25Z
Support mounting hostPath volumes for executors
commit 4ada724ab2f6e79553fa05475b216884f9ba127a
Author: madanadit <adit@...>
Date: 2018-04-05T20:57:45Z
Read mode for mounted volumes
commit 17258a3967b6d8b2170c3c7d7186449c3155fef1
Author: madanadit <adit@...>
Date: 2018-04-05T22:05:07Z
Refactor
commit 6fff716cbe42e184059660cf2009f594048a6420
Author: madanadit <adit@...>
Date: 2018-04-06T18:18:09Z
Fix style
commit f961b33aa34ad7908ac6d77ded251e18bf6e835a
Author: madanadit <adit@...>
Date: 2018-04-06T19:57:45Z
Add unit tests
commit af4d9baae8d637215d2be645b377fa5dbd714e94
Author: madanadit <adit@...>
Date: 2018-04-06T20:00:18Z
Update comment
commit 13e10493fa6f3b40c3eb406d0dc2f88d09ce20b8
Author: madanadit <adit@...>
Date: 2018-04-06T23:56:31Z
Fix unit tests
commit 7a25d76e386c0d8157eba0ccc0517bea3b7f7f0e
Author: madanadit <adit@...>
Date: 2018-04-16T22:41:26Z
Fix typo
commit a5a277a065dec156e976478de7767062bcf1da13
Author: madanadit <adit@...>
Date: 2018-04-17T21:51:58Z
Change configuration format
commit 772128e13895d6f313efa3cbc38947db51c3e493
Author: madanadit <adit@...>
Date: 2018-04-17T23:34:37Z
Fix build
commit a393b92c93d60bdcf96025ae24d6c1fbecf17f9b
Author: madanadit <adit@...>
Date: 2018-04-17T23:44:08Z
Fix test
commit b9b3dcb22ded628f0abe1caa0beb2b15da6ccc49
Author: madanadit <adit@...>
Date: 2018-04-18T00:39:14Z
Fetch properties correctly
commit 81a7811b06a0195e84ffeddc135875da6c500a7e
Author: madanadit <adit@...>
Date: 2018-04-18T05:21:11Z
Fix test cases
commit 95d1b0d8c681cd46f47c8ab1692172d0b3b0aba8
Author: madanadit <adit@...>
Date: 2018-04-18T06:42:56Z
Abstract tests
commit facde97b365a7acb02c41e5ef076a9ea0f1edff9
Author: madanadit <adit@...>
Date: 2018-04-18T06:59:43Z
Add readOnly option
commit ccdc7990ca8995ff86f46647f8a2949848f06380
Author: madanadit <adit@...>
Date: 2018-04-18T07:06:17Z
Fix test
commit 7c1be8aff51a462bf96012fafbbfec765424de53
Author: madanadit <adit@...>
Date: 2018-04-18T07:24:55Z
Driver hostPath volumes with tests
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
