GitHub user andrusha opened a pull request: https://github.com/apache/spark/pull/21260
[SPARK-23529][K8s] Support mounting hostPath volumes This PR continues #21095 and intersects with #21238. I've added volume mounts as a separate step and added PersistantVolumeClaim support. There is a fundamental problem with how we pass the options through spark conf to fabric8. For each volume type and all possible volume options we would have to implement some custom code to map config values to fabric8 calls. This will result in big body of code we would have to support and means that Spark will always be somehow out of sync with k8s. I think there needs to be a discussion how to proceed correctly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrusha/spark k8s-vol Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21260.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21260 ---- commit 51a877a7d86b6c0f18d381ba9300de18ccef9189 Author: madanadit <adit@...> Date: 2018-04-05T01:02:25Z Support mounting hostPath volumes for executors commit 4ada724ab2f6e79553fa05475b216884f9ba127a Author: madanadit <adit@...> Date: 2018-04-05T20:57:45Z Read mode for mounted volumes commit 17258a3967b6d8b2170c3c7d7186449c3155fef1 Author: madanadit <adit@...> Date: 2018-04-05T22:05:07Z Refactor commit 6fff716cbe42e184059660cf2009f594048a6420 Author: madanadit <adit@...> Date: 2018-04-06T18:18:09Z Fix style commit f961b33aa34ad7908ac6d77ded251e18bf6e835a Author: madanadit <adit@...> Date: 2018-04-06T19:57:45Z Add unit tests commit af4d9baae8d637215d2be645b377fa5dbd714e94 Author: madanadit <adit@...> Date: 2018-04-06T20:00:18Z Update comment commit 13e10493fa6f3b40c3eb406d0dc2f88d09ce20b8 Author: madanadit <adit@...> Date: 2018-04-06T23:56:31Z Fix unit tests commit 7a25d76e386c0d8157eba0ccc0517bea3b7f7f0e Author: madanadit <adit@...> Date: 2018-04-16T22:41:26Z Fix typo commit a5a277a065dec156e976478de7767062bcf1da13 Author: madanadit <adit@...> Date: 2018-04-17T21:51:58Z Change configuration format commit 772128e13895d6f313efa3cbc38947db51c3e493 Author: madanadit <adit@...> Date: 2018-04-17T23:34:37Z Fix build commit a393b92c93d60bdcf96025ae24d6c1fbecf17f9b Author: madanadit <adit@...> Date: 2018-04-17T23:44:08Z Fix test commit b9b3dcb22ded628f0abe1caa0beb2b15da6ccc49 Author: madanadit <adit@...> Date: 2018-04-18T00:39:14Z Fetch properties correctly commit 81a7811b06a0195e84ffeddc135875da6c500a7e Author: madanadit <adit@...> Date: 2018-04-18T05:21:11Z Fix test cases commit 95d1b0d8c681cd46f47c8ab1692172d0b3b0aba8 Author: madanadit <adit@...> Date: 2018-04-18T06:42:56Z Abstract tests commit facde97b365a7acb02c41e5ef076a9ea0f1edff9 Author: madanadit <adit@...> Date: 2018-04-18T06:59:43Z Add readOnly option commit ccdc7990ca8995ff86f46647f8a2949848f06380 Author: madanadit <adit@...> Date: 2018-04-18T07:06:17Z Fix test commit 7c1be8aff51a462bf96012fafbbfec765424de53 Author: madanadit <adit@...> Date: 2018-04-18T07:24:55Z Driver hostPath volumes with tests commit f482dfc370bace0c5315a2514604be2965bcbbaf Author: madanadit <adit@...> Date: 2018-04-19T23:56:58Z Refactor commit 136f5f65481f05cf10bd0c6968bf84c4363df53d Author: Andrew Korzhuev <andrew.korzhuev@...> Date: 2018-05-07T17:03:05Z Address comments, rewrite volume mount to step commit f1ada4c402a7f5ca5fd5428a28a457cd2400c6f2 Author: Andrew Korzhuev <andrew.korzhuev@...> Date: 2018-05-07T17:08:12Z Add persistant volume claim ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org