I would recommend looking into Spark’s native support for running on Kubernetes. One can just start the application against Kubernetes directly using spark-submit in cluster mode or starting the Spark context with the right parameters in client mode. See https://spark.apache.org/docs/latest/running-on-kubernetes.html
I would think that building Helm around this architecture of running Spark applications would be easier than running a Spark standalone cluster. But admittedly I’m not very familiar with the Helm technology – we just use spark-submit. -Matt Cheah From: Pat Ferrel <[email protected]> Date: Sunday, June 30, 2019 at 12:55 PM To: "[email protected]" <[email protected]> Subject: k8s orchestrating Spark service We're trying to setup a system that includes Spark. The rest of the services have good Docker containers and Helm charts to start from. Spark on the other hand is proving difficult. We forked a container and have tried to create our own chart but are having several problems with this. So back to the community… Can anyone recommend a Docker Container + Helm Chart for use with Kubernetes to orchestrate: Spark standalone Master several Spark Workers/Executors This not a request to use k8s to orchestrate Spark Jobs, but the service cluster itself. Thanks
smime.p7s
Description: S/MIME cryptographic signature
