It is not a standalone spark cluster. In some details, it deploys a Spark Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and an extra REST Service. When people submit Spark application to that REST Service, the REST Service will create a CRD inside the Kubernetes cluster. Then Spark Operator will pick up the CRD and launch the Spark application. The one click tool intends to hide these details, so people could just submit Spark and do not need to deal with too many deployment details.
On Tue, Feb 22, 2022 at 8:09 PM Bitfox <bit...@bitfox.top> wrote: > Can it be a cluster installation of spark? or just the standalone node? > > Thanks > > On Wed, Feb 23, 2022 at 12:06 PM bo yang <bobyan...@gmail.com> wrote: > >> Hi Spark Community, >> >> We built an open source tool to deploy and run Spark on Kubernetes with a >> one click command. For example, on AWS, it could automatically create an >> EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will >> be able to use curl or a CLI tool to submit Spark application. After the >> deployment, you could also install Uber Remote Shuffle Service to enable >> Dynamic Allocation on Kuberentes. >> >> Anyone interested in using or working together on such a tool? >> >> Thanks, >> Bo >> >>
