Hi, There are two distinct actions here; namely Deploy and Run.
Deployment can be done by command line script with autoscaling. In the newer versions of Kubernnetes you don't even need to specify the node types, you can leave it to the Kubernetes cluster to scale up and down and decide on node type. The second point is the running spark that you will need to submit. However, that depends on setting up access permission, use of service accounts, pulling the correct dockerfiles for the driver and the executors. Those details add to the complexity. Thanks view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: > Hi Spark Community, > > We built an open source tool to deploy and run Spark on Kubernetes with a > one click command. For example, on AWS, it could automatically create an > EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will > be able to use curl or a CLI tool to submit Spark application. After the > deployment, you could also install Uber Remote Shuffle Service to enable > Dynamic Allocation on Kuberentes. > > Anyone interested in using or working together on such a tool? > > Thanks, > Bo > >
