RE: [Fork] ]RE: One click to run Spark on Kubernetes

customers to focus on running Spark application on their EKS cluster. Thanks, Janak From: Mich Talebzadeh Sent: Wednesday, February 23, 2022 11:54 AM To: Agarwal, Janak Cc: Spark dev list Subject: RE: [EXTERNAL] [Fork] ]RE: One click to run Spark on Kubernetes CAUTION: This email originated from

Re: One click to run Spark on Kubernetes

It uses Helm to deploy Spark Operator and Nginx. For other parts like creating EKS, IAM role, node group, etc, it uses AWS SDK to provision those AWS resources. On Wed, Feb 23, 2022 at 11:28 AM Bjørn Jørgensen wrote: > So if I get this right you will make a Helm chart to >

Re: [Fork] ]RE: One click to run Spark on Kubernetes

-managed/self-managed EC2 based node groups. > > > > Thanks, > > Janak > > > > *From:* Mich Talebzadeh > *Sent:* Wednesday, February 23, 2022 10:46 AM > *To:* Agarwal, Janak > *Cc:* Spark dev list > *Subject:* RE: [EXTERNAL] [Fork] ]RE: One click to

Re: One click to run Spark on Kubernetes

So if I get this right you will make a Helm chart to deploy Spark and some other stuff on K8S? ons. 23. feb. 2022 kl. 17:49 skrev bo yang : > Hi Sarath, let's follow up offline on this. > > On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy < > sarath.annare...@gmail.com> wrote:

RE: [Fork] ]RE: One click to run Spark on Kubernetes

[Fork] ]RE: One click to run Spark on Kubernetes CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hi Janak, Are you talking about EKS Fargate? Thanks

Re: [Fork] ]RE: One click to run Spark on Kubernetes

Want to give EMR on EKS a > spin as you decide your path forward? > > > > > > Best, > > Janak > > > > *From:* Sarath Annareddy > *Sent:* Wednesday, February 23, 2022 7:41 AM > *To:* bo yang > *Cc:* Mich Talebzadeh ; Spark Dev List < >

Re: One click to run Spark on Kubernetes

Hi Sarath, let's follow up offline on this. On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy wrote: > Hi bo > > How do we start? > > Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc > > > Thanks > Sarath > > > Sent from my iPhone > > On Feb 23, 2022, at 10:27 AM, bo yang

Re: One click to run Spark on Kubernetes

Hi bo How do we start? Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc Thanks Sarath Sent from my iPhone > On Feb 23, 2022, at 10:27 AM, bo yang wrote: > >  > Hi Sarath, thanks for your interest and willing to contribute! The project > supports local development

Re: One click to run Spark on Kubernetes

Hi Sarath, thanks for your interest and willing to contribute! The project supports local development using MiniKube. Similarly there is a one click command with one extra argument to deploy all components in MiniKube, and people could use that to develop on their local MacBook. On Wed, Feb 23,

Re: One click to run Spark on Kubernetes

Hi bo I am interested to contribute. But I don’t have free access to any cloud provider. Not sure how I can get free access. I know Google, aws, azure only provides temp free access, it may not be sufficient. Guidance is appreciated. Sarath Sent from my iPhone > On Feb 23, 2022, at 2:01

Re: One click to run Spark on Kubernetes

Right, normally people start with simple script, then add more stuff, like permission and more components. After some time, people want to run the script consistently in different environments. Things will become complex. That is why we want to see whether people have interest for such a "one

Re: One click to run Spark on Kubernetes

Hi, There are two distinct actions here; namely Deploy and Run. Deployment can be done by command line script with autoscaling. In the newer versions of Kubernnetes you don't even need to specify the node types, you can leave it to the Kubernetes cluster to scale up and down and decide on node

Re: One click to run Spark on Kubernetes

Merging another email from Prasad. It could co-exist with livy. Livy is similar like the REST Service + Spark Operator. Unfortunately Livy is not very active right now. To Amihay, the link is: https://github.com/datapunchorg/punch. On Tue, Feb 22, 2022 at 8:53 PM amihay gonen wrote: > Can you

Re: One click to run Spark on Kubernetes

We do not have SaaS yet. Now it is an open source project we build in our part time , and we welcome more people working together on that. You could specify cluster size (EC2 instance type and number of instances) and run it for 1 hour. Then you could run one click command to destroy the cluster.

Re: One click to run Spark on Kubernetes

Hi Bo Yang, Would it be something along the lines of Apache livy? Thanks, Prasad On Tue, Feb 22, 2022 at 10:22 PM bo yang wrote: > It is not a standalone spark cluster. In some details, it deploys a Spark > Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) > and an extra

Re: One click to run Spark on Kubernetes

It is not a standalone spark cluster. In some details, it deploys a Spark Operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) and an extra REST Service. When people submit Spark application to that REST Service, the REST Service will create a CRD inside the Kubernetes cluster.

One click to run Spark on Kubernetes

Hi Spark Community, We built an open source tool to deploy and run Spark on Kubernetes with a one click command. For example, on AWS, it could automatically create an EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will be able to use curl or a CLI tool to submit Spark