It uses Helm to deploy Spark Operator and Nginx. For other parts like creating EKS, IAM role, node group, etc, it uses AWS SDK to provision those AWS resources.
On Wed, Feb 23, 2022 at 11:28 AM Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > So if I get this right you will make a Helm <https://helm.sh> chart to > deploy Spark and some other stuff on K8S? > > ons. 23. feb. 2022 kl. 17:49 skrev bo yang <bobyan...@gmail.com>: > >> Hi Sarath, let's follow up offline on this. >> >> On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy < >> sarath.annare...@gmail.com> wrote: >> >>> Hi bo >>> >>> How do we start? >>> >>> Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc >>> >>> >>> Thanks >>> Sarath >>> >>> >>> Sent from my iPhone >>> >>> On Feb 23, 2022, at 10:27 AM, bo yang <bobyan...@gmail.com> wrote: >>> >>> >>> Hi Sarath, thanks for your interest and willing to contribute! The >>> project supports local development using MiniKube. Similarly there is a one >>> click command with one extra argument to deploy all components in MiniKube, >>> and people could use that to develop on their local MacBook. >>> >>> >>> On Wed, Feb 23, 2022 at 7:41 AM Sarath Annareddy < >>> sarath.annare...@gmail.com> wrote: >>> >>>> Hi bo >>>> >>>> I am interested to contribute. >>>> But I don’t have free access to any cloud provider. Not sure how I can >>>> get free access. I know Google, aws, azure only provides temp free access, >>>> it may not be sufficient. >>>> >>>> Guidance is appreciated. >>>> >>>> Sarath >>>> >>>> Sent from my iPhone >>>> >>>> On Feb 23, 2022, at 2:01 AM, bo yang <bobyan...@gmail.com> wrote: >>>> >>>> >>>> >>>> Right, normally people start with simple script, then add more stuff, >>>> like permission and more components. After some time, people want to run >>>> the script consistently in different environments. Things will become >>>> complex. >>>> >>>> That is why we want to see whether people have interest for such a "one >>>> click" tool to make things easy. >>>> >>>> >>>> On Tue, Feb 22, 2022 at 11:31 PM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> There are two distinct actions here; namely Deploy and Run. >>>>> >>>>> Deployment can be done by command line script with autoscaling. In the >>>>> newer versions of Kubernnetes you don't even need to specify the node >>>>> types, you can leave it to the Kubernetes cluster to scale up and down >>>>> and >>>>> decide on node type. >>>>> >>>>> The second point is the running spark that you will need to submit. >>>>> However, that depends on setting up access permission, use of service >>>>> accounts, pulling the correct dockerfiles for the driver and the >>>>> executors. >>>>> Those details add to the complexity. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: >>>>> >>>>>> Hi Spark Community, >>>>>> >>>>>> We built an open source tool to deploy and run Spark on Kubernetes >>>>>> with a one click command. For example, on AWS, it could automatically >>>>>> create an EKS cluster, node group, NGINX ingress, and Spark Operator. >>>>>> Then >>>>>> you will be able to use curl or a CLI tool to submit Spark application. >>>>>> After the deployment, you could also install Uber Remote Shuffle Service >>>>>> to >>>>>> enable Dynamic Allocation on Kuberentes. >>>>>> >>>>>> Anyone interested in using or working together on such a tool? >>>>>> >>>>>> Thanks, >>>>>> Bo >>>>>> >>>>>> > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 >
