Hi Sarath, let's follow up offline on this. On Wed, Feb 23, 2022 at 8:32 AM Sarath Annareddy <sarath.annare...@gmail.com> wrote:
> Hi bo > > How do we start? > > Is there a plan? Onboarding, Arch/design diagram, tasks lined up etc > > > Thanks > Sarath > > > Sent from my iPhone > > On Feb 23, 2022, at 10:27 AM, bo yang <bobyan...@gmail.com> wrote: > > > Hi Sarath, thanks for your interest and willing to contribute! The project > supports local development using MiniKube. Similarly there is a one click > command with one extra argument to deploy all components in MiniKube, and > people could use that to develop on their local MacBook. > > > On Wed, Feb 23, 2022 at 7:41 AM Sarath Annareddy < > sarath.annare...@gmail.com> wrote: > >> Hi bo >> >> I am interested to contribute. >> But I don’t have free access to any cloud provider. Not sure how I can >> get free access. I know Google, aws, azure only provides temp free access, >> it may not be sufficient. >> >> Guidance is appreciated. >> >> Sarath >> >> Sent from my iPhone >> >> On Feb 23, 2022, at 2:01 AM, bo yang <bobyan...@gmail.com> wrote: >> >> >> >> Right, normally people start with simple script, then add more stuff, >> like permission and more components. After some time, people want to run >> the script consistently in different environments. Things will become >> complex. >> >> That is why we want to see whether people have interest for such a "one >> click" tool to make things easy. >> >> >> On Tue, Feb 22, 2022 at 11:31 PM Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Hi, >>> >>> There are two distinct actions here; namely Deploy and Run. >>> >>> Deployment can be done by command line script with autoscaling. In the >>> newer versions of Kubernnetes you don't even need to specify the node >>> types, you can leave it to the Kubernetes cluster to scale up and down and >>> decide on node type. >>> >>> The second point is the running spark that you will need to submit. >>> However, that depends on setting up access permission, use of service >>> accounts, pulling the correct dockerfiles for the driver and the executors. >>> Those details add to the complexity. >>> >>> Thanks >>> >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Wed, 23 Feb 2022 at 04:06, bo yang <bobyan...@gmail.com> wrote: >>> >>>> Hi Spark Community, >>>> >>>> We built an open source tool to deploy and run Spark on Kubernetes with >>>> a one click command. For example, on AWS, it could automatically create an >>>> EKS cluster, node group, NGINX ingress, and Spark Operator. Then you will >>>> be able to use curl or a CLI tool to submit Spark application. After the >>>> deployment, you could also install Uber Remote Shuffle Service to enable >>>> Dynamic Allocation on Kuberentes. >>>> >>>> Anyone interested in using or working together on such a tool? >>>> >>>> Thanks, >>>> Bo >>>> >>>>
