Hi John, Thank you for sharing your experience with us! Great work at ApacheCon!
Making Nemo work on Google Dataproc is a great idea. If Beam users don't use cloud dataflow, they can definitely try out Nemo as an open-source runner. Thanks! -Gon On Sat, Sep 21, 2019 at 9:53 AM John Youngseok Yang <johnya...@gmail.com> wrote: > Hi Nemoers, > > Just wanted to share my trip to ApacheCon NA 2019. :) > Overall it was great fun. I met and talked with many folks working on a > number of different Apache big data projects. > Several people who attended Wonwook's talk at Beam summit EU already knew > about Nemo. Kudos to Wonwook for the excellent job! > > People really liked how Nemo transparently integrates with existing > programming libraries (Beam, Spark), and cluster resource managers (YARN, > Mesos). > This seems to validate the decision we made early on to make Nemo > compatible with other Apache big data projects including REEF. > People also liked the optimizations, and some of them were curious whether > the optimization options can be configured automatically for them. > I suppose this validates the "smart compiler" works some of us are > currently working on. > > I did 2 talks at the conference. > One was the main Nemo talk, and the other was a demo of running Nemo on > Google DataProc in a Beam session. > The main talk was primarily about how Nemo optimizes distributed execution. > On the next day, Beam organizers generously let me do a demo talk to > demonstrate how easy it is to spin up new cloud instances, and run Beam > programs using the Nemo runner. > > After seeing the demo one of the attendees there suggested that I file a PR > to a GCP repository ( > https://github.com/GoogleCloudPlatform/dataproc-initialization-actions) to > automate the installation process, and we can do a blog post on using Nemo > on Google Dataproc. > I'll work on this to make Nemo more accessible to GCP users. > > Thank Nemoers for all your work! > > Cheers, > John > -- Byung-Gon Chun
