Let's move the scheduling off-list. On Tue, Jul 2, 2019 at 12:50 AM 송원욱 <wsong0...@gmail.com> wrote:
> Ooh, that’s going to bit a tricky. We’re landing in SEA in the morning of > the 9th and flying back early on the 16th. The conference is held from the > 10th until the 12th. Any other dates that could be possible, or should we > try pushing it for the evening of the 15th, or should we try giving it an > another go maybe next time? > > > On 2 Jul 2019, at 1:08 AM, Davor Bonaci <da...@apache.org> wrote: > > > > Would love to meet if you are in the area. > > > > Scheduling wise, I’ll be landing at SEA around 6 pm on the 15th, so 16th > > and onwards would be better. Evening on the 15th can work, but it is > > pushing it. > > > > Davor > > > > On Mon, Jul 1, 2019 at 8:40 AM 송원욱 <won...@apache.org> wrote: > > > >> Hi! > >> > >> I got back from the Beam Summit Europe 2019 that happened last week in > >> Berlin, and I had lots of interesting conversations and feedbacks from > the > >> people that I've met there. I thought I would share some of them with > the > >> dev list. By the way you can check out the talk on youtube > >> <https://youtu.be/DKxYE8YWF_o>! > >> > >> First of all, a lot of people were *very* interested in Apache Nemo! > and a > >> lot of people from the Beam community were very excited to hear about a > new > >> runner with primary support for their language! A few reasons for their > >> interest had been that since Beam does not actually get involved in the > >> runtime layer, where the actual scheduling or communication or > distributed > >> computation happens, they were interested in the optimizations that can > be > >> done in such layers. > >> > >> Second, with all the support from the TFX team, as well as the Beam SQL > >> team, it would bring loads of new possibilities for Nemo by supporting > the > >> *portability* *layer* of Beam, which supports applications written with > any > >> languages among Java, Python, and Go (and more in the future!). The > >> portability layer is getting more and more mature, and I think it's > about > >> time to support the portability layer for Nemo as well, as not a lot of > >> runners support it so far and it would give Nemo a head start. > >> > >> Another thing that I've noticed is that a lot of people are still very > much > >> interested in *batch* processing rather than stream processing. From the > >> people that I've talked to, I've learned that people found stream > >> processing to be quite pricey and that they haven't found stream > processing > >> worth the price that they were paying (for example, Spotify runs all of > >> their data processing workloads as batch). I guess Nemo could be a good > >> candidate to run batch processing, as Spark often suffers from problems > as > >> large-scale shuffle and data skew problems, if not provided with > machines > >> with enough memory, whereas Nemo is able to provide the optimizations > for > >> such problems. I've also found the people were interested if Nemo > supports > >> Kubernetes, which is a topic that we should definitely look into. > >> > >> I've also had many questions from the engineers from *Seznam.cz *and > >> *shopify.com > >> <http://shopify.com>* where they run their own datacenters to process > >> their > >> data (I think). They have been facing exactly the same problems as > >> illustrated above (large-scale shuffle, data skew, frequent data > reloading > >> for broadcasted data, utilizing transient resources, etc.), and have had > >> questions about running their data processing workloads on their large > >> amounts of data that they are facing every day (upto 40TB/day). I should > >> definitely follow up with them to see how they are doing, if they are > >> trying to use Nemo in their production, to provide help if needed and to > >> see Nemo's performance with real workloads. > >> > >> Lastly, I have been talking with Pablo (from Beam) about the trip to > >> *Seattle* and Renton, Washington next week regarding the USENIX ATC '19 > >> conference, and have had a chat about organizing a lunch and maybe a > small > >> talk with the Googlers there as well! I've also heard that Davor is also > >> based in Seattle, so I have been thinking that it would be a great > >> opportunity for us to meet in person. 😀The date would be probably the > >> *15th > >> of July*, so please keep the date in mind if you would be interested! > >> > >> Cheers, > >> Wonook > >> > >
