Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-24 Thread Alexey Romanenko
I don’t see any updates on my calendar. Does it work for others? > On 19 Sep 2019, at 17:16, Ismaël Mejía wrote: > > 25/09 looks ok. I just updated the meeting invitation to the new > date.I will prepare a mini agenda in the shared minute document in the > meantime. > I cannot see the old invit

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-24 Thread Vishwas Bm
Hi Ismael, Can you also add me to this meeting. I would also like to contribute. Regards, Vishwas On 2019/09/19 12:13:40, Etienne Chauchot wrote: > Hi Rahul and Xinyu,I just added you to the list of guests in the meeting. > Time is 5pm GMT +2. That being said, for some > reason last meeting s

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-19 Thread Ismaël Mejía
25/09 looks ok. I just updated the meeting invitation to the new date.I will prepare a mini agenda in the shared minute document in the meantime. I cannot see the old invitees, can someone please confirm me they see the date updated. Thanks, Ismaël On Thu, Sep 19, 2019 at 2:13 PM Etienne Chauchot

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-19 Thread Etienne Chauchot
Hi Rahul and Xinyu,I just added you to the list of guests in the meeting. Time is 5pm GMT +2. That being said, for some reason last meeting scheduled was 08/28. Ismael initially created the meeting, I do not have the rights to add a new date. Ismael can you add a date ? I suggest 09/25. WDYT ? B

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-18 Thread rahul patwari
Hi, I would love to join the call. Can you also share the meeting invitation with me? Thanks, Rahul On Wed 18 Sep, 2019, 11:48 PM Xinyu Liu, wrote: > Alexey and Etienne: I'm very happy to join the sync-up meeting. Please > forward the meeting info to me. I am based in California, US and hopefu

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-18 Thread Xinyu Liu
Alexey and Etienne: I'm very happy to join the sync-up meeting. Please forward the meeting info to me. I am based in California, US and hopefully the time will work :). Thanks, Xinyu On Wed, Sep 18, 2019 at 6:39 AM Etienne Chauchot wrote: > Hi Xinyu, > > Thanks for offering help ! My comments a

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-18 Thread Etienne Chauchot
Hi Xinyu, Thanks for offering help ! My comments are inline: Le vendredi 13 septembre 2019 à 12:16 -0700, Xinyu Liu a écrit : > Hi, Etienne, > The slides are very informative! Thanks for sharing the details about how the > Beam API are mapped into Spark > Structural Streaming. Thanks ! > We (Lin

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-18 Thread Etienne Chauchot
Hi Rui,Thanks for proposing to contribute to this new runner ! Here are the pointers:- SS runner branch: https://github.com/apache/beam/tree/spark-runner_structured-streaming- spark design doc for multiple watermarks support: https://docs.google.com/document/d/1IAH9UQJPUiUCLd7H6dazRK2k1szDX38SnM6

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-17 Thread Rui Wang
+1 on merging the runner into master, which make it more discoverable and easy to contribute (I am also interested in contributing). -Rui On Tue, Sep 17, 2019 at 3:36 AM Alexey Romanenko wrote: > Hi Xinyu, > > Great to hear that you wish to contribute into new Spark runner! We used > to have th

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-17 Thread Alexey Romanenko
Hi Xinyu, Great to hear that you wish to contribute into new Spark runner! We used to have the sync meetings about all Spark runners in general every two weeks, so feel free to let know us if you want to participate too. Also, as one of the contributors for Structural Streaming Spark runner (ye

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-13 Thread Xinyu Liu
Hi, Etienne, The slides are very informative! Thanks for sharing the details about how the Beam API are mapped into Spark Structural Streaming. We (LinkedIn) are also interested in trying the new SparkRunner to run Beam pipeine in batch, and contribute to it too. From my understanding, seems the f

Re: Pointers on Contributing to Structured Streaming Spark Runner

2019-09-11 Thread Rui Wang
Hello Etienne, Your slide mentioned that streaming mode development is blocked because Spark lacks supporting multiple-aggregations in its streaming mode but design is ongoing. Do you have a link or something else to their design discussion/doc? -Rui On Wed, Sep 11, 2019 at 5:10 PM Etienne Chau