Re: Add myself to jira to update Avro dependency

2019-10-11 Thread Kenneth Knowles
Thanks for doing this for us! Kenn On Fri, Oct 11, 2019 at 5:10 PM Ahmet Altay wrote: > Welcome to the project. I added your username to JIRA. > > On Fri, Oct 11, 2019 at 5:08 PM Jordanna wrote: > >> This is Jordanna Chord from Google. I'd like to update Apache Beam to >> 1.9.1 of the Avro

Re: Apply Confluence permission for fix `Python Tips` wiki command issue

2019-10-11 Thread jincheng sun
Hi Thomas, Thank you for granting access, I have updated the `Python Tips`. Best, Jincheng Thomas Weise 于2019年10月10日周四 下午9:38写道: > Thanks for spotting this. > > Access granted (for user "jincheng"). > > > On Thu, Oct 10, 2019 at 12:01 AM jincheng sun > wrote: > >> Hi all, >> >> The docker

Re: Add myself to jira to update Avro dependency

2019-10-11 Thread Ahmet Altay
Welcome to the project. I added your username to JIRA. On Fri, Oct 11, 2019 at 5:08 PM Jordanna wrote: > This is Jordanna Chord from Google. I'd like to update Apache Beam to > 1.9.1 of the Avro library. Can someone add me as a contributor for Beam's > Jira issue tracker so that I can

Add myself to jira to update Avro dependency

2019-10-11 Thread Jordanna
This is Jordanna Chord from Google. I'd like to update Apache Beam to 1.9.1 of the Avro library. Can someone add me as a contributor for Beam's Jira issue tracker so that I can assign/create the ticket? Thank;s ~jordanna chord username: jchord

Re: Outreachy Applicant

2019-10-11 Thread Omshi Samal
Hi Rui, It would be great if I could get some advice on where to get started - I'm a bit new to this. Thank you! :) Best, Omshi On Wed, Oct 9, 2019, 11:24 PM Omshi Samal wrote: > Hello! > > My username is omshi. > > > > On Wed, Oct 9, 2019 at 10:38 PM Rui Wang wrote: > >> Welcome! Can

Re: So much green

2019-10-11 Thread Robert Bradshaw
Very nice to see, thanks for sharing. On Fri, Oct 11, 2019 at 5:44 AM Maximilian Michels wrote: > > Glad to see that we have fixed the recent flakes. Let's keep up the good > work :) > > -Max > > On 10.10.19 23:37, Kenneth Knowles wrote: > > All the cells in the pull request template are green

Re: Python thread pool executor for Apache Beam

2019-10-11 Thread Robert Bradshaw
Can we use a lower default timeout to mitigate this issue in the short term (I'd imagine one second or possibly smaller would be sufficient for our use), and get a fix upstream in the long term? On Fri, Oct 11, 2019 at 9:38 AM Luke Cwik wrote: > > I'm looking for a thread pool that re-uses

Python thread pool executor for Apache Beam

2019-10-11 Thread Luke Cwik
I'm looking for a thread pool that re-uses threads that are idle before creating new ones and has an API that is compatible with the concurrent.futures ThreadPoolExecutor[1]. To my knowledge, the concurrent.futures ThreadPool creates new threads for tasks up until the thread pool limit before

Re: Feedback on how we use Apache Beam in my company

2019-10-11 Thread Pierre Vanacker
Thanks Etienne & Matthias ! Why not, it kinda depends on the location :) What meetup / summit do you have in mind ? Pierre De : Matthias Baetens Répondre à : "u...@beam.apache.org" Date : vendredi 11 octobre 2019 à 09:45 À : "u...@beam.apache.org" Cc : dev Objet : Re: Feedback on how we

Re: Feedback on how we use Apache Beam in my company

2019-10-11 Thread Pierre Vanacker
Nice, thanks. I just registered, see you there ! Pierre De : Alexey Romanenko Répondre à : "u...@beam.apache.org" Date : vendredi 11 octobre 2019 à 15:29 À : "u...@beam.apache.org" Cc : dev Objet : Re: Feedback on how we use Apache Beam in my company Hi Pierre, If you are in Paris region

Re: Feedback on how we use Apache Beam in my company

2019-10-11 Thread Alexey Romanenko
Hi Pierre, If you are in Paris region (I can guess because of Dailymotion name =) ) then it would be great to chat about that at next (2nd) Paris Beam meetup, which will be held very soon, October 17th. https://www.meetup.com/Paris-Apache-Beam-Meetup/events/264545288/

Re: [spark structured streaming runner] merge to master?

2019-10-11 Thread Etienne Chauchot
Hi Kenn, Comments inline On 10/10/2019 21:02, Kenneth Knowles wrote: +1 I think our experiences with things that go to master early have been very good. So I am in favor ASAP. We can exclude it from releases easily until it is ready for end users. I have mixed emotions on exclusion: I'd

Re: [spark structured streaming runner] merge to master?

2019-10-11 Thread Etienne Chauchot
Hi Robert comments inline: On 10/10/2019 20:49, Robert Bradshaw wrote: On Thu, Oct 10, 2019 at 12:39 AM Etienne Chauchot wrote: Hi guys, You probably know that there has been for several months an work developing a new Spark runner based on Spark Structured Streaming framework. This work is

Re: [spark structured streaming runner] merge to master?

2019-10-11 Thread Etienne Chauchot
I think that it is important to also provide in the build a jar that only contains the old runner, for people who want to ship only one. Etienne On 10/10/2019 15:40, Alexey Romanenko wrote: +1 for merging this new runner too (even if it’s not 100% ready for the moment) in case if it doesn’t

Re: So much green

2019-10-11 Thread Maximilian Michels
Glad to see that we have fixed the recent flakes. Let's keep up the good work :) -Max On 10.10.19 23:37, Kenneth Knowles wrote: All the cells in the pull request template are green right now and have been for most of the afternoon & evening. I just thought I would share this wonderful fact

Re: Resizing Beam IOITs

2019-10-11 Thread Michał Walenia
Hi, thanks for the suggestion. I think it's reasonable to include a small configuration for fast testing. I'll add such a config to the PR. Have a good day, Michal On Wed, Oct 9, 2019 at 5:05 AM Chamikara Jayalath wrote: > > > On Tue, Oct 8, 2019 at 6:52 AM Michał Walenia > wrote: > >> Hi

Re: Beam 2.15.0 SparkRunner issues

2019-10-11 Thread Ismaël Mejía
Thanks for reporting back Tim and great you found that it was a GC issue. In the meantime I filled BEAM-8384 [1] for the `spark.default.parallelism` issue. The Spark runner (both classic and structured streaming) only in exceptional cases tries to improve the default configuration values, but if

Re: Feedback on how we use Apache Beam in my company

2019-10-11 Thread Matthias Baetens
This is great, Pierre! Thank you for sharing, very interesting. Would you and your team be interested to talk about your use case at a meetup (or Summit) in the future? :) All the best, Matthias On Wed, 9 Oct 2019 at 15:59, Etienne Chauchot wrote: > Very nice ! > > Thanks > > ccing dev list >

So much green

2019-10-11 Thread Kenneth Knowles
All the cells in the pull request template are green right now and have been for most of the afternoon & evening. I just thought I would share this wonderful fact with everyone. Seeing it really makes an impression! Kenn