[DISCUSS] Bump the version of GRPC from 1.21.0 to 1.22.0+ (May be the latest 1.26.0?)

2019-12-23 Thread jincheng sun
Hi folks, When submitting a Python word count job to a Flink session/standalone cluster repeatedly, the meta space usage of the task manager of the Flink cluster will continuously increase (about 40MB each time). The reason is that the Beam classes are loaded with the user class loader(child-first

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread jincheng sun
Congrats Kasia! Best, Jincheng Ankur Goenka 于2019年12月24日周二 上午7:48写道: > Congratulations Kasia! > > On Mon, Dec 23, 2019 at 2:57 PM Chamikara Jayalath > wrote: > >> Congrats! >> >> On Mon, Dec 23, 2019 at 2:15 PM Yichi Zhang wrote: >> >>> Congrats Kasia! >>> >>> On Mon, Dec 23, 2019 at 2:07 PM

Re: [BEAM-9015] Adding pyXX-cloud instead of pyXX-gcp and pyXX-aws

2019-12-23 Thread Robert Bradshaw
Makes sense to me. On Mon, Dec 23, 2019 at 3:33 PM Pablo Estrada wrote: > > Hi all, > a couple of contributors [1][2] have been kind enough to add support for s3 > filesystem[3] for the Python SDK. Part of this involved adding a tox task > called py37-aws, to install the relevant dependencies a

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Ankur Goenka
Congratulations Kasia! On Mon, Dec 23, 2019 at 2:57 PM Chamikara Jayalath wrote: > Congrats! > > On Mon, Dec 23, 2019 at 2:15 PM Yichi Zhang wrote: > >> Congrats Kasia! >> >> On Mon, Dec 23, 2019 at 2:07 PM Rui Wang wrote: >> >>> Congrats! >>> >>> >>> -Rui >>> >>> On Mon, Dec 23, 2019 at 1:39

[BEAM-9015] Adding pyXX-cloud instead of pyXX-gcp and pyXX-aws

2019-12-23 Thread Pablo Estrada
Hi all, a couple of contributors [1][2] have been kind enough to add support for s3 filesystem[3] for the Python SDK. Part of this involved adding a tox task called py37-aws, to install the relevant dependencies and run unit tests for it (in a mocked-out environment). To avoid running a full extra

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Chamikara Jayalath
Congrats! On Mon, Dec 23, 2019 at 2:15 PM Yichi Zhang wrote: > Congrats Kasia! > > On Mon, Dec 23, 2019 at 2:07 PM Rui Wang wrote: > >> Congrats! >> >> >> -Rui >> >> On Mon, Dec 23, 2019 at 1:39 PM Udi Meiri wrote: >> >>> Congrats Kasia! >>> >>> On Mon, Dec 23, 2019 at 1:23 PM Kyle Weaver wro

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Yichi Zhang
Congrats Kasia! On Mon, Dec 23, 2019 at 2:07 PM Rui Wang wrote: > Congrats! > > > -Rui > > On Mon, Dec 23, 2019 at 1:39 PM Udi Meiri wrote: > >> Congrats Kasia! >> >> On Mon, Dec 23, 2019 at 1:23 PM Kyle Weaver wrote: >> >>> Congrats Kasia! And thanks for sharing, Pablo. >>> >>> On Mon, Dec 23

Re: [Spark Structured Streaming runner] perfs and encoders

2019-12-23 Thread Udi Meiri
That's great! The optimizations seem to have helped a lot! On Mon, Dec 23, 2019 at 6:49 AM Etienne Chauchot wrote: > Hi all, > > good news ! > > I did some refactoring of the encoders to improve maintenability and > replace as much as possible string generated code with compiled code and > the p

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Rui Wang
Congrats! -Rui On Mon, Dec 23, 2019 at 1:39 PM Udi Meiri wrote: > Congrats Kasia! > > On Mon, Dec 23, 2019 at 1:23 PM Kyle Weaver wrote: > >> Congrats Kasia! And thanks for sharing, Pablo. >> >> On Mon, Dec 23, 2019 at 4:16 PM Pablo Estrada wrote: >> >>> Hi everyone, >>> >>> Please join me a

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Udi Meiri
Congrats Kasia! On Mon, Dec 23, 2019 at 1:23 PM Kyle Weaver wrote: > Congrats Kasia! And thanks for sharing, Pablo. > > On Mon, Dec 23, 2019 at 4:16 PM Pablo Estrada wrote: > >> Hi everyone, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Kasia Kucharczyk >> >

Re: [ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Kyle Weaver
Congrats Kasia! And thanks for sharing, Pablo. On Mon, Dec 23, 2019 at 4:16 PM Pablo Estrada wrote: > Hi everyone, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Kasia Kucharczyk > > Kasia has contributed to Beam in many ways, including the performance > testing

[ANNOUNCE] New committer: Kasia Kucharczyk

2019-12-23 Thread Pablo Estrada
Hi everyone, Please join me and the rest of the Beam PMC in welcoming a new committer: Kasia Kucharczyk Kasia has contributed to Beam in many ways, including the performance testing infrastructure, and has even spoken at events about Beam. In consideration of Kasia's contributions, the Beam PMC

Re: External transform API in Java SDK

2019-12-23 Thread Heejong Lee
On Fri, Dec 20, 2019 at 11:38 AM Luke Cwik wrote: > What do side inputs look like? > A user needs to first pass PCollections for side inputs into the external transform in addition to ordinary input PCollections and define PCollectionViews inside the external transform something like: PCollecti

Re: [VOTE] Release 2.17.0, release candidate #2

2019-12-23 Thread Luke Cwik
+1, I validated the Java quickstarts for the runners and the issues I have brought up have been moved to a future release. On Fri, Dec 20, 2019 at 8:09 PM Ahmet Altay wrote: > +1, I validated the python2 quick starts using wheels. Thank you for > pushing the release this far. > > On Thu, Dec 19,

Re: Contributor permissions

2019-12-23 Thread Luke Cwik
Welcome, I have added you as a contributor to the BEAM JIRA project and assigned BEAM-8987 and BEAM-8897 to you. Feel free to unassign any JIRAs from yourself and pickup and unassigned JIRAs. On Sat, Dec 21, 2019 at 10:47 AM Shashanka Balakuntala < shbalakunt...@gmail.com> wrote: > Hi, > > Yes I

[Spark Structured Streaming runner] perfs and encoders

2019-12-23 Thread Etienne Chauchot
Hi all, good news ! I did some refactoring of the encoders to improve maintenability and replace as much as possible string generated code with compiled code and the perf results are awesome ! Best Etienne

Re: [PROPOSAL] python precommit timeouts

2019-12-23 Thread jincheng sun
Big +1 on it. Thanks a lot for the improvement Udi ! It also make sense to have this timeout for other tests like Max said. I'm thinking whether there are any timeout configurations for Jenkins, in this case, the timeout config could be applied to all tests if necessary. Best, Jincheng Robert