Re: A new reworked Elasticsearch 7+ IO module

2020-03-06 Thread Jean-Baptiste Onofre
Hi, I think WARN makes sense and the safest approach. It allows users to be notify and eventually update or back on previous Beam IO version. Regards JB > Le 6 mars 2020 à 18:49, Kenneth Knowles a écrit : > > Since the user provides backendVersion, here are some possible levels of > things

Re: GroupIntoBatches not Working properly for Direct Runner Java

2020-03-06 Thread Kenneth Knowles
Can you reproduce it if you replace your Pubsub source with a TestStream and verify with PAssert [1]? This would enable you to easily build a unit test. You could even open a pull request adding that to the test suite for GroupIntoBatches [2]. That would be an excellent contribution to Beam. Kenn

Re: No space left on device - beam-jenkins 1 and 7

2020-03-06 Thread Alan Myrvold
Did a one time cleanup of tmp files owned by jenkins older than 3 days. Agree that we need a longer term solution. Passing recent tests on all executors except jenkins-12, which has not scheduled recent builds for the past 13 days. Not scheduling:

Re: No space left on device - beam-jenkins 1 and 7

2020-03-06 Thread Ahmet Altay
+Alan Myrvold is doing a one time cleanup. I agree that we need to have a solution to automate this task or address the root cause of the buildup. On Thu, Mar 5, 2020 at 2:47 AM Michał Walenia wrote: > Hi there, > it seems we have a problem with Jenkins workers again. Nodes 1 and 7 both > fail

Re: Snowflake connector

2020-03-06 Thread Chamikara Jayalath
Absolutely. Please create a JIRA and coordinate with Elias and any others that would like to contribute to this. Thanks, Cham On Fri, Mar 6, 2020 at 10:46 AM Shashanka Balakuntala < shbalakunt...@gmail.com> wrote: > Hi Chamikara and Elias, > This seems like an interesting feature. Can I start

Re: Jenkins jobs not running for my PR 10438

2020-03-06 Thread Tomo Suzuki
Thanks, Luke! On Fri, Mar 6, 2020 at 13:34 Tomo Suzuki wrote: > Hi Beam committers, > (Thank you, Michał!) > > Would you trigger the precommit checks for > https://github.com/apache/beam/pull/11063 > with the following 6 commands ? > Run Java PostCommit > Run Java HadoopFormatIO Performance

Re: Jenkins jobs not running for my PR 10438

2020-03-06 Thread Luke Cwik
Done On Fri, Mar 6, 2020 at 10:35 AM Tomo Suzuki wrote: > Hi Beam committers, > (Thank you, Michał!) > > Would you trigger the precommit checks for > https://github.com/apache/beam/pull/11063 > with the following 6 commands ? > Run Java PostCommit > Run Java HadoopFormatIO Performance Test >

Re: [DISCUSS] @Experimental annotations - processes and alternatives

2020-03-06 Thread Kenneth Knowles
OK I tried to make a tiny bit of progress on this, with `grep --ignore-case --line-number --recursive '@experimental' .` there are 578 occurrences (includes website and comments). Via `| cut -d ':' -f 1 | sort | uniq | wc -l` there are 377 distinct code files. So that's a big project but easily

Re: Snowflake connector

2020-03-06 Thread Shashanka Balakuntala
Hi Chamikara and Elias, This seems like an interesting feature. Can I start working on this? *Regards* Shashanka Balakuntala Srinivasa On Sat, Mar 7, 2020 at 12:00 AM Chamikara Jayalath wrote: > I don't think we have this but contributions are welcome. > > Thanks, > Cham > > On Tue, Mar 3,

Re: Jenkins jobs not running for my PR 10438

2020-03-06 Thread Tomo Suzuki
Hi Beam committers, (Thank you, Michał!) Would you trigger the precommit checks for https://github.com/apache/beam/pull/11063 with the following 6 commands ? Run Java PostCommit Run Java HadoopFormatIO Performance Test Run BigQueryIO Streaming Performance Test Java Run Dataflow ValidatesRunner

Re: Snowflake connector

2020-03-06 Thread Chamikara Jayalath
I don't think we have this but contributions are welcome. Thanks, Cham On Tue, Mar 3, 2020 at 4:46 AM Elias Djurfeldt wrote: > Hi all, > > I've stumbled upon a use case where I might need a SnowflakeIO in Python. > Has anyone worked on this before or are there any discussions surrounding > it?

Re: A new reworked Elasticsearch 7+ IO module

2020-03-06 Thread Kenneth Knowles
Since the user provides backendVersion, here are some possible levels of things to add in expand() based on that (these are extra niceties beyond the agreed number of releases to remove) - WARN for backendVersion < n - reject for backendVersion < n with opt-in pipeline option to keep it working

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

2020-03-06 Thread Chamikara Jayalath
Congrats Kamil! On Fri, Mar 6, 2020 at 6:44 AM Katarzyna Kucharczyk wrote: > Great news! Congratulations Kamil!  > > On Tue, Mar 3, 2020 at 9:05 PM Łukasz Gajowy wrote: > >> Congratulations mate! Well deserved. :) >> >> pon., 2 mar 2020 o 17:01 Elias Djurfeldt >> napisał(a): >> >>> Congrats

Re: Contributing Twister2 runner to Apache Beam

2020-03-06 Thread Pulasthi Supun Wickramasinghe
I understand that the discussion is on a more broad level than the Twister2 runner. From my experience developing the runner the main advantage of being inside the beam project was the easy access to the wide range of tests and other core/utility code as Kyle pointed out. Unmerging runners that

Re: [ANNOUNCE] New Committer: Kamil Wasilewski

2020-03-06 Thread Katarzyna Kucharczyk
Great news! Congratulations Kamil!  On Tue, Mar 3, 2020 at 9:05 PM Łukasz Gajowy wrote: > Congratulations mate! Well deserved. :) > > pon., 2 mar 2020 o 17:01 Elias Djurfeldt > napisał(a): > >> Congrats Kamil!! >> >> On Mon, 2 Mar 2020 at 16:16, Karolina Rosół >> wrote: >> >>>

Re: A new reworked Elasticsearch 7+ IO module

2020-03-06 Thread Etienne Chauchot
Hi all, it's been 3 weeks since the survey on ES versions the users use. The survey received very few responses: only 9 responses for now (multiple versions possible of course). The responses are the following: ES2: 0 clients, ES5: 1, ES6: 5, ES7: 8 It tends to go toward a drop of ES2

Re: Upcoming Apache Beam meetups in Warsaw

2020-03-06 Thread Michał Walenia
Hi, I'd also like to bump the meetup - I had great fun and interesting conversations on the previous one and I'm looking forward to having more. See you in Warsaw! If you'd like to present on the meetup (or on another one), please let us know! :) Cheers! Michal On Tue, Mar 3, 2020 at 11:17 AM