Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

2020-04-09 Thread Alex Van Boxel
We forgot something ... ... it/she/he needs a *name*! _/ _/ Alex Van Boxel On Fri, Apr 10, 2020 at 6:19 AM Kenneth Knowles wrote: > Looking forward to the guide. I enjoy doing (bad) drawings as a way to > relax. And I want them to be properly on brand :-) > > Kenn > > On Thu, Apr 9, 2

Re: Usage metrics for Beam

2020-04-09 Thread Robert Bradshaw
Yes, it's hard to know what can conclusively be drawn from the raw totals. I do think trends and ratios (e.g. Py2 vs. Py3) will, however, roughly reflect underlying usage (which itself is ambiguously defined). On Thu, Apr 9, 2020 at 7:30 PM Kenneth Knowles wrote: > Yea, interpreting the raw abso

Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

2020-04-09 Thread Kenneth Knowles
Looking forward to the guide. I enjoy doing (bad) drawings as a way to relax. And I want them to be properly on brand :-) Kenn On Thu, Apr 9, 2020 at 10:35 AM Maximilian Michels wrote: > Awesome. What a milestone! The mascot is a real eye catcher. Thank you > Julian and Aizhamal for making it h

Re: Usage metrics for Beam

2020-04-09 Thread Kenneth Knowles
Yea, interpreting the raw absolute number is tricky. You can probably manage to see certain kinds of trends if you just look at relative numbers. Kenn On Thu, Apr 9, 2020 at 6:42 PM Austin Bennett wrote: > @Robert Bradshaw , you sent that pypi link [1] the > other day in response to something

Re: Usage metrics for Beam

2020-04-09 Thread Austin Bennett
@Robert Bradshaw , you sent that pypi link [1] the other day in response to something else, which is what prompted me to ask Gris about Maven (based on that link [2], @* Kenneth Knowles ). I recall talking to someone about Maven download statistics at ApacheCon. Perhaps these are not the only

Re: [VOTE] Release 2.20.0, release candidate #2

2020-04-09 Thread Robert Bradshaw
+1, the artifacts and signatures all look good, and I also checked that the Python wheels work with a simple pipeline in a fresh virtual environment. On Thu, Apr 9, 2020 at 5:11 PM Ahmet Altay wrote: > +1 - validated python quickstarts batch/streaming with python 2.7. > > Thank you Rui! > > On T

Re: [VOTE] Release 2.20.0, release candidate #2

2020-04-09 Thread Ahmet Altay
+1 - validated python quickstarts batch/streaming with python 2.7. Thank you Rui! On Thu, Apr 9, 2020 at 12:28 PM Valentyn Tymofieiev wrote: > +1. Checked mobile gaming batch examples, and a streaming quickstart on > Dataflow, on Python 3.7 using Linux wheels. > > On Thu, Apr 9, 2020 at 11:13 A

Re: Usage metrics for Beam

2020-04-09 Thread Kenneth Knowles
I found some info from 2010 [1] that it was available to anyone with deploy permission. The instructions still work. Kenn [1] https://blog.sonatype.com/2010/12/now-available-central-download-statistics-for-oss-projects/ On Thu, Apr 9, 2020 at 3:41 PM Robert Bradshaw wrote: > For Python, there'

Re: Usage metrics for Beam

2020-04-09 Thread Robert Bradshaw
For Python, there's https://pypistats.org/packages/apache-beam . It's unclear how accurate these are, and how many of these downloads represent users vs. tools (e.g. setting up environments for continuous testing). On Thu, Apr 9, 2020 at 3:29 PM Griselda Cuevas wrote: > Hi folks - I'm interested

Usage metrics for Beam

2020-04-09 Thread Griselda Cuevas
Hi folks - I'm interested in knowing more about Beam's adoption through user downloads. Do you know what's the protocol to access Maven and check on Java downloads? Also - do you have any other recos on how to measure the project's adoption evolution? Thanks! G

Re: Dataflow Streaming ValidatesRunner

2020-04-09 Thread Kenneth Knowles
We (Beam) used to run all ValidatesRunner tests with the --streaming flag forced, because we didn't have autodetection of unbounded data in the pipeline. That functionality was lost in the migration to gradle IIRC. It is still valuable, but also expensive. The quota issue Luke mentioned could be re

Re: Dataflow Streaming ValidatesRunner

2020-04-09 Thread Luke Cwik
There are a limited number of streaming pipelines that run as post commits on Jenkins but it isn't comprehensive compared to the validates runner set. Googlers regularly import the github code into Google and find issues with implementations that have been merged. Sometimes its bugs in implementat

Re: Dataflow Streaming ValidatesRunner

2020-04-09 Thread Reuven Lax
You can also call Create and call setIsBoundedInternal(IsBounded.UNBOUNDED) on the resulting PCollection, which will force the streaming runner to be used. On Thu, Apr 9, 2020 at 2:25 PM Steve Niemitz wrote: > Ah yeah I forgot that you can force a pipeline into streaming mode with > that flag. >

Re: Dataflow Streaming ValidatesRunner

2020-04-09 Thread Steve Niemitz
Ah yeah I forgot that you can force a pipeline into streaming mode with that flag. It sounds like the story here is there are tests for the streaming worker, but they run "on the side" in Google's environment? My concern is it seems like (publically at least) there's no test coverage on the strea

Re: Dataflow Streaming ValidatesRunner

2020-04-09 Thread Luke Cwik
You can use Create in streaming pipelines as well but you want to ensure that --streaming is passed as a flag. You could update the existing test target and force --streaming to be inserted for example here: https://github.com/lukecwik/incubator-beam/blob/8097972b4d0ed759aa45f6710ac02b982c6e8deb/ru

Re: Add account as contributor to the Beam JIRA

2020-04-09 Thread Luke Cwik
Welcome, I haved added you as a contributor and assigned the JIRA to you. On Thu, Apr 9, 2020 at 1:06 PM Paul Fisher wrote: > Hello! > > I (pfishgoogle / pf...@google.com) fixed > https://issues.apache.org/jira/browse/BEAM-9731. Can I be added as a > contributor to the Beam JIRA? > > Thanks! > >

Add account as contributor to the Beam JIRA

2020-04-09 Thread Paul Fisher
Hello! I (pfishgoogle / pf...@google.com) fixed https://issues.apache.org/jira/browse/BEAM-9731. Can I be added as a contributor to the Beam JIRA? Thanks! -- ‘Creep’ is a bad song. Sent From My Thom’s Computer smime.p7s Description: S/MIME Cryptographic Signature

Re: [VOTE] Release 2.20.0, release candidate #2

2020-04-09 Thread Valentyn Tymofieiev
+1. Checked mobile gaming batch examples, and a streaming quickstart on Dataflow, on Python 3.7 using Linux wheels. On Thu, Apr 9, 2020 at 11:13 AM Rui Wang wrote: > Hi everyone, > Please review and vote on the release candidate #2 for the version 2.20.0, > as follows: > [ ] +1, Approve the rele

[VOTE] Release 2.20.0, release candidate #2

2020-04-09 Thread Rui Wang
Hi everyone, Please review and vote on the release candidate #2 for the version 2.20.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Re: Request edit access to Beam wiki

2020-04-09 Thread Pablo Estrada
Hi Ning! I've given you edit privileges for the wiki. Thanks! -P. On Thu, Apr 9, 2020 at 10:33 AM Ning Kang wrote: > Hi all, > > I've been developing some screendiff integration tests for Interactive > Beam and would like to add a few instructions in the wiki. > > Could I get edit access to the

Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

2020-04-09 Thread Maximilian Michels
Awesome. What a milestone! The mascot is a real eye catcher. Thank you Julian and Aizhamal for making it happen. On 06.04.20 22:05, Aizhamal Nurmamat kyzy wrote: > I am happy to announce that this vote has passed, with 13 approving +1 > votes, 5 of which are binding PMC votes. > > We have the fin

Request edit access to Beam wiki

2020-04-09 Thread Ning Kang
Hi all, I've been developing some screendiff integration tests for Interactive Beam and would like to add a few instructions in the wiki. Could I get edit access to the wiki? [image: h3rak7q86QR.png] My user ID to the confluence site is: ningk Thanks! Ning.

Dataflow Streaming ValidatesRunner

2020-04-09 Thread Steve Niemitz
I was trying to run a @ValidatesRunner test for the streaming dataflow runner, but I actually can't find any way to run them in streaming. It looks like all the tests are set up using the Create transform, which generates a batch pipeline. Are there actually no @ValidatesRunner tests for the stre

Re: A new reworked Elasticsearch 7+ IO module

2020-04-09 Thread Etienne Chauchot
Hi Kenn, The user does not specify the backendVersion targeted (at least on the current version of the IO) it is transparent to him: the IO detects the version with a REST call and adapts its behavior. But, anyway, I agree, we need to put at least a WARN if detected version is 2. As the new IO