Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

Thanks everyone for supporting it. Yes, it's very slow to get tickets resolved by infra. I propose a minor improvement to reduce interactions with infra. So far, we have granted maintainer permission(read & write) to release managers' personal accounts. This step needs help from infra to add new

Re: Poor Python 3.x performance on Dataflow?

To close the loop here, the regression reported here is not specific to Beam or Dataflow. The difference in performance is caused by a 'regression' in the deprecated numpy random number generator, which we use to generate synthetic input for the load test pipeline. Since new releases of numpy don'

Re: Labels on PR

What do you exactly mean with github grep... where is it an issue. I find it useful for searching here: [image: Screen Shot 2020-02-13 at 06.11.33.png] OK, you get some false positives, but then the color coding works. You can't search on a category so this looks like the only alternative. I was

Re: Python2.7 Beam End-of-Life Date

On Wed, Feb 12, 2020 at 1:29 AM Ismaël Mejía wrote: > I am with Chad on this, we should probably extend it a bit more, even if it > makes us struggle a bit at least we have some workarounds as Robert > suggests, > and as Chad said there are still many people playing the python 3 catchup > game, >

Re: daily dataflow job failing today

On Wed, Feb 12, 2020 at 12:54 PM Ismaël Mejía wrote: > Independently of the bug in the dependency release the fact that the Beam > Python > SDK does not have pinned fixed dependency numbers is error-prone. We may > continue to have this kind of problems until we fix this (with other > dependencie

Re: Google's support framework for community-led meetups

Hi Maria, This might be useful in user@ as well? Happy to walk you through editing webpage submitting a PR. Then up to appropriate committer as to whether to approve. Write me off list and we can find time. A less focused walkthrough (including way more than you need just for that, in case wan

Re: FnAPI proto backwards compatibility

On Wed, Feb 12, 2020 at 12:04 PM Robert Bradshaw wrote: > On Wed, Feb 12, 2020 at 11:08 AM Luke Cwik wrote: > > > > We can always detect on the runner/SDK side whether there is an unknown > field[1] within a payload and fail to process it but this is painful in two > situations: > > 1) It doesn'

Re: Labels on PR

Alex would you consider removing the descriptions from the labels? It seems that github greps not only the text of the label but also the text of the description producing false positives, e.g. if I search the label `io` it resolves not only all the IOs but also results like `core` because it match

Re: daily dataflow job failing today

Independently of the bug in the dependency release the fact that the Beam Python SDK does not have pinned fixed dependency numbers is error-prone. We may continue to have this kind of problems until we fix this (with other dependencies too). In the Java SDK we do not accept such type of dynamic dep

Re: daily dataflow job failing today

Related: we have dependencies on avro, avro-python3, and fastavro. fastavro supports both python 2 and 3. Could we reduce this dependency list and depend only on fastavro? If we need avro and avro-python3 for the purposes of testing only, we can move them to test only dependencies. +Chamikara Jaya

Re: FnAPI proto backwards compatibility

On Wed, Feb 12, 2020 at 11:08 AM Luke Cwik wrote: > > We can always detect on the runner/SDK side whether there is an unknown > field[1] within a payload and fail to process it but this is painful in two > situations: > 1) It doesn't provide for a good error message since you can't say what the

Re: daily dataflow job failing today

Thank you Valentyn! On Wed, Feb 12, 2020 at 11:32 AM Valentyn Tymofieiev wrote: > Yes, otherwise all Python tests will continue to fail until Avro comes up > with a new release. Sent: https://github.com/apache/beam/pull/10844 > > On Wed, Feb 12, 2020 at 11:08 AM Ahmet Altay wrote: > >> Should w

Google's support framework for community-led meetups

Hi everyone, I have been working closely with Gris Cuevas to develop a framework to increase transparency on how Google Open Source supports community-led events. Below you will find a table that identifies different event goals, community roles, and the type of support Google Open Source can off

Re: Request to be added to maintainters in Jira.

What is your JIRA id? Also, note that there is an ongoing issue that prevents many people from running tests themselves on their PRs[1] and requires asking on the dev@ mailing list for someone with the appropriate set of permissions to launch the tests for you. 1: https://issues.apache.org/jira/b

Re: daily dataflow job failing today

Yes, otherwise all Python tests will continue to fail until Avro comes up with a new release. Sent: https://github.com/apache/beam/pull/10844 On Wed, Feb 12, 2020 at 11:08 AM Ahmet Altay wrote: > Should we update Beam's setup.py to skip this avro-python3 version? > > On Wed, Feb 12, 2020 at 10:5

Request to be added to maintainters in Jira.

Hi Beam developers, I have been working on adding Beam Python tests since last November. It is inconvenient for me right now since I can't run tests, comment on open issues, ask or answer questions on the forum. For example, I have a PR that may fix BEAM-9003, but I can't run the test or see the t

Re: FnAPI proto backwards compatibility

On Wed, Feb 12, 2020 at 7:57 AM Robert Bradshaw wrote: > On Tue, Feb 11, 2020 at 7:25 PM Kenneth Knowles wrote: > > > > On Tue, Feb 11, 2020 at 8:38 AM Robert Bradshaw > wrote: > >> > >> On Mon, Feb 10, 2020 at 7:35 PM Kenneth Knowles > wrote: > >> > > >> > On the runner requirements side: if

Re: daily dataflow job failing today

Should we update Beam's setup.py to skip this avro-python3 version? On Wed, Feb 12, 2020 at 10:57 AM Alan Krumholz wrote: > makes sense. I'll add this workaround for now. > Thanks so much for your help! > > On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev > wrote: > >> Alan, Dataflow worker

Re: daily dataflow job failing today

makes sense. I'll add this workaround for now. Thanks so much for your help! On Wed, Feb 12, 2020 at 10:33 AM Valentyn Tymofieiev wrote: > Alan, Dataflow workers preinstall Beam SDK dependencies, including (a > working version) of avro-python3. So after reading your email once again, I > think i

Re: daily dataflow job failing today

Alan, Dataflow workers preinstall Beam SDK dependencies, including (a working version) of avro-python3. So after reading your email once again, I think in your case you were not able to install Beam SDK locally. So a workaround for you would be to `pip install avro-python3==1.9.1` or `pip install p

Re: daily dataflow job failing today

Ah, there's already https://issues.apache.org/jira/browse/AVRO-2737 and it received attention. On Wed, Feb 12, 2020 at 10:19 AM Valentyn Tymofieiev wrote: > Opened https://issues.apache.org/jira/browse/AVRO-2738 > > On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev > wrote: > >> Here's a sho

Re: daily dataflow job failing today

Opened https://issues.apache.org/jira/browse/AVRO-2738 On Wed, Feb 12, 2020 at 10:14 AM Valentyn Tymofieiev wrote: > Here's a short repro: > > :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch > root@04b45a100d16:/# pip install avro-python3 > Collecting avro-python3 > Downloading av

Re: daily dataflow job failing today

Here's a short repro: :~$ docker run -it --entrypoint=/bin/bash python:3.7-stretch root@04b45a100d16:/# pip install avro-python3 Collecting avro-python3 Downloading avro-python3-1.9.2.tar.gz (37 kB) ERROR: Command errored out with exit status 1: command: /usr/local/bin/python -c 'import

Re: daily dataflow job failing today

Yes, it is a bug in the recent Avro release. We should report it to the Avro maintainers. The workaround is to downgrade avro-python3 to 1.9.1, for example via requirements.txt. On Wed, Feb 12, 2020 at 10:06 AM Steve Niemitz wrote: > avro-python3 1.9.2 was released on pypi 4 hours ago, and added

Re: daily dataflow job failing today

avro-python3 1.9.2 was released on pypi 4 hours ago, and added pycodestyle as a dependency, probably related? On Wed, Feb 12, 2020 at 1:03 PM Luke Cwik wrote: > +dev > > There was recently an update to add autoformatting to the Python SDK[1]. > > I'm seeing this during testing of a PR as well.

Re: daily dataflow job failing today

+dev There was recently an update to add autoformatting to the Python SDK[1]. I'm seeing this during testing of a PR as well. 1: https://lists.apache.org/thread.html/448bb5c2d73fbd74eec7aacb5f28fa2f9d791784c2e53a2e3325627a%40%3Cdev.beam.apache.org%3E On Wed, Feb 12, 2020 at 9:57 AM Alan Krumho

Re: [PROPOSAL] Preparing for Beam 2.20.0 release

+1. Thank you. On Tue, Feb 11, 2020 at 11:01 PM Rui Wang wrote: > Hi all, > > The next (2.20.0) release branch cut is scheduled for 02/26, according to > the calendar > > . > I would like to v

Re: Cross-language pipelines status

On Wed, Feb 12, 2020 at 8:10 AM Alexey Romanenko wrote: > > AFAIK, there's no official guide for cross-language pipelines. But there >> are examples and test cases you can use as reference such as: >> >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.p

Re: Cross-language pipelines status

> AFAIK, there's no official guide for cross-language pipelines. But there are > examples and test cases you can use as reference such as: > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py > >

Re: FnAPI proto backwards compatibility

On Tue, Feb 11, 2020 at 7:25 PM Kenneth Knowles wrote: > > On Tue, Feb 11, 2020 at 8:38 AM Robert Bradshaw wrote: >> >> On Mon, Feb 10, 2020 at 7:35 PM Kenneth Knowles wrote: >> > >> > On the runner requirements side: if you have such a list at the pipeline level, it is an opportunity for the li

Re: Cross-language pipelines status

Thank you for response! > AFAIK, there's no official guide for cross-language pipelines. But there are > examples and test cases you can use as reference such as: > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py > >

Re: Labels on PR

The prefix is just extra characters makes readibility worse, notice that the full category (e.g. ios/runners/etc) will match because we have an extra tag equivalent to the prefix, so filtering is still possible. you rarely need to filter for more than one criteria, that's why github does not allow

Re: Labels on PR

Ismael, I saw that you removed the prefix. I still want to have some grouping between the subtypes, so I color coded them. I also added all the labels in the file. We now have 62 labels. _/ _/ Alex Van Boxel On Wed, Feb 12, 2020 at 12:03 PM Ismaël Mejía wrote: > Forgot to mention, older PRs

Re: Labels on PR

Forgot to mention, older PRs will look not classified, up to you guys if you want to do manually. All new PRs will be automatically labeled. On Wed, Feb 12, 2020 at 12:02 PM Ismaël Mejía wrote: > For info Alex's PR to suport autolabeler was merged today and INFRA > enabled the plugin. > I create

Re: Labels on PR

For info Alex's PR to suport autolabeler was merged today and INFRA enabled the plugin. I created an artificial PR to check it was autolabeled correctly. It is working perfectly now. Thanks Alex ! On Tue, Feb 11, 2020 at 5:23 PM Robert Bradshaw wrote: > +1 to finding the right balance. > > I do

Re: Dynamic timers now supported!

Great to know you get it working on Dataflow easily Reuven. As a new feature it looks great! Agree with Kenn maybe worth to open a new thread to discuss the changes still needed to support this in portable runners. On Mon, Feb 10, 2020 at 8:25 PM Kenneth Knowles wrote: > I think the (lack of) p

Re: Python2.7 Beam End-of-Life Date

I am with Chad on this, we should probably extend it a bit more, even if it makes us struggle a bit at least we have some workarounds as Robert suggests, and as Chad said there are still many people playing the python 3 catchup game, so worth to support those users. But maybe it is worth to evalua

No space left on apache-beam-jenkins-7

Hi there, it seems we have an error on one of the Jenkins workers, I created a Jira to track this. Who can take care of this? https://issues.apache.org/jira/browse/BEAM-9302 Michal -- Michał Walenia Polidea | Software Engineer M: +48 791 432 002 <+48791432002> E: mic