Re: Companies using Beam?

2020-05-26 Thread Ahmet Altay
On Tue, May 26, 2020 at 6:02 PM Aizhamal Nurmamat kyzy wrote: > > > Aizhamal, what is the plan for your document? Would it be fine if we add >> this powered_by page now, and edit as we go? >> > Yes. The document was created as a plan to add all those sections, but it > would be great to start

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Thomas Weise
> > I think the "set_version.sh" script could be called in the release > scripts to remove the -SNAPSHOT suffix on the release branch. > > I would expect the release branch to have the next -SNAPSHOT version (not the case currently):

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Ahmet Altay
On Tue, May 26, 2020 at 11:50 AM Maximilian Michels wrote: > Don't think so. Feel free to create one. > I think a related jira would be ( https://issues.apache.org/jira/browse/BEAM-9181). Removing the snapshot form javadoc versions. > > We already have a script which updates the version to a

Re: Companies using Beam?

2020-05-26 Thread Aizhamal Nurmamat kyzy
Aizhamal, what is the plan for your document? Would it be fine if we add > this powered_by page now, and edit as we go? > Yes. The document was created as a plan to add all those sections, but it would be great to start adding them as soon as possible. > Ahmet > > On Tue, May 26, 2020 at 2:30

[RELEASE VOTE RESULT] Release 2.21.0, release candidate #1

2020-05-26 Thread Kyle Weaver
I'm happy to announce that we have approved this release. There are 3 approving votes, all of which are binding: * Ahmet Altay * Luke Cwik * Robert Bradshaw Thanks for everyone's help! I'm going to finalize the release and send out the official release announcement tomorrow. Kyle

Re: Companies using Beam?

2020-05-26 Thread Ahmet Altay
Another example is Arrow's powered by page. ( https://arrow.apache.org/powered_by/). I like two aspects of this page. First it is simple to add things, there is a pull request link at the top of the list. Second, it lists products and companies alike. For Beam, we can list TFX as one of these

Re: DataflowRunner | Cross-language

2020-05-26 Thread Robert Bradshaw
On Tue, May 26, 2020 at 4:12 PM Sergei Sokolenko wrote: > small correction. While the new runner will be available with Beam 2.21, > the Cross-Language support will be available in 2.22. > There will be limitations in the initial set of connectors you can use > with Cross-Lang. But at least you

Re: DoFnSignature#isStateful deprecated

2020-05-26 Thread Luke Cwik
I believe DoFnSignature#isStateful is remnant of a bad API name choice and was renamed to usesState. I would remove DoFnSignature#isStateful as it does not seem to be used anywhere. Does DoFnSignatures#usesValueState return true if the DoFn says it needs @RequiresTimeSortedInput because of how a

Re: DataflowRunner | Cross-language

2020-05-26 Thread Sergei Sokolenko
small correction. While the new runner will be available with Beam 2.21, the Cross-Language support will be available in 2.22. There will be limitations in the initial set of connectors you can use with Cross-Lang. But at least you will have something to test with, starting in 2.22 On Tue, May

Re: Companies using Beam?

2020-05-26 Thread Aizhamal Nurmamat kyzy
> Oops, just saw you linked it earlier Aizhamal. Is it still open for input? > > Yes, it is! Feel free to add suggestions right in the doc. The doc will eventually need some cleanup/reorg, but now is a good time to add suggestions. Thanks Matthias! > On Mon, 25 May 2020 at 09:20, Matthias

Re: Interest

2020-05-26 Thread Aizhamal Nurmamat kyzy
Hi Dule, Thank you for the introduction and your interest to work on Apache Beam documentation with Season of Docs. To participate in the program you need to follow the guides here [1] [2]. If you are new to the program, we suggest: 1. Start by studying our proposed project ideas and

Re: [VOTE] Release 2.21.0, release candidate #1

2020-05-26 Thread Kyle Weaver
Thanks Kenn. Which experimental feature are you referring to? On Tue, May 26, 2020 at 10:00 AM Kenneth Knowles wrote: > https://issues.apache.org/jira/browse/BEAM-10015 is a correctness > issue, basically an experimental feature (I hope marked as such) not really > working at all. It probably

Re: Subscribe Dev Maillist

2020-05-26 Thread Peter Huang
Hi Pablo, Thanks. I just did it :) Best Regards Peter Huang On Tue, May 26, 2020 at 11:01 AM Pablo Estrada wrote: > Hi Peter! > You'll need to email dev-subscr...@beam.apache.org > Welcome! > -P. > > On Tue, May 26, 2020 at 9:07 AM Peter Huang > wrote: > >> >>

Re: Code Coverage Tracking

2020-05-26 Thread Tyson Hamilton
Huh.. I just realized it is closed already, now I'm confused. On 2020/05/26 15:08:21, Tyson Hamilton wrote: > Yes, this would be a good choice for the fixit. The tracking issue exists > already too: > > https://issues.apache.org/jira/browse/BEAM-1132 > > On 2020/05/26 14:00:00, Kenneth

Re: Code Coverage Tracking

2020-05-26 Thread Tyson Hamilton
Yes, this would be a good choice for the fixit. The tracking issue exists already too: https://issues.apache.org/jira/browse/BEAM-1132 On 2020/05/26 14:00:00, Kenneth Knowles wrote: > We tried coveralls (I also used it in a few past projects) and it did not > really work for us. A service

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Maximilian Michels
Don't think so. Feel free to create one. We already have a script which updates the version to a non-snapshot version: https://github.com/apache/beam/blob/master/release/src/main/scripts/set_version.sh However, it seems that this is merely a variant of this script which we use to cut the release

Re: DataflowRunner | Cross-language

2020-05-26 Thread Sergei Sokolenko
More info will be forthcoming after Beam 2.21 is out. There will be a docs page describing how it all works. On Thu, May 21, 2020 at 11:18 PM Paweł Urbanowicz < pawel.urbanow...@polidea.com> wrote: > Hello, community, > > I found information that Google is working on supporting Dataflow runner >

Re: DataflowRunner | Cross-language

2020-05-26 Thread Chamikara Jayalath
We are working on making Kafka IO available to Python streaming users on Dataflow through cross-language transforms. There's no ETA for the availability of the framework in general for Dataflow yet. Thanks, Cham On Thu, May 21, 2020 at 11:18 PM Paweł Urbanowicz < pawel.urbanow...@polidea.com>

Re: Subscribe Dev Maillist

2020-05-26 Thread Pablo Estrada
Hi Peter! You'll need to email dev-subscr...@beam.apache.org Welcome! -P. On Tue, May 26, 2020 at 9:07 AM Peter Huang wrote: > >

Re: Contributor access to Beam JIRA

2020-05-26 Thread Pablo Estrada
Welcome Ashwin! And thanks for the new contributions! I've noticed some of them already, and they're appreciated. Best -P. On Tue, May 26, 2020 at 7:57 AM Ismaël Mejía wrote: > Done. welcome! > > On Mon, May 25, 2020 at 5:12 PM Ashwin Ramaswami > wrote: > > > > I'd like to work on some issues,

Re: deploy_travis.sh ?

2020-05-26 Thread Ahmet Altay
Thank you for updating the readme. This file was removed as part of a change that removed the need to share personal credentials with travis in order to deploy directly to svn. On Tue, May 26, 2020 at 8:03 AM Kyle Weaver wrote: > > If my understanding is correct, I will at least update the >

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Jacek Laskowski
Hi Max, > I think you bring up a good point, for the sake of release build reproducibility, we may want to remove the snapshot suffix for the source release. Wish I could be as clear as yourself with this. Yes, that's what I've been bothered about. Is there a JIRA issue for this already? I've

Re: 2.22.0 Release Update

2020-05-26 Thread Steve Niemitz
On Tue, May 26, 2020 at 12:20 PM Brian Hulette wrote: > > On Mon, May 25, 2020 at 1:30 PM Steve Niemitz wrote: > >> I tried to validate the release branch on dataflow, but it looks like >> there's no tag yet for the 2.22 docker containers [1] >> > > I haven't built any Dataflow containers yet

Re: 2.22.0 Release Update

2020-05-26 Thread Brian Hulette
On Mon, May 25, 2020 at 1:30 PM Steve Niemitz wrote: > I tried to validate the release branch on dataflow, but it looks like > there's no tag yet for the 2.22 docker containers [1] > I haven't built any Dataflow containers yet since BEAM-9971 and BEAM-10057 are actively being worked on and

Interest

2020-05-26 Thread dule martins
Hello! Trust this mail meets you well, i went through your project idea and found interest in the following project idea: 2. Update of the runner comparison page / capability matrix

Subscribe Dev Maillist

2020-05-26 Thread Peter Huang

Re: [DISCUSS] Dealing with @Ignored tests

2020-05-26 Thread Tyson Hamilton
A lot of these are good fixit tasks (shameless plug [1]), * archeology on the @Ignored tests to normalize them with proper labelling * checkstyle validation for @Ignored tests * spring cleaning on open JIRA issues Having a Jira issue for the first 2 of these and adding the beam-fixit label

Re: [REVIEW][please pause website changes] Migrated the Beam website to Hugo

2020-05-26 Thread Aizhamal Nurmamat kyzy
Hi, Alexey, Yes, now we can keep updating the website as before. Here is a contributor guide: https://github.com/apache/beam/blob/master/website/CONTRIBUTE.md Thanks for checking :) Aizhamal On Tue, May 26, 2020 at 8:39 AM Alexey Romanenko wrote: > Thank you to everybody who worked on this

Re: [REVIEW][please pause website changes] Migrated the Beam website to Hugo

2020-05-26 Thread Alexey Romanenko
Thank you to everybody who worked on this migration. I just wanted to clarify - does it mean, since it’s finished now, we can update a website as before? > On 15 May 2020, at 01:33, Aizhamal Nurmamat kyzy wrote: > > Thank you Ahmet, Brian, Robert and everyone else who spent time working on

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Maximilian Michels
If you really want to work with the source code, I'd recommend using the released source code: https://beam.apache.org/get-started/downloads/#releases Even there the version in gradle.properties says x.y.z-SNAPSHOT. You may want to remove the -SNAPSHOT suffix. I understand that this is confusing

Re: Code Coverage Tracking

2020-05-26 Thread Luke Cwik
Its integrated and available as part of the Jenkins UI: https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PreCommit_Java_Cron/lastBuild/jacoco/ On Tue, May 26, 2020 at 8:10 AM Tyson Hamilton wrote: > Huh.. I just realized it is closed already, now I'm confused. > > On

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Kyle Weaver
> When we release the version, the RC suffix is dropped. I think this might not actually be true, at least for the git tag, since we just copy the tag from the accepted RC without changing anything. However, it might not matter because RC2 artifacts should be identical to the final release

Re: deploy_travis.sh ?

2020-05-26 Thread Kyle Weaver
> If my understanding is correct, I will at least update the documentation/README in beam-wheels. Sounds good. On Sun, May 24, 2020 at 7:33 PM Austin Bennett wrote: > Thanks Ashwin, > > Exactly. The readme doesn't match the current state of the code. > > > On Sun, May 24, 2020, 2:34 PM Ashwin

Re: Contributor access to Beam JIRA

2020-05-26 Thread Ismaël Mejía
Done. welcome! On Mon, May 25, 2020 at 5:12 PM Ashwin Ramaswami wrote: > > I'd like to work on some issues, assign issues to myself, and help close out > old issues that are already resolved. Could you give me contributor access to > the Beam JIRA?

Re: More metadata in Coder Proto

2020-05-26 Thread Kenneth Knowles
Thought I'd mention a valuable enhancement that has been proposed a couple of times: when inferring a coder, choose a deterministic coder when one is needed. Our current behavior of picking a coder first and then crashing if we picked the wrong one is suboptimal, for no real reason. xlang does

Re: Code Coverage Tracking

2020-05-26 Thread Kenneth Knowles
We tried coveralls (I also used it in a few past projects) and it did not really work for us. A service that I really like is codecov.io (and they have a chrome plugin to overlay coverage on PRs) but the granularity of GitHub permissions was a problem in the past (most GitHub apps require blanket

Re: Event Calendar?

2020-05-26 Thread Kenneth Knowles
+1 for merging release schedule into a general Apache Beam calendar. Right now the calendar is a bit buried so we should make it more discoverable. Kenn On Thu, May 21, 2020 at 3:07 AM Maximilian Michels wrote: > Would it make sense to combine it with the Apache Beam release calendar? > > >

Re: [VOTE] Release 2.21.0, release candidate #1

2020-05-26 Thread Kenneth Knowles
https://issues.apache.org/jira/browse/BEAM-10015 is a correctness issue, basically an experimental feature (I hope marked as such) not really working at all. It probably has a fairly small audience for now. I will not -1 because of it but I will -0. If there is another RC this should be included.

Re: Apache Beam Fixit 2020-06-08

2020-05-26 Thread Kenneth Knowles
This is the perfect time for improving engineering excellence. Reliability is especially important right now. Improved greenness, reduced flakiness, improved coverage (and highlighting the coverage signal), triaging ignored tests removing dead code. These are all clear wins for any project. For

[BEAM-10054] Pipeline stalls with DirectRunner

2020-05-26 Thread Maximilian Michels
Could somebody familiar with the Python SDK take a look at this problem? It manifests in the Direct Runner stalling execution. Tests are passing but I'm unsure about the context of the commit which introduced the change (linked in the PR): https://github.com/apache/beam/pull/11777 Thanks, Max

Re: What's the purpose of version=2.20.0-RC2 in gradle.properties?

2020-05-26 Thread Jacek Laskowski
Hi Max, > You probably want to work with the release artifacts, instead of cloning > the development branch. I'm not sure I understand. I did the following to work with the sources of v2.20.0. Am I missing something? git fetch --all --tags --prune git checkout -b v2.20.0 v2.20.0 The last