Re: [VOTE] Release 2.29.0, release candidate #1

+1 (binding) I ran some python quick start examples. Most validations in the sheet were already done :) Thank you all! On Thu, Apr 22, 2021 at 9:15 AM Kyle Weaver wrote: > +1 (non-) > > Ran Python wordcount on Flink and Spark. > > On Wed, Apr 21, 2021 at 5:20 PM Brian Hulette wrote: > >> +1

Flaky test issue report

This is your daily summary of Beam's current flaky tests. These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. BEAM-12200: SamzaStoreStateInternalsTest is flaky

P1 issues report

This is your daily summary of Beam's current P1 issues, not including flaky tests. See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. BEAM-12205: Dataflow pipelines broken NoSuchMethodError DoFnInvoker.invokeSetup()

Re: Issues and PR names and descriptions (or should we change the contribution guide)

Case study: https://github.com/apache/beam/pull/14618/commits To get back to the question of Issue and PR titles: - When doing Jira triage, fix issue titles and issue type to be meaningful. These autogenerate release notes, and also we use these to find duplicates, etc. - Any committer can

Re: [QUESTION] Dockerized Integration Tests with Java/Gradle

Thanks all for the warm welcome and documentation. I'll peruse the links sent and ask follow-ups once I've digested the info. Thanks again, Evan On Thu, Apr 22, 2021 at 12:11 PM Brian Hulette wrote: > Welcome Evan! > > Note we do have some examples in Beam of running IO integration tests >

Re: Issues and PR names and descriptions (or should we change the contribution guide)

I think I am wrong about this. It seems like for squashed/rebased commits it is still GitHub that is committer? But it does seem to have the metadata about who did the squash & merge. This pattern of storing important metadata outside of git is not a good direction. Kenn On Thu, Apr 22, 2021 at

Re: Issues and PR names and descriptions (or should we change the contribution guide)

That is unfortunate that GitHub is the committer of merge commits :-/ though I suppose you have the author field you can use. It is unfortunate the this is a different field based on method. Kenn On Thu, Apr 22, 2021 at 12:39 PM Ismaël Mejía wrote: > I was not referring to author identity but

Re: Issues and PR names and descriptions (or should we change the contribution guide)

I was not referring to author identity but to committer identity that matters to know who accepted to merge something but it seems we are not really using this much because github is the 'committer' of merge commits too :S maybe something we can improve as part of this discussion. git show

Re: Issues and PR names and descriptions (or should we change the contribution guide)

Author identity is preserved. Here's an output of 'git log' commit 93ecc1d3a4b997b2490c4439972ffaf09125299f Merge: 2e9ee8c005 4e3decbb4e <-- a merge commit that merges 2 commit, 4e3decbb4e and it's parent. Author history is preserved on 4e3decbb4e Author: Ismaël Mejía

Re: Issues and PR names and descriptions (or should we change the contribution guide)

In the past github squash and merge did not preserve the committer identity correctly, is it still the case? If so we should not be using it. https://github.com/isaacs/github/issues/1368 On Thu, Apr 22, 2021 at 8:41 PM Robert Bradshaw wrote: > > On Thu, Apr 22, 2021 at 11:29 AM Valentyn

Re: Issues and PR names and descriptions (or should we change the contribution guide)

On Thu, Apr 22, 2021 at 11:29 AM Valentyn Tymofieiev wrote: > I always squash-and-merge even when there is only 1 commit. This avoids > the necessity to edit the commit message to remove not so helpful "Merge > pull request xxx" message. Is there any harm to recommend squash by default > in the

Re: Issues and PR names and descriptions (or should we change the contribution guide)

I always squash-and-merge even when there is only 1 commit. This avoids the necessity to edit the commit message to remove not so helpful "Merge pull request xxx" message. Is there any harm to recommend squash by default in the upcoming squash bot even for single commit PRs? On Thu, Apr 22, 2021

Re: Issues and PR names and descriptions (or should we change the contribution guide)

On Thu, Apr 22, 2021 at 9:33 AM Kenneth Knowles wrote: > > On Thu, Apr 22, 2021 at 7:04 AM Alexey Romanenko > wrote: > >> Thanks Ismael for bringing this on the table again. Kind of my >> “favourite” topic, unfortunately, that I raised a couple of times… Let me >> share some of my thoughts on

Re: [ANNOUNCE] New committer: Yichi Zhang

Thanks everyone! It's my honor and I hope I can make more contributions in the future! On Thu, Apr 22, 2021 at 10:11 AM Yichi Zhang wrote: > Thanks, Brian! > > On Thu, Apr 22, 2021 at 9:11 AM Brian Hulette wrote: > >> Congratulations Yichi! >> >> On Thu, Apr 22, 2021 at 8:05 AM Robert Burke

Re: Issues and PR names and descriptions (or should we change the contribution guide)

On Thu, Apr 22, 2021 at 7:04 AM Alexey Romanenko wrote: > Thanks Ismael for bringing this on the table again. Kind of my “favourite” > topic, unfortunately, that I raised a couple of times… Let me share some of > my thoughts on this. > > First of all, as Beam developers, honestly we have to

Re: Issues and PR names and descriptions (or should we change the contribution guide)

It's hard to change existing behavior without a change in circumstances or tooling. I created https://github.com/apache/beam/pull/14619 , which will add its opinion about squash or merge as a comment once a PR has ben LGTM'd. The most important thing is not the advice it gives, but the fact that

Re: [VOTE] Release 2.29.0, release candidate #1

+1 (non-) Ran Python wordcount on Flink and Spark. On Wed, Apr 21, 2021 at 5:20 PM Brian Hulette wrote: > +1 (non-binding) > > I ran a python pipeline exercising the DataFrame API, and another > exercising SQLTransform in Python, both on Dataflow. > > On Wed, Apr 21, 2021 at 12:55 PM Kenneth

Re: [ANNOUNCE] New committer: Yichi Zhang

Congratulations Yichi! On Thu, Apr 22, 2021 at 8:05 AM Robert Burke wrote: > Congratulations Yichi! > > On Thu, Apr 22, 2021, 7:17 AM Alexey Romanenko > wrote: > >> Congratulations, well deserved! >> >> On 22 Apr 2021, at 10:03, Jan Lukavský wrote: >> >> Congrats Yichi! >> On 4/22/21 4:58 AM,

Re: [QUESTION] Dockerized Integration Tests with Java/Gradle

Welcome Evan! Note we do have some examples in Beam of running IO integration tests against testcontainers [1] that startup "fakes". We do this for Kafka [2], Kinesis [3], and there's a test that does this for both Kafka and Pubsub [4]. Is that the kind of thing you had in mind? It looks like

Re: [QUESTION] Dockerized Integration Tests with Java/Gradle

Hi Evan, Great to hear that you are going to contribute to Beam. Welcome! For integration tests we mostly use k8s. Did you take a look on current implementation of ITs for ElasticsearchIO (e.g. [1]) and how it runs on Jenkins [2]? Also, perhaps worth to mention our very good guide about

Re: [ANNOUNCE] New committer: Yichi Zhang

Congratulations Yichi! On Thu, Apr 22, 2021, 7:17 AM Alexey Romanenko wrote: > Congratulations, well deserved! > > On 22 Apr 2021, at 10:03, Jan Lukavský wrote: > > Congrats Yichi! > On 4/22/21 4:58 AM, Ahmet Altay wrote: > > Congratulations Yichi!  > > On Wed, Apr 21, 2021 at 6:48 PM

Re: [PROPOSAL] Upgrade Cassandra driver from 3.x to 4.x in CassandraIO

Thanks, it looks promising! I just have a couple things to ask. 1) Could you briefly summarise and add here or/and to design doc all breaking changes for users that you expect (if any)? Can we avoid them, at least, maybe temporary? For example, we used to deprecate an old public API and keep

[QUESTION] Dockerized Integration Tests with Java/Gradle

Hi folks! I'm Evan, and I'm fairly new to developing the Beam SDK. I've been a user for a number of years and have done some private SDK customizations along the way for my day job, but have recently been given the green light to contribute back to the OSS repo  In particular, I've worked with

Re: Issues and PR names and descriptions (or should we change the contribution guide)

Thanks Ismael for bringing this on the table again. Kind of my “favourite” topic, unfortunately, that I raised a couple of times… Let me share some of my thoughts on this. First of all, as Beam developers, honestly we have to agree if we care about our commits history or not. If not (or not so

Re: [ANNOUNCE] New committer: Yichi Zhang

Congratulations, well deserved! > On 22 Apr 2021, at 10:03, Jan Lukavský wrote: > > Congrats Yichi! > > On 4/22/21 4:58 AM, Ahmet Altay wrote: >> Congratulations Yichi!  >> >> On Wed, Apr 21, 2021 at 6:48 PM Chamikara Jayalath > > wrote: >> Congrats Yichi! >>

Should WindowFn have a mininal Duration?

Hi, I have come across a "problem" while implementing some toy Pipeline. I would like to split input PCollection into two parts - droppable data (delayed for more than allowed lateness from the end of the window) from the rest. I will not go into details, as that is not relevant, the problem

Re: [ANNOUNCE] New committer: Yichi Zhang

Congrats Yichi! On 4/22/21 4:58 AM, Ahmet Altay wrote: Congratulations Yichi!  On Wed, Apr 21, 2021 at 6:48 PM Chamikara Jayalath mailto:chamik...@google.com>> wrote: Congrats Yichi! On Wed, Apr 21, 2021 at 6:14 PM Heejong Lee mailto:heej...@google.com>> wrote: