Re: Ordered PCollections eventually?

Awesome, thanks Pablo! On Mon, May 10, 2021 at 4:05 PM Pablo Estrada wrote: > CDC would also benefit. I am working on a proposal for this that is > concerned with streaming pipelines, and per-key ordered delivery. I will > share with you as soon as I have a draft. > Best > -P. > > On Mon, May

Re: Transform-specific thread pools in Python

On Mon, May 10, 2021 at 8:01 AM Stephan Hoyer wrote: > Hi Beam devs, > > I've been exploring recently how to optimize IO bound steps for my Python > Beam pipelines, and have come up with a solution that I think might make > sense to upstream into Beam's Python SDK. > > It appears that Beam

Re: Ordered PCollections eventually?

CDC would also benefit. I am working on a proposal for this that is concerned with streaming pipelines, and per-key ordered delivery. I will share with you as soon as I have a draft. Best -P. On Mon, May 10, 2021 at 2:56 PM Reuven Lax wrote: > There has been talk, but nothing concrete. > > On

Flaky test issue report

This is your daily summary of Beam's current flaky tests. These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. BEAM-12311: Python PostCommit are close to timeout

P1 issues report

This is your daily summary of Beam's current P1 issues, not including flaky tests. See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. BEAM-12321: Failure in test_run_packable_combine_per_key and

Re: P0 (outage) report

This is very important, but is not an "outage" per https://beam.apache.org/contribute/jira-priorities/ I have adjusted the Jira. This is P1 and release blocker. Release blockers are indicated by setting Fix Version field to an unreleased version. Kenn On Mon, May 10, 2021 at 10:58 AM Beam Jira

Ordered PCollections eventually?

Hi All, I was wondering if there had been any plans for creating ordered PCollections in the Beam model? Or if there might be plans for them in the future? I know that Beam SQL and Beam DataFrames would directly benefit from an ordered PCollection. Regards, Sam

Re: [DISCUSS] Warn when KafkaIO is used as a bounded source

Just added more details on BEAM-6466 . In short, BEAM-6466 looks more like a FR instead of a bug to me. On Fri, Apr 30, 2021 at 12:48 PM Pablo Estrada wrote: > I suppose a production-ready

Re: Flake trends - better?

Any suggestions on how to clean this up? We can organize a cleanup to reduce the numbers a bit. Ideally we need to find a way to prevent the future growth but a temporary reduction might make it easier for us to keep reviewing and closing new issues. On Mon, May 10, 2021 at 9:11 AM Brian Hulette

Re: Extremely Slow DirectRunner

Hi Evan, What do you mean startup delay? Is it the time that from you start the pipeline to the time that you notice the first output record from PubSub? On Sat, May 8, 2021 at 12:50 AM Ismaël Mejía wrote: > Can you try running direct runner with the option >

P0 (outage) report

This is your daily summary of Beam's current outages. See https://beam.apache.org/contribute/jira-priorities/#p0-outage for the meaning and expectations around P0 issues. BEAM-12316: LGPL in bundled dependencies (https://issues.apache.org/jira/browse/BEAM-12316)

Re: Flake trends - better?

In addition to stale flake jiras, I think there are also many tracking tests that were disabled years ago due to flakiness. On Sat, May 8, 2021 at 1:39 PM Kenneth Knowles wrote: > Oh the second chart is not automatically associated with the board/filter. > Here is the correct link: >

Re: [PROPOSAL] Vendored bytebuddy dependency release

If nothing breaks, and we check perf, then absolutely this seems good. Kenn On Mon, May 10, 2021 at 12:38 AM Ismaël Mejía wrote: > Most issues on the previous migration were related to changes on behavior > of class-loading on Java 11. It seems Oracle is taking a more backwards > compatible on

Re: LGPL-2.1 in beam-vendor-grpc

+1 fully agree. Regards JB > Le 10 mai 2021 à 16:02, Jan Lukavský a écrit : > > +1 for blocking the release - I think we should not release something about > which we _know_ that it might be legally problematic. And we should > definitely create a check in the build process that will warn

Transform-specific thread pools in Python

Hi Beam devs, I've been exploring recently how to optimize IO bound steps for my Python Beam pipelines, and have come up with a solution that I think might make sense to upstream into Beam's Python SDK. It appears that Beam runners (at least the Cloud Dataflow runner) typically use only a single

Re: LGPL-2.1 in beam-vendor-grpc

+1 to bock a release because of this. Agree with Jan’s arguments. — Alexey > On 10 May 2021, at 16:02, Jan Lukavský wrote: > > +1 for blocking the release - I think we should not release something about > which we _know_ that it might be legally problematic. And we should > definitely create

Re: LGPL-2.1 in beam-vendor-grpc

+1 for blocking the release - I think we should not release something about which we _know_ that it might be legally problematic. And we should definitely create a check in the build process that will warn about such issues in the future.  Jan On 5/10/21 3:44 PM, Ismaël Mejía wrote: Tomo

Re: LGPL-2.1 in beam-vendor-grpc

Tomo just confirmed in the ticket that if we update the gRPC vendored version we won't need the JBoss dependency anymore so we should be good to go with the upgrade. The open question is if this should be blocking for the upcoming Beam 2.31.0 release or we can fix it afterwards. On Mon, May 10,

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

I was investigating the strange timeout ( https://github.com/apache/beam/pull/14474) but was occupied with something else lately. Let me try the new version today to see any improvements. On Mon, May 10, 2021 at 4:57 AM Ismaël Mejía wrote: > I just saw that gRPC 1.37.1 is out now (and with

Re: LGPL-2.1 in beam-vendor-grpc

We have been discussing about updating the vendored dependency in BEAM-11227 , if I remember correctly the newer version of gRPC does not require the jboss dependency, so probably is the best upgrade path, can you confirm Tomo Suzuki

Beam Dependency Check Report (2021-05-08)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue chromedriver-binary 88.0.4324.96.0

Re: LGPL-2.1 in beam-vendor-grpc

Also we have very similar discussion about it in https://issues.apache.org/jira/browse/LEGAL-572 Just to be clear about the context of it, it's not a legal requirement of Apache Licence, it's Apache Software Foundation policy, that we should not limit our users in using our software. If the LGPL

Re: LGPL-2.1 in beam-vendor-grpc

Hi You can take a look on https://www.apache.org/legal/resolved.html Regards JB > Le 10 mai 2021 à 12:56, Elliotte Rusty Harold a écrit : > > Anyone have a link to the official Apache policy about this? Thanks. > >> On Mon, May 10, 2021 at 10:07 AM Jan Lukavský wrote: >> >> Hi, >> >>

Re: LGPL-2.1 in beam-vendor-grpc

Yeah LGPL is cat X license. So it should not be embedded and distributed. Regards JB > Le 10 mai 2021 à 12:07, Jan Lukavský a écrit : > > Hi, > > we are bundling dependencies with LGPL-2.1, according to license header in > META-INF/maven/org.jboss.modules/jboss-modules/pom.xml. I think is

LGPL-2.1 in beam-vendor-grpc

Hi, we are bundling dependencies with LGPL-2.1, according to license header in META-INF/maven/org.jboss.modules/jboss-modules/pom.xml. I think is might be an issue, already reported here: [1]. I created [2] to track it on our side.  Jan [1] https://issues.apache.org/jira/browse/FLINK-22555

Re: Upgrading vendored gRPC from 1.26.0 to 1.36.0

I just saw that gRPC 1.37.1 is out now (and with aarch64 support for python!) that made me wonder about this, what is the current status of upgrading the vendored dependency Tomo? On Thu, Apr 8, 2021 at 4:16 PM Tomo Suzuki wrote: > We observed the cron job of Java Precommit for the master

Re: [PROPOSAL] Vendored bytebuddy dependency release

Most issues on the previous migration were related to changes on behavior of class-loading on Java 11. It seems Oracle is taking a more backwards compatible on latest releases, so let's hope everything will go well. In the meantime I tested the upgrade locally and tests are passing ok so we should