Failing Python builds & AppEngine application

Hey, A strange thing happened a few hours ago. All python builds (e.g. [1]) started failing because of: google.api_core.exceptions.NotFound: 404 The project apache-beam-testing does not exist or it does not contain an active Cloud Datastore or Cloud Firestore database. Please visit http://console

Re: Failing Python builds & AppEngine application

It was me! I disabled the App in the premise that it only hosted the old perf graphs that were replaced with Grafana. Thanks for fixing the issue. Is there anything else on the app, or is there more migration to Grafana required, or cleanup unfinished? On Thu, Aug 6, 2020, 4:30 AM Damian Gadomsk

Re: Monitoring performance for releases

Robert, this is not too far off what I'm proposing. We can always create JIRA issues for performance regressions and mark them with a Fix Version. Especially, the time of the release is a good time to re-evaluate whether some gross performance regressions can be detected. Of course, if it's a m

MQTT python IO connector

Hi all, I'm looking to use MQTT as the messaging broker in my streaming python pipeline - is there any current efforts with this currently happening? Would be very interested in knowing this before I attempt a custom io connector.

Re: Use Coder message for cross-lang ExternalConfigurationPayload?

The PR for this is up now: https://github.com/apache/beam/pull/12481 Any volunteers to help review? We may want a separate reviewer for Python and Java changes. Brian On Wed, Aug 5, 2020 at 9:00 AM Brian Hulette wrote: > What I'm working on changes ExternalConfigurationPayload [1] to this: > >

Status of dynamic worker scaling with Kafka consumers

Hi Folks When processing events from Kafka, it seems that, from my reading, the distribution of partitions maps directly to the worker via the concept of 'splits' : https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java#L

Re: Use Coder message for cross-lang ExternalConfigurationPayload?

I can help review the Java change. -Rui On Thu, Aug 6, 2020 at 9:53 AM Brian Hulette wrote: > The PR for this is up now: https://github.com/apache/beam/pull/12481 > Any volunteers to help review? We may want a separate reviewer for Python > and Java changes. > > Brian > > On Wed, Aug 5, 2020 a

Re: MQTT python IO connector

There currently isn't one in Python but there is one in Java. Consider contributing a cross-lang connector to MQTT. There are several other dev threads about doing this for other IOs and examples you can look at like Kafka. On Thu, Aug 6, 2020 at 9:43 AM Zeolla, Nicholas, Vodafone Group < nicholas

Re: BigQuery writer error: Clustering fields can only be set when TimePartitioning is set

Thanks, I filed https://issues.apache.org/jira/browse/BEAM-10652 for this. I imagine this would be a pretty straight-forward PR (just remove the check at [1]?). The only tricky part would be making sure we test it adequately. [1] https://github.com/apache/beam/blob/5e0e798ddd827fd212ac89b8c6f6f2cf

Beam Summit Status Report - 8/6

Hi folks, I wanted to provide you with the Beam Summit Status report from today's meeting. If you would like to join the next public meeting on Wednesday, August 19th at 11:30 AM PST please let me know and I will send a calendar invite over to you! Also don't forget to register for the Summit

Re: Broken links in code velocity dashboard

Damian, or anyone else, do you know if there were other changes to the dashboard? I started to see closed PRs in the currently open PRs list (e.g. https://github.com/apache/beam/pull/12349, https://github.com/apache/beam/pull/12374). Not sure what is causing it, but it seems like a new issue. On

Re: NanosInstant not being recognised by BigQueryIO.Write

Created https://issues.apache.org/jira/browse/BEAM-10655 Please add any additional information there. If you hope to provide a fix that's welcome too. Also, if you need to get the JIRA assigned to you, you can request JIRA access through the Beam dev list (you don't need to be a committer for this

Re: [PROPOSAL] Preparing for Beam 2.24.0 release

+1 - Thank you Daniel!! On Wed, Jul 29, 2020 at 4:30 PM Daniel Oliveira wrote: > > You probably meant 2.24.0. > > Thanks, yes I did. Mark "Fix Version/s" as "2.24.0" everyone. :) > > On Wed, Jul 29, 2020 at 4:14 PM Valentyn Tymofieiev > wrote: > >> +1, Thanks Daniel! >> >> On Wed, Jul 29, 2020

Memory Issue When Running Beam On Flink

Hi, We recently started using Apache Beam version 2.20.0 running on Flink version 1.9 deployed on kubernetes to process unbounded streams of data. However, we noticed that the memory consumed by stateful Beam is steadily increasing over time with no drops no matter what the current bandwidth is. W

Re: [PROPOSAL] Preparing for Beam 2.24.0 release

Awesome! -Rui On Thu, Aug 6, 2020 at 4:14 PM Ahmet Altay wrote: > +1 - Thank you Daniel!! > > On Wed, Jul 29, 2020 at 4:30 PM Daniel Oliveira > wrote: > >> > You probably meant 2.24.0. >> >> Thanks, yes I did. Mark "Fix Version/s" as "2.24.0" everyone. :) >> >> On Wed, Jul 29, 2020 at 4:14 PM

Re: Stateful Pardo Question

On Tue, Aug 4, 2020 at 1:08 PM jmac...@godaddy.com wrote: > So, after some additional digging, it appears that Beam does not > consistently check for timer expiry before calling process. The result is > that it may be the case that the watermark has moved beyond your timer > expiry, and if youre