Re: Usability regression using SDF Unbounded Source wrapper + DirectRunner

Hi Steve, We have one wrapper optimization[1] merged in and it will be released with 2.27.0. Would you like to verify whether it helps improve the performance on DirectRunner? [1] https://github.com/apache/beam/pull/13592 On Mon, Dec 28, 2020 at 12:17 PM Boyuan Zhang wrote: > Hi Antonio, > >

Re: Usability regression using SDF Unbounded Source wrapper + DirectRunner

Hi Antonio, Thanks for the data! I want to elaborate more on where the overhead could come from when on Flink. - with --experiments=use_deprecated_read --fasterrCopy=true, I am able to > achieve 13K TPS This execution uses UnboundedSource path, where the checkpoint frequency for source

Re: Need help with the Go Sdk

Looked at the PR, and added another comment: the gogradle.lock file was deleted, which would definitely cause the errors in the original post. On Mon, Dec 28, 2020, 12:04 PM Robert Burke wrote: > > It seems this is related to the specific PR rather than the usual cause > for that error. If one

Re: Need help with the Go Sdk

It seems this is related to the specific PR rather than the usual cause for that error. If one runs into similar errors, it's usually faster to get a resolution if you include the PR where it's happening since errors are rarely context free. Otherwise, incase someone finds this thread looking for

Re: Need help with the Go Sdk

Commented on the PR (https://github.com/apache/beam/pull/12938). On Mon, Dec 28, 2020 at 11:41 AM Sonam Ramchand < sonam.ramch...@venturedive.com> wrote: > Hi Devs, > For ':sdks:go:resolveBuildDependencies' task, I have been getting: > > Exception in resolution, message is: > Cannot resolve

Need help with the Go Sdk

Hi Devs, For ':sdks:go:resolveBuildDependencies' task, I have been getting: Exception in resolution, message is: Cannot resolve dependency:github.com/ajstarks/deck: commit='LATEST_COMMIT', urls=[https://github.com/ajstarks/deck.git, g...@github.com:ajstarks/deck.git] Resolution stack is: +-

Re: [VOTE] Release 2.27.0, release candidate #1

Thank you Kyle and Ismael! In that case, I'll move forward to build RC2. Best -P. On Mon, Dec 28, 2020 at 10:53 AM Kyle Weaver wrote: > > I am not sure what's the status of this work - should this block the > release until we can get those containers out? > > We can release Flink 1.11 and Flink

Re: [VOTE] Release 2.27.0, release candidate #1

> I am not sure what's the status of this work - should this block the release until we can get those containers out? We can release Flink 1.11 and Flink 1.12 containers and I'm *pretty sure* they would work correctly, but to be totally sure we'd have to upgrade our portable Flink tests to cover

Re: [VOTE] Release 2.27.0, release candidate #1

Good catch Ismael. Thanks! I've created https://issues.apache.org/jira/browse/INFRA-21238 to request the repositories to be created. I am not sure what's the status of this work - should this block the release until we can get those containers out? cc: @Kyle Weaver Best -P. On Mon, Dec 28,

Re: [VOTE] Release 2.26.0, release candidate #1

That was due to having the image part of the release changes for 2.27.0 in a bit too early (and partially complete at the time). My understanding is the separate java8 and java11 containers weren't to be available until 2.27.0, so 2.26.0 is the last release with the single beam_java_sdk build. On

Re: [VOTE] Release 2.27.0, release candidate #1

It seems we are not publishing the latest versions of the Flink Job Server (Flink 1.11 and 1.12) docker images These do not exist: https://hub.docker.com/r/apache/beam_flink1.11_job_server https://hub.docker.com/r/apache/beam_flink1.12_job_server but this does and has the good 2.27.0_rc1 tag:

Re: [VOTE] Release 2.26.0, release candidate #1

It seems the tag of the docker image for java8 was not updated after the release went out, can somebody please fix this. https://hub.docker.com/r/apache/beam_java8_sdk/tags?page=1=last_updated On Sat, Dec 12, 2020 at 7:19 AM Jean-Baptiste Onofre wrote: > +1 (binding) > > Sorry for the delay.

ElasticsearchIO - Read multiple PubSub topics and write to different Indices

Hi all, I am writing to Elasticsearch using Beam (Google Dataflow). The pipeline is ingesting data from PubSub subscription and writing them to Elasticsearch index (using this Dataflow Template ).

Beam Dependency Check Report (2020-12-28)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue chromedriver-binary 87.0.4280.88.0