Re: [VOTE] Release 2.52.0, release candidate #3

As mentioned in another thread [1], there is a recently detected vulnerability in pyarrow [2]. It appears to be a concern for Beam users that we can mitigate in the upcoming release. We can reassess early next week in case there is a revised assessment for severity for this vulnerability. In the

Re: [VOTE] Release 2.52.0, release candidate #3

+1 (binding). Tested multi-lang Java/Python jobs. Thanks, Cham On Fri, Nov 10, 2023, 12:28 PM Svetak Sundhar via dev wrote: > +1 Non Binding -- tested Python SDK batch. > > > Svetak Sundhar > > Data Engineer > s vetaksund...@google.com > > > > On Fri, Nov 10, 2023 at 2:58 PM Danny McCormick

Re: [VOTE] Release 2.52.0, release candidate #3

+1 Non Binding -- tested Python SDK batch. Svetak Sundhar Data Engineer s vetaksund...@google.com On Fri, Nov 10, 2023 at 2:58 PM Danny McCormick via dev wrote: > > Note: the release guide >

Re: [VOTE] Release 2.52.0, release candidate #3

> Note: the release guide and blog post say the RC image has a tag

Re: [Python SDK] PyArrow Critical Vulnerability

>From https://pypi.org/project/pyarrow-hotfix/ : pyarrow_hotfix must be imported in your application or library code for it to take effect. Just installing the package is not sufficient: For Beam users, that means that the pipeline code running on the workers would need to import this module on

Re: [VOTE] Release 2.52.0, release candidate #3

+1 (non-binding) Tested the Go SDK on Dataflow with own use cases. Note: the release guide and blog post

Re: Adding Dead Letter Queues to Beam IOs

Thanks - the general ideas seem solid, I added some questions/comments as well. On Fri, Nov 10, 2023 at 1:32 PM Robert Bradshaw via dev wrote: > Thanks. I added some comments to the doc and open PR. > > On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev > wrote: > > > > Hi All, > > > > I've

Re: Adding Dead Letter Queues to Beam IOs

Thanks. I added some comments to the doc and open PR. On Wed, Nov 8, 2023 at 12:44 PM John Casey via dev wrote: > > Hi All, > > I've written up a design for adding DLQs to existing Beam IOs. It's been > through a round of reviews with some Dataflow folks at Google, but I'd > appreciate any

Re: Upgrading Avro dependencies

I guess I'm a bit confused as to why specifically generateTestAvroJava seems to use the wrong version. I see our version specific generated code, but this action appears to be inherited from the plugin, and is configured with whichever avro version is provided. Given that I tried to just change to

Re: [Python SDK] PyArrow Critical Vulnerability

Hi Piotr, thanks for bringing this to the list. There is a FR to support pyarrow https://github.com/apache/beam/issues/28410 . I looked into it briefly in https://github.com/apache/beam/pull/28437 but saw some test failures and it has been on back burner. Given the news about vulnerability it

Re: Upgrading Avro dependencies

Hi John, This old Avro version in Beam is a very long story. Briefly, since initially it was toughly integrated into Java SDK “core” module then it was not possible to upgrade an Avro version without breaking changes for users (because of some Avro incompatible changes, as you have noticed

Upgrading Avro dependencies

Hi All, There was a CVE detected in Avro 1.8.2 (CVE-2023-39410), so I'm trying to upgrade to avro 1.11.3. Unfortunately, it seems that our auto-generated Avro test classes aren't being generated properly with this new version. I've updated our avro generation plugin as well, but for whatever

Re: [VOTE] Release 2.52.0, release candidate #3

+1 (binding) Artifacts and signatures look good, validated one of the Python wheels in a fresh install. On Fri, Nov 10, 2023 at 7:23 AM Alexey Romanenko wrote: > > +1 (binding) > > Java SDK with Spark runner > > — > Alexey > > On 9 Nov 2023, at 16:44, Ritesh Ghorse via dev wrote: > > +1

Re: [VOTE] Release 2.52.0, release candidate #3

+1 (binding) Java SDK with Spark runner — Alexey > On 9 Nov 2023, at 16:44, Ritesh Ghorse via dev wrote: > > +1 (non-binding) > > Validated Python SDK quickstart batch and streaming. > > Thanks! > > On Thu, Nov 9, 2023 at 9:25 AM Jan Lukavský > wrote: >> +1

Beam High Priority Issue Report (46)

This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/29386 [Failing Test]: