[VOTE] Release 2.58.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #2 for the version 2.58.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Re: [VOTE] Release 2.58.0, release candidate #1

framework. > > Filed bug: https://github.com/apache/beam/issues/31970 > > I am working on a fix right now. Requesting to release rc3 with either a > revert of affected commit or include my upcoming fix. > > Thanks, > - Jeff > > On Wed, Jul 17, 2024 at 5:35 PM Jack Mc

Re: [VOTE] Release 2.58.0, release candidate #1

runner and runner v2): > > https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/main/it/google-cloud-platform > > Thanks, > > Yi > > On Wed, Jul 17, 2024 at 5:35 PM Jack McCluskey via dev < > dev@beam.apache.org> wrote: > >> Hi everyone, >>

[VOTE] Release 2.58.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.58.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Beam 2.58.0 Release Branch Cut

Hey everyone, I just cut the 2.58.0 release branch ( https://github.com/apache/beam/tree/release-2.58.0) and will be working on getting the release branch stabilized. There are currently two release blockers on the release milestone ( https://github.com/apache/beam/milestone/22) with fixes

Beam 2.58.0 Release

Hey everyone, The next release (2.58.0) branch cut is scheduled for July 10, 2024, 2 weeks from today, according to the release calendar [1]. I'd like to perform this release; I will cut the branch on that date, and cherrypick release-blocking fixes afterwards, if any. Please help with the

Re: Image Embedding Support for MLTransform

It would help if I actually included the link to the new doc! https://docs.google.com/document/d/1n0ZyXcRjxqIb5f-wZcMBXdrkgjEfZHYu-zf8bpzE0dA/edit?usp=sharing On Wed, May 29, 2024 at 10:26 AM Jack McCluskey wrote: > Hey everyone, > > I've put together a very small overview doc on what steps

Image Embedding Support for MLTransform

Hey everyone, I've put together a very small overview doc on what steps need to be taken to add support for generating image embeddings to the Python SDK's MLTransform suite. It's mostly organized as a reference for me while I get familiar with the framework and is relatively general, but it may

Re: [RESULT] [VOTE] Release 2.53.0, release candidate #2

elsewhere). >> >> On Thu, Jan 4, 2024, 11:23 AM Jack McCluskey via dev >> wrote: >> >>> Hey everyone, >>> >>> Following up on this, I do need help from a PMC member for the PMC-only >>> finalization steps ( >>> https

Re: [RESULT] [VOTE] Release 2.53.0, release candidate #2

Hey everyone, Following up on this, I do need help from a PMC member for the PMC-only finalization steps ( https://github.com/apache/beam/blob/master/contributor-docs/release-guide.md#pmc-only-finalization ) Thanks, Jack McCluskey On Thu, Jan 4, 2024 at 10:27 AM Jack McCluskey wrote: > Hey

[RESULT] [VOTE] Release 2.53.0, release candidate #2

Hey everyone, I'm happy to announce that we have unanimously approved this release. There are nine approving votes, three of which are binding: * Jan Lukavský (binding) * Chamikara Jayalath (binding) * Robert Burke (binding) * XQ Hu * Danny McCormick * Bruno Volpato * Svetak Sundhar * Yi Hu *

Re: [VOTE] Release 2.53.0, release candidate #2

;>> Jan >>>> On 12/27/23 14:13, Danny McCormick via dev wrote: >>>> >>>> +1 (non-binding) >>>> >>>> Tested with some example ML notebooks. >>>> >>>> Thanks, >>>> Danny >>>> >

[VOTE] Release 2.53.0, release candidate #2

Happy holidays everyone, Please review and vote on the release candidate #2 for the version 2.53.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and

Re: [VOTE] Release 2.53.0, release candidate #1

;> >> On Thu, Dec 21, 2023 at 4:30 PM Danny McCormick via dev < >> dev@beam.apache.org> wrote: >> >>> +1 (non-binding) >>> >>> Tested against a few Beam ML notebooks using the direct runner and >>> Dataflow runner. >>

[VOTE] Release 2.53.0, release candidate #1

Happy holidays everyone, Please review and vote on the release candidate #1 for the version 2.53.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and

Re: Beam 2.53.0 Release Branch Cut

Hey everyone, Just wanted to give an update on the release. There's one potentially blocking issue left (https://github.com/apache/beam/issues/29818) that I'd like to get resolved, or at the very least, understood as a non-issue, before building RC1. Thanks, Jack McCluskey On Wed, Dec 13, 2023

Beam 2.53.0 Release Branch Cut

Hey everyone, I just cut the 2.53.0 release branch ( https://github.com/apache/beam/tree/release-2.53.0) and will be working on getting the release branch stabilized. There are two release blockers on the release milestone ( https://github.com/apache/beam/milestone/17) 1. A transform upgrade

State-Aware Element Batching in Python

Hey everyone, This is long-overdue, but I wanted to share a doc ( https://docs.google.com/document/d/1Rin_5Vm3qT1Mkb5PcHgTDrjXc3j0Admzi3fEGEHB2-4/edit?usp=sharing) that outlines a new implementation of BatchElements in the Python SDK that is state-aware, and therefore allows for batching elements

Beam 2.53.0 Release

Hey everyone, the next release (2.53.0) branch cut is scheduled for Dec 13, 2023, 2 weeks from today, according to the release calendar [1]. I'd like to perform this release; I will cut the branch on that date, and cherrypick release-blocking fixes afterwards, if any. Please help with the

The Current State of Beam Python Type Hinting

Hey everyone, I put together a small doc explaining how Beam Python type hinting works + where the module needs to go in the future with changes to Python itself. This is over at https://s.apache.org/beam-python-type-hinting-overview and I'll be putting it into a few places for discoverability as

Re: [Discuss] Idea to increase RC voting participation

I'm +1 on helping explain what we mean by "validate the RC" since we're really just asking users to see if their existing use cases work along with our typical slate of tests. I don't know if offloading that work to our active validators is the right approach though, documentation/screen share of

Re: [ANNOUNCE] New PMC Member: Valentyn Tymofieiev

Congrats Valentyn! On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming Valentyn > Tymofieiev as our newest PMC member. > > Valentyn has been contributing to Beam since 2017. Notable highlights > include his work on the

Re: [ANNOUNCE] New PMC Member: Robert Burke

Congrats Robert! On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming Robert Burke < > lostl...@apache.org> as our newest PMC member. > > Robert has been a part of the Beam community since 2017. He is our > resident Gopher,

Re: [ANNOUNCE] New PMC Member: Alex Van Boxel

Congrats, Alex! On Tue, Oct 3, 2023 at 2:49 PM XQ Hu via dev wrote: > Configurations, Alex! > > On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming Alex Van Boxel < >> alexvanbo...@apache.org> as our newest PMC

[RFC] Bootloader Buffered Logging

Hey everyone, I've written a small design doc around implementing some buffered logging for the Beam boot.go scripts that is available at https://s.apache.org/beam-buffered-logging. This should help surface errors that occur during worker set-up (like issues with dependency installation) that

Re: Go SDK Review Requested | README Rewrite

I've made a pass on it, also looping in Robert Burke for another set of eyes on it as well. On Sun, Jul 9, 2023 at 3:20 PM Damon Douglas wrote: > Good day, everyone, > > PR 27415 is a suggested > rewrite of the Go SDK README. Could someone review it?

Re: [DISCUSS] Enable Github Discussions?

Also going to be -1 on this one, I'm not sure we pick anything up from adding a forum apart from adding another place that needs to be checked. On Tue, Jul 4, 2023 at 4:03 AM Jan Lukavský wrote: > -1 > > Totally agree with Byron and Alexey. > > Jan > On 7/3/23 21:18, Byron Ellis via dev wrote:

Re: [Proposal] Remove of Fix the Beam Dependency Check Report Job

terms of actionability vs spam volume. > > On Tue, May 30, 2023 at 7:24 AM Jack McCluskey via dev < > dev@beam.apache.org> wrote: > >> Hi everyone, >> >> Just bumping this again now that the long weekend is behind us. If no one >> advocates for fixing the job in t

Re: [RESULT] [VOTE] Release 2.48.0, release candidate #2

Completely forgot to circle back to this, these should published now On Wed, May 31, 2023 at 11:10 AM Jack McCluskey wrote: > I'll get that going shortly. > > On Wed, May 31, 2023 at 11:00 AM Ritesh Ghorse via dev < > dev@beam.apache.org> wrote: > >> Could someone from the `beammaintainers

Re: [RESULT] [VOTE] Release 2.48.0, release candidate #2

I'll get that going shortly. On Wed, May 31, 2023 at 11:00 AM Ritesh Ghorse via dev wrote: > Could someone from the `beammaintainers DockerHub team` please help deploy > the docker images for release [1] > > Thanks! > > [1] >

Client-Side Throttling in Apache Beam

Hey everyone, While working on some remote model handler code I hit a point where I needed to understand how Beam IOs interpret and action on being throttled by an external service. This turned into a few discussions and then a small write-up doc (

Re: [Proposal] Remove of Fix the Beam Dependency Check Report Job

Hi everyone, Just bumping this again now that the long weekend is behind us. If no one advocates for fixing the job in the next few days I'll assume a lazy consensus and remove it. I also want to point out a typo in the subject, it should be "Remove *or* Fix." Thanks, Jack McCluskey On Thu,

[Proposal] Remove of Fix the Beam Dependency Check Report Job

Hey everyone, The Beam Dependency Check Report email (like https://lists.apache.org/thread/tc9v1d66rx77wzvrjnkcf0jo3rxtmrhn) has not had a successful incarnation since July 21st, 2022. I've done a little bit of digging into the problem and have found that the issue lies in a query

Repository Deletion Postmortem

Hey everyone, Now that 2.47.0 is out, I've got a short write-up on how the repository was very briefly deleted back when I was cutting the release branch at https://s.apache.org/beam-repo-deletion-postmortem. This includes the rough order of operations, the recovery, and then the action items

[RESULT] [VOTE] Release 2.47.0, release candidate #3

I'm happy to announce that we have unanimously approved this release. There are nine approving votes, four of which are binding: * Ahmet Altay * Jean-Baptiste Onofré * Chamikara Jayalath * Jan Lukavský Thank you to the following non-binding voters for helping with validation: * Anand Inguva *

Re: [VOTE] Release 2.47.0, release candidate #3

sing the RC. >>>> Created https://github.com/apache/beam/issues/26576. >>>> >>>> Thanks, >>>> Cham >>>> >>>> On Fri, May 5, 2023 at 11:11 PM Austin Bennett >>>> wrote: >>>> >>>>> +1 ( non-bi

[VOTE] Release 2.47.0, release candidate #3

Hi everyone, Please review and vote on the release candidate #3 for the version 2.47.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Re: [VOTE] Release 2.47.0, release candidate #2

) >> >> On Wed, May 3, 2023 at 2:01 PM Jack McCluskey via dev < >> dev@beam.apache.org> wrote: >> >>> I have just now realized I didn't replace the release version or RC >>> number from the template's first line in either vote email. Oops. To be >&g

Re: [PROPOSAL] Preparing for 2.48.0 Release

I'd suggest shooting for 2.48.0 so we're ahead of the end-of-support date. We're also supporting 5 different Python versions in 2.47.0, it's probably for the best to try and pare that down. On Thu, May 4, 2023 at 11:25 AM Anand Inguva via dev wrote: > Thanks Ritesh!! > > Python 3.7 support is

Re: [Proposal] Automate Release Signing

+1 to automating release signing. As it stands now, this step requires a PMC member to add a new release manager's GPG key which can add time to getting a release started. This also results in the public key used to sign each release changing from one version to the next, as different release

Re: [VOTE] Release 2.47.0, release candidate #2

I have just now realized I didn't replace the release version or RC number from the template's first line in either vote email. Oops. To be clear this is 2.47.0 RC2 On Wed, May 3, 2023 at 1:58 PM Jack McCluskey wrote: > Hi everyone, > > Please review and vote on the release candidate #3 for the

[VOTE] Release 2.47.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #3 for the version 1.2.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Re: [VOTE] Release 2.47.0, release candidate #1

lend/beam-samples/ > (Java SDK v8/v11/v17, Spark 3.x runner). > > --- > Alexey > > On 28 Apr 2023, at 16:06, Jack McCluskey via dev > wrote: > > There was a GCP outage that impacted pushing containers to GCR, I expected > it to impact Java containers specifically but

Re: [VOTE] Release 2.47.0, release candidate #1

and signatures all look good, and I validated a couple of >> Python pipelines in a fresh install. >> >> Assuming all the tests (including the Dataflow ones) pass (modulo the two >> mentioned above; seems a fair justification to not block on those) I'm +1 >> (binding) on this r

Re: [VOTE] Release 2.47.0, release candidate #1

3:23 PM Valentyn Tymofieiev > wrote: > >> Thanks, Jack! >> >> re [12]: >> >> I am seeing some test errors - have they been investigated? >> Also, did all test suites run? I think I am not seeing output of some of >> the suites, like >> >>

Re: [VOTE] Release 2.47.0, release candidate #1

k! > > re [12]: > > I am seeing some test errors - have they been investigated? > Also, did all test suites run? I think I am not seeing output of some of > the suites, like > > Run Python Dataflow V2 ValidatesRunner > > > > On Wed, Apr 26, 2023 at 9:14 PM Jack

[VOTE] Release 2.47.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #3 for the version 1.2.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Re: [ANNOUNCE] New committer: Anand Inguva

Congratulations, Anand! On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Congratulations! > > On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský wrote: > >> Congrats Anand! >> On 4/21/23 20:05, Robert Burke wrote: >> >> Congratulations Anand! >> >> On Fri,

Re: [PROPOSAL] Preparing for 2.47.0 Release

d, Apr 12, 2023 at 8:10 PM Ahmet Altay wrote: >> >>> Jack, how is the release coming along? >>> >>> On Tue, Apr 4, 2023 at 12:23 PM Jack McCluskey via dev < >>> dev@beam.apache.org> wrote: >>> >>>> Hey everyone, >>>> >>>

Re: Beam Release DockerHub Group

+1 to simplifying the infra side, especially with an aim towards automating the processes we can. The more we can streamline and simplify the better. On Mon, Apr 17, 2023 at 11:18 AM Danny McCormick via dev < dev@beam.apache.org> wrote: > Hey everyone, in an effort to reduce the burden of

Re: [PROPOSAL] Preparing for 2.47.0 Release

We're making good progress on finding and fixing bugs. Not quite to building an RC candidate yet, but so far nothing that seems to be a difficult fix. On Wed, Apr 12, 2023 at 8:10 PM Ahmet Altay wrote: > Jack, how is the release coming along? > > On Tue, Apr 4, 2023 at 12:23 PM Jack

Re: Jenkins Flakes

Is there a technical reason we can't migrate Java code coverage over to the Codecov tool/Actions like we have with Go and Python? On Tue, Apr 11, 2023 at 11:25 AM Moritz Mack wrote: > Yes, sorry Robert for being so unspecific. With everywhere I meant Java > only, my bad! > > > > On 11.04.23,

Re: Jenkins Flakes

+1 for getting Jenkins back into a happier state, getting release blockers resolved ahead of building an RC has been severely hindered by Jenkins not picking up tests or running them properly. On Tue, Apr 11, 2023 at 10:24 AM Danny McCormick via dev < dev@beam.apache.org> wrote: > *;tldr - I

2.47.0 Branch Cut

Hey folks, The 2.47.0 release branch has been cut! Hopefully the rest of this release is extremely uneventful. Thanks, Jack McCluskey -- Jack McCluskey SWE - DataPLS PLAT/ Dataflow ML RDU jrmcclus...@google.com

Re: 2.47.0 Release Update and Brief Code Freeze

Hey folks, Thank you for your patience and understanding. The repo has been returned to a good state and the code freeze has been lifted. I'll be *actually* cutting the release branch shortly. Thanks, Jack McCluskey On Wed, Apr 5, 2023 at 4:52 PM Jack McCluskey wrote: > Hey everyone, > >

2.47.0 Release Update and Brief Code Freeze

Hey everyone, While I was working on cutting the release branch this afternoon I hit a snag with the release script, and in trying to revert the halfway-completed work on that the entire Beam repo was (briefly) deleted. This has been fixed, but as a consequence of this and the permissions on the

Re: [PROPOSAL] Preparing for 2.47.0 Release

Hey everyone, I need a PMC member's help adding my pubkey to https://dist.apache.org/repos/dist/release/beam/KEYS as well as adding PyPI user jrmccluskey to the maintainers of the Apache Beam package. These are the last steps I have to do to complete prep for the release. Thanks, Jack McCluskey

[PROPOSAL] Preparing for 2.47.0 Release

Hey all, The next (2.47.0) release branch cut is scheduled for April 5th, 2023, according to the release calendar [1]. I will be performing this release. My plan is to cut the branch on that date, and cherrypick release-blocking fixes afterwards, if any. Please help me make sure the release

Re: Launch Dataflow Flex Templates from Go

As far as I know, Shivam's solution is the correct one for launching a template from Go. You're just routing your calls through a Dataflow-specific API library rather than going through the API discovery service. On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark wrote: > Jack added the Go

Re: [Go SDK] Direct Runner Replacement: Prism

Congratulations on getting the runner to a state you're happy contributing to the main repo! I'm happy to help review PRs and get sub-packages in. Anything that helps developers and users test Beam pipelines more effectively is a welcome inclusion. Thanks, Jack McCluskey P.S. I'm glad the Prism

TensorFlow Hub Integration in RunInference

Hey everyone, I've put together some thoughts on how we would allow users to load pre-trained models from TensorFlow Hub into RunInference pipelines in a small doc: https://docs.google.com/document/d/11WpWQ_x3dpgRTvkzKS47lCXA_-PGzUBAIKNJQfLpJTg/edit?usp=sharing

Re: [Python][Bikeshed] typehint vs. type-hint vs. "type hint"

I'm in agreement that how we refer to type hints in documentation should be standardized across the board. It's a good practice for both style and clarity. Seems like it wouldn't be too hard to update our docstrings either, based on a quick search of the repo. On Mon, Nov 7, 2022 at 9:00 AM Brian

Re: [ANNOUNCE] New committer: Ritesh Ghorse

Congrats Ritesh! On Thu, Nov 3, 2022 at 10:12 PM Danny McCormick via dev wrote: > Congrats Ritesh! This is definitely well deserved! > > On Thu, Nov 3, 2022 at 8:08 PM Robert Burke wrote: > >> Woohoo! Well done Ritesh! :D >> >> On Thu, Nov 3, 2022, 5:04 PM Anand Inguva via dev >> wrote: >>

Re: Custom Inference Fns in RunInference

Thank you to everyone who chimed in on the doc! The discussion was very productive, and I have since updated the design doc with some more detail based on feedback and some suggestions. Any and all feedback is welcome! Thanks, Jack McCluskey On Fri, Sep 16, 2022 at 2:45 PM Jack McCluskey

Custom Inference Fns in RunInference

Hey everyone, I'm back with a brief design doc discussing ways that users could provide custom inference functions for RunInference model handlers, which is available at https://docs.google.com/document/d/1YYGsF20kminz7j9ifFdCD5WQwVl8aTeCo0cgPjbdFNU/edit?usp=sharing

Re: [RFC] State & Timers API Design for Go SDK

Great write-up on state and timers! The solution you chose feels very in-line with how the Go SDK works. Make sure the design doc makes it onto the wiki once you've addressed any feedback! On Thu, Jul 28, 2022 at 1:49 PM Kerry Donny-Clark via dev < dev@beam.apache.org> wrote: > I think this a