Beam 2.58.0 Release

2024-08-08 Thread Jack McCluskey via user
We are happy to present the new 2.58.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release. For more information about changes in 2.58.0, check out the detailed release notes (https://github.com/apache/beam/milestone/22.) I/Os

Beam 2.53.0 Release

2024-01-05 Thread Jack McCluskey via user
We are happy to present the new 2.53.0 release of Beam. This release includes both improvements and new functionality. For more information on changes in 2.53.0, check out the detailed release notes (https://github.com/apache/beam/milestone/17). Highlights * Python streaming users that use 2.47.0

Re: Getting Started With Implementing a Runner

2023-06-22 Thread Jack McCluskey via user
Hey Joey, The best resource to look at, at the moment, is likely Robert Burke's Prism runner that he is implementing ( https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners/prism). Runners are pretty complicated and there are a number of primitives that you need to have implemented o

Re: Launch Dataflow Flex Templates from Go

2023-02-15 Thread Jack McCluskey via user
As far as I know, Shivam's solution is the correct one for launching a template from Go. You're just routing your calls through a Dataflow-specific API library rather than going through the API discovery service. On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark wrote: > Jack added the Go templa

Re: Java prefix external pipeline with go sdk

2022-08-02 Thread Jack McCluskey via user
Okay so I have gone back and confirmed that the translation failed error also originates from the direct runner ( https://github.com/apache/beam/blob/8da6363b6fef8da5e73976be2b1277a776c05239/sdks/go/pkg/beam/runners/direct/direct.go#L337) so you should try to execute on your runner of choice. I'm g

Re: Java prefix external pipeline with go sdk

2022-08-01 Thread Jack McCluskey via user
Hey Sergiusz, I'm looking into the cross-language problems you're running into. I can't say with 100% certainty for the first issue, but the problem with the second is that you're running on the Go direct runner, which does not support cross-language transforms. You'll need to start up a more full