Re: Introducing beam.MLTransform

2023-05-15 Thread Ahmet Altay via dev
Thank you for writing this, Anand! I think this is a good way to leverage existing libraries and also a good usability improvement. On Wed, May 10, 2023 at 11:04 AM XQ Hu via dev wrote: > Agree with Danny. Thanks for writing this! > > On Wed, May 10, 2023 at 10:35 AM Danny McCormick via dev < >

[Benchmarks/Proposal] Loading models with multi_process_shared.py for RunInference

2023-05-15 Thread Danny McCormick via dev
Hey everyone, Right now, using RunInference with large models and on GPUs has several performance gaps. I put together a document focusing on one: when running inference with large models, we often OOM because we load several copies of the model at once. My document explores using the multi_proces

Re: [DISCUSS] Idempotent initialization of file systems

2023-05-15 Thread Robert Bradshaw via dev
On Mon, May 15, 2023 at 8:38 AM Moritz Mack wrote: > > Hi all, > > I was just looking into an old issue again, SerializablePipelineOptions > calling FileSystems.setDefaultPipelineOptions on deserialization [1]. This > applies to various runners including Flink and Spark, but not Dataflow as far

[ANNOUNCE] Beam 2.47.0 Released

2023-05-15 Thread Jack McCluskey
The Apache Beam Team is pleased to announce the release of version 2.47.0. You can download the release here: https://beam.apache.org/get-started/downloads/ This release includes bug fixes, features, and improvements detailed on the Beam Blog: https://beam.apache.org/blog/beam-2.47.0/ and the Gi

[DISCUSS] Idempotent initialization of file systems

2023-05-15 Thread Moritz Mack
Hi all, I was just looking into an old issue again, SerializablePipelineOptions calling FileSystems.setDefaultPipelineOptions on deserialization [1]. This applies to various runners including Flink and Spark, but not Dataflow as far as I know. Problem: Current initialization of FileSystems thr

Beam High Priority Issue Report (32)

2023-05-15 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/26621 [Failing Test]: be