Infrastructure-as-Code to provision a private GKE autopilot kubernetes cluster and strimzi kafka

2023-03-01 Thread Damon Douglas via dev
Hello Everyone, I created a PR to provide to the Beam community terraform code to provision a private Google Kubernetes Engine and kubernetes manifests to provision an internally TCP load balanced strimzi.io Kafka cluster. This solution helped me a lot when I needed a repeatable solution to spin

Re: Consuming one PCollection before consuming another with Beam

2023-03-01 Thread Reuven Lax via dev
I'm not sure I understand this use case well. What are you planning on doing with the BQ dataset if it were processed first? Were you planning on caching information in memory? Storing data in Beam state? Something else? On Wed, Mar 1, 2023 at 10:43 AM Kenneth Knowles wrote: > > > On Tue, Feb

Re: Consuming one PCollection before consuming another with Beam

2023-03-01 Thread Kenneth Knowles
On Tue, Feb 28, 2023 at 5:14 PM Sahil Modak wrote: > The number of keys/data in BQ would not be constant and grow with time. > > A rough estimate would be around 300k keys with an average size of 5kb per > key. Both the count of the keys and the size of the key would be feature > dependent

Beam High Priority Issue Report (37)

2023-03-01 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P1 Issues: https://github.com/apache/beam/issues/25669 [Bug]: Different