Re: [ANNOUNCE] New Committer: XQ Hu

Congratulations! 🎉🎉🎉 Well deserved! On Mon, Jun 24, 2024, 4:32 PM XQ Hu via dev wrote: > Thanks a lot! Happy to keep working with all of you! > > On Mon, Jun 24, 2024 at 6:22 PM Valentyn Tymofieiev via dev < > dev@beam.apache.org> wrote: > >> Congratulations and thank you for all your co

Web APIs I/O Connector Guide

Hello Everyone, The Beam website now has a published guide on a new transform that supports reads and writes with Web APIs such as REST or gRPC. https://beam.apache.org/documentation/io/built-in/webapis/ Best, Damon

Throttle PTransform

Hello Everyone, The following describes a Throttle PTransform that holds element throughput to minimize downstream API overusage. Thank you for reading and your valuable input. https://s.apache.org/beam-throttle-transform Best, Damon

Re: [ANNOUNCE] New Committer: Svetak Sundhar

Congratulations Svetak!!! 😁😁😁 On Thu, Feb 15, 2024 at 8:09 PM Jan Lukavský wrote: > Congrats Svetak! > On 2/14/24 16:11, Yi Hu via dev wrote: > > Congrats, Svetak! > > On Wed, Feb 14, 2024 at 9:50 AM John Casey via dev > wrote: > >> Congrats Svetak! >> >> On Wed, Feb 14, 2024 at 9:00 AM Ahmed A

[prism] Programmatic Cancel, and Drain

Good day, Everyone, We are implementing a Cancel and Drain feature for the Prism runner. Please see https://github.com/apache/beam/issues/29669 that contains everything that goes into a design document. For more information about Prism, please see https://github.com/apache/beam/tree/master/sdks/go

Re: Unable to run - execute java examples

Good day, Vineet, Welcome to Beam 👋. We also have starter kits. I've pasted the link below which references various supported languages: https://beam.apache.org/blog/beam-starter-projects/ These might allow a developer to have what is minimally needed to get started. Please feel free to reach o

Re: [LAZY CONSENSUS] Create separate repository for Swift SDK

+1 On Mon, Sep 25, 2023, 9:04 AM Kenneth Knowles wrote: > Hi all, > > I propose to unblock Byron's work by creating a new repository for the > Beam Swift SDK. This will be the first of its kind, and break from > tradition of having Beam be kind of a mini-mono-repo. > > Discussion of the Swift SD

Re: [ANNOUNCE] New committer: Ahmed Abualsaud

Well deserved! Congratulations, Ahmed! I'm so happy for you. On Thu, Aug 24, 2023, 5:46 PM Byron Ellis via dev wrote: > Congratulations! > > On Thu, Aug 24, 2023 at 5:34 PM Robert Burke wrote: > >> Congratulations Ahmed!! >> >> On Thu, Aug 24, 2023, 4:08 PM Chamikara Jayalath via dev < >> dev@b

Request for submissions | Can Beam slow down a PCollection?

Hello Everyone, As part of a long term research and development into throttling API requests within Beam pipelines, we are asking for your opinion in the form of code snippets. The following document briefly reviews the problem statement and some instructions on how you could submit your code snip

Re: ByteBuddy ClassLoadingStrategy.Default.INJECTION vs getClassLoadingStrategy

he testing scenario a bit tricky. We do >> have tests that model this scenario but they did not catch this I guess. >> >> On Mon, Jul 17, 2023 at 1:19 AM Damon Douglas >> wrote: >> >>> Good day, everyone, >>> >>> For clarity, I organize the fol

ByteBuddy ClassLoadingStrategy.Default.INJECTION vs getClassLoadingStrategy

Good day, everyone, For clarity, I organize the following into situation, background, assessment, and proposal. Best, Damon - Situation Issue #26981 reports an IllegalArgumentException associated with the ByteBuddy dependency throwing the message " must

Go SDK Review Requested | README Rewrite

Good day, everyone, PR 27415 is a suggested rewrite of the Go SDK README. Could someone review it? Thank you, Damon

Go PR Review Requested | [API Overuse Study] Implement Go environment variable type

Hello Everyone, PR 27407 is a Go implementation of an environment variable type. This PR is one of many that implements the support infrastructure needed to collect data on Beam's reading from and writing to APIs (see PR's linked issue for context). Whi

Re: [DISCUSS] Enable Github Discussions?

I'm very strong +1 for replacing the use of Email with GitHub Discussions. Thank you for bringing this up. On Fri, Jun 30, 2023 at 7:38 AM Danny McCormick via dev wrote: > Thanks for starting this discussion! > > I'm a weak -1 for this proposal. While I think that GH Discussions can be > a good

[20498] Remove SuppressWarnings nullness from SelectByteBuddyHelpers

Hello Everyone, *For those experienced with Beam*: PR/26413 [1] begins the long journey of addressing [2] which is a subtask of [3]. 千里之行，始於足下 :-) *For those beginning on the learning path*: The pull request referenced in the email is not directly related to Beam but involves a Java annotation

Re: [ANNOUNCE] New committer: Damon Douglas

at 1:00 PM Robert Burke wrote: >>> >>>> Congratulations Damon!!! >>>> >>>> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles wrote: >>>> >>>>> Hi all, >>>>> >>>>> Please join me and the rest of the

Re: [ANNOUNCE] New committer: Anand Inguva

Congratulations Anand! On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev wrote: > Congratulations Anand! > > On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev < > dev@beam.apache.org> wrote: > >> Congrats Anand! >> >> On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev >> wrote: >>

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

I like that we are expanding annotations in the hopes of balancing safety and freedom and it's helpful to see the other projects. My confession 🙈🙉🙊 is that I stopped using the Experimental annotation because I reasoned that there's already a license saying "don't sue us, it's your fault". I stopp

A simple PR FYI and maybe helpful tips on how to add a new Java module

Hello Everyone, Despite how really simple this PR is, I thought I would take this time to relay some helpful tips I learned to hopefully help those learning Beam and wanting to contribute new modules. *For those who know Beam*: https://github.com/apache/beam/pull/25951 adds the Java IO modules s

Re: Infrastructure-as-Code to provision a private GKE autopilot kubernetes cluster and strimzi kafka

ster/.test-infra/terraform/google-cloud-platform/google-kubernetes-engine 2) https://github.com/apache/beam/tree/master/.test-infra/kafka/strimzi Best, Damon On Wed, Mar 1, 2023 at 4:12 PM Damon Douglas wrote: > Hello Everyone, > > I created a PR to provide to the Beam community terraf

Re: Refactor BigQuery SchemaTransforms naming

Thank you, Ahmed. This is ready now. On Fri, Mar 3, 2023 at 9:18 AM Ahmed Abualsaud wrote: > Thank you Damon, I left a few comments. > > On Fri, Mar 3, 2023 at 11:14 AM Damon Douglas via dev > wrote: > >> Hello Everyone, >> >> This PR brings BigQuery Schema t

Refactor BigQuery SchemaTransforms naming

Hello Everyone, This PR brings BigQuery Schema transforms in line with the others in terms of naming conventions and use of AutoService. https://github.com/apache/beam/pull/25706 Best, Damon

Infrastructure-as-Code to provision a private GKE autopilot kubernetes cluster and strimzi kafka

Hello Everyone, I created a PR to provide to the Beam community terraform code to provision a private Google Kubernetes Engine and kubernetes manifests to provision an internally TCP load balanced strimzi.io Kafka cluster. This solution helped me a lot when I needed a repeatable solution to spin

[Go SDK] Bump github.com/tetratelabs/wazero version to 1.0.0-pre.9

Hello Beam Gophers, This PR patches the Go SDK wasm example, bumping github.com/tetratelabs/wazero version to 1.0.0-pre.9. https://github.com/apache/beam/pull/25560 Best, Damon

Re: Launch Dataflow Flex Templates from Go

Good day, Ashok, I was able to submit a Dataflow job using this code snippet. Change the bracketed <> values with what makes sense to your Google Cloud environment. Could you let me know if this works for you? Best, Damon params := &dataflow.LaunchTemplateParameters{ Environment: &dataflow

Re: Beam Release 2.46

Hello Danny, Do you mind if I shadow you while you do this? Best, Damon On Thu, Feb 9, 2023 at 3:17 PM Kenneth Knowles wrote: > Excellent! Keep that release train rolling. > > On Thu, Feb 9, 2023 at 9:28 AM Ahmet Altay via dev > wrote: > >> Thank you Danny! >> >> On Wed, Feb 8, 2023 at 6:46

Re: Refactor Kubernetes Kafka External load balancer dependency in tests

quired. Due to the nuances of the jenkins setup and its particulars with gcloud and kubectl, I was not able to test and replicate this in my own environment. Best, Damon On Wed, Jan 25, 2023 at 12:12 PM Damon Douglas wrote: > Hello Everyone, > > Thank you for your feedback. I started

A practical guide to Schema | Java SDK Edition

Hello Everyone, *If you are new to Beam please consider yourself a welcomed participant in this conversation. At the end of this email is a longer explanation to help understand this email. I love talking about Beam so feel free to reach out to me if anything isn't clear and not knowing what to a

Re: Refactor Kubernetes Kafka External load balancer dependency in tests

ure is not necessary and it has >> triggered security alerts quite often... Would like to help if needed. >> >> Best, >> Yi >> >> On Mon, Jan 23, 2023 at 1:25 PM Damon Douglas via dev < >> dev@beam.apache.org> wrote: >> >>> Hello Everyo

Refactor Kubernetes Kafka External load balancer dependency in tests

Hello Everyone, I would like to share with the community a proposal [1] to refactor a Kubernetes Kafka External Load Balancer dependency in tests. It fixes [2]. The referenced document summarizes the situation, background, assessment, and recommendation. Best, Damon *References* 1.[Public fa

Re: A Declarative API for Apache Beam

Hello Robert, I'm replying to say that I've been waiting for something like this ever since I started learning Beam and I'm grateful you are pushing this forward. Best, Damon On Wed, Dec 14, 2022 at 2:05 PM Robert Bradshaw wrote: > While Beam provides powerful APIs for authoring sophisticated

[Beam Playground] Local Development Environment: Kubernetes vs Docker Compose

Hello Everyone, *Even if this is your first day learning Beam, please feel welcome to vote.* *Please cast your single question answer on your preference* for Kubernetes [1] versus Docker Compose [2] in local development of the Beam Playground [3]. The form provides short and long versioned expla

CsvIO Write Support

Hello Everyone, *For those new to Beam, even if this is your first day, consider yourselves a welcome contributor to this conversation. I remember what it was like first learning Beam on my own and I am passionate about everyone's learning experience. Below are definitions/references and a sugge

Re: Gradle Task Configuration Avoidance

and an easier thing to look at over the command line output. >> >> On Thu, Dec 8, 2022 at 8:26 AM Damon Douglas via dev >> wrote: >> >>> Thank you, Kerry, for your kind and encouraging words! >>> >>> Kenn, I wondered as well whether there exist

Re: Announcement: Github Actions Self-Hosted Runners Testing

ggers are synced? > > On Fri, Dec 9, 2022 at 11:03 AM Damon Douglas via dev > wrote: > >> Thank you for this information. For Playground Backend and Frontend, are >> self-hosted runners no longer needed since we are using Cloud Build? >> >> *For those who would l

Re: Announcement: Github Actions Self-Hosted Runners Testing

Thank you for this information. For Playground Backend and Frontend, are self-hosted runners no longer needed since we are using Cloud Build? *For those who would like more explanation, please see below.* Playground is our https://play.beam.apache.org/ that allows us to execute Apache Beam code

Re: Gradle Task Configuration Avoidance

ual feeling of "I don't quite understand, and don't have time to >> get context" I can read all the context in the mail. >> This error message had confused me, so I really appreciate the cleanup >> and explanation. >> >> On Fri, Dec 2, 2022, 7:28 PM Da

[Proposal] | Move FileIO and TextIO from :sdks:java:core to :sdks:java:io:file

Hello Everyone, *If you identify yourself on the Beam learning journey, even if this is your first day, please see yourself as a welcome participant in this conversation and consider reviewing the bottom portion of this email for guidance.* *The Short Version (For those with Java Beam SDK knowled

Re: Configuration Driven File Writes

orm whose config is ([common_parameters], [xml-params], > [csv-params], ...) rather than separate schema transforms for each. > (2) Is there a plan to do a similar thing for the Read side? > > On Fri, Dec 2, 2022 at 9:48 AM Damon Douglas > wrote: > > > > Hello Everyone

Gradle Task Configuration Avoidance

Hello Everyone, *If you are new to Beam and coming from non-Java language conventions, it is likely you are new to gradle. At the end of this email is a list of definitions and references to help understand this email.* *Short Version (For those who know gradle)*: A pull request [1] may fix the

Configuration Driven File Writes

Hello Everyone, *For those new to Beam, even if this is your first day, consider yourselves a welcome contributor to this conversation. I remember what it was like first learning Beam on my own and I am passionate about everyone's learning experience. Below are definitions/references and a sugge

Configuration driven Pub/Sub Writes

Hello Everyone, *For those new to Beam, even if this is your first day, consider yourselves a welcome contributor to this conversation. I remember what it was like first learning Beam on my own and I am passionate about everyone's learning experience. Below are definitions/references and a sugge

Re: [DISCUSS] Jenkins -> GitHub Actions ?

l the terraform code that provisions the triggers; typically applied once per GCP project setup 4. See on GitHub progress as the trigger executes and success/failure On Fri, Nov 18, 2022, 4:20 PM Damon Douglas wrote: > I wonder if some integration tests could be offloaded to their respective

Re: [DISCUSS] Jenkins -> GitHub Actions ?

I wonder if some integration tests could be offloaded to their respective cloud provider. For example, the Google cloud related integration tests could be executed on Cloud build. Cloud Build's service account, or custom, could have the minimally necessary IAM roles to access Google Cloud resource

Re: One Configuration, Many File Write Formats

mat > (json, avro, xml, csv, parquet) and retrieve the relevant file writing > PTransform. I also like how the same configuration can be re-used for > different file formats. Looking forward to seeing its implementation :) > > Ahmed > > On Mon, Nov 14, 2022 at 11:53 AM Damon Dougl

Pub/Sub Client | Add Java getSchema and Go GetSchema methods

Hello Everyone, *For those new to Beam, even if this is your first day, consider yourselves a welcome contributor to this conversation. Below are definitions/references and a suggested learning path to understand this email.* Proposal | Java For Java, I would like to add the following to Pubsub

SchemaTransformProvider | Java class naming convention

Hello Everyone, Do we like the following Java class naming convention for SchemaTransformProviders [1]? The proposal is: (Read|Write)SchemaTransformProvider *For those new to Beam, even if this is your first day, consider yourselves a welcome contributor to this conversation. Below are defini

One Configuration, Many File Write Formats

Hello Everyone, I hope you are doing well. The following design document proposes, via a single configuration, a producer of a Beam File writing transform supporting multiple formats. bit.ly/fileioschematransformwriteprovider For those new to Beam and Schema, I've added a final section of sugges

Re: Experimental WebAssembly Example | Go Beam SDK

achin Agarwal wrote: > This is super interesting, thank you Damon! > > On Thu, Nov 10, 2022 at 10:51 AM Damon Douglas via dev < > dev@beam.apache.org> wrote: > >> Hello Everyone, >> >> I created https://github.com/apache/beam/pull/24081 to start a >&g

Experimental WebAssembly Example | Go Beam SDK

Hello Everyone, I created https://github.com/apache/beam/pull/24081 to start a conversation around WebAssembly support in Beam. WebAssembly is an experimental technology. According to WebAssembly.org, "Wasm is designed as a portable compilation target for programming languages". This PR is a si

Re: Go, Java, & Python Project Starter / Example Using Terraform to build Dataflow Custom Templates

20, which > doesn't yet exist: > https://github.com/GoogleCloudPlatform/professional-services/blob/main/examples/dataflow-custom-templates/go/go.mod#L3 > > The latest version is Go v1.19. > > Cheers, > Robert Burke > Beam Go Busybody > > On Tue, 18 Oct 2022 at 09:01

Go, Java, & Python Project Starter / Example Using Terraform to build Dataflow Custom Templates

/GoogleCloudPlatform/professional-services/tree/main/examples/dataflow-custom-templates Best, Damon -- *Damon Douglas* Strategic Cloud Engineer, Data & Analytics, Google Cloud damondoug...@google.com

Java & Python Project Starter / Example Using Terraform to build Dataflow Custom Templates

hat allows one to apply the terraform modules without installing or downloading any code on your local machine. https://github.com/GoogleCloudPlatform/professional-services/tree/main/examples/dataflow-custom-templates Best, Damon -- *Damon Douglas* Strategic Cloud Engineer, Data & Ana

Try Beam Katas Today

Hello Everyone, If you don't already know, there are helpful instructional tools for learning the Apache Beam SDKs called Beam Katas hosted on https://stepik.org. Similar to traditional Kata , they are meant to be repeated as practice. Before practicing the kat

Golang SDK Katas

Hello Everyone, I would like to start an initiative to contribute to the Apache Beam katas for the golang SDK. My JIRA username is `damondouglas`. I will be adding subtasks under https://issues.apache.org/jira/browse/BEAM-9676 Thank you for including me in this exciting project. Sincerely, Da

Golang Apache Beam volunteer to help

Hello Robert, I enjoyed meeting some of your colleagues last night at the Seattle Apache Beam Meetup at the new Google building. I found your contact information via https://beam.apache.org/roadmap/go-sdk/. It suggested I contact you to see where I could possibly volunteer to help with the Apach