Re: Apache beam Python ReadFromKafka Issue GroupByKey cannot be applied to an unbounded PCollection

Please use beam.io.fileio.WriteToFiles On Thu, Jul 25, 2024 at 2:48 AM Dhirendra Singh wrote: > Hello Beam Devs, > > Thank you for your help. > I have been trying to connect to the Kafka Enabled Azure event hub > using Dataflow beam code using Python. > Apache Beam version: 2.56.0 > Python

Re: [VOTE] Release 2.58.0, release candidate #1

+1 (non-binding). Tested it with a simple Dataflow ML pipeline and looks good: https://github.com/google/dataflow-ml-starter/actions/runs/9991815564/job/27615395240 On Wed, Jul 17, 2024 at 5:35 PM Jack McCluskey via dev wrote: > Hi everyone, > > Please review and vote on the release candidate

Re: Apache beam github repo collaborator

Welcome to Beam! You can start contributing now. Some useful docs: - https://github.com/apache/beam/blob/master/CONTRIBUTING.md - https://github.com/apache/beam/tree/master/contributor-docs - https://cwiki.apache.org/confluence/display/BEAM/Developer+Guides You can start with some good

Re: [ANNOUNCE] New Committer: XQ Hu

Thanks a lot! Happy to keep working with all of you! On Mon, Jun 24, 2024 at 6:22 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Congratulations and thank you for all your contributions to Beam! > > On Mon, Jun 24, 2024 at 1:49 PM Robert Burke wrote: > >> Congratulations XQ! >>

Re: [VOTE] Release 2.57.0, release candidate #1

+1 (non-binding). Tested this with a simple Dataflow ML pipeline with Python: https://github.com/google/dataflow-ml-starter/actions/runs/9619553472/job/26536415002 On Fri, Jun 21, 2024 at 4:18 PM Kenneth Knowles wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for

Re: [ACTION REQUESTED] Help me draft the Beam Board Report for June 2024

Thanks a lot for doing this for Beam. I added a few notes and feel free to change them. On Thu, May 23, 2024 at 2:57 PM Kenneth Knowles wrote: > The next Beam board report is due next Wednesday, June 12. Please draft it > together at https://s.apache.org/beam-draft-report-2024-06 > > The doc is

Re: default_sdk_harness_log_level multi-language support

Feel free to open a github issue to track this idea. For Dataflow, the potential workaround is to filter the logs: https://cloud.google.com/dataflow/docs/guides/filter-logs On Fri, May 24, 2024 at 5:01 AM Ferran Fernández Garrido < ffernandez@gmail.com> wrote: > Hi! > > I am trying to adjust

Re: Question: Java Apache Beam, mock external Clients initialized in Setup

I am not sure which part you want to test. If the processData part should be tested, you could refactor the code without use any Beam specific code and test the processing data logic. >From your example, it seems that you are calling some APIs, we recently added a new Web API IO:

Re: Python apache beam bigquery temporary tables

I think you hit this issue: https://stackoverflow.com/questions/64526500/using-writetobigquery-file-loads-in-a-streaming-pipeline-just-creates-a-lot-of-t I created https://github.com/apache/beam/issues/31329 to track this bug. Some workarounds: 1.

Re: Chronon + Beam

Welcome to Beam! Let us know if you have any questions when contributing to Beam! On Tue, May 7, 2024 at 7:05 PM Varant Zanoyan wrote: > Hi Beam community, > > Nikhil and I are two of the authors of the Chronon > project, maintained by Airbnb and Stripe. We're >

Re: Beam + Google Summer of Code 2024

Welcome to Beam! On Wed, May 1, 2024 at 4:13 PM Danny McCormick via dev wrote: > Hey everyone, > > It's my pleasure to announce 2 contributors have been accepted as GSoC > students for Beam this year! > > Ayush Pandey will be working on a project to implement RAG example > pipelines using Beam

Re: [VOTE] Release 2.56.0, release candidate #2

+1 (non-binding). Tested it using the dataflow ML pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/8862170843/job/24334816481 On Sat, Apr 27, 2024 at 7:42 AM Danny McCormick via dev wrote: > Hi everyone, > Please review and vote on the release candidate #2 for the version

Re: [VOTE] Release 2.56.0, release candidate #1

+1 (non binding). Tested the simple Dataflow ML job: https://github.com/google/dataflow-ml-starter/actions/runs/8824985423/job/24228468173 On Wed, Apr 24, 2024 at 2:01 PM Danny McCormick via dev wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 2.56.0, >

Re: tox issues in dev container

> wrote: > >> Yeah that was the tox command I was running >> >> On Fri, Apr 5, 2024, 4:37 PM XQ Hu via dev wrote: >> >>> >>> https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks >>> >>> This gene

Re: tox issues in dev container

https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-LintandFormattingChecks This generally works well. Have you checked this? On Fri, Apr 5, 2024 at 4:07 PM Joey Tran wrote: > I think I might be doing something silly with my environment. > > I'm trying to lint using tox in

Re: [VOTE] Patch Release 2.55.1, release candidate #2

+1 (non-binding). Tested this using a simple Dataflow ML pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/8541848483. On Wed, Apr 3, 2024 at 2:35 PM Jeff Kinard wrote: > +1. Validated running from local gradle JAR and staged maven JAR for > expansion-service. > > On Wed, Apr

Re: [VOTE] Release 2.55.0, release candidate #3

+1 (non-binding). Ran the simple ML pipeline without any issue: https://github.com/google/dataflow-ml-starter/actions/runs/8349158153 On Tue, Mar 19, 2024 at 11:55 AM Ritesh Ghorse via dev wrote: > +1 (non-binding) - Ran a few python batch examples on Direct and Dataflow > runner. > > Thanks! >

Re: [VOTE] Release 2.55.0, release candidate #2

OK. +1 (non-bindg). Tested the simple Dataflow ML pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/8308559154 On Mon, Mar 18, 2024 at 10:26 AM Yi Hu wrote: > Yes, there is only one cherry-pick compared to RC1 (#30637) regarding a > dependency change, not all artifacts are

Re: [VOTE] Release 2.55.0, release candidate #2

I do not see RC2 for Python SDK here: https://pypi.org/project/apache-beam/#history On Fri, Mar 15, 2024 at 10:54 AM Yi Hu via dev wrote: > +1 (non-binding). > > Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates > (Java SDK 11, Dataflow runner, details: >

Re: Update confluent dependencies version in kafka io

This sounds great! Feel free to create an issue to track this work! On Mon, Mar 11, 2024 at 9:36 AM Maciej Szwaja via dev wrote: > Hi, > > This is my first email to this list, hello everyone! > > I have a question regarding the confluent version dependency that beam, or > more specifically

Re: [ACTION REQUESTED] Help me draft the Beam Board Report for March 2024

Thanks for the ping! I added several notes and feel free to make more changes. On Mon, Mar 11, 2024 at 2:49 PM Kenneth Knowles wrote: > Ping! > > Would really love help from folks building stuff to report out on what > they've built, especially! > > Kenn > > On Tue, Mar 5, 2024 at 12:15 PM

Re: Issue building python SDK with M2 Mac

I do not have the problem to do this on my M1 by cloning the repo and using conda to create venv with python 3.11 and installing it with pip install -e ".[gcp,test]". It installs numpy with 1.26.4. On Thu, Mar 7, 2024 at 7:48 AM Joey Tran wrote: > Hey all, > > I'm trying to get a beam python

Re: [ANNOUNCE] New Committer: Svetak Sundhar

Great job, Svetak! Thanks for all your contributions to Beam!!! On Mon, Feb 12, 2024 at 4:44 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Congrats, Svetak! > > On Mon, Feb 12, 2024 at 11:20 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam

Re: Playground: File Explorer?

Can you provide which example you are referring to? I checked a few examples and usually we use beam.Map(print) to display some output values. On Wed, Feb 7, 2024 at 8:55 PM Joey Tran wrote: > Hey all, > > I've been really trying to use Playground for educating new Beam users but > it feels

Re: [VOTE] Release 2.54.0, release candidate #2

+1 (non-binding) Validated with a simple RunInference Python pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/7821639833/job/21339032997 On Wed, Feb 7, 2024 at 7:10 PM Yi Hu via dev wrote: > +1 (non-binding) > > Validated with Dataflow Template: >

Re: [VOTE] Release 2.54.0, release candidate #1

+1 validated by running the simple RunInference ML pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/7761835540/job/21171080332 On Fri, Feb 2, 2024 at 4:10 PM Robert Burke wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 2.54.0, > as

Re: Google Artifact Registry detects critical vuln CVE-2023-45853 in beam dataflow

FYI. The ongoing PR: https://github.com/apache/beam/pull/30011 will switch to the distroless images, which will have less vulnerabilities in the future. On Wed, Jan 24, 2024 at 12:32 PM Valentyn Tymofieiev wrote: > > Does the beam project generally attempt to address as many of these >

Re: Setting up beam locally

Assuming you run this under beam/sdks/python, since I use zsh, I have to do this `pip install -e ".[gcp,test]"`. On Fri, Jan 5, 2024 at 1:33 AM G Gautam wrote: > Hi everyone, > > Need help in setting beam locally. > > When trying to setup locally > on entering this command: pip install -e

Re: [VOTE] Release 2.53.0, release candidate #2

+1 (non-binding) Tested with the simple RunInference pipeline: https://github.com/google/dataflow-ml-starter/actions/runs/7332832875/job/19967521369 On Tue, Dec 26, 2023 at 3:29 PM Jack McCluskey via dev wrote: > Happy holidays everyone, > > Please review and vote on the release candidate #2

Re: [VOTE] Release 2.53.0, release candidate #1

+1 (non-binding) Tested apache-beam[gcp]==2.53.0rc1 with a simple RunInference pipeline using Dataflow Runner: https://github.com/google/dataflow-ml-starter/actions/runs/7296067726/job/19883348692 On Thu, Dec 21, 2023 at 4:30 PM Danny McCormick via dev wrote: > +1 (non-binding) > > Tested

Re: Unable to run - execute java examples

Welcome to Beam! To test your Java environment, you can just run this gradle task: ./gradlew :examples:java:wordCount On Sat, Dec 16, 2023 at 6:30 PM Vineet wrote: > Hi, > > I want to run the beam examples locally , building from source (for > contributing). > Hence after setting all

Re: Beam 2.52.0 Release

Thanks a lot! Great job, Team! On Fri, Nov 17, 2023 at 7:21 PM Danny McCormick via user < u...@beam.apache.org> wrote: > I am happy to announce that the 2.52.0 release of Beam has been finalized. > This release includes both improvements and new functionality. > > For more information on changes

Re: [VOTE] Release 2.52.0, release candidate #5

+1 (non binding) Tested the Python SDK RC5 using the ML pipeline under https://github.com/google/dataflow-ml-starter https://github.com/google/dataflow-ml-starter/actions/runs/6898545809/job/18768732434 ran well. On Thu, Nov 16, 2023 at 7:46 PM Robert Bradshaw via dev wrote: > +1 (binding) > >

Re: [PYTHON] partitioner utilities?

+1 on this idea. Thanks! On Thu, Oct 19, 2023 at 3:40 PM Joey Tran wrote: > Yeah, I already implemented these partitioners for my use case (I just > pasted the classnames/docstrings for them) and I used both combiners.Top > and combiners.Sample. > > In fact, before writing these partitioners I

Re: [Discuss] Idea to increase RC voting participation

+1. This is a great idea to try. @Danny McCormick FYI as our next release manager. On Wed, Oct 18, 2023 at 2:30 PM Johanna Öjeling via dev wrote: > When I have contributed to Apache Airflow, they have tagged all > contributors concerned in a GitHub issue when the RC is available and asked > us

Re: [YAML] Aggregations

+1 on your proposal. On Fri, Oct 20, 2023 at 4:59 PM Robert Bradshaw via dev wrote: > On Fri, Oct 20, 2023 at 11:35 AM Kenneth Knowles wrote: > > > > A couple other bits on having an expression language: > > > > - You already have Python lambdas at places, right? so that's quite a > lot more

Re: [ANNOUNCE] New Committer: Sam Whittle

Congratulations! On Mon, Oct 16, 2023 at 1:58 PM Ahmet Altay via dev wrote: > Congratulations Sam! > > On Mon, Oct 16, 2023 at 10:42 AM Byron Ellis via dev > wrote: > >> Congrats Sam! >> >> On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev < >> dev@beam.apache.org> wrote: >> >>>

Re: [ANNOUNCE] New Committer: Byron Ellis

Congratulations! On Mon, Oct 16, 2023 at 1:58 PM Ahmet Altay via dev wrote: > Congratulations Byron! > > On Mon, Oct 16, 2023 at 10:35 AM Tomo Suzuki via dev > wrote: > >> Congratulations! >> >> >> On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev < >> dev@beam.apache.org> wrote: >>

Re: [VOTE] Release 2.51.0, release candidate #1

+1 (non-binding). Tested the simple dataflow ML starter job with https://github.com/google/dataflow-ml-starter/actions/runs/6397130175/job/17364408813 . On Tue, Oct 3, 2023 at 2:29 PM Danny McCormick via dev wrote: > All Beam Python versions 2.50 and greater run exclusively on Dataflow > runner

Re: [ANNOUNCE] New PMC Member: Valentyn Tymofieiev

Congratulations, Valentyn! On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming Valentyn > Tymofieiev as our newest PMC member. > > Valentyn has been contributing to Beam since 2017. Notable highlights > include his work

Re: [ANNOUNCE] New PMC Member: Robert Burke

Congratulations, Robert! On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming Robert Burke < > lostl...@apache.org> as our newest PMC member. > > Robert has been a part of the Beam community since 2017. He is our > resident

Re: [ANNOUNCE] New PMC Member: Alex Van Boxel

Configurations, Alex! On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming Alex Van Boxel < > alexvanbo...@apache.org> as our newest PMC member. > > Alex has been with Beam since 2016, very early in the life of the project.

Re: Beam ML Use Cases - Google Summer of Code 2023

The blog looks great! Thanks for doing this and I hope you have learned a lot! Thanks a lot to Danny for your support! On Wed, Sep 13, 2023 at 12:58 PM Reeba Qureshi wrote: > Hi everyone > > I have completed Google Summer of Code 2023 with Apache Beam, where I > worked on developing real-world

Re: [VOTE] Release 2.50.0, release candidate #2

+1 Ran the simple Dataflow ML GPU batch job using https://github.com/google/dataflow-ml-starter with Python 2.50.0rc2 to validate the RC works well. On Sat, Aug 26, 2023 at 12:16 AM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > +1 > > Verified that the issue detected in RC0 has

Re: [ANNOUNCE] New committer: Ahmed Abualsaud

Congratulations, Ahmed! On Thu, Aug 24, 2023, 5:49 PM Ahmet Altay via dev wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new > committer: Ahmed Abualsaud (ahmedabuals...@apache.org). > > Ahmed has been part of the Beam community since January 2022, working >

[Request for Feedback] Mechanism for "Beam Website Feedback"

Hi all, Following what Ahmet proposed here ( https://lists.apache.org/thread/9l18gs7vgqz6g1cfc2m66rbkrcnr9qmj), we changed the "SEED FEEDBACK" button to open the Google form (

Re: [PROPOSAL] Preparing for 2.50.0 Release

Welcome to the Beam community! Our release managers usually follow this https://beam.apache.org/contribute/release-guide/#10-vote-and-validate-release-candidate to send the votes out and ask for any feedback regarding the release candidate. If you could help run any validation on your side and

Propose to add the new security section to the Beam releases

Hi All, We are proposing to explicitly add the security fixes to the Beam release notes. https://github.com/apache/beam/pull/27976 modified the template in CHANGES.md by adding this new section. Please let us know if you have any questions or feel free to comment PR:

Re: KafkaIO Parameter Issue | Runtime PipelineOptions | Apche Beam

I am not entirely sure what you try to do with PipelineOptions. Assuming you use Beam Java, https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/options/PipelineOptions.html documents how to construct PipelineOptions is via command-line argument parsing.

Re: [Discuss] Get rid of OWNERS files

+1. On Tue, Aug 8, 2023 at 10:55 AM Danny McCormick via dev wrote: > Hey everyone, I'd like to propose getting rid of OWNERS files from the > Beam repo. Right now, I don't think they are serving a meaningful purpose: > > - Many OWNERS files are outdated and point to people who are no longer >

Re: [DISCUSS] Enable Github Discussions?

+1 with GH discussion. If Airflow can do this https://github.com/apache/airflow/discussions, I think we can do this as well. On Mon, Jul 3, 2023 at 9:51 AM Alexey Romanenko wrote: > -1 > I understand that for some people, who maybe are not very familiar with > ASF and its “Apache Way” [1], it

Re: [beam-starter-typescript]: Missing place to create issue

I believe Robert is the owner for that project. On Mon, Jun 12, 2023 at 11:30 PM david-kh...@hotmail.com < david-kh...@hotmail.com> wrote: > Hi Beam community, > > > > I am David and new to the community. After tried to tweak some code from > beam-starter-ts, I have found some issues and want to

Re: Introducing beam.MLTransform

Agree with Danny. Thanks for writing this! On Wed, May 10, 2023 at 10:35 AM Danny McCormick via dev < dev@beam.apache.org> wrote: > Thanks Anand! I left a few comments, but overall I think this is a > great/well constructed proposal - this is a good way to quickly bring a > suite of

Re: [ANNOUNCE] New committer: Damon Douglas

Congrats Damon!!! On Mon, Apr 24, 2023 at 4:34 PM Danny McCormick via dev wrote: > Congrats Damon! > > On Mon, Apr 24, 2023 at 4:03 PM Ahmet Altay via dev > wrote: > >> Congratulations Damon! >> >> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke wrote: >> >>> Congratulations Damon!!! >>> >>> On

Re: [ANNOUNCE] New committer: Anand Inguva

Congratulations, Anand!!! On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev wrote: > Congratulations, Anand! > > On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev < > dev@beam.apache.org> wrote: > >> Congratulations! >> >> On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský wrote: >>