[ANNOUNCE] Beam 2.33.0 Release

The Apache Beam team is pleased to announce the release of version 2.33.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release her

Re: Python SDK and Kerberos

Thank you, this is useful. The HDFS client library that Beam uses supports Kerberos authentication [1], but it would require adding a flag to Beam to use it. I don't have experience with Kerberos, but it looks like running Kinit might be enough: "HTTPKerberosAuth normally uses the default principal

Re: Python SDK and Kerberos

et Altay wrote: > /cc +Udi Meiri > > On Wed, Feb 3, 2021 at 9:23 AM Doutre, Mark > wrote: > >> Hi, >> I’m trying to use the python sdk to write data to hdfs. However our >> cluster is kerberized. Is it possible to do this with the current sdk? If >> it is, how t

Re: BEAM-2217 NotImplementedError - DataflowRunner parsing Protos from PubSub (Python SDK)

ding the output type like so: Map(parse_message_blobs).with_output_types(typing.Any) That should prevent Beam from attempting to use the ProtoCoder. On Wed, Jun 10, 2020 at 1:47 PM Brian Hulette wrote: > +Udi Meiri - do you have any suggestions to resolve > this? > > It looks lik

[ANNOUNCE] Beam 2.18.0 Released

. -- Udi Meiri, on behalf of The Apache Beam team

Datastore support in Python 3

Just writing this to let Beam Python users that Datastore will be supported in Python 3 (JIRA issue ). It uses a different library (google-cloud-datastore), so it will require changes to your pipeline code. smime.p7s Description: S/MIME Cryptograph

Re: FileIO with GCP - KMS support

Hi Juan! I've recently replaced GCS copy with rewrite: https://github.com/apache/beam/pull/7682 It should be available in the next Beam release (2.11). On Wed, Feb 20, 2019 at 7:43 AM Juan Carlos Garcia wrote: > Sorry, i hit send before verifying the right name of the method: > > The correct me

Re: Struggling to write data from pubsub to gcs (python)

Hi Brice, I believe the issue you're seeing is due to that writing to GCS is not yet supported in streaming pipelines. Regards, Udi On Fri, Aug 3, 2018 at 6:28 AM Brice Giesbrecht wrote: > Hello beam gurus. I have modified the streaming wordcount example from 2.5 > ( > https://github.com/apach

Re: Documentation for Beam on Windows

n Wed, May 23, 2018 at 1:49 PM Udi Meiri wrote: > >> > >> Hi all, > >> > >> I was looking yesterday for a quickstart guide on how to use Beam on > Windows but saw that those guides are exclusively for Linux users. > >> > >> What documentat

Documentation for Beam on Windows

Hi all, I was looking yesterday for a quickstart guide on how to use Beam on Windows but saw that those guides are exclusively for Linux users. What documentation is available for people wanting to use Beam on Windows machines? Thanks! smime.p7s Description: S/MIME Cryptographic Signature

Re: I'm back and ready to help grow our community!

Welcome back and congrats again! On Thu, May 17, 2018 at 2:23 PM Dmitry Demeshchuk wrote: > While this may be a bit off topic, I still want to say this. > > Congratulations on your graduation, Gris! > > On Thu, May 17, 2018 at 2:19 PM, Griselda Cuevas wrote: > >> Hi Everyone, >> >> >> I was abs