Database IAM authentication failing from Google Dataflow instance

Hi all, I have a Python based application that is using Apache beam in batch mode and Google Dataflow as a worker. Yesterday, I was facing an issue passing environmental variable to Dataflow workers. I have temporarily commented uses of the non.public Python package which required environmental

RE: Processing data from Kafka. Python

It seems to be fixed by adding option to Java expansion service: "--experiments=use_deprecated_read" I have found connected ticket: https://issues.apache.org/jira/browse/BEAM-11991 Best regards, Stanislav Porotikov From: Поротиков Станислав Вячеславович via user Sent: Tuesday, December 19, 2023

How to set flow control for pubsubliteio?

How to change flow control config for pubsubliteio ? I saw the setting has been taken out as part of https://issues.apache.org/jira/browse/BEAM-14129 But without setup flow control correctly, my beam app is running super slow ingesting from pubsbulite and getting NO_CLIENT_TOKEN error on the

Re: Environmental variables not accessible in Dataflow pipeline

Dataflow VMs cannot know your local env variable. I think you should use custom container: https://cloud.google.com/dataflow/docs/guides/using-custom-containers. Here is a sample project: https://github.com/google/dataflow-ml-starter On Wed, Dec 20, 2023 at 4:57 AM Sofia’s World wrote: > Hello

Re: Environmental variables not accessible in Dataflow pipeline

Hello Sumit Thanks. Sorry...I guess if the value of the env variable is always the same u can pass it as job params?..though it doesn't sound like a viable option... Hth On Wed, 20 Dec 2023, 09:49 Sumit Desai, wrote: > Hi Sofia, > > Thanks for the response. For now, we have decided not to use

Re: Environmental variables not accessible in Dataflow pipeline

Hi Sofia, Thanks for the response. For now, we have decided not to use flex template. Is there a way to pass environmental variables without using any template? Thanks & Regards, Sumit Desai On Wed, Dec 20, 2023 at 3:16 PM Sofia’s World wrote: > Hi > My 2 cents. .have u ever considered using

Re: Environmental variables not accessible in Dataflow pipeline

Hi My 2 cents. .have u ever considered using flex templates to run your pipeline? Then you can pass all your parameters at runtime.. (Apologies in advance if it does not cover your use case...) On Wed, 20 Dec 2023, 09:35 Sumit Desai via user, wrote: > Hi all, > > I have a Python application

Environmental variables not accessible in Dataflow pipeline

Hi all, I have a Python application which is using Apache beam and Dataflow as runner. The application uses a non-public Python package 'uplight-telemetry' which is configured using 'extra_packages' while creating pipeline_options object. This package expects an environmental variable named