Hi all, I have a Python based application that is using Apache beam in batch mode and Google Dataflow as a worker. Yesterday, I was facing an issue passing environmental variable to Dataflow workers. I have temporarily commented uses of the non.public Python package which required environmental variables to function.
The first step of my pipeline is to read data from a database table as input PCollection. The library that I have used as Input connector requires DB build-in user and password and first step is getting executed successfully. Now, in second step, I want to update the DB rows (just 1 right now for testing) to IN_PROGRESS. Here, I am using an IAM user which I am also using outside of Dataflow. But I am getting an error in dataflow pipeline - *sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "xx.xx.xxx.xx", port 5432 failed: FATAL: AlloyDB IAM user authentication failed for user "{iam_user}". * I also tried creating a new IAM user corresponding to the service account I am using for workers and provided it with the same permissions as the IAM user outside of dataflow. But ,I am still seeing the same error. From logs, I can see DB IP ,DB name and IAM users are correctly being passed. Is there anything additional that I should be doing for an IAM user to successfully connect to DB? Thanks & Regards, Sumit Desai