Hello, I am a developer trying to use Apache Beam, and I have a nuanced problem I need help with. I have a pipeline which has to read in 40 million records from multiple Parquet files from AWS S3. The only way I can get the credentials I need for this particular bucket is to call an API, which I do before the pipeline executes, and then I store the credentials in the PipelineOptions for the pipeline to use during the read. However, the credentials are only valid for one hour, and my pipeline takes longer than one hour to run. So after an hour of execution, the pipeline fails with a credentials invalidation error. The only way I can refresh the credentials is by calling the API. Is there a way for me to do this in my pipeline while it's running?
Any help would be appreciated! Thanks and sincerely, Ramya ______________________________________________________________________ The information contained in this e-mail may be confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
