tmaoz opened a new issue, #58475: URL: https://github.com/apache/spark/issues/58475
Hello, In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark and spark-connect. We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark. We would like to have an option to change the credentials for connections where the pipeline configuration is coming from config files we have that are dynamically translated into Spark code. For writers this is not a problem as we can add our own retry mechanism that recreates the writer with the updated credentials. However, For readers, this is a problem because at the point of materialization, the reader is already baked into the pipeline with the original credentials and it does not have any way to change the credentials without restarting the entire ETL process, which could be running for several hours before the plan reaches the point where it needs to use the JDBC connection. I'm wondering if it would be possible to add some way of reconfiguring just the reader/writer credentials from PySpark code without having to write dedicated Java code, compile into a JAR and load when running Spark? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
