alamb opened a new issue, #25662: URL: https://github.com/apache/datafusion/issues/25662
## Is your feature request related to a problem or challenge? @pepijnve introduced support for `CREATE EXTERNAL CATALOG` in [PR #25112](https://github.com/apache/datafusion/pull/25112). Many external catalogs and table providers need credentials to connect to services such as Iceberg catalogs. The [discussion on external catalog registration](https://github.com/apache/datafusion/pull/25112#discussion_r4071629685) highlighted that shared secrets are currently passed through SQL options, raising concerns about exposing credentials when those options are displayed. A session-level secret store would let users reuse credentials across multiple connections and let applications inject credentials outside SQL. ## Describe the solution you'd like Add a `SecretsStore` extension point at the session level, with a simple in-memory map as the default implementation, and wire SQL `CREATE SECRET` and `DROP SECRET` statements to it. The initial implementation should support: - Registering, looking up, and removing named secrets programmatically, so applications can supply credentials outside SQL. - Creating and removing secrets through SQL using the same store. - Replacing the default store with an application-provided implementation. - Accessing the store from catalog and table providers, including their factories, so multiple connections can reuse a named secret. Keep the interface open to different provider integrations. The SQL syntax, representation of secrets, and mechanism for referencing them from connection options need to be designed as part of this work. Persistent storage and integrations with specific external secret managers can be follow-ups. Include an end-to-end example and tests showing two catalog or table provider connections using the same secret. Statement and plan formatting should not expose secret values. ## Describe alternatives you've considered Continue passing credentials directly in connection options or handle secrets independently in each integration. This leaves credential reuse and out-of-band injection to individual providers without a shared session-level interface. ## Additional context This is a follow-up to [PR #25112](https://github.com/apache/datafusion/pull/25112), based on the [SecretsStore proposal](https://github.com/apache/datafusion/pull/25112#discussion_r4071674001) and [review feedback](https://github.com/apache/datafusion/pull/25112/changes#r4075296586). The review specifically calls out making the store accessible to the various catalogs and providers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
