Hi All, As Sung highlighted in today's Comminity Sync call, there is a general problem with refreshing credentials vended in the context of a Staged Create operation (usually CTAS).
The same problem is beling discussed in the Iceberg community [1] Recap: * It is generally not possible for a client to refresh credentials received from the initial Staged Create REST API call because the related table is not created until commit, thus leading to 404 response from the regular /credentials endpoint * Conceptually, AuthZ checks required on refreshing these credentials are different between the initial request and the refresh. The initial request implies that no data files exist yet and as such resembles an ordinary CREATE TABLE request. When the refresh credential call comes in, some data files are likely to exist already. Therefore, the credential refresh request is more similar to an UPDATE from the AuthZ perspective. Optoins: * Wait for the Iceberg REST API spec change. This will allow associating a particular Stage Create "session" with a particular client. Yet, this does not address the AuthZ aspect at all. * Pre-register a Table Entity at Staged Create time. This will reserve the entity name for the anticipated commit and allow the Polaris Authorizer to distinguish subsequent requests for refreshing credentials from the initial "create" request. * Use a different credential refresh URI for Staged Create environments. This URI is part of the Staged Create REST response, IIRC, and can be expected to override other client-side configs. The URI can use custom path segments or query parameters to identity the specific Staged env. context. * Something else? Please share ideas. [1] https://lists.apache.org/thread/35lj7vhtqtwl5nv9rzpln4mw4fbh7gdp Thanks, Dmitri.
