Hi all, I agree with Yufei on the discovery problem, and with the profile shape JB described.
One thing on main: the names in polaris.storage.aws.<name> are process-wide, so any realm with the flag on can resolve any of them. A privilege to use a profile, checked at create and update, is what would close that. The secrets manager thread Dmitri pointed at can stay separate. Thanks, Vignesh On Sat, 14 Nov 2026 at 13:16, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Yufei, > > I like the "credential profile" idea. To build on what Dmitri said, I think > a profile should be decoupled from where the secret actually lives: a > profile is a named, server-side ref, and its backend can be the Quarkus > config (as today, wired to k8s secrets by the deployment admin) or a proper > secrets manager later (Vault, cloud secret managers, ...). That way we keep > the current behavior for single org deployments, and the secrets manager > discussion can move forward independently without blocking this. > > I would scope profiles to a realm and require a privilege to use a profile, > checked when a catalog (or connection) is created or updated. Today, for > storage, the name is only resolved when credentials are vended, and the > only control is the realm level RESOLVE_CREDENTIALS_BY_STORAGE_NAME flag. > Once usage is authorized, listing is straightforward: return the profiles > the caller is allowed to use, with metadata only (name, type, description), > never credential material. > > Unknown or unauthorized profiles should be rejected at create/update time, > not on the first credential vending. > > storageName is on the S3, Azure, GCS and FILE storage configs, but only AWS > resolves credentials from it today. If we generalize to profiles, we should > either support the other types or make it clear in the API and docs which > types use it. > > Regarding PR #5664, I'm fine moving forward as it follows the #3409 pattern > behind the same flag. However, I would avoid adding more storageName > consumers before we agree on the profile model. > > A short design doc would probably help us converge, I'm happy to help on > it. > > Regards > JB > > On Sat, Nov 14, 2026 at 1:01 AM Yufei Gu <[email protected]> wrote: > > > Hi everyone, > > > > While reviewing PR 5664, (https://github.com/apache/polaris/pull/5664) I > > found storageName requires the admin creating a connection to know a name > > configured on the server. That is awkward when catalog admins and Polaris > > deployment admins are different people. It’s very likely in a multi-realm > > Polaris deployment. > > > > Polaris already uses this pattern for storage credentials (#3409 ( > > https://github.com/apache/polaris/pull/3409)). I think we can keep it > and > > improve discovery. > > > > One idea is to treat these names as credential profiles and later let > users > > list the profiles they are allowed to use. They would select one and put > > its name into storageName. Credentials would stay on the server, and > > existing configurations would continue to work. > > > > This would let catalog admins choose credentials without inspecting > server > > configuration or asking deployment admins for each name. Both listing and > > using a profile should respect permissions. > > > > WDYT? > > > > Thanks, > > Yufei > > >
