Hi Srinivas, hi all, As Dmitri said, summer time indeed has slowed down the community bandwidth, but I'm happy to get back to this topic today.
Thanks for the thorough writeup and also the summary of the current status quo: very useful. I'm OK with both the two-phase framing and the fail-closed approach. I have however one concern on Phase 1 that I think is worth settling before we move forward: the storageName field is being **overloaded to do two independent jobs**. Today (per #3409) this field has exactly one purpose: select the deployment-time credential set. Phase 1 quietly adds a second: it becomes the identity key of the named-config map and the target of polaris.storage.name. Those are two different namespaces: a catalog-scoped config identity vs. a deployment-scoped credential selector; welding them together means you can't share one credential set across configs, and you can't rename a config without re-pointing its credentials. This surfaces as two concrete rough edges: 1. The default config's storageName is ambiguous. §5.1 excludes the default's name from the named map, but §5.4 resolves references only against that map. So if the default carries storageName = "foo" and a table sets polaris.storage.name = "foo", the lookup misses and we fail closed with a "dangling reference", even though a config named "foo" plainly exists. **The default config having a name at all is a code smell.** 2. The two usages can disagree. If named references are found in the hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd resolve a table to a named storage config, but wouldn't pick named credentials for it, falling back to the default provider chain. That's a silent fail-open path inside a design tailored for fail-closed. My proposal: decouple the two concepts, and use a real map of name -> StorageConfigInfo in Create/UpdateCatalogRequest; credential selection keeps using the current storageName field (we could maybe rename it). The map key becomes the reference identity; uniqueness comes for free. That fully decouples the two concerns: several configs can share one credential set, and the default no longer needs a name. WDYT? Thanks, Alex On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi Srinivas, > > Thanks for putting this document together. I think it provides an excellent > summary of this discussion and lays out clean path forward. > > As for phase 1 it looks like we could start implementing it, if other > reviews agree. Given that it's summer time it may take a week or two for > everybody to catch up on this thread, though. > > From a practical point of view, I tend to think that non-default storage > config can be use effectively only at the namespace level. For a new table > to be created via the IRC API, the request can only operate on the > name/config stored in the immediate namespace or higher. The table itself > cannot define the storage name unless we revisit the point of managing it > only via Polaris properties. > > It might be worth allowing Iceberg table properties to carry the storage > name too. Polaris can forward that information to its internal properties > on changes. However, this point does not affect Phase 1, I think. We can > add this kind of property forwarding later. > > Cheers, > Dmitri. > > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <[email protected]> > wrote: > > > I've put together an updated design proposal document detailing the storage > > configuration management proposal. Please provide your feedback on the > > proposal. > > > > > > https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing > > > > > > > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Adam, > > > > > > Could you give some more details, please, about where you think the > > Iceberg > > > spec disallows multiple storage per table? > > > > > > I never thought it was not allowed. Perhaps it was inconvenient or > > > impractical in existing clients, but the spec itself seems to be fairly > > > unbiased with respect to actual storage technology... Perhaps I missed > > > something, though. > > > > > > Thanks, > > > Dmitri. > > > > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <[email protected]> wrote: > > > > > > > Hi Yufei, > > > > > > > > Regarding #2: > > > > Would opting for a simple (assuming per table) storageConfigName or > > > > storageConfigId > > > > support future extensibility? > > > > Asking from a multiple-storage-per-table perspective as it's something > > > that > > > > the Iceberg spec does allow IMHO. > > > > > > > > Cheers, > > > > Adam > > > > > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> wrote: > > > > > > > > > Hi folks, thanks for the discussion. They are really helpful! > > > > > > > > > > I think there are two separate questions here: > > > > > > > > > > 1. How should we store storage configurations? As catalog properties > > or > > > > as > > > > > separate entities? > > > > > I'd lean toward separate entities in the long term. It provides a > > > cleaner > > > > > boundary between a catalog and its storage configurations, avoids > > > > bloating > > > > > the catalog object as the number of configs grows, and gives us a > > > better > > > > > foundation for future access control since entity level permissions > > are > > > > > easier to reason about. > > > > > > > > > > 2. How should a table or namespace refer to a storage configuration? > > > > > I'd prefer storing a simple reference on the entity itself rather > > than > > > > > introducing a separate relationship table. Both approaches support > > > > > hierarchical lookup and inheritance when a storage config is not > > > > specified > > > > > at the current level. However, I don't think we should allow multiple > > > > > storage configs to be associated with a single table or namespace. > > > Given > > > > > that cardinality, a simple storageConfigName or storageConfigId field > > > > seems > > > > > sufficient and easier to understand than a dedicated relationship > > > model. > > > > > > > > > > For example: > > > > > > > > > > Catalog > > > > > ├─ entity: StorageConfigA > > > > > ├─ entity: StorageConfigB > > > > > └─ entity: StorageConfigC > > > > > > > > > > Table1 storage property -> StorageConfigA > > > > > Table2 storage property -> StorageConfigB > > > > > Table3 storage property -> StorageConfigC > > > > > > > > > > To me, this keeps the model simple while still supporting shared > > > storage > > > > > configurations and future extensibility. > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Robert, > > > > > > > > > > > > The distinction between "attached" and "referenced" config makes > > > sense. > > > > > > > > > > > > My impression from JB's message on Mar 19 in this thread [1] is > > that > > > JB > > > > > is > > > > > > specifically proposing the "referenced" model, though. > > > > > > > > > > > > From my POV, both approaches can work, but the "referenced" model > > > might > > > > > be > > > > > > more intuitive to users. > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected]> > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > AFAIK the current Polaris code appears to resolve the storage > > > > > > > configuration from the leaf (table), upwards via the namespace > > > > elements > > > > > > to > > > > > > > the catalog, choosing the first one found. > > > > > > > I am not sure whether that functionality is already used > > anywhere? > > > > > > > Being able to use a storage configuration on a namespace or a > > leaf > > > > > > appears > > > > > > > appealing. > > > > > > > > > > > > > > OTOH I see a demand for named storage configurations. > > > > > > > > > > > > > > I wonder whether the mechanism to derive the storage > > configuration > > > > > could > > > > > > > support both "directly attached" and "referenced" storage > > > > > configurations: > > > > > > > A namespace element or leaf can have either a "directly attached" > > > > > storage > > > > > > > configuration or a reference to a named one. > > > > > > > > > > > > > > We'd need management-ish APIs to handle the storage > > configurations > > > > for > > > > > a > > > > > > > namespace or leaf. > > > > > > > These APIs, which do not exist yet, could start with "directly > > > > > attached" > > > > > > > storage configurations and once the authz questions are resolved, > > > > > extend > > > > > > to > > > > > > > include named, shared storage configurations. > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > To recap the discussion from the Community Sync today: My > > > > > understanding > > > > > > > is > > > > > > > > that the consensus was: > > > > > > > > > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If more > > than > > > > one > > > > > > > > Storage Config is present, they are to have different (within > > the > > > > > > > catalog) > > > > > > > > names. Rishi to open a new PR for this (scoped to REST API + > > > Sever > > > > > > > > changes). > > > > > > > > > > > > > > > > * Defer CLI changes until we have the full end-to-end system > > > > working > > > > > > (it > > > > > > > > should be usable via curl) > > > > > > > > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding > > > > > support > > > > > > > for > > > > > > > > multiple Storage Configs > > > > > > > > > > > > > > > > * The connection between storage names and storage access > > > > credentials > > > > > > is > > > > > > > to > > > > > > > > remain unchanged for now. It is effectively a deployment time > > > > concern > > > > > > > given > > > > > > > > the current OSS code. > > > > > > > > > > > > > > > > Please correct / add if I missed anything. > > > > > > > > > > > > > > > > From my POV, I think we still need to discuss how exactly > > > entities > > > > > > > (tables, > > > > > > > > views) will link to Storage Config before we can resume PR > > 4023. > > > > > > > > > > > > > > > > I propose storing the config name in the Entity properties (not > > > in > > > > > > Table > > > > > > > > Metadata properties). > > > > > > > > > > > > > > > > If clients also need a way to define it while interacting with > > > > > Polaris > > > > > > > over > > > > > > > > the IRC API, let's discuss how that can be done. > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Srinivas, > > > > > > > > > > > > > > > > > > I see that PR 4023 is active again. I'll try and post a > > fresh > > > > > review > > > > > > > > soon. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Hi JB and Dmitri, > > > > > > > > >> > > > > > > > > >> I raised PR #4023 < > > > https://github.com/apache/polaris/pull/4023> > > > > > > > > yesterday > > > > > > > > >> as part of the broader per-table storage config effort, and > > I > > > > > > believe > > > > > > > it > > > > > > > > >> implements exactly what you are proposing. > > > > > > > > >> > > > > > > > > >> Building on the named storage configs introduced in PR #3409 > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR > > allows > > > > > > > > namespaces > > > > > > > > >> and tables to specify a polaris.storage.name property. This > > > > acts > > > > > as > > > > > > > the > > > > > > > > >> "identifier" JB mentioned, allowing credential vending to > > > > resolve > > > > > > the > > > > > > > > >> correct, named storage config at runtime without introducing > > > new > > > > > > > > >> management > > > > > > > > >> APIs. > > > > > > > > >> > > > > > > > > >> Could you please take a look at the PR and let me know if > > this > > > > > > aligns > > > > > > > > with > > > > > > > > >> what you are looking for? > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
