Hi folks, While reviewing the storage configuration PR (#5721), we identified several issues with the Phase 1 design. We think the project would benefit from skipping Phase 1 and going directly to Phase 2, where storage configuration becomes a first-class entity. I’d like to hear what the community thinks.
As currently proposed, Phase 1 adds support for multiple storage configurations within the catalog’s existing StorageConfigInfo. Phase 2 would make storage configuration a separate entity with its own REST endpoints. Phase 1 was always intended as a transitional step. Issues we found with Phase 1: 1. *Default base location semantics.* Today, allowedLocations can be empty, and the catalog’s default-base-location is added to it automatically. With multiple storage configurations, for example, several buckets or a mix of S3 and Azure, the catalog default may not apply to every configuration. A catalog default pointing to S3 would not work for an Azure configuration. Each storage configuration would therefore need its own default base location. Without an applicable default, a create-table request that omits an explicit location would fail. 2. *Backward compatibility.* Adding a default base location to StorageConfigInfo and giving it precedence over the catalog-level property would change behavior for existing users, who expect the default to come from the catalog properties. Handling this cleanly would require deprecation and precedence rules, adding complexity to an interface we already plan to replace. 3. *Coarse-grained updates.* Because the configurations live inside the catalog, changing a single storage configuration means updating the whole catalog object. Supporting targeted updates would require REST endpoints similar to those proposed for Phase 2. 4. *No fine-grained access control.* Phase 1 ties storage configuration management to catalog administration. Users who need to manage a storage configuration would need catalog admin privileges. Separating those permissions would require additional design work. 5. *Migration cost and user churn.* Shipping Phase 1 would require us to later migrate configurations to the Phase 2 model. Users could see an interface introduced, deprecated, and replaced within a short period. We expect Phase 1 to require a similar amount of work to Phase 2, with much of that work eventually replaced. We may skip Phase 1 and implement storage configuration directly as a first-class entity with its own REST API. There are two possible approaches: - *Option A: Reuse the policy framework.* A storage configuration could be modeled as a new policy type that can be attached at the catalog, namespace, and table levels. If the community agrees that storage configuration fits the policy model, this may be the simplest path. - *Option B: Introduce a dedicated storage configuration entity.* We could follow a pattern similar to the Tag proposal (PR #5366), where tags live under a catalog and have their own CRUD endpoints. Applied here, users would create a catalog and then create one or more storage configurations under it. We also need to decide whether storage configurations should be catalog-scoped or top-level. Both choices have trade-offs. I’d especially appreciate feedback on these questions: 1. Are there objections to skipping Phase 1? 2. Do you prefer Option A (reuse the policy framework) or Option B (a dedicated entity)? 3. Should storage configurations be catalog-scoped or top-level? Yufei On Tue, Oct 20, 2026 at 2:33 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > Srinivas opened PR [5721] for the Management API change related to > supporting multiple storage configurations per catalog. > > Please take a look. > > Thanks, > Dmitri. > > [5721] https://github.com/apache/polaris/pull/5721 > > On Fri, Aug 28, 2026 at 12:35 PM Alexandre Dutra <[email protected]> > wrote: > > > Hi Srinivas, hi all, > > > > As Dmitri said, summer time indeed has slowed down the community > > bandwidth, but I'm happy to get back to this topic today. > > > > Thanks for the thorough writeup and also the summary of the current > > status quo: very useful. I'm OK with both the two-phase framing and > > the fail-closed approach. > > > > I have however one concern on Phase 1 that I think is worth settling > > before we move forward: the storageName field is being **overloaded to > > do two independent jobs**. > > > > Today (per #3409) this field has exactly one purpose: select the > > deployment-time credential set. Phase 1 quietly adds a second: it > > becomes the identity key of the named-config map and the target of > > polaris.storage.name. Those are two different namespaces: a > > catalog-scoped config identity vs. a deployment-scoped credential > > selector; welding them together means you can't share one credential > > set across configs, and you can't rename a config without re-pointing > > its credentials. > > > > This surfaces as two concrete rough edges: > > > > 1. The default config's storageName is ambiguous. §5.1 excludes the > > default's name from the named map, but §5.4 resolves references only > > against that map. So if the default carries storageName = "foo" and a > > table sets polaris.storage.name = "foo", the lookup misses and we fail > > closed with a "dangling reference", even though a config named "foo" > > plainly exists. **The default config having a name at all is a code > > smell.** > > > > 2. The two usages can disagree. If named references are found in the > > hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd > > resolve a table to a named storage config, but wouldn't pick named > > credentials for it, falling back to the default provider chain. That's > > a silent fail-open path inside a design tailored for fail-closed. > > > > My proposal: decouple the two concepts, and use a real map of name -> > > StorageConfigInfo in Create/UpdateCatalogRequest; credential selection > > keeps using the current storageName field (we could maybe rename it). > > The map key becomes the reference identity; uniqueness comes for free. > > That fully decouples the two concerns: several configs can share one > > credential set, and the default no longer needs a name. > > > > WDYT? > > > > Thanks, > > Alex > > > > On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > > Hi Srinivas, > > > > > > Thanks for putting this document together. I think it provides an > > excellent > > > summary of this discussion and lays out clean path forward. > > > > > > As for phase 1 it looks like we could start implementing it, if other > > > reviews agree. Given that it's summer time it may take a week or two > for > > > everybody to catch up on this thread, though. > > > > > > From a practical point of view, I tend to think that non-default > storage > > > config can be use effectively only at the namespace level. For a new > > table > > > to be created via the IRC API, the request can only operate on the > > > name/config stored in the immediate namespace or higher. The table > itself > > > cannot define the storage name unless we revisit the point of managing > it > > > only via Polaris properties. > > > > > > It might be worth allowing Iceberg table properties to carry the > storage > > > name too. Polaris can forward that information to its internal > properties > > > on changes. However, this point does not affect Phase 1, I think. We > can > > > add this kind of property forwarding later. > > > > > > Cheers, > > > Dmitri. > > > > > > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra < > > [email protected]> > > > wrote: > > > > > > > I've put together an updated design proposal document detailing the > > storage > > > > configuration management proposal. Please provide your feedback on > the > > > > proposal. > > > > > > > > > > > > > > > https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing > > > > > > > > > > > > > > > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Adam, > > > > > > > > > > Could you give some more details, please, about where you think the > > > > Iceberg > > > > > spec disallows multiple storage per table? > > > > > > > > > > I never thought it was not allowed. Perhaps it was inconvenient or > > > > > impractical in existing clients, but the spec itself seems to be > > fairly > > > > > unbiased with respect to actual storage technology... Perhaps I > > missed > > > > > something, though. > > > > > > > > > > Thanks, > > > > > Dmitri. > > > > > > > > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <[email protected]> > wrote: > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > Regarding #2: > > > > > > Would opting for a simple (assuming per table) storageConfigName > or > > > > > > storageConfigId > > > > > > support future extensibility? > > > > > > Asking from a multiple-storage-per-table perspective as it's > > something > > > > > that > > > > > > the Iceberg spec does allow IMHO. > > > > > > > > > > > > Cheers, > > > > > > Adam > > > > > > > > > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> > > wrote: > > > > > > > > > > > > > Hi folks, thanks for the discussion. They are really helpful! > > > > > > > > > > > > > > I think there are two separate questions here: > > > > > > > > > > > > > > 1. How should we store storage configurations? As catalog > > properties > > > > or > > > > > > as > > > > > > > separate entities? > > > > > > > I'd lean toward separate entities in the long term. It > provides a > > > > > cleaner > > > > > > > boundary between a catalog and its storage configurations, > avoids > > > > > > bloating > > > > > > > the catalog object as the number of configs grows, and gives > us a > > > > > better > > > > > > > foundation for future access control since entity level > > permissions > > > > are > > > > > > > easier to reason about. > > > > > > > > > > > > > > 2. How should a table or namespace refer to a storage > > configuration? > > > > > > > I'd prefer storing a simple reference on the entity itself > rather > > > > than > > > > > > > introducing a separate relationship table. Both approaches > > support > > > > > > > hierarchical lookup and inheritance when a storage config is > not > > > > > > specified > > > > > > > at the current level. However, I don't think we should allow > > multiple > > > > > > > storage configs to be associated with a single table or > > namespace. > > > > > Given > > > > > > > that cardinality, a simple storageConfigName or storageConfigId > > field > > > > > > seems > > > > > > > sufficient and easier to understand than a dedicated > relationship > > > > > model. > > > > > > > > > > > > > > For example: > > > > > > > > > > > > > > Catalog > > > > > > > ├─ entity: StorageConfigA > > > > > > > ├─ entity: StorageConfigB > > > > > > > └─ entity: StorageConfigC > > > > > > > > > > > > > > Table1 storage property -> StorageConfigA > > > > > > > Table2 storage property -> StorageConfigB > > > > > > > Table3 storage property -> StorageConfigC > > > > > > > > > > > > > > To me, this keeps the model simple while still supporting > shared > > > > > storage > > > > > > > configurations and future extensibility. > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Robert, > > > > > > > > > > > > > > > > The distinction between "attached" and "referenced" config > > makes > > > > > sense. > > > > > > > > > > > > > > > > My impression from JB's message on Mar 19 in this thread [1] > is > > > > that > > > > > JB > > > > > > > is > > > > > > > > specifically proposing the "referenced" model, though. > > > > > > > > > > > > > > > > From my POV, both approaches can work, but the "referenced" > > model > > > > > might > > > > > > > be > > > > > > > > more intuitive to users. > > > > > > > > > > > > > > > > [1] > > > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > AFAIK the current Polaris code appears to resolve the > storage > > > > > > > > > configuration from the leaf (table), upwards via the > > namespace > > > > > > elements > > > > > > > > to > > > > > > > > > the catalog, choosing the first one found. > > > > > > > > > I am not sure whether that functionality is already used > > > > anywhere? > > > > > > > > > Being able to use a storage configuration on a namespace > or a > > > > leaf > > > > > > > > appears > > > > > > > > > appealing. > > > > > > > > > > > > > > > > > > OTOH I see a demand for named storage configurations. > > > > > > > > > > > > > > > > > > I wonder whether the mechanism to derive the storage > > > > configuration > > > > > > > could > > > > > > > > > support both "directly attached" and "referenced" storage > > > > > > > configurations: > > > > > > > > > A namespace element or leaf can have either a "directly > > attached" > > > > > > > storage > > > > > > > > > configuration or a reference to a named one. > > > > > > > > > > > > > > > > > > We'd need management-ish APIs to handle the storage > > > > configurations > > > > > > for > > > > > > > a > > > > > > > > > namespace or leaf. > > > > > > > > > These APIs, which do not exist yet, could start with > > "directly > > > > > > > attached" > > > > > > > > > storage configurations and once the authz questions are > > resolved, > > > > > > > extend > > > > > > > > to > > > > > > > > > include named, shared storage configurations. > > > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > To recap the discussion from the Community Sync today: My > > > > > > > understanding > > > > > > > > > is > > > > > > > > > > that the consensus was: > > > > > > > > > > > > > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > > > > > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If > > more > > > > than > > > > > > one > > > > > > > > > > Storage Config is present, they are to have different > > (within > > > > the > > > > > > > > > catalog) > > > > > > > > > > names. Rishi to open a new PR for this (scoped to REST > API > > + > > > > > Sever > > > > > > > > > > changes). > > > > > > > > > > > > > > > > > > > > * Defer CLI changes until we have the full end-to-end > > system > > > > > > working > > > > > > > > (it > > > > > > > > > > should be usable via curl) > > > > > > > > > > > > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after > > adding > > > > > > > support > > > > > > > > > for > > > > > > > > > > multiple Storage Configs > > > > > > > > > > > > > > > > > > > > * The connection between storage names and storage access > > > > > > credentials > > > > > > > > is > > > > > > > > > to > > > > > > > > > > remain unchanged for now. It is effectively a deployment > > time > > > > > > concern > > > > > > > > > given > > > > > > > > > > the current OSS code. > > > > > > > > > > > > > > > > > > > > Please correct / add if I missed anything. > > > > > > > > > > > > > > > > > > > > From my POV, I think we still need to discuss how exactly > > > > > entities > > > > > > > > > (tables, > > > > > > > > > > views) will link to Storage Config before we can resume > PR > > > > 4023. > > > > > > > > > > > > > > > > > > > > I propose storing the config name in the Entity > properties > > (not > > > > > in > > > > > > > > Table > > > > > > > > > > Metadata properties). > > > > > > > > > > > > > > > > > > > > If clients also need a way to define it while interacting > > with > > > > > > > Polaris > > > > > > > > > over > > > > > > > > > > the IRC API, let's discuss how that can be done. > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Srinivas, > > > > > > > > > > > > > > > > > > > > > > I see that PR 4023 is active again. I'll try and post > a > > > > fresh > > > > > > > review > > > > > > > > > > soon. > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> Hi JB and Dmitri, > > > > > > > > > > >> > > > > > > > > > > >> I raised PR #4023 < > > > > > https://github.com/apache/polaris/pull/4023> > > > > > > > > > > yesterday > > > > > > > > > > >> as part of the broader per-table storage config > effort, > > and > > > > I > > > > > > > > believe > > > > > > > > > it > > > > > > > > > > >> implements exactly what you are proposing. > > > > > > > > > > >> > > > > > > > > > > >> Building on the named storage configs introduced in PR > > #3409 > > > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this > PR > > > > allows > > > > > > > > > > namespaces > > > > > > > > > > >> and tables to specify a polaris.storage.name > property. > > This > > > > > > acts > > > > > > > as > > > > > > > > > the > > > > > > > > > > >> "identifier" JB mentioned, allowing credential vending > > to > > > > > > resolve > > > > > > > > the > > > > > > > > > > >> correct, named storage config at runtime without > > introducing > > > > > new > > > > > > > > > > >> management > > > > > > > > > > >> APIs. > > > > > > > > > > >> > > > > > > > > > > >> Could you please take a look at the PR and let me know > > if > > > > this > > > > > > > > aligns > > > > > > > > > > with > > > > > > > > > > >> what you are looking for? > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
