Hi Srinivas, Thanks for putting this document together. I think it provides an excellent summary of this discussion and lays out clean path forward.
As for phase 1 it looks like we could start implementing it, if other reviews agree. Given that it's summer time it may take a week or two for everybody to catch up on this thread, though. >From a practical point of view, I tend to think that non-default storage config can be use effectively only at the namespace level. For a new table to be created via the IRC API, the request can only operate on the name/config stored in the immediate namespace or higher. The table itself cannot define the storage name unless we revisit the point of managing it only via Polaris properties. It might be worth allowing Iceberg table properties to carry the storage name too. Polaris can forward that information to its internal properties on changes. However, this point does not affect Phase 1, I think. We can add this kind of property forwarding later. Cheers, Dmitri. On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra <[email protected]> wrote: > I've put together an updated design proposal document detailing the storage > configuration management proposal. Please provide your feedback on the > proposal. > > > https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing > > > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Adam, > > > > Could you give some more details, please, about where you think the > Iceberg > > spec disallows multiple storage per table? > > > > I never thought it was not allowed. Perhaps it was inconvenient or > > impractical in existing clients, but the spec itself seems to be fairly > > unbiased with respect to actual storage technology... Perhaps I missed > > something, though. > > > > Thanks, > > Dmitri. > > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <[email protected]> wrote: > > > > > Hi Yufei, > > > > > > Regarding #2: > > > Would opting for a simple (assuming per table) storageConfigName or > > > storageConfigId > > > support future extensibility? > > > Asking from a multiple-storage-per-table perspective as it's something > > that > > > the Iceberg spec does allow IMHO. > > > > > > Cheers, > > > Adam > > > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> wrote: > > > > > > > Hi folks, thanks for the discussion. They are really helpful! > > > > > > > > I think there are two separate questions here: > > > > > > > > 1. How should we store storage configurations? As catalog properties > or > > > as > > > > separate entities? > > > > I'd lean toward separate entities in the long term. It provides a > > cleaner > > > > boundary between a catalog and its storage configurations, avoids > > > bloating > > > > the catalog object as the number of configs grows, and gives us a > > better > > > > foundation for future access control since entity level permissions > are > > > > easier to reason about. > > > > > > > > 2. How should a table or namespace refer to a storage configuration? > > > > I'd prefer storing a simple reference on the entity itself rather > than > > > > introducing a separate relationship table. Both approaches support > > > > hierarchical lookup and inheritance when a storage config is not > > > specified > > > > at the current level. However, I don't think we should allow multiple > > > > storage configs to be associated with a single table or namespace. > > Given > > > > that cardinality, a simple storageConfigName or storageConfigId field > > > seems > > > > sufficient and easier to understand than a dedicated relationship > > model. > > > > > > > > For example: > > > > > > > > Catalog > > > > ├─ entity: StorageConfigA > > > > ├─ entity: StorageConfigB > > > > └─ entity: StorageConfigC > > > > > > > > Table1 storage property -> StorageConfigA > > > > Table2 storage property -> StorageConfigB > > > > Table3 storage property -> StorageConfigC > > > > > > > > To me, this keeps the model simple while still supporting shared > > storage > > > > configurations and future extensibility. > > > > > > > > Yufei > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Robert, > > > > > > > > > > The distinction between "attached" and "referenced" config makes > > sense. > > > > > > > > > > My impression from JB's message on Mar 19 in this thread [1] is > that > > JB > > > > is > > > > > specifically proposing the "referenced" model, though. > > > > > > > > > > From my POV, both approaches can work, but the "referenced" model > > might > > > > be > > > > > more intuitive to users. > > > > > > > > > > [1] > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected]> > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > AFAIK the current Polaris code appears to resolve the storage > > > > > > configuration from the leaf (table), upwards via the namespace > > > elements > > > > > to > > > > > > the catalog, choosing the first one found. > > > > > > I am not sure whether that functionality is already used > anywhere? > > > > > > Being able to use a storage configuration on a namespace or a > leaf > > > > > appears > > > > > > appealing. > > > > > > > > > > > > OTOH I see a demand for named storage configurations. > > > > > > > > > > > > I wonder whether the mechanism to derive the storage > configuration > > > > could > > > > > > support both "directly attached" and "referenced" storage > > > > configurations: > > > > > > A namespace element or leaf can have either a "directly attached" > > > > storage > > > > > > configuration or a reference to a named one. > > > > > > > > > > > > We'd need management-ish APIs to handle the storage > configurations > > > for > > > > a > > > > > > namespace or leaf. > > > > > > These APIs, which do not exist yet, could start with "directly > > > > attached" > > > > > > storage configurations and once the authz questions are resolved, > > > > extend > > > > > to > > > > > > include named, shared storage configurations. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > To recap the discussion from the Community Sync today: My > > > > understanding > > > > > > is > > > > > > > that the consensus was: > > > > > > > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If more > than > > > one > > > > > > > Storage Config is present, they are to have different (within > the > > > > > > catalog) > > > > > > > names. Rishi to open a new PR for this (scoped to REST API + > > Sever > > > > > > > changes). > > > > > > > > > > > > > > * Defer CLI changes until we have the full end-to-end system > > > working > > > > > (it > > > > > > > should be usable via curl) > > > > > > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding > > > > support > > > > > > for > > > > > > > multiple Storage Configs > > > > > > > > > > > > > > * The connection between storage names and storage access > > > credentials > > > > > is > > > > > > to > > > > > > > remain unchanged for now. It is effectively a deployment time > > > concern > > > > > > given > > > > > > > the current OSS code. > > > > > > > > > > > > > > Please correct / add if I missed anything. > > > > > > > > > > > > > > From my POV, I think we still need to discuss how exactly > > entities > > > > > > (tables, > > > > > > > views) will link to Storage Config before we can resume PR > 4023. > > > > > > > > > > > > > > I propose storing the config name in the Entity properties (not > > in > > > > > Table > > > > > > > Metadata properties). > > > > > > > > > > > > > > If clients also need a way to define it while interacting with > > > > Polaris > > > > > > over > > > > > > > the IRC API, let's discuss how that can be done. > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Srinivas, > > > > > > > > > > > > > > > > I see that PR 4023 is active again. I'll try and post a > fresh > > > > review > > > > > > > soon. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Hi JB and Dmitri, > > > > > > > >> > > > > > > > >> I raised PR #4023 < > > https://github.com/apache/polaris/pull/4023> > > > > > > > yesterday > > > > > > > >> as part of the broader per-table storage config effort, and > I > > > > > believe > > > > > > it > > > > > > > >> implements exactly what you are proposing. > > > > > > > >> > > > > > > > >> Building on the named storage configs introduced in PR #3409 > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR > allows > > > > > > > namespaces > > > > > > > >> and tables to specify a polaris.storage.name property. This > > > acts > > > > as > > > > > > the > > > > > > > >> "identifier" JB mentioned, allowing credential vending to > > > resolve > > > > > the > > > > > > > >> correct, named storage config at runtime without introducing > > new > > > > > > > >> management > > > > > > > >> APIs. > > > > > > > >> > > > > > > > >> Could you please take a look at the PR and let me know if > this > > > > > aligns > > > > > > > with > > > > > > > >> what you are looking for? > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
