I've put together an updated design proposal document detailing the storage configuration management proposal. Please provide your feedback on the proposal.
https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Adam, > > Could you give some more details, please, about where you think the Iceberg > spec disallows multiple storage per table? > > I never thought it was not allowed. Perhaps it was inconvenient or > impractical in existing clients, but the spec itself seems to be fairly > unbiased with respect to actual storage technology... Perhaps I missed > something, though. > > Thanks, > Dmitri. > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <[email protected]> wrote: > > > Hi Yufei, > > > > Regarding #2: > > Would opting for a simple (assuming per table) storageConfigName or > > storageConfigId > > support future extensibility? > > Asking from a multiple-storage-per-table perspective as it's something > that > > the Iceberg spec does allow IMHO. > > > > Cheers, > > Adam > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> wrote: > > > > > Hi folks, thanks for the discussion. They are really helpful! > > > > > > I think there are two separate questions here: > > > > > > 1. How should we store storage configurations? As catalog properties or > > as > > > separate entities? > > > I'd lean toward separate entities in the long term. It provides a > cleaner > > > boundary between a catalog and its storage configurations, avoids > > bloating > > > the catalog object as the number of configs grows, and gives us a > better > > > foundation for future access control since entity level permissions are > > > easier to reason about. > > > > > > 2. How should a table or namespace refer to a storage configuration? > > > I'd prefer storing a simple reference on the entity itself rather than > > > introducing a separate relationship table. Both approaches support > > > hierarchical lookup and inheritance when a storage config is not > > specified > > > at the current level. However, I don't think we should allow multiple > > > storage configs to be associated with a single table or namespace. > Given > > > that cardinality, a simple storageConfigName or storageConfigId field > > seems > > > sufficient and easier to understand than a dedicated relationship > model. > > > > > > For example: > > > > > > Catalog > > > ├─ entity: StorageConfigA > > > ├─ entity: StorageConfigB > > > └─ entity: StorageConfigC > > > > > > Table1 storage property -> StorageConfigA > > > Table2 storage property -> StorageConfigB > > > Table3 storage property -> StorageConfigC > > > > > > To me, this keeps the model simple while still supporting shared > storage > > > configurations and future extensibility. > > > > > > Yufei > > > > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Robert, > > > > > > > > The distinction between "attached" and "referenced" config makes > sense. > > > > > > > > My impression from JB's message on Mar 19 in this thread [1] is that > JB > > > is > > > > specifically proposing the "referenced" model, though. > > > > > > > > From my POV, both approaches can work, but the "referenced" model > might > > > be > > > > more intuitive to users. > > > > > > > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected]> wrote: > > > > > > > > > Hi all, > > > > > > > > > > AFAIK the current Polaris code appears to resolve the storage > > > > > configuration from the leaf (table), upwards via the namespace > > elements > > > > to > > > > > the catalog, choosing the first one found. > > > > > I am not sure whether that functionality is already used anywhere? > > > > > Being able to use a storage configuration on a namespace or a leaf > > > > appears > > > > > appealing. > > > > > > > > > > OTOH I see a demand for named storage configurations. > > > > > > > > > > I wonder whether the mechanism to derive the storage configuration > > > could > > > > > support both "directly attached" and "referenced" storage > > > configurations: > > > > > A namespace element or leaf can have either a "directly attached" > > > storage > > > > > configuration or a reference to a named one. > > > > > > > > > > We'd need management-ish APIs to handle the storage configurations > > for > > > a > > > > > namespace or leaf. > > > > > These APIs, which do not exist yet, could start with "directly > > > attached" > > > > > storage configurations and once the authz questions are resolved, > > > extend > > > > to > > > > > include named, shared storage configurations. > > > > > > > > > > Thoughts? > > > > > > > > > > Robert > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > To recap the discussion from the Community Sync today: My > > > understanding > > > > > is > > > > > > that the consensus was: > > > > > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If more than > > one > > > > > > Storage Config is present, they are to have different (within the > > > > > catalog) > > > > > > names. Rishi to open a new PR for this (scoped to REST API + > Sever > > > > > > changes). > > > > > > > > > > > > * Defer CLI changes until we have the full end-to-end system > > working > > > > (it > > > > > > should be usable via curl) > > > > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding > > > support > > > > > for > > > > > > multiple Storage Configs > > > > > > > > > > > > * The connection between storage names and storage access > > credentials > > > > is > > > > > to > > > > > > remain unchanged for now. It is effectively a deployment time > > concern > > > > > given > > > > > > the current OSS code. > > > > > > > > > > > > Please correct / add if I missed anything. > > > > > > > > > > > > From my POV, I think we still need to discuss how exactly > entities > > > > > (tables, > > > > > > views) will link to Storage Config before we can resume PR 4023. > > > > > > > > > > > > I propose storing the config name in the Entity properties (not > in > > > > Table > > > > > > Metadata properties). > > > > > > > > > > > > If clients also need a way to define it while interacting with > > > Polaris > > > > > over > > > > > > the IRC API, let's discuss how that can be done. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Srinivas, > > > > > > > > > > > > > > I see that PR 4023 is active again. I'll try and post a fresh > > > review > > > > > > soon. > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > >> Hi JB and Dmitri, > > > > > > >> > > > > > > >> I raised PR #4023 < > https://github.com/apache/polaris/pull/4023> > > > > > > yesterday > > > > > > >> as part of the broader per-table storage config effort, and I > > > > believe > > > > > it > > > > > > >> implements exactly what you are proposing. > > > > > > >> > > > > > > >> Building on the named storage configs introduced in PR #3409 > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows > > > > > > namespaces > > > > > > >> and tables to specify a polaris.storage.name property. This > > acts > > > as > > > > > the > > > > > > >> "identifier" JB mentioned, allowing credential vending to > > resolve > > > > the > > > > > > >> correct, named storage config at runtime without introducing > new > > > > > > >> management > > > > > > >> APIs. > > > > > > >> > > > > > > >> Could you please take a look at the PR and let me know if this > > > > aligns > > > > > > with > > > > > > >> what you are looking for? > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
