Thanks Yufei, Here is the first PR https://github.com/apache/polaris/pull/5556
On Tue, Sep 8, 2026 at 5:03 PM Yufei Gu <[email protected]> wrote: > Hi Srinivas, Thanks for putting together the design doc and moving > this forward. The design doc looks good to me overall. Looking forward > to the PRs! > > Yufei > > On Fri, Aug 28, 2026 at 9:35 AM Alexandre Dutra <[email protected]> wrote: > > > > Hi Srinivas, hi all, > > > > As Dmitri said, summer time indeed has slowed down the community > > bandwidth, but I'm happy to get back to this topic today. > > > > Thanks for the thorough writeup and also the summary of the current > > status quo: very useful. I'm OK with both the two-phase framing and > > the fail-closed approach. > > > > I have however one concern on Phase 1 that I think is worth settling > > before we move forward: the storageName field is being **overloaded to > > do two independent jobs**. > > > > Today (per #3409) this field has exactly one purpose: select the > > deployment-time credential set. Phase 1 quietly adds a second: it > > becomes the identity key of the named-config map and the target of > > polaris.storage.name. Those are two different namespaces: a > > catalog-scoped config identity vs. a deployment-scoped credential > > selector; welding them together means you can't share one credential > > set across configs, and you can't rename a config without re-pointing > > its credentials. > > > > This surfaces as two concrete rough edges: > > > > 1. The default config's storageName is ambiguous. §5.1 excludes the > > default's name from the named map, but §5.4 resolves references only > > against that map. So if the default carries storageName = "foo" and a > > table sets polaris.storage.name = "foo", the lookup misses and we fail > > closed with a "dangling reference", even though a config named "foo" > > plainly exists. **The default config having a name at all is a code > > smell.** > > > > 2. The two usages can disagree. If named references are found in the > > hierarchy, but RESOLVE_CREDENTIALS_BY_STORAGE_NAME is off, we'd > > resolve a table to a named storage config, but wouldn't pick named > > credentials for it, falling back to the default provider chain. That's > > a silent fail-open path inside a design tailored for fail-closed. > > > > My proposal: decouple the two concepts, and use a real map of name -> > > StorageConfigInfo in Create/UpdateCatalogRequest; credential selection > > keeps using the current storageName field (we could maybe rename it). > > The map key becomes the reference identity; uniqueness comes for free. > > That fully decouples the two concerns: several configs can share one > > credential set, and the default no longer needs a name. > > > > WDYT? > > > > Thanks, > > Alex > > > > On Mon, Aug 10, 2026 at 7:04 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > > > Hi Srinivas, > > > > > > Thanks for putting this document together. I think it provides an > excellent > > > summary of this discussion and lays out clean path forward. > > > > > > As for phase 1 it looks like we could start implementing it, if other > > > reviews agree. Given that it's summer time it may take a week or two > for > > > everybody to catch up on this thread, though. > > > > > > From a practical point of view, I tend to think that non-default > storage > > > config can be use effectively only at the namespace level. For a new > table > > > to be created via the IRC API, the request can only operate on the > > > name/config stored in the immediate namespace or higher. The table > itself > > > cannot define the storage name unless we revisit the point of managing > it > > > only via Polaris properties. > > > > > > It might be worth allowing Iceberg table properties to carry the > storage > > > name too. Polaris can forward that information to its internal > properties > > > on changes. However, this point does not affect Phase 1, I think. We > can > > > add this kind of property forwarding later. > > > > > > Cheers, > > > Dmitri. > > > > > > On Wed, Aug 5, 2026 at 10:51 AM Srinivas Rishindra < > [email protected]> > > > wrote: > > > > > > > I've put together an updated design proposal document detailing the > storage > > > > configuration management proposal. Please provide your feedback on > the > > > > proposal. > > > > > > > > > > > > > https://docs.google.com/document/d/1TSaZ0G2oEoWq6lDMSPIIfHLwXxW9VZmhUNE3s_jfxgI/edit?usp=sharing > > > > > > > > > > > > > > > > On Thu, Jun 18, 2026 at 3:23 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Adam, > > > > > > > > > > Could you give some more details, please, about where you think the > > > > Iceberg > > > > > spec disallows multiple storage per table? > > > > > > > > > > I never thought it was not allowed. Perhaps it was inconvenient or > > > > > impractical in existing clients, but the spec itself seems to be > fairly > > > > > unbiased with respect to actual storage technology... Perhaps I > missed > > > > > something, though. > > > > > > > > > > Thanks, > > > > > Dmitri. > > > > > > > > > > On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <[email protected]> > wrote: > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > Regarding #2: > > > > > > Would opting for a simple (assuming per table) storageConfigName > or > > > > > > storageConfigId > > > > > > support future extensibility? > > > > > > Asking from a multiple-storage-per-table perspective as it's > something > > > > > that > > > > > > the Iceberg spec does allow IMHO. > > > > > > > > > > > > Cheers, > > > > > > Adam > > > > > > > > > > > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> > wrote: > > > > > > > > > > > > > Hi folks, thanks for the discussion. They are really helpful! > > > > > > > > > > > > > > I think there are two separate questions here: > > > > > > > > > > > > > > 1. How should we store storage configurations? As catalog > properties > > > > or > > > > > > as > > > > > > > separate entities? > > > > > > > I'd lean toward separate entities in the long term. It > provides a > > > > > cleaner > > > > > > > boundary between a catalog and its storage configurations, > avoids > > > > > > bloating > > > > > > > the catalog object as the number of configs grows, and gives > us a > > > > > better > > > > > > > foundation for future access control since entity level > permissions > > > > are > > > > > > > easier to reason about. > > > > > > > > > > > > > > 2. How should a table or namespace refer to a storage > configuration? > > > > > > > I'd prefer storing a simple reference on the entity itself > rather > > > > than > > > > > > > introducing a separate relationship table. Both approaches > support > > > > > > > hierarchical lookup and inheritance when a storage config is > not > > > > > > specified > > > > > > > at the current level. However, I don't think we should allow > multiple > > > > > > > storage configs to be associated with a single table or > namespace. > > > > > Given > > > > > > > that cardinality, a simple storageConfigName or > storageConfigId field > > > > > > seems > > > > > > > sufficient and easier to understand than a dedicated > relationship > > > > > model. > > > > > > > > > > > > > > For example: > > > > > > > > > > > > > > Catalog > > > > > > > ├─ entity: StorageConfigA > > > > > > > ├─ entity: StorageConfigB > > > > > > > └─ entity: StorageConfigC > > > > > > > > > > > > > > Table1 storage property -> StorageConfigA > > > > > > > Table2 storage property -> StorageConfigB > > > > > > > Table3 storage property -> StorageConfigC > > > > > > > > > > > > > > To me, this keeps the model simple while still supporting > shared > > > > > storage > > > > > > > configurations and future extensibility. > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Robert, > > > > > > > > > > > > > > > > The distinction between "attached" and "referenced" config > makes > > > > > sense. > > > > > > > > > > > > > > > > My impression from JB's message on Mar 19 in this thread [1] > is > > > > that > > > > > JB > > > > > > > is > > > > > > > > specifically proposing the "referenced" model, though. > > > > > > > > > > > > > > > > From my POV, both approaches can work, but the "referenced" > model > > > > > might > > > > > > > be > > > > > > > > more intuitive to users. > > > > > > > > > > > > > > > > [1] > > > > https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > AFAIK the current Polaris code appears to resolve the > storage > > > > > > > > > configuration from the leaf (table), upwards via the > namespace > > > > > > elements > > > > > > > > to > > > > > > > > > the catalog, choosing the first one found. > > > > > > > > > I am not sure whether that functionality is already used > > > > anywhere? > > > > > > > > > Being able to use a storage configuration on a namespace > or a > > > > leaf > > > > > > > > appears > > > > > > > > > appealing. > > > > > > > > > > > > > > > > > > OTOH I see a demand for named storage configurations. > > > > > > > > > > > > > > > > > > I wonder whether the mechanism to derive the storage > > > > configuration > > > > > > > could > > > > > > > > > support both "directly attached" and "referenced" storage > > > > > > > configurations: > > > > > > > > > A namespace element or leaf can have either a "directly > attached" > > > > > > > storage > > > > > > > > > configuration or a reference to a named one. > > > > > > > > > > > > > > > > > > We'd need management-ish APIs to handle the storage > > > > configurations > > > > > > for > > > > > > > a > > > > > > > > > namespace or leaf. > > > > > > > > > These APIs, which do not exist yet, could start with > "directly > > > > > > > attached" > > > > > > > > > storage configurations and once the authz questions are > resolved, > > > > > > > extend > > > > > > > > to > > > > > > > > > include named, shared storage configurations. > > > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > To recap the discussion from the Community Sync today: My > > > > > > > understanding > > > > > > > > > is > > > > > > > > > > that the consensus was: > > > > > > > > > > > > > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > > > > > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If > more > > > > than > > > > > > one > > > > > > > > > > Storage Config is present, they are to have different > (within > > > > the > > > > > > > > > catalog) > > > > > > > > > > names. Rishi to open a new PR for this (scoped to REST > API + > > > > > Sever > > > > > > > > > > changes). > > > > > > > > > > > > > > > > > > > > * Defer CLI changes until we have the full end-to-end > system > > > > > > working > > > > > > > > (it > > > > > > > > > > should be usable via curl) > > > > > > > > > > > > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after > adding > > > > > > > support > > > > > > > > > for > > > > > > > > > > multiple Storage Configs > > > > > > > > > > > > > > > > > > > > * The connection between storage names and storage access > > > > > > credentials > > > > > > > > is > > > > > > > > > to > > > > > > > > > > remain unchanged for now. It is effectively a deployment > time > > > > > > concern > > > > > > > > > given > > > > > > > > > > the current OSS code. > > > > > > > > > > > > > > > > > > > > Please correct / add if I missed anything. > > > > > > > > > > > > > > > > > > > > From my POV, I think we still need to discuss how exactly > > > > > entities > > > > > > > > > (tables, > > > > > > > > > > views) will link to Storage Config before we can resume > PR > > > > 4023. > > > > > > > > > > > > > > > > > > > > I propose storing the config name in the Entity > properties (not > > > > > in > > > > > > > > Table > > > > > > > > > > Metadata properties). > > > > > > > > > > > > > > > > > > > > If clients also need a way to define it while > interacting with > > > > > > > Polaris > > > > > > > > > over > > > > > > > > > > the IRC API, let's discuss how that can be done. > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Srinivas, > > > > > > > > > > > > > > > > > > > > > > I see that PR 4023 is active again. I'll try and post > a > > > > fresh > > > > > > > review > > > > > > > > > > soon. > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> Hi JB and Dmitri, > > > > > > > > > > >> > > > > > > > > > > >> I raised PR #4023 < > > > > > https://github.com/apache/polaris/pull/4023> > > > > > > > > > > yesterday > > > > > > > > > > >> as part of the broader per-table storage config > effort, and > > > > I > > > > > > > > believe > > > > > > > > > it > > > > > > > > > > >> implements exactly what you are proposing. > > > > > > > > > > >> > > > > > > > > > > >> Building on the named storage configs introduced in > PR #3409 > > > > > > > > > > >> <https://github.com/apache/polaris/pull/3409>, this > PR > > > > allows > > > > > > > > > > namespaces > > > > > > > > > > >> and tables to specify a polaris.storage.name > property. This > > > > > > acts > > > > > > > as > > > > > > > > > the > > > > > > > > > > >> "identifier" JB mentioned, allowing credential > vending to > > > > > > resolve > > > > > > > > the > > > > > > > > > > >> correct, named storage config at runtime without > introducing > > > > > new > > > > > > > > > > >> management > > > > > > > > > > >> APIs. > > > > > > > > > > >> > > > > > > > > > > >> Could you please take a look at the PR and let me > know if > > > > this > > > > > > > > aligns > > > > > > > > > > with > > > > > > > > > > >> what you are looking for? > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
