Hi Yufei, Regarding #2: Would opting for a simple (assuming per table) storageConfigName or storageConfigId support future extensibility? Asking from a multiple-storage-per-table perspective as it's something that the Iceberg spec does allow IMHO.
Cheers, Adam On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> wrote: > Hi folks, thanks for the discussion. They are really helpful! > > I think there are two separate questions here: > > 1. How should we store storage configurations? As catalog properties or as > separate entities? > I'd lean toward separate entities in the long term. It provides a cleaner > boundary between a catalog and its storage configurations, avoids bloating > the catalog object as the number of configs grows, and gives us a better > foundation for future access control since entity level permissions are > easier to reason about. > > 2. How should a table or namespace refer to a storage configuration? > I'd prefer storing a simple reference on the entity itself rather than > introducing a separate relationship table. Both approaches support > hierarchical lookup and inheritance when a storage config is not specified > at the current level. However, I don't think we should allow multiple > storage configs to be associated with a single table or namespace. Given > that cardinality, a simple storageConfigName or storageConfigId field seems > sufficient and easier to understand than a dedicated relationship model. > > For example: > > Catalog > ├─ entity: StorageConfigA > ├─ entity: StorageConfigB > └─ entity: StorageConfigC > > Table1 storage property -> StorageConfigA > Table2 storage property -> StorageConfigB > Table3 storage property -> StorageConfigC > > To me, this keeps the model simple while still supporting shared storage > configurations and future extensibility. > > Yufei > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Robert, > > > > The distinction between "attached" and "referenced" config makes sense. > > > > My impression from JB's message on Mar 19 in this thread [1] is that JB > is > > specifically proposing the "referenced" model, though. > > > > From my POV, both approaches can work, but the "referenced" model might > be > > more intuitive to users. > > > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > Cheers, > > Dmitri. > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected]> wrote: > > > > > Hi all, > > > > > > AFAIK the current Polaris code appears to resolve the storage > > > configuration from the leaf (table), upwards via the namespace elements > > to > > > the catalog, choosing the first one found. > > > I am not sure whether that functionality is already used anywhere? > > > Being able to use a storage configuration on a namespace or a leaf > > appears > > > appealing. > > > > > > OTOH I see a demand for named storage configurations. > > > > > > I wonder whether the mechanism to derive the storage configuration > could > > > support both "directly attached" and "referenced" storage > configurations: > > > A namespace element or leaf can have either a "directly attached" > storage > > > configuration or a reference to a named one. > > > > > > We'd need management-ish APIs to handle the storage configurations for > a > > > namespace or leaf. > > > These APIs, which do not exist yet, could start with "directly > attached" > > > storage configurations and once the authz questions are resolved, > extend > > to > > > include named, shared storage configurations. > > > > > > Thoughts? > > > > > > Robert > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > To recap the discussion from the Community Sync today: My > understanding > > > is > > > > that the consensus was: > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > * Allow multiple Storage Config objects per Catalog. If more than one > > > > Storage Config is present, they are to have different (within the > > > catalog) > > > > names. Rishi to open a new PR for this (scoped to REST API + Sever > > > > changes). > > > > > > > > * Defer CLI changes until we have the full end-to-end system working > > (it > > > > should be usable via curl) > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding > support > > > for > > > > multiple Storage Configs > > > > > > > > * The connection between storage names and storage access credentials > > is > > > to > > > > remain unchanged for now. It is effectively a deployment time concern > > > given > > > > the current OSS code. > > > > > > > > Please correct / add if I missed anything. > > > > > > > > From my POV, I think we still need to discuss how exactly entities > > > (tables, > > > > views) will link to Storage Config before we can resume PR 4023. > > > > > > > > I propose storing the config name in the Entity properties (not in > > Table > > > > Metadata properties). > > > > > > > > If clients also need a way to define it while interacting with > Polaris > > > over > > > > the IRC API, let's discuss how that can be done. > > > > > > > > WDYT? > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <[email protected] > > > > > > wrote: > > > > > > > > > Hi Srinivas, > > > > > > > > > > I see that PR 4023 is active again. I'll try and post a fresh > review > > > > soon. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > [email protected]> > > > > > wrote: > > > > > > > > > >> Hi JB and Dmitri, > > > > >> > > > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023> > > > > yesterday > > > > >> as part of the broader per-table storage config effort, and I > > believe > > > it > > > > >> implements exactly what you are proposing. > > > > >> > > > > >> Building on the named storage configs introduced in PR #3409 > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows > > > > namespaces > > > > >> and tables to specify a polaris.storage.name property. This acts > as > > > the > > > > >> "identifier" JB mentioned, allowing credential vending to resolve > > the > > > > >> correct, named storage config at runtime without introducing new > > > > >> management > > > > >> APIs. > > > > >> > > > > >> Could you please take a look at the PR and let me know if this > > aligns > > > > with > > > > >> what you are looking for? > > > > >> > > > > > > > > > > > > > > >
