Hi folks, thanks for the discussion. They are really helpful! I think there are two separate questions here:
1. How should we store storage configurations? As catalog properties or as separate entities? I'd lean toward separate entities in the long term. It provides a cleaner boundary between a catalog and its storage configurations, avoids bloating the catalog object as the number of configs grows, and gives us a better foundation for future access control since entity level permissions are easier to reason about. 2. How should a table or namespace refer to a storage configuration? I'd prefer storing a simple reference on the entity itself rather than introducing a separate relationship table. Both approaches support hierarchical lookup and inheritance when a storage config is not specified at the current level. However, I don't think we should allow multiple storage configs to be associated with a single table or namespace. Given that cardinality, a simple storageConfigName or storageConfigId field seems sufficient and easier to understand than a dedicated relationship model. For example: Catalog ├─ entity: StorageConfigA ├─ entity: StorageConfigB └─ entity: StorageConfigC Table1 storage property -> StorageConfigA Table2 storage property -> StorageConfigB Table3 storage property -> StorageConfigC To me, this keeps the model simple while still supporting shared storage configurations and future extensibility. Yufei On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Robert, > > The distinction between "attached" and "referenced" config makes sense. > > My impression from JB's message on Mar 19 in this thread [1] is that JB is > specifically proposing the "referenced" model, though. > > From my POV, both approaches can work, but the "referenced" model might be > more intuitive to users. > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > Cheers, > Dmitri. > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected]> wrote: > > > Hi all, > > > > AFAIK the current Polaris code appears to resolve the storage > > configuration from the leaf (table), upwards via the namespace elements > to > > the catalog, choosing the first one found. > > I am not sure whether that functionality is already used anywhere? > > Being able to use a storage configuration on a namespace or a leaf > appears > > appealing. > > > > OTOH I see a demand for named storage configurations. > > > > I wonder whether the mechanism to derive the storage configuration could > > support both "directly attached" and "referenced" storage configurations: > > A namespace element or leaf can have either a "directly attached" storage > > configuration or a reference to a named one. > > > > We'd need management-ish APIs to handle the storage configurations for a > > namespace or leaf. > > These APIs, which do not exist yet, could start with "directly attached" > > storage configurations and once the authz questions are resolved, extend > to > > include named, shared storage configurations. > > > > Thoughts? > > > > Robert > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > To recap the discussion from the Community Sync today: My understanding > > is > > > that the consensus was: > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > * Allow multiple Storage Config objects per Catalog. If more than one > > > Storage Config is present, they are to have different (within the > > catalog) > > > names. Rishi to open a new PR for this (scoped to REST API + Sever > > > changes). > > > > > > * Defer CLI changes until we have the full end-to-end system working > (it > > > should be usable via curl) > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding support > > for > > > multiple Storage Configs > > > > > > * The connection between storage names and storage access credentials > is > > to > > > remain unchanged for now. It is effectively a deployment time concern > > given > > > the current OSS code. > > > > > > Please correct / add if I missed anything. > > > > > > From my POV, I think we still need to discuss how exactly entities > > (tables, > > > views) will link to Storage Config before we can resume PR 4023. > > > > > > I propose storing the config name in the Entity properties (not in > Table > > > Metadata properties). > > > > > > If clients also need a way to define it while interacting with Polaris > > over > > > the IRC API, let's discuss how that can be done. > > > > > > WDYT? > > > > > > Cheers, > > > Dmitri. > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Srinivas, > > > > > > > > I see that PR 4023 is active again. I'll try and post a fresh review > > > soon. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > [email protected]> > > > > wrote: > > > > > > > >> Hi JB and Dmitri, > > > >> > > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023> > > > yesterday > > > >> as part of the broader per-table storage config effort, and I > believe > > it > > > >> implements exactly what you are proposing. > > > >> > > > >> Building on the named storage configs introduced in PR #3409 > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows > > > namespaces > > > >> and tables to specify a polaris.storage.name property. This acts as > > the > > > >> "identifier" JB mentioned, allowing credential vending to resolve > the > > > >> correct, named storage config at runtime without introducing new > > > >> management > > > >> APIs. > > > >> > > > >> Could you please take a look at the PR and let me know if this > aligns > > > with > > > >> what you are looking for? > > > >> > > > > > > > > > >
