Hi All, Actually, another option is to use catalog-based feature flags for edge case settings.
Our feature flag tools support per-entity settings in general, but in storage integration code specifically, the catalog entity is _not_ available where the flags need to be checked. If we were to refactor the code to expose the catalog entity to storage integration code, then special case flags could be defined as feature flags and so we could avoid bloating the Storage Config in the Admin REST API. WDYT? Thanks, Dmitri. On Thu, Jan 22, 2026 at 11:51 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi JB, > > I'm also in favour of smaller configs in general. However, as far as > storage integration is concerned, unfortunately, the differences among the > multitude of providers require us to have some input about their feature > set. > > The input could come from admin users (the config suggestion, following > the existing pattern in Polaris). Polaris can have reasonable defaults for > most of the storage configs, but as user feedback shows, there are > deployments where it may not work well, so we may still need input from the > admin user in some special cases. > > We could also interrogate the service endpoints in runtime in order to > discover what the backend is, but I'm not sure this approach is preferable. > It will complicate code a lot IMHO (reminds me of PC bootloaders ;) ). > > Do people have other ideas about handling storage provider differences? > > Cheers, > Dmitri. > > On Thu, Jan 22, 2026 at 3:29 AM Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi Yong, >> >> I think it's reasonable to check using the endpoint URL. >> We have to be sure it's consistent for users (to have the same check >> behavior as before). If so, I don't think we need a new flag for that >> (personally, I'm more in favor to limit at max the number of flags / >> config). >> >> Regards >> JB >> >> On Wed, Jan 21, 2026 at 4:43 AM Yong Zheng <[email protected]> wrote: >> >> > Hello all, >> > >> > While working on https://github.com/apache/polaris/issues/3440, I >> noticed >> > the way on how we are currently determining whether a S3-compatible >> storage >> > is a bit odd where we are determining this by checking if a region >> property >> > of catalog and account id of IAM role are being set. Now back to the >> > reported issue, where the reporter is using assuming role with an >> > S3-compatible backend without KMS and a catalog region property was set >> (as >> > we didn't mention this anywhere). By doing so, it falls back to the >> > wildcard KMS policy which is not valid for certain S3-compatible >> storage (I >> > am assuming the reporter is using MinIO or something equivalent). >> > >> > In this case, both account id and region are set but they are actually >> > valid settings (but according to the code, the comment said it should >> not >> > be valid: >> > >> https://github.com/apache/polaris/blob/0b54f7046295ff19d434f9f0bd47b0b612be51a5/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java#L294 >> > ). >> > >> > I think it may be better to determine is a S3-compatible storage is AWS >> or >> > not by looking at endpoint URL instead (sample PR: >> > https://github.com/apache/polaris/pull/3496). Let me know what your >> guys >> > think. >> > >> > Thanks, >> > Yong Zheng >> > >> >
