Hi I agree with Yufei here. Let's keep things simple, just checking on the fly works :)
Regards JB On Wed, Nov 18, 2026 at 3:04 AM Yufei Gu <[email protected]> wrote: > > Thanks for chiming in, Dmitri. I don’t think we need to tie location > validation to deployment or migration. A catalog admin should be able to > run the check against a live Polaris server before turning off the flag. > > Yufei > > > On Tue, Nov 17, 2026 at 5:57 PM Dmitri Bourlatchkov via dev < > [email protected]> wrote: > > > Hi Yufei, > > > > The Python CLI can probably work in this use case too. However, as you > > notes it will require changes to the Management API. > > > > An Admin Tool command is standalone and is not attached to a particular API > > version. It only depends on the MetsStore data. > > > > Since validating locations is a task that needs to run on the backend at > > deployment or migration time, I tend to think that the Admin Tool is a > > better fit... and will probably be easier to implement and decommission > > if/when Polaris evolves beyond the need for the > > current ALLOW_UNSTRUCTURED_TABLE_LOCATION feature flag. > > > > Cheers, > > Dmitri. > > > > On Tue, Nov 17, 2026 at 2:01 PM Yufei Gu <[email protected]> wrote: > > > > > Thanks, Arnab, for putting this together, and Dmitri for the suggestions. > > > Would it make sense to offer this through the Polaris CLI? The person > > > changing a catalog’s settings should also be able to check the impact, > > > using the same credentials and workflow. Requiring database access makes > > > that harder, especially when catalog admins and deployment admins are > > > different people. > > > > > > A management API could support checking one catalog or an entire realm, > > > with the CLI displaying the results. Validation would still run on the > > > server and reuse the existing logic. For a realm check, we should ensure > > > the caller can inspect every catalog so the report is complete. > > > > > > The Admin Tool could still be useful when the server is unavailable, but > > > the CLI seems easier for everyday use. > > > > > > Yufei > > > > > > > > > On Mon, Nov 16, 2026 at 8:27 AM Dmitri Bourlatchkov via dev < > > > [email protected]> wrote: > > > > > > > HI Arnab, > > > > > > > > Good idea about an Admin command for this check! > > > > > > > > Since it works both for the structured and unstructured cases, I'd > > > propose > > > > naming it "check-entity-locations". > > > > > > > > The "--assume-structured" is probably not necessary. The Admin Tool can > > > get > > > > it from its RealmConfig, which is settable independently of the Server > > > > (both as a -D java CLI parameter and .properties entry). The tool > > should > > > > document this in the "help" message. > > > > > > > > AdminToolProducers will probably need adjustments to produce a real > > > > RealmConfig. > > > > > > > > It might be a bit inconvenient for end users, but it will ensure that > > the > > > > validation code shared between the Server and Admin Tool uses > > consistent > > > > sources (RealmConfig) for this value. > > > > > > > > For JSON output, I'd suggest one line per JSON object. This will > > > hopefully > > > > make shell-based processing easier. > > > > > > > > As the next step we could probably enhance the command to also perform > > > > location overlap checks to reconcile with and double-check runtime > > > overlap > > > > validation. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Fri, Nov 13, 2026 at 2:45 PM Arnab Karmakar <[email protected]> > > > > wrote: > > > > > > > > > Hi all, > > > > > As we're now very close to releasing 1.9.0, I think we can resume > > > > > the discussion about the command naming and detailed output format > > for > > > a > > > > > read-only admin report of tables and views that are incompatible with > > > > > structured locations (tracked in [5917]). > > > > > > > > > > The documentation for this behavior has already landed in [5922] and > > > I'd > > > > > like to start working on the command, and I need some > > > consensus/pointers > > > > on > > > > > the command name and output format. Here is a concrete proposal. > > > > > > > > > > Command name > > > > > > > > > > polaris-admin-tool.jar check-structured-locations \ > > > > > --realm <realm> [--catalog <catalog>] \ > > > > > [--assume-structured | --as-if-flag-false] \ > > > > > [--format text|json] > > > > > > > > > > > > > > > - --realm: required. The command walks every catalog in the realm > > > > unless > > > > > --catalog narrows it to one. > > > > > - --assume-structured: evaluates the structured-location rules as > > if > > > > > ALLOW_UNSTRUCTURED_TABLE_LOCATION=false, even while the flag is > > > still > > > > > true. This is the pre-flight check before disabling the flag. > > > > > - Without that option, the command uses the effective flag value > > and > > > > > still reports affected entities when the flag is already false. > > > > > - The command is read-only. It never changes configuration, > > catalog > > > > > metadata or stored data. It does not block disabling the flag and > > > does > > > > > not > > > > > repair anything. > > > > > - I'm open to alternatives such as validate-locations or > > > > > report-unstructured-locations. > > > > > > > > > > Output format > > > > > > > > > > The default is human-readable text. It prints the current effective > > > > setting > > > > > per catalog, then one line per violation, then a summary: > > > > > > > > > > Realm: my-realm > > > > > Catalog: prod ALLOW_UNSTRUCTURED_TABLE_LOCATION=true (catalog > > > > > override) evaluating as: structured > > > > > > > > > > TYPE ENTITY LOCATION REASON > > > > > TABLE ns2.orders s3://bucket/prod/ns1/orders outside > > > > > namespace location s3://bucket/prod/ns2 > > > > > VIEW ns2.orders_v s3://bucket/legacy/orders_v outside > > > > > namespace location s3://bucket/prod/ns2 > > > > > > > > > > Summary: catalogs=1 entities_checked=412 incompatible=2 > > > > > > > > > > --format json prints one object per finding, so operators can script > > > > > remediation: > > > > > > > > > > > > > > > > > > > > > > > > {"realm":"my-realm","catalog":"prod","namespace":["ns2"],"name":"orders","type":"TABLE", > > > > > "location":"s3://bucket/prod/ns1/orders", > > > > > "namespaceLocations":["s3://bucket/prod/ns2"], > > > > > "reason":"OUTSIDE_NAMESPACE_LOCATION"} > > > > > > > > > > Exit codes > > > > > > > > > > - 0: no incompatible entities found. > > > > > - 1: the report completed and found incompatible entities. > > > > > - A different non-zero code for errors. > > > > > > > > > > Implementation > > > > > > > > > > The command will reuse the server's structured-location validation > > (the > > > > > check introduced in [5856]). That way the report matches what the > > > server > > > > > enforces. The Admin Tool docs will get usage and sample output. > > > > > > > > > > Feedback and guidance are really welcome. Please let me know if I’m > > > > missing > > > > > anything or if there’s anything I should consider. > > > > > > > > > > Thanks > > > > > > > > > > Arnab > > > > > > > > > > > > > > > [5856]: https://github.com/apache/polaris/pull/5856 > > > > > > > > > > [5917]: https://github.com/apache/polaris/issues/5917 > > > > > [5922]: https://github.com/apache/polaris/pull/5922 > > > > > > > > > > On Thu, Oct 29, 2026 at 10:27 PM Yufei Gu <[email protected]> > > > wrote: > > > > > > > > > > > Two issues were filed: > > > > > > - #5916: Document the impact of disabling > > > > > ALLOW_UNSTRUCTURED_TABLE_LOCATION > > > > > > <https://github.com/apache/polaris/issues/5916> — assigned to > > > > milestone > > > > > > *1.9.0*. > > > > > > - #5917: Add an admin report for incompatible table/view locations > > > > > > <https://github.com/apache/polaris/issues/5917> — optional > > read-only > > > > > > tooling, planned as a follow-up after 1.9.0. > > > > > > Yufei > > > > > > > > > > > > > > > > > > On Thu, Oct 29, 2026 at 12:20 AM Jean-Baptiste Onofré < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > I agree with Alex and Russell: we should not block true -> false. > > > > > > > > > > > > > > Let's start with prep step before the 1.9.0 release as you > > > proposed: > > > > > > > 1. We document the caveat in the configuration docs and upgrade > > > > notes. > > > > > > > 2. The report command to the admin tool makes sense. A read-only > > > > > > > command that walks the catalog (or realm) and lists the > > > tables/views > > > > > > > whose locations would fail the structured check. That can come > > in a > > > > > > > follow-up after 1.9.0 release. > > > > > > > > > > > > > > Regards > > > > > > > JB > > > > > > > > > > > > > > On Wed, Oct 28, 2026 at 5:40 PM Yufei Gu <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > I got that a flawed feature isn't a bug :-). My main concern is > > > > that > > > > > it > > > > > > > > breaks other important features, like credential vending and > > > table > > > > > > > updates, > > > > > > > > which occur much more frequently than updating the flag. Are we > > > OK > > > > > with > > > > > > > > this? Maybe it's fine given that only the catalog admin has the > > > > > > > permission. > > > > > > > > Let's document this caveat first. Having a tool to check all > > > > > > > > locations under a catalog is also a good idea. > > > > > > > > > > > > > > > > Yufei > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 28, 2026 at 8:23 AM Russell Spitzer < > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I beat you to it Alex :) I see we are thinking alike. > > > > > > > > > > > > > > > > > > On Wed, Oct 28, 2026 at 10:21 AM Alex Dutra < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Yufei, > > > > > > > > > > > > > > > > > > > > I agree that the situation is tricky. > > > > > > > > > > > > > > > > > > > > Option 1 is indeed expensive, but option 2 would actually > > > > prevent > > > > > > > > > > someone who cleaned up their warehouse so that all tables > > > > comply, > > > > > > > from > > > > > > > > > > genuinely switching to structured mode. Conceptually, this > > > > feels > > > > > > > wrong > > > > > > > > > > to me, because we are preventing a transition from a less > > > > secure > > > > > > > setup > > > > > > > > > > to a more secure setup. > > > > > > > > > > > > > > > > > > > > Another concern I have is a practical one: if the flag is > > set > > > > at > > > > > > the > > > > > > > > > > realm level, how is Polaris going to detect the change? > > > > > > > > > > > > > > > > > > > > Instead, how about a mix of documentation + a dedicated > > tool > > > > that > > > > > > > > > > could report all the tables that *would* fail if structured > > > > mode > > > > > is > > > > > > > > > > switched on? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Alex > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 28, 2026 at 1:19 AM Yufei Gu < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > While reviewing #5856 ( > > > > > > https://github.com/apache/polaris/pull/5856), > > > > > > > I > > > > > > > > > > > noticed that we allow changing > > > > > ALLOW_UNSTRUCTURED_TABLE_LOCATION > > > > > > > from > > > > > > > > > > true > > > > > > > > > > > to false without checking existing table/view locations. > > > This > > > > > can > > > > > > > break > > > > > > > > > > > tables that were valid under the previous setting. > > > > > > > > > > > > > > > > > > > > > > For example: > > > > > > > > > > > > > > > > > > > > > > - Create ns1.t1 at its default location with the flag > > > > > enabled. > > > > > > > > > > > - Rename it to ns2.t2. The data stays under ns1. > > > > > > > > > > > - Turn the flag off. Loading the table with > > credentials > > > > can > > > > > > now > > > > > > > fail > > > > > > > > > > > with a 403 because its location is outside ns2. > > > > > > > > > > > > > > > > > > > > > > The catalog update succeeds, but the problem shows up > > when > > > > > > someone > > > > > > > > > > accesses > > > > > > > > > > > the table later. > > > > > > > > > > > > > > > > > > > > > > I see two options: > > > > > > > > > > > > > > > > > > > > > > 1. Verify every table/view before allowing the change, > > > and > > > > > > > reject it > > > > > > > > > > if > > > > > > > > > > > any location is incompatible. This could be very > > > expensive > > > > > for > > > > > > > large > > > > > > > > > > > catalogs. We can consider it as phase 2. > > > > > > > > > > > 2. Block true → false altogether for now. This should > > > also > > > > > > cover > > > > > > > > > > > removing the property when that makes the effective > > > value > > > > > > > false. We > > > > > > > > > > can > > > > > > > > > > > still allow false → true. > > > > > > > > > > > > > > > > > > > > > > I’d lean toward option 2 for now. It seems better to > > reject > > > > the > > > > > > > > > > > configuration change than let it succeed and break > > existing > > > > > > tables > > > > > > > > > > > afterward. > > > > > > > > > > > > > > > > > > > > > > Any concerns with this approach? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
