Hi Yufei,

The Python CLI can probably work in this use case too. However, as you
notes it will require changes to the Management API.

An Admin Tool command is standalone and is not attached to a particular API
version. It only depends on the MetsStore data.

Since validating locations is a task that needs to run on the backend at
deployment or migration time, I tend to think that the Admin Tool is a
better fit... and will probably be easier to implement and decommission
if/when Polaris evolves beyond the need for the
current ALLOW_UNSTRUCTURED_TABLE_LOCATION feature flag.

Cheers,
Dmitri.

On Tue, Nov 17, 2026 at 2:01 PM Yufei Gu <[email protected]> wrote:

> Thanks, Arnab, for putting this together, and Dmitri for the suggestions.
> Would it make sense to offer this through the Polaris CLI? The person
> changing a catalog’s settings should also be able to check the impact,
> using the same credentials and workflow. Requiring database access makes
> that harder, especially when catalog admins and deployment admins are
> different people.
>
> A management API could support checking one catalog or an entire realm,
> with the CLI displaying the results. Validation would still run on the
> server and reuse the existing logic. For a realm check, we should ensure
> the caller can inspect every catalog so the report is complete.
>
> The Admin Tool could still be useful when the server is unavailable, but
> the CLI seems easier for everyday use.
>
> Yufei
>
>
> On Mon, Nov 16, 2026 at 8:27 AM Dmitri Bourlatchkov via dev <
> [email protected]> wrote:
>
> > HI Arnab,
> >
> > Good idea about an Admin command for this check!
> >
> > Since it works both for the structured and unstructured cases, I'd
> propose
> > naming it "check-entity-locations".
> >
> > The "--assume-structured" is probably not necessary. The Admin Tool can
> get
> > it from its RealmConfig, which is settable independently of the Server
> > (both as a -D java CLI parameter and .properties entry). The tool should
> > document this in the "help" message.
> >
> > AdminToolProducers will probably need adjustments to produce a real
> > RealmConfig.
> >
> > It might be a bit inconvenient for end users, but it will ensure that the
> > validation code shared between the Server and Admin Tool uses consistent
> > sources (RealmConfig) for this value.
> >
> > For JSON output, I'd suggest one line per JSON object. This will
> hopefully
> > make shell-based processing easier.
> >
> > As the next step we could probably enhance the command to also perform
> > location overlap checks to reconcile with and double-check runtime
> overlap
> > validation.
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, Nov 13, 2026 at 2:45 PM Arnab Karmakar <[email protected]>
> > wrote:
> >
> > > Hi all,
> > > As we're now very close to releasing 1.9.0, I think we can resume
> > > the discussion about the command naming and detailed output format for
> a
> > > read-only admin report of tables and views that are incompatible with
> > > structured locations (tracked in [5917]).
> > >
> > > The documentation for this behavior has already landed in [5922] and
> I'd
> > > like to start working on the command, and I need some
> consensus/pointers
> > on
> > > the command name and output format. Here is a concrete proposal.
> > >
> > > Command name
> > >
> > > polaris-admin-tool.jar check-structured-locations \
> > >   --realm <realm> [--catalog <catalog>] \
> > >   [--assume-structured | --as-if-flag-false] \
> > >   [--format text|json]
> > >
> > >
> > >    - --realm: required. The command walks every catalog in the realm
> > unless
> > >    --catalog narrows it to one.
> > >    - --assume-structured: evaluates the structured-location rules as if
> > >    ALLOW_UNSTRUCTURED_TABLE_LOCATION=false, even while the flag is
> still
> > >    true. This is the pre-flight check before disabling the flag.
> > >    - Without that option, the command uses the effective flag value and
> > >    still reports affected entities when the flag is already false.
> > >    - The command is read-only. It never changes configuration, catalog
> > >    metadata or stored data. It does not block disabling the flag and
> does
> > > not
> > >    repair anything.
> > >    - I'm open to alternatives such as validate-locations or
> > >    report-unstructured-locations.
> > >
> > > Output format
> > >
> > > The default is human-readable text. It prints the current effective
> > setting
> > > per catalog, then one line per violation, then a summary:
> > >
> > > Realm: my-realm
> > > Catalog: prod   ALLOW_UNSTRUCTURED_TABLE_LOCATION=true (catalog
> > > override)   evaluating as: structured
> > >
> > > TYPE   ENTITY          LOCATION                        REASON
> > > TABLE  ns2.orders      s3://bucket/prod/ns1/orders     outside
> > > namespace location s3://bucket/prod/ns2
> > > VIEW   ns2.orders_v    s3://bucket/legacy/orders_v     outside
> > > namespace location s3://bucket/prod/ns2
> > >
> > > Summary: catalogs=1 entities_checked=412 incompatible=2
> > >
> > > --format json prints one object per finding, so operators can script
> > > remediation:
> > >
> > >
> > >
> >
> {"realm":"my-realm","catalog":"prod","namespace":["ns2"],"name":"orders","type":"TABLE",
> > >  "location":"s3://bucket/prod/ns1/orders",
> > >  "namespaceLocations":["s3://bucket/prod/ns2"],
> > >  "reason":"OUTSIDE_NAMESPACE_LOCATION"}
> > >
> > > Exit codes
> > >
> > >    - 0: no incompatible entities found.
> > >    - 1: the report completed and found incompatible entities.
> > >    - A different non-zero code for errors.
> > >
> > > Implementation
> > >
> > > The command will reuse the server's structured-location validation (the
> > > check introduced in [5856]). That way the report matches what the
> server
> > > enforces. The Admin Tool docs will get usage and sample output.
> > >
> > > Feedback and guidance are really welcome. Please let me know if I’m
> > missing
> > > anything or if there’s anything I should consider.
> > >
> > > Thanks
> > >
> > > Arnab
> > >
> > >
> > > [5856]: https://github.com/apache/polaris/pull/5856
> > >
> > > [5917]: https://github.com/apache/polaris/issues/5917
> > > [5922]: https://github.com/apache/polaris/pull/5922
> > >
> > > On Thu, Oct 29, 2026 at 10:27 PM Yufei Gu <[email protected]>
> wrote:
> > >
> > > > Two issues were filed:
> > > > - #5916: Document the impact of disabling
> > > ALLOW_UNSTRUCTURED_TABLE_LOCATION
> > > > <https://github.com/apache/polaris/issues/5916> — assigned to
> > milestone
> > > > *1.9.0*.
> > > > - #5917: Add an admin report for incompatible table/view locations
> > > > <https://github.com/apache/polaris/issues/5917> — optional read-only
> > > > tooling, planned as a follow-up after 1.9.0.
> > > > Yufei
> > > >
> > > >
> > > > On Thu, Oct 29, 2026 at 12:20 AM Jean-Baptiste Onofré <
> [email protected]
> > >
> > > > wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > I agree with Alex and Russell: we should not block true -> false.
> > > > >
> > > > > Let's start with prep step before the 1.9.0 release as you
> proposed:
> > > > > 1. We document the caveat in the configuration docs and upgrade
> > notes.
> > > > > 2. The report command to the admin tool makes sense. A read-only
> > > > > command that walks the catalog (or realm) and lists the
> tables/views
> > > > > whose locations would fail the structured check. That can come in a
> > > > > follow-up after 1.9.0 release.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Wed, Oct 28, 2026 at 5:40 PM Yufei Gu <[email protected]>
> > wrote:
> > > > > >
> > > > > > I got that a flawed feature isn't a bug :-). My main concern is
> > that
> > > it
> > > > > > breaks other important features, like credential vending and
> table
> > > > > updates,
> > > > > > which occur much more frequently than updating the flag. Are we
> OK
> > > with
> > > > > > this? Maybe it's fine given that only the catalog admin has the
> > > > > permission.
> > > > > > Let's document this caveat first. Having a tool to check all
> > > > > > locations under a catalog is also a good idea.
> > > > > >
> > > > > > Yufei
> > > > > >
> > > > > >
> > > > > > On Wed, Oct 28, 2026 at 8:23 AM Russell Spitzer <
> > > > > [email protected]>
> > > > > > wrote:
> > > > > >
> > > > > > > I beat you to it Alex :) I see we are thinking alike.
> > > > > > >
> > > > > > > On Wed, Oct 28, 2026 at 10:21 AM Alex Dutra <[email protected]
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Yufei,
> > > > > > > >
> > > > > > > > I agree that the situation is tricky.
> > > > > > > >
> > > > > > > > Option 1 is indeed expensive, but option 2 would actually
> > prevent
> > > > > > > > someone who cleaned up their warehouse so that all tables
> > comply,
> > > > > from
> > > > > > > > genuinely switching to structured mode. Conceptually, this
> > feels
> > > > > wrong
> > > > > > > > to me, because we are preventing a transition from a less
> > secure
> > > > > setup
> > > > > > > > to a more secure setup.
> > > > > > > >
> > > > > > > > Another concern I have is a practical one: if the flag is set
> > at
> > > > the
> > > > > > > > realm level, how is Polaris going to detect the change?
> > > > > > > >
> > > > > > > > Instead, how about a mix of documentation + a dedicated tool
> > that
> > > > > > > > could report all the tables that *would* fail if structured
> > mode
> > > is
> > > > > > > > switched on?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Alex
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Oct 28, 2026 at 1:19 AM Yufei Gu <
> [email protected]
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > While reviewing #5856 (
> > > > https://github.com/apache/polaris/pull/5856),
> > > > > I
> > > > > > > > > noticed that we allow changing
> > > ALLOW_UNSTRUCTURED_TABLE_LOCATION
> > > > > from
> > > > > > > > true
> > > > > > > > > to false without checking existing table/view locations.
> This
> > > can
> > > > > break
> > > > > > > > > tables that were valid under the previous setting.
> > > > > > > > >
> > > > > > > > > For example:
> > > > > > > > >
> > > > > > > > >    - Create ns1.t1 at its default location with the flag
> > > enabled.
> > > > > > > > >    - Rename it to ns2.t2. The data stays under ns1.
> > > > > > > > >    - Turn the flag off. Loading the table with credentials
> > can
> > > > now
> > > > > fail
> > > > > > > > >    with a 403 because its location is outside ns2.
> > > > > > > > >
> > > > > > > > > The catalog update succeeds, but the problem shows up when
> > > > someone
> > > > > > > > accesses
> > > > > > > > > the table later.
> > > > > > > > >
> > > > > > > > > I see two options:
> > > > > > > > >
> > > > > > > > >    1. Verify every table/view before allowing the change,
> and
> > > > > reject it
> > > > > > > > if
> > > > > > > > >    any location is incompatible. This could be very
> expensive
> > > for
> > > > > large
> > > > > > > > >    catalogs. We can consider it as phase 2.
> > > > > > > > >    2. Block true → false altogether for now. This should
> also
> > > > cover
> > > > > > > > >    removing the property when that makes the effective
> value
> > > > > false. We
> > > > > > > > can
> > > > > > > > >    still allow false → true.
> > > > > > > > >
> > > > > > > > > I’d lean toward option 2 for now. It seems better to reject
> > the
> > > > > > > > > configuration change than let it succeed and break existing
> > > > tables
> > > > > > > > > afterward.
> > > > > > > > >
> > > > > > > > > Any concerns with this approach?
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to