Hi all, Thanks for the replies and it really helps align expectations.
I agree there is a spec/contract gap on GET …/applicable-policies (parameters are declared but ignored today), but no metastore-backed pagination is possible for this call. The only option would be the service-layer approach (full compute, stable sort, cursor). I also take Yufei’s and Yong’s point that we should wait for a solid use case before spending server work on that, and JB’s suggestion that we can leave it open until someone needs it rather than treating it as a scalability fix. I’ve noted that on GitHub #5311 <https://github.com/apache/polaris/issues/5311> and will hold off on implementation for now. If a concrete use case shows up, I’m happy to pick it up using the sort order and cursor shape JB described (PolicyIdentifier-based order; opaque token with last sort key rather than offset). Thanks, Arnab On Fri, Oct 30, 2026 at 10:46 PM Yong Zheng <[email protected]> wrote: > Hello, > > I also think we should wait for a use-case for now. This can be a very > expensive operation and we implicitly called out this in the CLI for why we > are not exporting this due to the compute overhead. > > Thanks, > Yong > > On Fri, Oct 30, 2026 at 11:12 AM Yufei Gu <[email protected]> wrote: > > > > Yufei, which one you mean? > > > > Hi JB, I assume this is the question for Arnab. > > > > Yufei > > > > > > On Fri, Oct 30, 2026 at 2:49 AM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi guys, > > > > > > On Yufei's first point, I read "app-level pagination" in Arnab's > proposal > > > as service-layer pagination on the server (compute, sort, slice), not > CLI > > > pagination. > > > Yufei, which one you mean? > > > If you mean server-side, I agree that we should only do it for a real > use > > > case. > > > > > > About the questions: > > > 1. I'm fine with app-level pagination as a v1. The endpoint already > > > declares page-token, page-size and next-page-token in the spec, so > > silently > > > returning everything is a contract gap. I agree with Yufei "wait for a > > use > > > case" argument, because this version doesn't reduce server work for > large > > > hierarchies. I suggest to treat that as contract compliance rather > than a > > > scalability fix. If nobody has a concrete need soon, we could leave it > > open > > > :) > > > 2. I agree with Yufei about the order. PolicyIdentifier (namespace > > levels, > > > then name) should already be unique. I would not add inheritable flag > or > > > source unless we find a real collision. We should document the order in > > the > > > spec so clients can rely on it. > > > 3. No objection to not reuse the metastore PageToken encoding, since > > there > > > is no single store listing behind this call. I think the token must be > > > opaque and self-validating, so we can have a clean 400 when the token > is > > > not correct (rather than an exception). Also, because the effective set > > is > > > recomputed on every request, I would encode the last-returned sort key > > > rather than an offset. I believe it's more stable if policies are > > attached > > > or detached between pages. > > > 4. I would keep this thread for the server side only. The CLI follow-up > > can > > > be a separate issue once the server returns next-page-token. > > > > > > Regards > > > JB > > > > > > On Fri, Oct 30, 2026 at 2:11 AM Yufei Gu <[email protected]> wrote: > > > > > > > Hi Arnab, > > > > > > > > Thanks for picking this up. A few clarifications: > > > > > > > > 1. Does app-level pagination refer to CLI pagination? If so, I agree > > > with > > > > that approach. If it involves a server-side change, listPolicies and > > > > listGenericTables would be good to have, but I suggest waiting until > we > > > > have a solid use case before moving forward with > getApplicablePolicies. > > > > > > > > 2. Sorting by PolicyIdentifier (namespace, then name) should already > > be > > > > unique since the merge yields one entry per policy. Adding policy > type > > > as a > > > > tie-breaker works as well. > > > > > > > > Best regards, > > > > Yufei > > > > > > > > > > > > On Wed, Oct 28, 2026 at 10:54 PM Arnab Karmakar < > [email protected]> > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > Following up on GitHub issue #5311 > > > > > <https://github.com/apache/polaris/issues/5311> (policy / > > > generic-table > > > > > list pagination): listGenericTables (#5533 > > > > > <https://github.com/apache/polaris/issues/5533>) and listPolicies > > > (#5557 > > > > > <https://github.com/apache/polaris/issues/5557>) are merged. The > > only > > > > > remaining gap is GET /polaris/v1/{prefix}/applicable-policies, > which > > > > > declares page-token, page-size, and next-page-token in the spec but > > > still > > > > > returns the full assembled result set. > > > > > > > > > > I plan to implement the remaining work and would like consensus on > > how > > > > > pagination should behave for this endpoint before opening a PR. > > > > > > > > > > Background > > > > > ---------- > > > > > Unlike listPolicies, getApplicablePolicies does not map to a single > > > > > backing-store listing. PolicyCatalog#getEffectivePolicies walks the > > > > entity > > > > > path (catalog / namespace / table), loads policy mappings per > level, > > > > merges > > > > > inheritable vs non-inheritable policies, and returns a > > > > > List<ApplicablePolicy> that PolicyCatalogHandler currently wraps > in a > > > > > HashSet (no stable order, no continuation token). > > > > > > > > > > Proposal (for discussion) > > > > > ------------------------- > > > > > 1. Page boundary: After computing the full effective set for the > > > request > > > > > (same semantics as today), apply pagination in the service > layer: > > > > > sort into a deterministic order, return page-size items, and > > expose > > > > > next-page-token when more items remain. The page token would > > encode > > > > > an opaque cursor (e.g. offset into the sorted snapshot), not a > > > > > metastore PageToken. > > > > > > > > > > This honors the REST contract and LIST_PAGINATION_MAX_PAGE_SIZE > > > > > behavior without changing inheritance semantics. It does not > > reduce > > > > > server work for large hierarchies. > > > > > > > > > > 2. Stable ordering: Replace HashSet assembly with a sorted list > > before > > > > > slicing. Candidate sort key (lexicographic): > > > > > policy identifier (namespace levels, then policy name), then > > policy > > > > > type, then inheritable flag / source if needed for uniqueness. > > > > > The response schema keeps uniqueItems. > > > > > > > > > > 3. Adapter/handler: Thread pageToken and pageSize through > > > > > PolicyCatalogAdapter#getApplicablePolicies into > > > > > PolicyCatalogHandler#getApplicablePolicies, mirroring > listPolicies > > > > > (PageToken.build, rejectIncompleteListing, next-page-token on > the > > > > > response). > > > > > > > > > > Questions for the list > > > > > ---------------------- > > > > > 1. Is application-level pagination (full compute + sort + slice) an > > > > > acceptable v1 for this endpoint, given there is no single store > > listing > > > > to > > > > > paginate? > > > > > > > > > > 2. What total order should clients rely on across pages? Is sorting > > > > > by PolicyIdentifier (namespace + name) then policy type sufficient, > > or > > > > > should we document a different key? > > > > > > > > > > 3. Any objection to encoding the continuation as an opaque > in-memory > > > > cursor > > > > > rather than reusing metastore PageToken encoding? > > > > > > > > > > 4. Scope: Should this thread cover only applicable-policies, or > also > > > any > > > > > CLI follow-up (#5385-style) once the server returns > next-page-token? > > > > > > > > > > Please let me know if this direction sounds reasonable. > > > > > > > > > > Thanks, > > > > > Arnab > > > > > > > > > > > > > > >
