Hi guys, On Yufei's first point, I read "app-level pagination" in Arnab's proposal as service-layer pagination on the server (compute, sort, slice), not CLI pagination. Yufei, which one you mean? If you mean server-side, I agree that we should only do it for a real use case.
About the questions: 1. I'm fine with app-level pagination as a v1. The endpoint already declares page-token, page-size and next-page-token in the spec, so silently returning everything is a contract gap. I agree with Yufei "wait for a use case" argument, because this version doesn't reduce server work for large hierarchies. I suggest to treat that as contract compliance rather than a scalability fix. If nobody has a concrete need soon, we could leave it open :) 2. I agree with Yufei about the order. PolicyIdentifier (namespace levels, then name) should already be unique. I would not add inheritable flag or source unless we find a real collision. We should document the order in the spec so clients can rely on it. 3. No objection to not reuse the metastore PageToken encoding, since there is no single store listing behind this call. I think the token must be opaque and self-validating, so we can have a clean 400 when the token is not correct (rather than an exception). Also, because the effective set is recomputed on every request, I would encode the last-returned sort key rather than an offset. I believe it's more stable if policies are attached or detached between pages. 4. I would keep this thread for the server side only. The CLI follow-up can be a separate issue once the server returns next-page-token. Regards JB On Fri, Oct 30, 2026 at 2:11 AM Yufei Gu <[email protected]> wrote: > Hi Arnab, > > Thanks for picking this up. A few clarifications: > > 1. Does app-level pagination refer to CLI pagination? If so, I agree with > that approach. If it involves a server-side change, listPolicies and > listGenericTables would be good to have, but I suggest waiting until we > have a solid use case before moving forward with getApplicablePolicies. > > 2. Sorting by PolicyIdentifier (namespace, then name) should already be > unique since the merge yields one entry per policy. Adding policy type as a > tie-breaker works as well. > > Best regards, > Yufei > > > On Wed, Oct 28, 2026 at 10:54 PM Arnab Karmakar <[email protected]> > wrote: > > > Hello, > > > > Following up on GitHub issue #5311 > > <https://github.com/apache/polaris/issues/5311> (policy / generic-table > > list pagination): listGenericTables (#5533 > > <https://github.com/apache/polaris/issues/5533>) and listPolicies (#5557 > > <https://github.com/apache/polaris/issues/5557>) are merged. The only > > remaining gap is GET /polaris/v1/{prefix}/applicable-policies, which > > declares page-token, page-size, and next-page-token in the spec but still > > returns the full assembled result set. > > > > I plan to implement the remaining work and would like consensus on how > > pagination should behave for this endpoint before opening a PR. > > > > Background > > ---------- > > Unlike listPolicies, getApplicablePolicies does not map to a single > > backing-store listing. PolicyCatalog#getEffectivePolicies walks the > entity > > path (catalog / namespace / table), loads policy mappings per level, > merges > > inheritable vs non-inheritable policies, and returns a > > List<ApplicablePolicy> that PolicyCatalogHandler currently wraps in a > > HashSet (no stable order, no continuation token). > > > > Proposal (for discussion) > > ------------------------- > > 1. Page boundary: After computing the full effective set for the request > > (same semantics as today), apply pagination in the service layer: > > sort into a deterministic order, return page-size items, and expose > > next-page-token when more items remain. The page token would encode > > an opaque cursor (e.g. offset into the sorted snapshot), not a > > metastore PageToken. > > > > This honors the REST contract and LIST_PAGINATION_MAX_PAGE_SIZE > > behavior without changing inheritance semantics. It does not reduce > > server work for large hierarchies. > > > > 2. Stable ordering: Replace HashSet assembly with a sorted list before > > slicing. Candidate sort key (lexicographic): > > policy identifier (namespace levels, then policy name), then policy > > type, then inheritable flag / source if needed for uniqueness. > > The response schema keeps uniqueItems. > > > > 3. Adapter/handler: Thread pageToken and pageSize through > > PolicyCatalogAdapter#getApplicablePolicies into > > PolicyCatalogHandler#getApplicablePolicies, mirroring listPolicies > > (PageToken.build, rejectIncompleteListing, next-page-token on the > > response). > > > > Questions for the list > > ---------------------- > > 1. Is application-level pagination (full compute + sort + slice) an > > acceptable v1 for this endpoint, given there is no single store listing > to > > paginate? > > > > 2. What total order should clients rely on across pages? Is sorting > > by PolicyIdentifier (namespace + name) then policy type sufficient, or > > should we document a different key? > > > > 3. Any objection to encoding the continuation as an opaque in-memory > cursor > > rather than reusing metastore PageToken encoding? > > > > 4. Scope: Should this thread cover only applicable-policies, or also any > > CLI follow-up (#5385-style) once the server returns next-page-token? > > > > Please let me know if this direction sounds reasonable. > > > > Thanks, > > Arnab > > >
