Hi Arnab, Thanks for picking this up. A few clarifications:
1. Does app-level pagination refer to CLI pagination? If so, I agree with that approach. If it involves a server-side change, listPolicies and listGenericTables would be good to have, but I suggest waiting until we have a solid use case before moving forward with getApplicablePolicies. 2. Sorting by PolicyIdentifier (namespace, then name) should already be unique since the merge yields one entry per policy. Adding policy type as a tie-breaker works as well. Best regards, Yufei On Wed, Oct 28, 2026 at 10:54 PM Arnab Karmakar <[email protected]> wrote: > Hello, > > Following up on GitHub issue #5311 > <https://github.com/apache/polaris/issues/5311> (policy / generic-table > list pagination): listGenericTables (#5533 > <https://github.com/apache/polaris/issues/5533>) and listPolicies (#5557 > <https://github.com/apache/polaris/issues/5557>) are merged. The only > remaining gap is GET /polaris/v1/{prefix}/applicable-policies, which > declares page-token, page-size, and next-page-token in the spec but still > returns the full assembled result set. > > I plan to implement the remaining work and would like consensus on how > pagination should behave for this endpoint before opening a PR. > > Background > ---------- > Unlike listPolicies, getApplicablePolicies does not map to a single > backing-store listing. PolicyCatalog#getEffectivePolicies walks the entity > path (catalog / namespace / table), loads policy mappings per level, merges > inheritable vs non-inheritable policies, and returns a > List<ApplicablePolicy> that PolicyCatalogHandler currently wraps in a > HashSet (no stable order, no continuation token). > > Proposal (for discussion) > ------------------------- > 1. Page boundary: After computing the full effective set for the request > (same semantics as today), apply pagination in the service layer: > sort into a deterministic order, return page-size items, and expose > next-page-token when more items remain. The page token would encode > an opaque cursor (e.g. offset into the sorted snapshot), not a > metastore PageToken. > > This honors the REST contract and LIST_PAGINATION_MAX_PAGE_SIZE > behavior without changing inheritance semantics. It does not reduce > server work for large hierarchies. > > 2. Stable ordering: Replace HashSet assembly with a sorted list before > slicing. Candidate sort key (lexicographic): > policy identifier (namespace levels, then policy name), then policy > type, then inheritable flag / source if needed for uniqueness. > The response schema keeps uniqueItems. > > 3. Adapter/handler: Thread pageToken and pageSize through > PolicyCatalogAdapter#getApplicablePolicies into > PolicyCatalogHandler#getApplicablePolicies, mirroring listPolicies > (PageToken.build, rejectIncompleteListing, next-page-token on the > response). > > Questions for the list > ---------------------- > 1. Is application-level pagination (full compute + sort + slice) an > acceptable v1 for this endpoint, given there is no single store listing to > paginate? > > 2. What total order should clients rely on across pages? Is sorting > by PolicyIdentifier (namespace + name) then policy type sufficient, or > should we document a different key? > > 3. Any objection to encoding the continuation as an opaque in-memory cursor > rather than reusing metastore PageToken encoding? > > 4. Scope: Should this thread cover only applicable-policies, or also any > CLI follow-up (#5385-style) once the server returns next-page-token? > > Please let me know if this direction sounds reasonable. > > Thanks, > Arnab >
