Thanks for the suggestion and feedback JB. Here is the PR: https://github.com/apache/polaris/pull/5557
Thanks, Yong On 2026/09/14 14:11:02 Jean-Baptiste Onofré wrote: > Hi Yong, > > That's a good call (I think I mentioned that we should start a > discussion in a PR I reviewed). > > BasePersistence.listFullEntities already takes an entityFilter + > PageToken, and AtomicOperationMetaStoreManager already uses that exact > pattern elsewhere (the task listing , catalog_role, etc). So this is a > valid path. The missing part is that > PolarisMetaStoreManager.listFullEntities hardcodes entity, so > PolicyCatalog can't reach the filter-aware version. I think we have to > add a small filter-aware overload at the manager layer to thread a > policyType predicate down. > That's a small approach we can implement easily. > > Regards > JB > > On Sun, Sep 13, 2026 at 3:14 AM Yong Zheng <[email protected]> wrote: > > > > Hello, > > > > I would like to start this ML for discussing if we should implement the > > pagination support for policy APIs (GET policy only for this ML). Here is a > > reference report issue for this matter: > > https://github.com/apache/polaris/issues/5311. > > > > Currently we have following paths which should support pagination: > > > > 1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies > > 2. GET /polaris/v1/{prefix}/applicable-policies > > > > For GET policies, we are returning everything via > > PageToken.readEverything() > > (https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41) > > > > For GET applicable-policies, we are computing effective policy by walking > > the entity hierarchy in memory rather than issuing an single backing-store > > listing, we may need some discussion around this. But it may be worth to > > start a different ML for this unless preferred by the community. > > > > Now to honor the pagination for GET policy, there are 2 cases which we will > > need to support: > > > > case 1: No filter on policyType > > > > This is the simple case where we just need to thread the token and matching > > to listSemanticModels (ref: > > https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92). > > > > case 2: Filter on policyType > > > > When a policyType is been set, the current code is calling > > listFullEntitiesAll and perform filter in memory (ref: > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188). > > > > For case 2, I would like to see what community thinks regarding what we > > should perform: > > 1. Ignore the pagination when a filter is set? > > - In this case, we stay with current behavior > > 2. Pagination at the post-filter? > > - We do pagination on the filtered response > > 3. Add a filter-aware store mthod? > > - Something likes listEntitiesByPolicyType that we handle pagination and > > avoid current full fetch and in-memory filter > > > > Thanks, > > Yong >
