Hi guys, I think Prithvi and Yufei have the right framing. My take is that the cache's job is to skip expensive grant-record load, not really to skip the database entirely (that's a larger concern). And that's the purpose to any cache that must stay consistent across pods. However, Robert's complaint about 23 cold-path queries is legitimate and traces the "real" code: AtomicOperationMetaStoreManager.loadResolvedEntityById() issues 1 or 2 grant-record queries per entity, and it's called once per entity in a resolved path with no batching. That's N+1, not related to caching cost. Imho, we have independent fixes here. I like Dmitri's idea about skipping grant-record lookups entirely when an external authorizer (Ranger) owns authorization, but I believe it would need extra plumbing to know "no internal grants needed" at the persistence layer, else I suspect we will load the persistence layer anyway.
I might be wrong, but: 1. I think grant-record loading is never actually batched, even in the "batch" API ;). BasePersistence.loadAllGrantRecordsOnSecurable()/loadAllGrantRecordsOnGrantee() only take a single entity id. AtomicOperationMetaStoreManager.loadResolvedEntities() does batch the entity-row lookup via lookupEntities(), but then toResolvedPolarisEntity() loops over each entity calling the single-entity grant methods. So, if I do the math, a "batched" 5-entity load still fires up to 10 grant queries. Same story in the single-entity loadResolvedEntityById() used by the cold-cache-miss path. I believe we can fix that with a JDBC composite-key IN query. The good news is that QueryGenerator already has the template for this (generateSelectQueryWithEntityIds()): a loadAllGrantRecordsOnSecurables(List<PolarisEntityId>) variant against ModelGrantRecord would be a direct copy of that pattern. 2. The Resolver.bulkValidate() has a batched version-check. bulkValidate() does one batched loadEntitiesChangeTracking call to get a fresh version for the whole toValidate set. But for every entity that's actually stale, it calls cache.getAndRefreshIfNeeded() one at a time, and that method calls refreshResolvedEntity(), which re-runs lookupEntityVersions() again per entity :/ I would suggest to evaluate how we can already improve that. I'm quite sure bulkValidate() refresh branch can be optimized (probably using the existing getOrLoadResolvedEntitties()). So before going to deep in cache refactoring, I think we can already do nice improvements. Regards JB On Tue, Sep 8, 2026 at 4:23 PM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi Prithvi, > > Thanks for the analysis! It is very helpful. > > The grant record lookups appear to be specific to the internal RBAC model. > I wonder if they are necessary with external Authorizers (e.g. Ranger). I > suppose those queries will return empty results in the external case anyway. > > What do people think about refactoring the resolution code path to take the > Authorizer "kind" into account? I believe Sung's prior work was moving in > that direction. After PR [5194] we should be able to avoid the grant record > lookup for authorizers that do not use them, I hope. > > [5194] https://github.com/apache/polaris/pull/5194 > > Cheers, > Dmitri. > > On Fri, Sep 4, 2026 at 12:46 PM Prithvi S <[email protected]> > wrote: > > > Hi Dmitri, > > > > Thanks for raising this. > > > > You are right that even on a cache hit, resolve still talks to JDBC. That > > is by design: InMemoryEntityCache is not a "skip the database" cache. For > > JDBC it is a "skip the expensive load" cache, with a version check as the > > invalidation path. > > I traced Resolver.resolveAll() with JDBC and compared a warmed > > InMemoryEntityCache against cache == null on the same catalog. The workload > > was a loadTable-like resolve (principal P1 with PRINCIPAL_ROLE:ALL, catalog > > "test", path N1/N2/T2). > > > > On a warm cache, that resolve issued 1 SELECT: > > > > SELECT id, catalog_id, entity_version, grant_records_version > > FROM ENTITIES > > WHERE (catalog_id, id) IN ((?, ?), ... ) AND realm_id = ? > > > > Without the cache, the same resolve issued 23 SELECTs: 9 full entity rows > > (including properties / internal_properties) and 14 grant-record lookups. > > Scenario Cache SELECTs Payload > > N1/N2/T2 (small properties) no-cache 23 947 B > > N1/N2/T2 cold-cache 23 947 B > > N1/N2/T2 warm-cache 1 144 B > > N1/N2/T1 (64KiB properties) no-cache 23 66 KB > > N1/N2/T1 warm-cache 1 144 B > > deep path (6 segments) no-cache 29 959 B > > deep path (6 segments) warm-cache 1 192 B > > > > So the version check is real, but it replaces the full entity JSON and the > > grant-record queries. Cold cache looks the same as no cache; the gain is on > > subsequent resolves of the same entities. > > > > I have not measured a full REST request or a stale-cache / high-churn path, > > so I do not want to claim an end-to-end speedup from this :) > > For the resolve path itself, the query shape is clear. > > > > WDYT? > > > > Thanks, > > Prithvi > > > > On Fri, Sep 4, 2026 at 6:50 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > InMemoryEntityCache is currently used in Apache Polaris with JDBC > > > Persistence. > > > > > > However, as far as I can tell, even for cached entities, the resolve will > > > still issue a JDBC query to confirm that the latest version was returned > > > from the cache. > > > > > > So, the question is: does this cache provide any considerable efficiency > > > gain? > > > > > > Has anyone measured the performance difference with and without this > > cache? > > > > > > Would anyone be willing to carry out such a test? > > > > > > WDYT? > > > > > > Thanks, > > > Dmitri. > > > > >
