yuqi1129 opened a new issue, #11736: URL: https://github.com/apache/gravitino/issues/11736
**Version**: main branch ### Describe what's wrong The shared `EntityChangeLogPoller` (consumed today by jcasbin caches, and planned to back entity-store cache multi-node invalidation) has three independent paths that cause a change-log row's invalidation to be **permanently lost**, leaving stale data on remote nodes with no self-healing. **1. Auto-increment id cursor has a commit-ordering gap.** The cursor is `WHERE id > highWater ORDER BY id` (`EntityChangeLogBaseSQLProvider#selectEntityChanges`). Auto-increment ids are assigned at INSERT time but become visible at COMMIT time, so a lower id can commit *after* a higher id: ``` Txn A inserts change id=100 (not yet committed) Txn B inserts change id=101, commits first poller sees 101 -> advances highWater to 101 Txn A commits id=100 next poll: WHERE id > 101 -> row 100 is never read -> its invalidation is lost ``` **2. Cursor advances even when a listener throws.** In `EntityChangeLogPoller#doPollChanges`, listener exceptions are caught and `entityPollHighWaterId` is advanced unconditionally. A single transient listener failure permanently drops that batch's invalidations. **3. `expireAfterAccess` turns a missed invalidation into permanent staleness.** `CaffeineEntityCache` builds the cache with `expireAfterAccess` (TTL resets on every read). A continuously-read hot key whose invalidation was lost via (1) or (2) never expires -> stale indefinitely. Cold keys self-heal; hot keys do not. ### Error message and/or stacktrace No crash/stacktrace -- silent stale reads. Observable as a remote node returning an outdated entity/relation after another node committed an ALTER/DROP/grant/revoke/setOwner. ### How to reproduce - Version: main branch, multi-node deployment sharing one backend DB. - (1): drive concurrent writes from multiple nodes so change-log inserts commit out of id order; observe a remote node never invalidating one of the changes. - (2): make a registered `EntityChangeLogListener` throw on one batch; observe the cursor advancing past it and the invalidation never being retried. - (3): keep reading a hot key on a remote node after its invalidation was missed; observe it never expiring despite the configured TTL. ### Additional context Suggested fixes (one PR can cover all three): 1. Cursor by `created_at` with a trailing overlap window + id-dedup, or a lagging high-water mark, to close the commit-ordering gap. 2. Do not advance the high-water past a batch whose listener failed (or retry that batch); keep listeners idempotent. 3. Use `expireAfterWrite` (alone or in addition to `expireAfterAccess`) so a missed invalidation degrades to bounded staleness instead of permanent. Found during design of entity-store cache multi-node invalidation; problems (1) and (2) already affect the existing jcasbin cache path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
