sundapeng opened a new pull request, #10573: URL: https://github.com/apache/paimon/pull/10573
### Purpose `RESTTokenFileIO` keeps data tokens in the process-wide `_TOKEN_CACHE`, and one lock per key in `_TOKEN_LOCKS`, keyed by `<identity>:<table>`. Neither is ever pruned. The identity is the access key id for DLF auth, which changes on every rotation of STS or ECS RAM role credentials, and the token itself for bearer auth. So a long-running process adds an entry for every table it touches after each rotation and keeps the expired ones forever, about 2 KB each: 20,000 tables and 5 identities reach 100,000 entries and 393 MB RSS. The Java `RESTTokenFileIO` has no such cache. This pull request drops expired tokens, and the idle locks of keys without a token, when caching a token finds the cache at a size threshold. The threshold starts at 1024 and then doubles with the live set, so pruning stays amortized O(1). A lock that a refresh still holds is kept; dropping an idle lock at worst lets two threads refresh the same key once. ### Tests - `test_token_cache_drops_expired_tokens_of_rotated_identities` covers dropped tokens, dropped idle locks, a kept held lock and the new threshold. The test file passes on Python 3.11 and 3.13. - A local churn script loads tokens for 20,000 tables under each of 10 identities, with a 1 s token lifetime. On master the cache grows to 200,000 entries and 711 MB RSS. With this change it stays between 21,000 and 34,000 entries and RSS levels off at about 185 MB. - flake8 with `dev/cfg.ini` passes. ### API and Format No. ### Documentation No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
