matanper opened a new issue, #68971: URL: https://github.com/apache/doris/issues/68971
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: `be/src/io/cache/fs_file_cache_storage.h:63-66` (one `_file_reader_list`, one `_file_name_to_reader` map and one `std::shared_mutex _mtx` per process), and `be/src/io/cache/fs_file_cache_storage.cpp:99-148` (`get_file_reader` and `contains_file_reader` take `_mtx` shared, while `insert_file_reader` and `remove_file_reader` take it exclusive). ### Description `FDCache` is the process-wide cache of open file-cache block readers (up to `file_cache_max_file_reader_cache_size`, default 1,000,000). Every read of a cached block goes through it: - each lookup (`FSFileCacheStorage::read` → `get_file_reader`) takes the single lock shared; - each miss inserts under the exclusive lock, and so does the eviction of the oldest entry; - each block removal (`FSFileCacheStorage::remove` → `remove_file_reader`) also takes the exclusive lock. With a write-through file cache under heavy load, blocks are added and evicted all the time. Readers on unrelated keys then queue behind those exclusive holders, and a waiting writer also blocks new shared lockers. On a BE with write-through on, `FDCache::get_file_reader` was one of the top off-CPU frames of compaction threads. There is also a small bug. The capacity check in `insert_file_reader` is `==` (`fs_file_cache_storage.cpp:119`). The config is mutable, so if it is lowered below the current size at runtime, the cache never shrinks: it evicts nothing and keeps growing. ### Solution Shard the map. Lookups (`get_file_reader`, `contains_file_reader`) take only their key's shard lock (e.g. 64 shards). The eviction order stays global, in a list guarded by its own mutex. Insert and remove always take that mutex before a shard lock, and lookups never take it. Capacity and oldest-first eviction therefore behave as before. Insert evicts with `>=` in a loop, so a lowered capacity takes effect on the next insert. ### How to Reproduce? Cloud mode with a file cache that load output is written into (write-through), or any other source of heavy cache churn. Run sustained loads plus compaction or queries that read the cache, and take an off-CPU profile of the BE. Threads wait on `FDCache::_mtx` in `get_file_reader`. ### Anything Else? The contention is from an off-CPU profile. We have not yet measured the patched build on a cluster. The patch adds unit tests: eviction order across shards, re-insert keeps the existing reader, remove frees a slot, shrinking the capacity, and an 8-reader / 4-writer concurrent stress test. These pass together with the existing `BlockFileCacheTest.fd_cache*` tests under ASAN. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
