matanper opened a new issue, #68972: URL: https://github.com/apache/doris/issues/68972
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: - `be/src/io/cache/cached_remote_file_reader.cpp:790-796`: on a failed read, the direct-read loop just `break`s, with the comment `// TODO: maybe read failed because block evict, should handle error`. - `be/src/io/cache/cached_remote_file_reader.cpp:178-185`: `_insert_file_reader` uses `_cache_file_readers.emplace(...)`, so a block that is downloaded again cannot replace the stale entry. - `be/src/io/cache/block_file_cache.h:164-166`: a block whose only extra reference is the reader's direct-read map (`_owned_by_cached_reader`) is `releasable()`, so it can be evicted. ### Description With `enable_read_cache_file_directly` (default true), `CachedRemoteFileReader` keeps a map from offset to `FileBlockSPtr` (`_cache_file_readers`), and reads those blocks without going through the cache lock. These blocks are evictable on purpose. So eviction can remove a block and delete its file while a reader still has the block in its map. The entry is never dropped: 1. The next direct read of that range calls `FileBlock::read`. That fails to open the deleted file, logs `open file failed` in `FSFileCacheStorage::read` (`fs_file_cache_storage.cpp:384/397`), and the loop breaks. The read then falls back to the indirect path. 2. The indirect path downloads the block again and calls `_insert_file_reader` with the new block. `emplace` does not overwrite the existing key, so the stale block stays. 3. Every later read of that offset by the same reader repeats steps 1 and 2: a failed open, a warning, and a fallback. Results are still correct, because the reader falls back. But each read of the range pays a failed `open()` plus the indirect path. On busy BEs this shows up as bursts of `open file failed` warnings for the same files, which is how we found it. The indirect path already heals a stale `DOWNLOADED` entry on `NOT_FOUND` (`need_self_heal`, `cached_remote_file_reader.cpp:1031-1066`), but the direct path does not. ### Solution - When a direct read fails, drop the entry from `_cache_file_readers`. Do it after releasing the shared lock, and only if the entry still maps the same block, then fall back as today. - In `_insert_file_reader`, use `insert_or_assign`, so a block downloaded again replaces the stale one. - Add a bvar that counts dropped stale blocks. ### How to Reproduce? 1. Keep `enable_read_cache_file_directly` on (the default) and use a small file cache. 2. Open a reader on a segment and read a range, so the block is cached and recorded in the reader's direct-read map. 3. Evict or clear that block from the cache (for example, cache pressure from other data, or a file cache clear). 4. Read the same range again with the same reader, several times. Each read logs `open file failed` and goes through the indirect path. We have a unit test, `direct_read_drops_evicted_block_and_maps_replacement`, that fails on the current code. ### Anything Else? Not covered here: stale metadata after a restart. Cache loaders trust the stored block metadata without checking that the file exists. Readers recover from that through the remote fallback. Related but separate: #68348 (blocks marked deleting while busy are never reclaimed; `_cache_file_readers` is one of the holders it lists). ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
