liaoxin01 opened a new pull request, #69980:
URL: https://github.com/apache/doris/pull/69980
### What problem does this PR solve?
Issue Number: None
Related PR: #64995
Problem Summary:
A cloud BE logs a burst of WARN lines with a full stack trace like
meet error status: [NOT_FOUND]File not found in global packed index:
data/<tablet>/<rowset>_0.idx
while compaction reads its input rowsets. In one instance a 4-hour BE log
had 42k
such lines over 21k paths in 158 tablets, yet every compaction succeeded.
Reproduce: write a rowset whose segment file is smaller than
small_file_threshold_bytes (so it is packed) but whose inverted index file is
larger (so it is uploaded as a regular object), then compact or query it.
Each
open of the regular .idx file prints the WARN with a stack, twice per
compaction
(eligibility check and the merge itself).
Root cause: #64995 made PackedFileSystem::open_file_impl fall back to
PackedFileManager::get_packed_slice_location when a path is not in
_index_map,
so that the file cache preload can read back a segment of the rowset being
written, whose rowset meta does not exist yet. The fallback was added
unconditionally, so it also runs for a PackedFileSystem built from a rowset
meta. That index map already holds every packed slice of the rowset, so a
miss
means a regular file, and the probe can only fail. The failure is a
Status::NotFound, and NOT_FOUND records a stack trace and logs a WARN as
soon as
it is constructed, before the caller falls back to opening the regular file.
Fix:
- Only consult the manager when the index map is not initialized, i.e. on the
write path, the same split exists_impl and file_size_impl already make.
- Return NOT_FOUND without a stack trace from get_packed_slice_location: on
the
write path a miss is also expected, for files too large to be packed.
After the fix, reading committed rowsets no longer takes the global index
lock
or logs anything for regular files; the write-path preload still finds packed
slices through the manager.
### Release note
Fix noisy "File not found in global packed index" WARN logs with stack traces
when reading rowsets that mix packed and regular files in cloud mode.
### Check List (For Author)
- Test: Unit Test
- Added
PackedFileSystemTest.OpenFileOfRowsetBeingWrittenUsesManagerLocation
and PackedFileSystemTest.OpenRegularFileOfOpenedRowsetIgnoresManager;
PackedFileSystemTest.* (24 tests) pass on an ASAN UT build.
- PackedFileManagerTest.GetMergeFileIndexForUnknownPathReturnsNotFound
passes.
PackedFileManagerTest.AppendSmallFileSuccess crashes in
FileCacheFactory::get_by_path when the suite is run on its own; it
crashes
the same way without this change.
- Behavior changed: Yes. A PackedFileSystem built from rowset meta no longer
looks paths up in the in-memory global packed index.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
