SwaraliJoshi commented on PR #7552:
URL: https://github.com/apache/hbase/pull/7552#issuecomment-4249390055
>but it is still a bit strange to me that why we close a store file reader
and then we still try to select for compaction?
So here we are actually seeing that the file is sent for compaction
(DateTieredCompaction) and simultaneously when the file is marked as compacted,
the reader is closed
*newCompactedfiles.forEach(HStoreFile::markCompactedAway);*
Here we are marking them compacted while a parallel thread is marking all
the marked files closed.
Whereas in the next step we have :
*compactedfiles = ImmutableList.sortedCopyOf(storeFileComparator,
Iterables.concat(compactedfiles, newCompactedfiles));*
Here the sorting happens based on timestamp (*reader.getMaximumTimestamp()*)
where we are seeing the NPE.
The root cause of the NPE is a race condition between store file reader
closure and compaction bookkeeping. During compaction, the system performs two
logically separate steps:
* Compaction bookkeeping
The StoreFileManager (e.g. DefaultStoreFileManager) removes compacted store
files from its internal active file set via addCompactionResults(...).
* Reader cleanup
The compacted HStoreFile’s reader is eventually closed (via
closeStoreFile(...)), either synchronously or asynchronously.
Before this fix, these two steps were not strictly ordered. As a result, it
was possible for a store file’s reader to be closed while the file was still
visible to compaction logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
