[
https://issues.apache.org/jira/browse/HBASE-18059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037151#comment-16037151
]
Appy commented on HBASE-18059:
------------------------------
Looking at things more closely after [~tianjingyun]'s comment. HBASE-15236 is
year old, so had to dig deep again.
If the following invariant always holds, then you're right. [~stack] can you
please confirm, since my knowledge of seqId isn't much.
"Seq id of cells in memstore will always be great than that of any store file
or bulk loaded file."
-----
Detailed analysis:
KVHeap uses
[KVScannerComparator|https://github.com/apache/hbase/blob/e65d8653e566bbbae03578a1f9ad858cabcb48bc/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java#L191].
It first compares the cells from two scanners using a CellComparator. Only
when the comparison is equal, it uses scanner.getScannerOrder().
Thinking about the case when two scanners in a KeyValueHeap might return the
same key, KVScannerComparator will resolve ties as follows:
- memstore scanners vs storefile(SF) scanner:
-- if cell in SF has seqId: Its seqId should be less than that of memstore's
cell. Memstore scanner will win.
-- if cell in SF does NOT have seqId: SF'cell [defaults to
seqId=0|https://github.com/apache/hbase/blob/e65d8653e566bbbae03578a1f9ad858cabcb48bc/hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java#L142],
memstore scanner will win. (Note that seqId of cells in hfiles are removed on
major compaction if older than certain time, default is 5 days)
- memstore vs bulk loaded file(BLF)
-- memstore cell's will have higher seqId, so memstore scanner will win.
- SF vs SF -> Either cells' seqId will be used to tie break, or
getScannerOrder() (determined by store files' seq ids).
- SF vs BLF
-- if both cells have seqId -> one with higher seqId wins.
-- BLF's cell has seqId and SF's cell doesn't -> BLF scanner wins since SF's
cell's seqId defaults to 0. (This will happen when bulk load file is new but SF
is result of major compaction where cell's seqId were erased).
-- SF's cell has seqId but BLF's cell doesn't -> can't happen. BLF always have
seqId.
- BLF vs BLF
-- Cell's seqid will be used to tie break.
> The scanner order for memstore scanners are wrong
> -------------------------------------------------
>
> Key: HBASE-18059
> URL: https://issues.apache.org/jira/browse/HBASE-18059
> Project: HBase
> Issue Type: Bug
> Components: regionserver, scan, Scanners
> Affects Versions: 2.0.0
> Reporter: Duo Zhang
> Assignee: Jingyun Tian
> Priority: Critical
> Fix For: 2.0.0
>
>
> This is comments for KeyValueScanner.getScannerOrder
> {code:title=KeyValueScanner.java}
> /**
> * Get the order of this KeyValueScanner. This is only relevant for
> StoreFileScanners and
> * MemStoreScanners (other scanners simply return 0). This is required for
> comparing multiple
> * files to find out which one has the latest data. StoreFileScanners are
> ordered from 0
> * (oldest) to newest in increasing order. MemStoreScanner gets LONG.max
> since it always
> * contains freshest data.
> */
> long getScannerOrder();
> {code}
> As now we may have multiple memstore scanners, I think the right way to
> select scanner order for memstore scanner is to ordered from Long.MAX_VALUE
> in decreasing order.
> But in CompactingMemStore and DefaultMemStore, the scanner order for memstore
> scanner is also start from 0, which will be messed up with StoreFileScanners.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
