[ https://issues.apache.org/jira/browse/HBASE-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833704#comment-16833704 ]
Eshcar Hillel commented on HBASE-22324: --------------------------------------- nice work. nitpicking: the two test methods are not 100% the same one uses "this" in the c'tor and the other does not. Other than that +1. Thanks again. > loss a mass of data when the sequenceId of cells greater than Integer.Max, > because MemStoreMergerSegmentsIterator can not merge segments > ------------------------------------------------------------------------------------------------------------------------------------------ > > Key: HBASE-22324 > URL: https://issues.apache.org/jira/browse/HBASE-22324 > Project: HBase > Issue Type: Bug > Affects Versions: 2.1.0, 2.2.0 > Reporter: chenyang > Priority: Blocker > Labels: patch > Fix For: 2.1.0 > > Attachments: HBASE-22324.branch-2.1.0001.patch > > > if your memstore type is CompactingMemStore,MemStoreMergerSegmentsIterator > can not merge memstore segments when the seqId of cells greater than > Integer.Max, as a result, lossing a mass of data. the reason is that > MemStoreMergerSegmentsIterator use Integer.Max as readPt when create Scanner, > but the seqId of cell may be greater than Integer.MAX_VALUE, it`s type is > long. code as below: > {code:java} > public MemStoreMergerSegmentsIterator(List<ImmutableSegment> segments, > CellComparator comparator, > int compactionKVMax) throws IOException { > super(compactionKVMax); > // create the list of scanners to traverse over all the data > // no dirty reads here as these are immutable segments > AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners); > //bug, should use Long.MAX_VALUE > heap = new KeyValueHeap(scanners, comparator); > } > SegmentScanner.java code as below > protected void updateCurrent() { > Cell startKV = current; > Cell next = null; > try { > while (iter.hasNext()) { > next = iter.next(); > // here, if seqId>readPoint(Integer.MAX_VALUE), never read cell, as a > result, lossing lots of cells > if (next.getSequenceId() <= this.readPoint) { > current = next; > return;// skip irrelevant versions > } > if (stopSkippingKVsIfNextRow && // for backwardSeek() stay in the > startKV != null && // boundaries of a single row > segment.compareRows(next, startKV) > 0) { > current = null; > return; > } > } // end of while > current = null; // nothing found > } finally { > if (next != null) { > // in all cases, remember the last KV we iterated to, needed for > reseek() > last = next; > } > } > } > MemStoreCompactorSegmentsIterator has the same bug > public MemStoreCompactorSegmentsIterator(List<ImmutableSegment> segments, > CellComparator comparator, int compactionKVMax, HStore store) throws > IOException { > super(compactionKVMax); > List<KeyValueScanner> scanners = new ArrayList<KeyValueScanner>(); > AbstractMemStore.addToScanners(segments, Integer.MAX_VALUE, scanners); > //bug, should use Long.MAX_VALUE > // build the scanner based on Query Matcher > // reinitialize the compacting scanner for each instance of iterator > compactingScanner = createScanner(store, scanners); > refillKVS(); > }{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)