[ https://issues.apache.org/jira/browse/HBASE-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191791#comment-15191791 ]
Lars Hofhansl commented on HBASE-15392: --------------------------------------- Here's the comment that we can add to optimize: {code} * This works together with ColumnTrackers and Filters. ColumnTrackers may issue seek hints, * such as seek to next column, next row, or an arbitrary seek key. * Optimize intercepts these and decides whether a seek is the most efficient _actual_ way * to get us to the requested cell. * It does this looking at the next indexed key of the current HFile as a heuristic, this key * is then compared with the seek key (this is optimized here to avoid actually creating the * seek key, it just does the compare as if we had a seek key). If the seek gets us onto the * next block we seek, otherwise we just INCLUDE or SKIP, and let the ColumnTrackers or Filters * go through the next Cell, and so on) * * The ColumnTrackers and Filters must behave correctly in all cases, i.e. if past the Cells * they care about they must issues a SKIP or SEEK. That's the right behavior anyway. {code} In the end it's very simple, really: When ColumnTracker or Filter suggest that we _could_ do a seek, we compare the next indexed key (which may or may not be real, it comes from an HFile) with the seek key (which is generated from the current Cell, and not real), if the seek key is larger, we seek, since we'll end on another block for sure, if the seek key is smaller we assume that skipping on the current block is cheaper. In my previous email meant that we create the _seek key_ this way. Not next indexed key. The seek key isn't real. Checkout StoreScanner.seekToNextRow. {code} protected boolean seekToNextRow(KeyValue kv) throws IOException { return reseek(matcher.getKeyForNextRow(kv)); } {code} Look how {{matcher.getKeyForNextRow(kv)}} creates a fake KV out of the current KV, such that it sorts after all Cells of the current row (empty cell and family and type=minimum), look at how Cells are compared when the cell is empty and type=minimum. That fake key is the key that is used for seeking. When we do the seek, that's where we land: Right after the last Cell of the current row and before (but not on) the first Cell of the next row. The compares in optimize() do the _exact same_ compare (just that for optimization the seek key is not actually created, that would be too expensive, it's _identical_ to compare matcher.getKeyForNextRow(kv) with next indexed key). > Single Cell Get reads two HFileBlocks > ------------------------------------- > > Key: HBASE-15392 > URL: https://issues.apache.org/jira/browse/HBASE-15392 > Project: HBase > Issue Type: Sub-task > Components: BucketCache > Reporter: stack > Assignee: stack > Attachments: 15392-0.98-looksee.txt, 15392.wip.patch, > 15392v2.wip.patch, 15392v3.wip.patch, 15392v4.patch, 15392v5.patch, > HBASE-15392_suggest.patch, gc.png, gc.png, io.png, no_optimize.patch, > no_optimize.patch, reads.png, reads.png, two_seeks.txt > > > As found by Daniel "SystemTap" Pol, a simple Get results in our reading two > HFileBlocks, the one that contains the wanted Cell, and the block that > follows. > Here is a bit of custom logging that logs a stack trace on each HFileBlock > read so you can see the call stack responsible: > {code} > 2016-03-03 22:20:30,191 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner: > START LOOP > 2016-03-03 22:20:30,192 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner: > QCODE SEEK_NEXT_COL > 2016-03-03 22:20:30,192 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex: > STARTED WHILE > 2016-03-03 22:20:30,192 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache: > OUT OF L2 > 2016-03-03 22:20:30,192 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read > offset=31409152, len=2103 > 2016-03-03 22:20:30,192 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine: > offset=31409152, length=2103 > 2016-03-03 22:20:30,193 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl: > From Cache [blockType=DATA, fileOffset=2055421, headerSize=33, > onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020, > prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053, > getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true, > buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54], > dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231, > fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, > includesTags=false, compressAlgo=NONE, compressTags=false, > cryptoContext=[cipher=NONE keyHash=NONE]]] > 2016-03-03 22:20:30,193 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl: > Cache hit return [blockType=DATA, fileOffset=2055421, headerSize=33, > onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020, > prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053, > getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true, > buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54], > dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231, > fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, > includesTags=false, compressAlgo=NONE, compressTags=false, > cryptoContext=[cipher=NONE keyHash=NONE]]] > java.lang.Throwable > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$CellBasedKeyBlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:324) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:831) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.reseekTo(HFileReaderImpl.java:812) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:288) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:198) > at > org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:321) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:279) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:806) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:795) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:624) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5703) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5849) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5622) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5598) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5584) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2187) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2123) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33512) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 22:20:30,193 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex: > READ [blockType=DATA, fileOffset=2055421, headerSize=33, > onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020, > prevBlockOffset=2053364, isUseHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053, > getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true, > buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@e19fbd54], > dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995139035672819231, > fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, > includesTags=false, compressAlgo=NONE, compressTags=false, > cryptoContext=[cipher=NONE keyHash=NONE]]] > 2016-03-03 22:20:30,193 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex: > DATA BLOCK IS TRUE BREAK > 2016-03-03 22:20:30,193 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileBlockIndex: > RETURNING > 2016-03-03 22:20:30,194 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner: > QCODE INCLUDE > 2016-03-03 22:20:30,194 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner: > QCODE SKIP > 2016-03-03 22:20:30,194 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.CombinedBlockCache: > OUT OF L2 > 2016-03-03 22:20:30,194 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.BucketCache: Read > offset=30932992, len=2103 > 2016-03-03 22:20:30,194 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] bucket.FileIOEngine: > offset=30932992, length=2103 > 2016-03-03 22:20:30,194 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl: > From Cache [blockType=DATA, fileOffset=2057478, headerSize=33, > onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020, > prevBlockOffset=2055421, isUseHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053, > getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true, > buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@6063ac95], > dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995698996184959679, > fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, > includesTags=false, compressAlgo=NONE, compressTags=false, > cryptoContext=[cipher=NONE keyHash=NONE]]] > 2016-03-03 22:20:30,194 TRACE > [B.defaultRpcServer.handler=20,queue=2,port=16020] hfile.HFileReaderImpl: > Cache hit return [blockType=DATA, fileOffset=2057478, headerSize=33, > onDiskSizeWithoutHeader=2024, uncompressedSizeWithoutHeader=2020, > prevBlockOffset=2055421, isUseHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, onDiskDataSizeWithHeader=2053, > getOnDiskSizeWithHeader=2057, totalChecksumBytes=4, isUnpacked=true, > buf=[org.apache.hadoop.hbase.nio.SingleByteBuff@6063ac95], > dataBeginsWith=\x00\x00\x00)\x00\x00\x01`\x00\x16user995698996184959679, > fileContext=[usesHBaseChecksum=true, checksumType=CRC32C, > bytesPerChecksum=16384, blocksize=65536, encoding=NONE, includesMvcc=true, > includesTags=false, compressAlgo=NONE, compressTags=false, > cryptoContext=[cipher=NONE keyHash=NONE]]] > java.lang.Throwable > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1515) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readNextDataBlock(HFileReaderImpl.java:906) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.isNextBlock(HFileReaderImpl.java:1106) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.positionForNextBlock(HFileReaderImpl.java:1100) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1118) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1139) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:152) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:114) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:628) > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5703) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5849) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5622) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5598) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5584) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2187) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2123) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33512) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > 2016-03-03 22:20:30,195 INFO > [B.defaultRpcServer.handler=20,queue=2,port=16020] regionserver.StoreScanner: > QCODE DONE > {code} > We are in StoreScanner#next. > Matcher does SEEK_NEXT_COL. > We pull in the block that has our Cell in it. > Matcher does INCLUDE > Then Matcher does SKIP > SKIP has us go read the next block. > Then Matcher does DONE. > Why ain't I getting to DONE after INCLUDE? > Any clues? > This code has been like this a while. -- This message was sent by Atlassian JIRA (v6.3.4#6332)