[ https://issues.apache.org/jira/browse/HBASE-21418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681082#comment-16681082 ]
Jeongdae Kim edited comment on HBASE-21418 at 11/9/18 2:41 PM: --------------------------------------------------------------- In the second patch, I tried to collect stats of columns with a bit of cost(stats collected only when flushing or compacting, because there are already a lot of comparisions in there), and use it to decide whether doing reseek or skip, when reseeking to next row, and use maxVersions when reseeking to next column as a heuristic. Please take a look again [~lhofhansl]. (Ah. I forgot to restore your comments about the deprecated HINT_LOOKAHEAD attribute. I will do it after your comments.) Thanks. was (Author: jeongdae kim): In the second patch, I tried to collect stats of columns with a bit of cost(stats collected only when flushing or compacting, because there are already a lot of comparisions in there), and use it to decide whether doing reseek or skip, when reseeking to next row, and use maxVersions when reseeking to next column as a heuristic. Please take a look again [~lhofhansl]. Thanks. > Reduce a number of reseek operations in MemstoreScanner when seek point is > close to the current row. > ---------------------------------------------------------------------------------------------------- > > Key: HBASE-21418 > URL: https://issues.apache.org/jira/browse/HBASE-21418 > Project: HBase > Issue Type: Improvement > Components: scan, Scanners > Affects Versions: 1.2.5 > Reporter: Jeongdae Kim > Assignee: Jeongdae Kim > Priority: Minor > Labels: performance > Attachments: HBASE-21418.branch-1.2.001.patch, > HBASE-21418.branch-1.2.001.patch, HBASE-21418.branch-1.2.002.patch > > > We observed “responseTooSlow” logs for Get requests in our production > clusters. even some get requests were responded after 10 seconds. > Affected get requests were done with the timerange, and target rows have many > columns that have some versions. > We reproduced this issue, and found this behavior happens only when scanning > in the memstore. after flushing the HStore, this slow response issue for Get > disappeared and all same get requests are responded very quickly. > > We investigated this case, and found this performance difference between > memstore scanner and hfile scanner is caused by the number of reseek > operations executed while scanning. When a store scanner needs to reseek the > next column, Hfile scanner wisely decide whether it have to reseek or not by > checking the seek point is in current block, whereas memstore scanner just do > reseek without decision unlike Hfile scanner. In our case, almost all columns > in the memstore have older timestamp than scan(get)’s timerange, and so many > reseek operations occur as much as about the number of columns. This results > in increasing the response time of Get requests sporadically. > > To improve the reseek operation of the memstore scanner, i think it’s better > skipping than seeking when reseek requested, if seek point is quite close to > current cell that the scanner is pointing now.(Actually, i changed > MatchCode.SEEK_NEXT_COL to MatchCode.Skip in our case, and the response time > of Get was 6x faster than before) But we can’t decide whether seek point is > close to the current cell or not, because memstore scannner has no > information such as next block index. > Before HBASE-13109, Scan.HINT_LOOKAHEAD was introduced to handle like this > case, and it may be deprecated someday. But, i think that hint is still be > useful for the memstore scanner to try to skip first, before reseeking, and > with this option we can make reseek operations of memstore scanner smarter. > > I tested this patch in our case, and got the same result as i changed > matchcode (mentioned above). -- This message was sent by Atlassian JIRA (v7.6.3#76005)