[
https://issues.apache.org/jira/browse/HBASE-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387468#comment-14387468
]
stack commented on HBASE-13291:
-------------------------------
# The vint mvcc on end of keyvalue is expensive to parse; we can't make use of
the unsafe toLong which seems a good deal faster than get bytes and bit
noodling going by micro-benchmarking (I see 10x diff doing unsafe #getLong over
get byte and left shifting). This fact and the way we have tags tagged on end
(perhaps) frustrates go-fast efforts.
# Doing a getLong instead of two getInts and then bit noodling is faster than
doing two getInts (this is using unsafe -- I saw almost 10x diff). Added it in.
# Bulk of our time is in StoreFileScanner#_next
# Next is ScanQueryMatcher#match. It is not being very smart. It uses the Cell
Interface to pull out elements to compare only each getXXXoffset or
getXXXlength starts over from scratch reparsing key, value, row, etc. lengths
extracting ints so as to do some offset math. This is a killer.The worst
offender is the call from #match to #isCellTTLExpired In here we check to see
if tags are enabled. We do this by parsing key and value ints to figure end of
KV to poke for tags. All scans are paying for that time when DLR sets an MVCC
on a KV.
# If lots of cells being returned, then we will be doing lots of array
resizing. These show in the profilings.
The HFileReaderV3 and then HFileReaderV2 with an abstract under this makes it
hard to navigate what is going on. Need to flatten now we are up on v3 file
format and all is on by default.
> Lift the scan ceiling
> ---------------------
>
> Key: HBASE-13291
> URL: https://issues.apache.org/jira/browse/HBASE-13291
> Project: HBase
> Issue Type: Improvement
> Components: Scanners
> Affects Versions: 1.0.0
> Reporter: stack
> Assignee: stack
> Attachments: 13291.inlining.txt, Screen Shot 2015-03-26 at 12.12.13
> PM.png, Screen Shot 2015-03-26 at 3.39.33 PM.png, hack_to_bypass_bb.txt,
> nonBBposAndInineMvccVint.txt, q (1).png, traces.7.svg, traces.filterall.svg,
> traces.nofilter.svg, traces.small2.svg, traces.smaller.svg
>
>
> Scanning medium sized rows with multiple concurrent scanners exhibits
> interesting 'ceiling' properties. A server runs at about 6.7k ops a second
> using 450% of possible 1600% of CPUs when 4 clients each with 10 threads
> doing scan 1000 rows. If I add '--filterAll' argument (do not return
> results), then we run at 1450% of possible 1600% possible but we do 8k ops a
> second.
> Let me attach flame graphs for two cases. Unfortunately, there is some
> frustrating dark art going on. Let me try figure it... Filing issue in
> meantime to keep score in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
