[
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378625#comment-14378625
]
Jonathan Lawlor commented on HBASE-11544:
-----------------------------------------
Spoke offline with [~stack] and [~lhofhansl]. An idea of a solution to the
extra object creations that is outlined below:
* NextState should not be an object, but instead an enum. This is what it was
initially but it was changed into an object to avoid repeated calculation of
size
* A ScannerContext or ScannerTracker class can be introduced. A single
ScannerContext/ScannerTracker instance would be instantiated per call to
RSRpcServices#scan (so only one object creation per batch versus potentially
millions of object creations)
** ScannerContext would encapsulate all limits to be enforced during the server
side scan and would also encapsulate all limit tracking
The result would be that the RegionScanner interface would look something like
{code}
NextState nextRaw(List<Cell> result) throws IOException;
NextState nextRaw(List<Cell> result, ScannerContext context) throws IOException;
{code}
where the second call would be used when limits on the row need to be enforced.
This would allow us to communicate state explicitly, avoid excessive object
creation, and avoid repeated size calculations. How does this sound? I assume
this would be implemented as an addendum to this issue. [~stack] and
[~lhofhansl], please correct me if any details were missed here.
> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return
> batch even if it means OOME
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Lawlor
> Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Allocation_Hot_Spots.html,
> HBASE-11544-branch_1_0-v1.patch, HBASE-11544-branch_1_0-v2.patch,
> HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch,
> HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch,
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v7.patch,
> HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch, gc.j.png, h.png,
> hits.j.png, m.png, mean.png, net.j.png, q (2).png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000. Dataset has
> large cells. I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the
> client whatever we've gathered once we pass out a certain size threshold
> rather than keep accumulating till we OOME.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
