[
https://issues.apache.org/jira/browse/HBASE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483438#comment-13483438
]
Kannan Muthukkaruppan commented on HBASE-6728:
----------------------------------------------
Lars wrote: <<<Will this work right if I set scanner caching to 1000 and then
deal with 2mb rows? In that case every response will be 2g, and it would always
block and never make any progress, right?>>
Yes, we considered that, and that's the reason for not using a simple counting
semaphore that's initialize to the max size. We want the implementation to
allow one request to exceed the queue size. We set the default at 1G, but we
can exceed the limit by 1 requests' size amount.
>From
>http://svn.apache.org/viewvc/hbase/branches/0.89-fb/src/main/java/org/apache/hadoop/hbase/util/SizeBasedThrottler.java?view=markup&pathrev=1385058
> :
{code}
15 * This implementation allows you to set the value of internal
16 * counter to be greater than threshold. It happens
17 * when internal counter is lower than threshold and
18 * increase method is called with parameter 'delta' big enough
19 * so that sum of delta and internal counter is greater than
20 * threshold. This is not a bug, this is a feature.
21 * It solves some problems:
22 * - thread calling increase with big parameter will not be
23 * starved by other threads calling increase with small
24 * arguments.
25 * - thread calling increase with argument greater than
26 * threshold won't deadlock. This is useful when throttling
27 * queues - you can submit object that is bigger than limit.
28 *
29 * This implementation introduces small costs in terms of
30 * synchronization (no synchronization in most cases at all), but is
31 * vulnerable to races. For details see documentation of
32 * increase method.
{code}
> [89-fb] prevent OOM possibility due to per connection responseQueue being
> unbounded
> -----------------------------------------------------------------------------------
>
> Key: HBASE-6728
> URL: https://issues.apache.org/jira/browse/HBASE-6728
> Project: HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Michal Gregorczyk
> Fix For: 0.94.3
>
> Attachments: 6728.94, 6728-trunk.txt
>
>
> The per connection responseQueue is an unbounded queue. The request handler
> threads today try to send the response in line, but if things start to
> backup, the response is sent via a per connection responder thread. This
> intermediate queue, because it has no bounds, can be another source of OOMs.
> [Have not looked at this issue in trunk. So it may or may not be applicable
> there.]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
