[
https://issues.apache.org/jira/browse/HBASE-11747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655771#comment-16655771
]
Andrew Purtell commented on HBASE-11747:
----------------------------------------
[~stack] We had an interesting production problem where our monitoring would
call getClusterStatus, enough monitoring endpoints would do this at once, and
the total heap demand for assembling the responses on some of our larger
clusters would crash the active master with OOME. Branch-2 has API changes to
getClusterStatus that help, you can filter for specific items of interest.
However at sufficient scale it just won't work. We need incremental reporting,
incremental query.
Related, over on
https://issues.apache.org/jira/browse/PHOENIX-4974?focusedCommentId=16655747&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16655747
{quote}
At some point we should think about optimizing
{{getRegionLocator(...).getAllRegionLocations()}} for high scale installations.
For example, we could support incremental queries. Given a timestamp, return
only the changes in region location since that time. Then the client can
initialize its cache with an expensive retrieval once, and refresh its cache
from there with lighter weight incremental queries. Requires reimplementing the
equivalent of the old Region Historian over in HBase though. We could keep
historical locations with fairly short TTL in META. Will need splittable META
for high scale anyhow.
{quote}
> ClusterStatus (heartbeat) is too bulky
> ---------------------------------------
>
> Key: HBASE-11747
> URL: https://issues.apache.org/jira/browse/HBASE-11747
> Project: HBase
> Issue Type: Sub-task
> Reporter: Virag Kothari
> Priority: Critical
> Attachments: exceptiontrace
>
>
> Following exception on 0.98 with 1M regions on cluster with 160 region servers
> {code}
> Caused by: java.io.IOException: Call to regionserverhost:port failed on local
> exception: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1482)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1454)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.getClusterStatus(MasterProtos.java:42555)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$5.getClusterStatus(HConnectionManager.java:2132)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$16.call(HBaseAdmin.java:2166)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$16.call(HBaseAdmin.java:2162)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
> ... 43 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message was too large. May be malicious. Use
> CodedInputStream.setSizeLimit() to increase the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
