[
https://issues.apache.org/jira/browse/HBASE-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945754#comment-13945754
]
Devaraj Das commented on HBASE-10785:
-------------------------------------
Looks good. Some nits:
1. We don't need the config (once we have tested this on real clusters)
2. The tableName argument in locateRegion is always going to be META. Might as
well not pass it and just declare it within the method.
> Metas own location should be cached
> -----------------------------------
>
> Key: HBASE-10785
> URL: https://issues.apache.org/jira/browse/HBASE-10785
> Project: HBase
> Issue Type: Improvement
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: hbase-10785_v1.patch, hbase-10785_v2.patch
>
>
> With ROOT table gone, we no longer cache the location of the meta table (in
> MetaCache) in 96+. I've checked 94 code, and there we cache meta, but not
> root.
> However, not caching the metas own location means that we are doing a
> zookeeper request every time we want to look up a regions location from meta.
> This means that there is a significant spike in zk requests whenever a region
> server goes down.
> This affects trunk,0.98 and 0.96 as well as hbase-10070 branch. I've
> discovered the issue in hbase-10070 because of the integration test
> (HBASE-10572) results in 150K requests to zk in 10min.
> A thread dump from one of the runs have 100+ threads from client in this
> stack trace:
> {code}
> "TimeBoundedMultiThreadedReaderThread_20" prio=10
> tid=0x00007f852c2f2000 nid=0x57b6 in Object.wait() [0x00007f85059e7000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at
> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
> - locked <0x00000000ea71aa78> (a
> org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1149)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:337)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:684)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1853)
> at
> org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:186)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:60)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1126)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1112)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1220)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1129)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:321)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.call(RpcRetryingCallerWithReadReplicas.java:257)
> - locked <0x00000000e9bcf238> (a
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:818)
> at
> org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.queryKey(MultiThreadedReader.java:288)
> at
> org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.readKey(MultiThreadedReader.java:249)
> at
> org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.runReader(MultiThreadedReader.java:192)
> at
> org.apache.hadoop.hbase.util.MultiThreadedReader$HBaseReaderThread.run(MultiThreadedReader.java:150)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
