[
https://issues.apache.org/jira/browse/HBASE-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458898#comment-13458898
]
Jean-Daniel Cryans commented on HBASE-6841:
-------------------------------------------
I saw while jstack'ing that we're spending a lot of time in
ensureZookeeperTrackers:
{noformat}
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
- locked <0x000000057cad4918> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1016)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
- locked <0x000000057cad4808> (a
org.apache.hadoop.hbase.MasterAddressTracker)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
- locked <0x000000057cac1258> (a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:827)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:903)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:958)
- locked <0x00000005848519f0> (a java.lang.Object)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1492)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1377)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:702)
{noformat}
Then it would do other stuff like:
{noformat}
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
- locked <0x0000000580dd7158> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1016)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
- locked <0x0000000580dd7048> (a
org.apache.hadoop.hbase.MasterAddressTracker)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
...
{noformat}
Then we finish creating the HTable:
{noformat}
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:467)
at
org.apache.hadoop.hbase.zookeeper.ClusterId.readClusterIdZNode(ClusterId.java:62)
at org.apache.hadoop.hbase.zookeeper.ClusterId.getId(ClusterId.java:50)
at org.apache.hadoop.hbase.zookeeper.ClusterId.hasId(ClusterId.java:44)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1287)
- locked <0x000000057d819bf8> (a java.lang.String)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1250)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1237)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:946)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
...
{noformat}
Then the actual pre-fetching:
{noformat}
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1007)
- locked <0x000000057ed6cf90> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy11.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:218)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:120)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:206)
{noformat}
Then we close the connection:
{noformat}
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
- locked <0x000000057f29bdc8> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1229)
at org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:672)
- locked <0x000000057e9291d8> (a org.apache.zookeeper.ZooKeeper)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.close(RecoverableZooKeeper.java:553)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.close(ZooKeeperWatcher.java:452)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1751)
at
org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:273)
- locked <0x0000000583b59498> (a
org.apache.hadoop.hbase.client.HConnectionManager$1)
at
org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:257)
- locked <0x0000000583b59498> (a
org.apache.hadoop.hbase.client.HConnectionManager$1)
at
org.apache.hadoop.hbase.client.HConnectionManager.access$300(HConnectionManager.java:133)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.close(HConnectionManager.java:1759)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:365)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:903)
...
{noformat}
Then do it all over again:
{noformat}
- locked <0x000000057f16eab8> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:284)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:83)
- locked <0x000000057f16dae0> (a
org.apache.hadoop.hbase.MasterAddressTracker)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:591)
- locked <0x000000057f149dc8> (a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:827)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:903)
{noformat}
> Meta prefetching is slower than doing multiple meta lookups
> -----------------------------------------------------------
>
> Key: HBASE-6841
> URL: https://issues.apache.org/jira/browse/HBASE-6841
> Project: HBase
> Issue Type: Improvement
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.2
>
>
> I got myself into a situation where I needed to truncate a massive table
> while it was getting hits and surprisingly the clients were not recovering.
> What I see in the logs is that every time we prefetch .META. we setup a new
> HConnection because we close it on the way out. It's awfully slow.
> We should just turn it off or make it useful. jstacks coming up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
