This definitely sounds like a serious problem. Agreed, it should be closed immediately. Also, a minor compaction shouldn't trigger any stats collection logic. Please file a JIRA and we'll get this fixed in the next release. As always, patches are welcome. Thanks, James
On Wed, Dec 28, 2016 at 7:37 PM 김연섭 <[email protected]> wrote: > In every minor compaction job of HBase, > > > > > > > > > > > > > > > > org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidePostDepth() > is called, > > and SYSTEM.CATALOG table is open to get guidepost width via > > htable = env.getTable( > > > > > > > > > > > > > > > > > > > > SchemaUtil.*getPhysicalTableName* > (PhoenixDatabaseMetaData.*SYSTEM_CATALOG_NAME_BYTES*, env > .getConfiguration())); > > This function call creates one zookeeper connection to get cluster id. > > DefaultStatisticsCollector doesn't close this zookeeper connection > immediately after get guidepost width, and the zookeeper connection remains > alive until HRegion is closed. > > This is not a problem with small number of Regions, but when number of > Region is large and upsert operation is frequent, the number of zookeeper > connection gradually increases to hundreds, and the zookeeper server nodes > experience short of available TCP/IP ports. > > I think this zookeeper connection should be closed immediately after get > guidepost width. > > My Phoenix environment is, > > - HBase : v1.1.8 > > - Phoenix : v4.9.0-HBase-1.1 > > - 80 nodes of hbase > > The following is the Dump stack of zookeeper connection creation > > > at java.lang.Thread.dumpStack(Thread.java:1329) > > at > org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:48) > > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1663) > > at > org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104) > > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:885) > > at > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:638) > > at > org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:99) > > at > org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:89) > > at > org.apache.hadoop.hbase.client.CoprocessorHConnection.getConnectionForEnvironment(CoprocessorHConnection.java:61) > > at > org.apache.hadoop.hbase.client.HTableWrapper.createWrapper(HTableWrapper.java:69) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:514) > > at > org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:503) > > at > org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidepostDepth(DefaultStatisticsCollector.java:123) > > at > org.apache.phoenix.schema.stats.DefaultStatisticsCollector.init(DefaultStatisticsCollector.java:299) > > at > org.apache.phoenix.schema.stats.DefaultStatisticsCollector.getInternalScanner(DefaultStatisticsCollector.java:293) > > at > org.apache.phoenix.schema.stats.DefaultStatisticsCollector.createCompactionScanner(DefaultStatisticsCollector.java:285) > > at > org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:692) > > at > org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:681) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > > at > org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) > > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39) > > at > org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:205) > > at > org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.preCompact(UngroupedAggregateRegionObserver.java:681) > > at > org.apache.hadoop.hbase.coprocessor.BaseRegionObserver.preCompact(BaseRegionObserver.java:195) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$9.call(RegionCoprocessorHost.java:612) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1708) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1785) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1757) > > at > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCompact(RegionCoprocessorHost.java:607) > > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.postCreateCoprocScanner(Compactor.java:247) > > at > org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:91) > > at > org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:119) > > at > org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1221) > > at > org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1845) > > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:521) > > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:557) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > > > > > >
