omg, looks like -Djava.net.preferIPv4Stack=true helped... probably some weird macos update...
2015-07-20 23:56 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>: > I degrade project with hbase testing utility to these versions: > <hadoop.version>2.5.0-cdh5.2.0</hadoop.version> > <hadoop.mr.version>2.5.0-mr1-cdh5.2.0</hadoop.mr.version> > <hbase.version>0.98.6-cdh5.2.0</hbase.version> > > It works. > > I upgrade to these: > > <hadoop.version>2.6.0-cdh5.4.4</hadoop.version> > <hadoop.mr.version>2.6.0-mr1-cdh5.4.4</hadoop.mr.version> > <hbase.version>1.0.0-cdh5.4.4</hbase.version> > > it hangs... > > 2015-07-20 23:27 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>: > >> I see these lines: >> >> 2015-07-20 21:27:21,791 INFO [RegionOpenAndInitThread-hbase:namespace-1] >> regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion >> hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => >> 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => >> 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => >> 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => >> 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = >> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp >> Table name == hbase:namespace >> >> 2015-07-20 21:27:21,802 INFO [RegionOpenAndInitThread-hbase:namespace-1] >> regionserver.HRegion (HRegion.java:doClose(1425)) - Closed >> hbase:namespace,,1437420441756.a69bbbcaf4a82786964da8d9cc62bea3. >> >> 2015-07-20 21:27:21,861 INFO >> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] hbase.MetaTableAccessor >> (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2 >> >> 2015-07-20 21:27:21,864 WARN >> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] zookeeper.ZKTableStateManager >> (ZKTableStateManager.java:setTableState(100)) - Moving table >> hbase:namespace state from ENABLING to ENABLED >> >> 2015-07-20 21:27:21,866 INFO >> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] handler.CreateTableHandler >> (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation >> successful >> >> >> then it hungs and prints thread dump in few minutes. >> >> >> 2015-07-20 21:25 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>: >> >>> >>> But do you see that thread printing anything in the logs? >>> >>> -- >>> Cloudera, Inc. >>> >>> >>> On Mon, Jul 20, 2015 at 12:07 PM, Serega Sheypak < >>> serega.shey...@gmail.com> wrote: >>> >>>> This is testing utiliy, it has few bytes of data to load. Running on >>>> oracle-jdk8 >>>> >>>> java.lang.Thread.run(Thread.java:745) >>>> Thread 178 (JvmPauseMonitor): >>>> State: TIMED_WAITING >>>> Blocked count: 3 >>>> Waited count: 398 >>>> Stack: >>>> java.lang.Thread.sleep(Native Method) >>>> >>>> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) >>>> java.lang.Thread.run(Thread.java:745) >>>> >>>> Thread 177 (JvmPauseMonitor): >>>> State: TIMED_WAITING >>>> Blocked count: 1 >>>> Waited count: 398 >>>> Stack: >>>> java.lang.Thread.sleep(Native Method) >>>> >>>> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) >>>> java.lang.Thread.run(Thread.java:745) >>>> >>>> 2015-07-20 19:30 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>: >>>> >>>>> -user@hbase >>>>> >>>>> Hi Serega, >>>>> >>>>> The RunTimeException is pointing to a timeout of nearly 3min. Have >>>>> tried to find in the master log lines what is causing that 3min pause? do >>>>> you see any log line related to the JvmPauseMonitor? (perhaps some GC >>>>> going >>>>> on) >>>>> >>>>> thanks, >>>>> esteban. >>>>> >>>>> -- >>>>> Cloudera, Inc. >>>>> >>>>> >>>>> On Mon, Jul 20, 2015 at 12:28 AM, Serega Sheypak < >>>>> serega.shey...@gmail.com> wrote: >>>>> >>>>>> @Sean, thanks. I saw sometimes Cloudera guys help here. I also used >>>>>> Cloudera community forum. >>>>>> >>>>>> @Jean-Marc, nothing special, just maven-plugin wrapper around >>>>>> miniHbaseCluster >>>>>> Here is the code where failure happen: new >>>>>> MiniHBaseCluster(configuration, >>>>>> 1); >>>>>> >>>>>> There are nothing special from my side. I'm surprised, it always >>>>>> worked >>>>>> since CDH 4.4, just bump dependency versions, fix code to follow API >>>>>> changes and that' all. >>>>>> >>>>>> 2015-07-20 4:08 GMT+02:00 Jean-Marc Spaggiari < >>>>>> jean-m...@spaggiari.org>: >>>>>> >>>>>> > Hi Serega, >>>>>> > >>>>>> > What kind of tests are your trying to run? The HBase test suite? Or >>>>>> > something you developed yourself? >>>>>> > >>>>>> > JM >>>>>> > >>>>>> > 2015-07-19 17:47 GMT-04:00 Serega Sheypak <serega.shey...@gmail.com >>>>>> >: >>>>>> > >>>>>> > > Hi, bumped my testing stuff to CDH 5.4.4 and got failure while >>>>>> running >>>>>> > > tests. Here is a log >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,607 INFO >>>>>> [SERGEYs-MBP:51977.activeMasterManager] >>>>>> > > master.TableNamespaceManager >>>>>> (TableNamespaceManager.java:start(85)) - >>>>>> > > Namespace table not found. Creating... >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,620 INFO [ProcessThread(sid:0 cport:-1):] >>>>>> > > server.PrepRequestProcessor >>>>>> (PrepRequestProcessor.java:pRequest(645)) - >>>>>> > Got >>>>>> > > user-level KeeperException when processing >>>>>> sessionid:0x14ea84290170000 >>>>>> > > type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error >>>>>> > > Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = >>>>>> NoNode for >>>>>> > > /hbase/table-lock/hbase:namespace >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,630 INFO >>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>>>> > > handler.CreateTableHandler (CreateTableHandler.java:process(189)) >>>>>> - >>>>>> > Create >>>>>> > > table hbase:namespace >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,644 INFO >>>>>> [RegionOpenAndInitThread-hbase:namespace-1] >>>>>> > > regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating >>>>>> > HRegion >>>>>> > > hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', >>>>>> BLOOMFILTER => >>>>>> > > 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS >>>>>> => >>>>>> > > 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', >>>>>> COMPRESSION => >>>>>> > > 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', >>>>>> BLOCKCACHE => >>>>>> > > 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = >>>>>> > > >>>>>> > > >>>>>> > >>>>>> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp >>>>>> > > Table name == hbase:namespace >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,655 INFO >>>>>> [RegionOpenAndInitThread-hbase:namespace-1] >>>>>> > > regionserver.HRegion (HRegion.java:doClose(1425)) - Closed >>>>>> > > hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b. >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,702 INFO >>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>>>> > > hbase.MetaTableAccessor >>>>>> (MetaTableAccessor.java:addRegionsToMeta(1169)) - >>>>>> > > Added 2 >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,704 WARN >>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>>>> > > zookeeper.ZKTableStateManager >>>>>> > (ZKTableStateManager.java:setTableState(100)) >>>>>> > > - Moving table hbase:namespace state from ENABLING to ENABLED >>>>>> > > >>>>>> > > 2015-07-19 23:40:18,706 INFO >>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>>>> > > handler.CreateTableHandler >>>>>> (CreateTableHandler.java:completed(219)) - >>>>>> > > Table, hbase:namespace, creation successful >>>>>> > > >>>>>> > > Process Thread Dump: Thread dump because: Master not initialized >>>>>> after >>>>>> > > 200000ms seconds >>>>>> > > >>>>>> > > 186 active threads >>>>>> > > >>>>>> > > Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0): >>>>>> > > >>>>>> > > State: WAITING >>>>>> > > >>>>>> > > Blocked count: 11 >>>>>> > > >>>>>> > > Waited count: 11 >>>>>> > > >>>>>> > > Waiting on >>>>>> > > >>>>>> > > >>>>>> > >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac >>>>>> > > >>>>>> > > Stack: >>>>>> > > >>>>>> > > sun.misc.Unsafe.park(Native Method) >>>>>> > > >>>>>> > > >>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>>> > > >>>>>> > > java.lang.Thread.run(Thread.java:745) >>>>>> > > >>>>>> > > Thread 249 (CatalogJanitor-SERGEYs-MBP:51977): >>>>>> > > >>>>>> > > >>>>>> > > ==== >>>>>> > > >>>>>> > > >>>>>> > > 2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster >>>>>> > > (MiniHBaseCluster.java:init(229)) - Error starting cluster >>>>>> > > >>>>>> > > java.lang.RuntimeException: Master not initialized after 200000ms >>>>>> seconds >>>>>> > > >>>>>> > > at >>>>>> > > >>>>>> > > >>>>>> > >>>>>> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225) >>>>>> > > >>>>>> > > at >>>>>> > > >>>>>> > > >>>>>> > >>>>>> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436) >>>>>> > > >>>>>> > > at >>>>>> > >>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224) >>>>>> > > >>>>>> > > at >>>>>> > > >>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93) >>>>>> > > >>>>>> > > at >>>>>> > > >>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80) >>>>>> > > >>>>>> > > at >>>>>> > > >>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67) >>>>>> > > >>>>>> > > >>>>>> > > fails here: MINI_HBASE_CLUSTER = new >>>>>> MiniHBaseCluster(configuration, 1); >>>>>> > > >>>>>> > > >>>>>> > > What it could be? Have no idea. >>>>>> > > >>>>>> > >>>>>> >>>>> >>>>> >>>> >>> >> >