So here's the run after I resolved all the set up issues:
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-trunk-smoketest-22/14/testReport/
Here's what I see timing out:
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-trunk-smoketest-22/14/testReport/org.apache.bigtop.itest.hbase.smoke/TestHBaseCompression/testNoCompression/
http://bigtop01.cloudera.org:8080/view/Hadoop%200.22/job/Bigtop-trunk-smoketest-22/14/testReport/org.apache.bigtop.itest.hbase.smoke/TestHBaseCompression/testGzipCompression/
Which is basically simply:
$ hbase org.apache.hadoop.hbase.util.CompressionTest
hdfs://ip-10-32-33-167.ec2.internal:17020/user/root/snappy-output/testfile.gz
gz
$ hbase org.apache.hadoop.hbase.util.CompressionTest
hdfs://ip-10-32-33-167.ec2.internal:17020/user/root/snappy-output/testfile.none
none
And yes, they are hanging when executed by hand as well. Which is
weird too, since the
test itself actually completes, and exits
org.apache.hadoop.hbase.util.CompressionTest.main
and then everything freezes over with the following stack trace:
$ jstack 8412
2011-11-03 18:33:37
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f4e6c001000 nid=0x213a
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"DestroyJavaVM" prio=10 tid=0x00007f4edc009800 nid=0x2108 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"LeaseChecker" daemon prio=10 tid=0x00007f4edc7e6800 nid=0x211c
waiting on condition [0x00007f4e62fe0000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1476)
at java.lang.Thread.run(Thread.java:619)
"LRU Statistics #0" prio=10 tid=0x00007f4edc7dd800 nid=0x211a waiting
on condition [0x00007f4e631e2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f4e88acc968> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00007f4edc7d8800
nid=0x2119 in Object.wait() [0x00007f4e632e3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f4e88b1e570> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:568)
- locked <0x00007f4e88b1e570> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x00007f4edc0ff000 nid=0x2113
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x00007f4edc0fd000 nid=0x2112
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x00007f4edc0fa800 nid=0x2111
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f4edc0f8800 nid=0x2110
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007f4edc0f6800
nid=0x210f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f4edc0d8800 nid=0x210e in
Object.wait() [0x00007f4ed031a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f4e896b0620> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00007f4e896b0620> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x00007f4edc0d6800 nid=0x210d
in Object.wait() [0x00007f4ed041b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f4e896b0700> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00007f4e896b0700> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x00007f4edc0d2000 nid=0x210c runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f4edc018000
nid=0x2109 runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f4edc019800
nid=0x210a runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f4edc078000
nid=0x210b runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f4edc10a000 nid=0x2114
waiting on condition
JNI global references: 1268
Does this ring a bell, is this a problem we should pursue?
Thanks,
Roman.
