Having fixed the groomserver problem I did the following:
1) clean out /tmp files 2) format Hadoop namenode 3) start Hadoop 4) start HBase/Zookeeper 5) start Hama 6) launch Hama examples rand -m 10 -r 10 2000 2000 30.5% matrixA The outcome is puzzling. A 3-second long diary shows up in the Hama log file reporting nothing unusual. But the terminal reported problems with HBase. When I Googled this problem all I saw was someone who had multiple versions of HBase on their system. I am using a fresh VM with Ubuntu 8.04, Hadoop 0.20.1, Zookeeper 3.2.1, HBase 0.20.2, Hama 0.2.0 and JDK 1.6.0_17. No older versions of anything was ever installed. BTW: I thought the problem might be related to my running HBase in standalone mode so I just switched to HBase pseudo-distributed mode but I see the same problems. Any help appreciated. -- Alan Hama log ================================================================================================================================== Fri Dec 11 09:23:03 EST 2009 Starting master on ngc ulimit -n 1024 2009-12-11 09:23:05,512 INFO org.apache.hama.HamaMaster: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting HamaMaster STARTUP_MSG: host = ngc/127.0.1.1 STARTUP_MSG: args = [start] STARTUP_MSG: version = 0.20.1 STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009 ************************************************************/ 2009-12-11 09:23:05,939 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=HamaMaster, port=40000 2009-12-11 09:23:06,634 INFO org.apache.hama.HamaMaster: Cleaning up the system directory 2009-12-11 09:23:06,710 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2009-12-11 09:23:06,716 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 40000: starting 2009-12-11 09:23:06,721 INFO org.apache.hama.HamaMaster: Starting RUNNING 2009-12-11 09:23:06,721 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 40000: starting Hama terminal /hadoop-0.20.1-test.jar:/home/ngc/Desktop/hama/bin/../lib/hbase-0.20.0.jar:/home/ngc/Desktop/hama/bin/../lib/hbase-0.20.0-test.jar:/home/ngc/Desktop/hama/bin/../lib/jasper- compiler-5.5.12.jar:/home/ngc/Desktop/hama/bin/../lib/jasper-runtime-5.5.12.jar:/home/ngc/Desktop/hama/bin/../lib/javacc.jar:/home/ngc/Desktop/hama/bin/../lib/jetty-6.1.14.jar:/home/ngc/Desktop/hama/bin/../lib/jetty-util-6.1.14.jar:/home/ngc/Desktop/hama/bin/../lib/jruby-complete-1.2.0.jar:/home/ngc/Desktop/hama/bin/../lib/json.jar:/home/ngc/Desktop/hama/bin/../lib/junit-3.8.1.jar:/home/ngc/Desktop/hama/bin/../lib/libthrift-r771587.jar:/home/ngc/Desktop/hama/bin/../lib/log4j-1.2.13.jar:/home/ngc/Desktop/hama/bin/../lib/log4j-1.2.15.jar:/home/ngc/Desktop/hama/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/ngc/Desktop/hama/bin/../lib/xmlenc-0.52.jar:/home/ngc/Desktop/hama/bin/../lib/zookeeper-3.2.1.jar:/home/ngc/Desktop/hama/bin/../lib/jetty-ext/*.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/annotations.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/ant.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-analysis-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-commons-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-tree-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-util-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/asm-xml-3.0.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/bcel.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/dom4j-full.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/findbugs-ant.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/findbugsGUI.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/findbugs.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/jsr305.jar:/home/ngc/Desktop/hama/bin/../lib/findbugs/plugin/coreplugin.jar:/home/ngc/Desktop/hadoop-0.20.1/conf:/home/ngc/Desktop/hbase-0.20.2/conf 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/ngc/Desktop/jre1.6.0_17/lib/i386/client:/home/ngc/Desktop/jre1.6.0_17/lib/i386:/home/ngc/Desktop/jre1.6.0_17/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.24-23-generic 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:user.name=ngc 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/ngc 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/ngc/Desktop/hama 09/12/11 09:25:12 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@1d8957f 09/12/11 09:25:12 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false 09/12/11 09:25:12 INFO zookeeper.ClientCnxn: Attempting connection to server localhost/127.0.0.1:2181 09/12/11 09:25:12 INFO zookeeper.ClientCnxn: Priming connection to java.nio.channels.SocketChannel[connected local=/127.0.0.1:37044 remote=localhost/127.0.0.1:2181] 09/12/11 09:25:12 INFO zookeeper.ClientCnxn: Server connection successful 09/12/11 09:26:30 WARN client.HConnectionManager$TableServers: Testing for table existence threw exception org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts. Exceptions: java.io.IOException: java.io.IOException: java.lang.NullPointerException at java.lang.Class.searchMethods(Unknown Source) at java.lang.Class.getMethod0(Unknown Source) at java.lang.Class.getMethod(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) (repeated many times) HBase log 2009-12-11 09:22:56,110 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=0.9374695MB (983008), Free=198.75003MB (208404512), Max=199.6875MB (209387520), Counts: Blocks=1, Access=10, Hit=9, Miss=1, Evictions=0, Evicted=0, Ratios: Hit Ratio=89.99999761581421%, Miss Ratio=10.000000149011612%, Evicted/Run=NaN 2009-12-11 09:23:35,186 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 2.0 2009-12-11 09:23:35,839 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 127.0.1.1:37976, regionname: -ROOT-,,0, startKey: <>} 2009-12-11 09:23:35,876 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: 127.0.1.1:37976, regionname: -ROOT-,,0, startKey: <>} complete 2009-12-11 09:23:35,886 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: 127.0.1.1:37976, regionname: .META.,,1, startKey: <>} 2009-12-11 09:23:35,893 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 0 row(s) of meta region {server: 127.0.1.1:37976, regionname: .META.,,1, startKey: <>} complete 2009-12-11 09:23:35,893 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned 2009-12-11 09:23:56,084 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=0.9374695MB (983008), Free=198.75003MB (208404512), Max=199.6875MB (209387520), Counts: Blocks=1, Access=12, Hit=11, Miss=1, Evictions=0, Evicted=0, Ratios: Hit Ratio=91.66666865348816%, Miss Ratio=8.33333358168602%, Evicted/Run=NaN 2009-12-11 09:24:35,188 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead, average load 2.0 2009-12-11 09:24:35,840 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 127.0.1.1:37976, regionname: -ROOT-,,0, startKey: <>} 2009-12-11 09:24:35,864 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scan of 1 row(s) of meta region {server: 127.0.1.1:37976, regionname: -ROOT-,,0, startKey: <>} complete 2009-12-11 09:24:35,889 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {server: 127.0.1.1:37976, regionname: .META.,,1, startKey: <>} 2009-12-11 09:24:35,904 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scan of 0 row(s) of meta region {server: 127.0.1.1:37976, regionname: .META.,,1, startKey: <>} complete 2009-12-11 09:24:35,904 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META. region(s) scanned 2009-12-11 09:24:56,066 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=0.9374695MB (983008), Free=198.75003MB (208404512), Max=199.6875MB (209387520), Counts: Blocks=1, Access=14, Hit=13, Miss=1, Evictions=0, Evicted=0, Ratios: Hit Ratio=92.85714030265808%, Miss Ratio=7.14285746216774%, Evicted/Run=NaN 2009-12-11 09:25:02,258 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit to 4 2009-12-11 09:25:12,715 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 37976, call null([...@1c01ba9) from 127.0.1.1:55801: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at java.lang.Class.searchMethods(Unknown Source) at java.lang.Class.getMethod0(Unknown Source) at java.lang.Class.getMethod(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) (repeated many times)
