Here is one way of figuring out whether hdfs experiences the same issue is the following:
check out http://svn.apache.org/repos/asf/hadoop/common/branches/branch-1 use the command to find out which tests create MiniDFSCluster: find . -name '*.java' -exec grep 'new MiniDFSCluster(' {} \; -print run one of the tests: ant test-core -Dtestcase=TestCopyCommitter See if there is similar exception. On my Mac, I got: [junit] Running org.apache.hadoop.tools.distcp2.mapred.TestCopyCommitter [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 12.283 sec On Sun, Jul 14, 2013 at 9:58 AM, David Williams <[email protected]>wrote: > Hi Ted, > > I'd be interested to know more about an hdfs issue. I was go to look > further into the line of reasoning you mentioned about the call > to getAddress. > > MiniDFSCluster.java, line 426: > String ipAddr = dn.getSelfAddr().getAddress().getHostAddress(); > > So that one of getSelfAddr or getAddress returned NULL. Is there > something new about hdfs or are the two related? > > > > > > On Jul 13, 2013, at 9:41 PM, Ted Yu wrote: > > Most likely this is an hdfs issue. > > On Sat, Jul 13, 2013 at 4:08 PM, David Williams <[email protected] > > wrote: > >> Hi Ted, >> >> I updated the dependencies and ran the tests again, and on my Mac OSX >> they pass and on CentOS I get the same error: >> >> >> $ lein test >> Retrieving org/apache/hbase/hbase/0.94.9/hbase-0.94.9.pom from central >> Retrieving org/apache/hbase/hbase/0.94.9/hbase-0.94.9.jar from central >> Retrieving org/apache/hbase/hbase/0.94.9/hbase-0.94.9-tests.jar from >> central >> >> lein test hbase.config-test >> >> lein test hbase.table-test >> Starting DataNode 0 with dfs.data.dir: >> /home/dwilliams/Desktop/Repos/hbase/target/test-data/1140edc6-7242-40cd-8ed8-05847fb14949/dfscluster_1e40ce89-1986-450b-ba6d-983caa9aeb78/dfs/data/data1,/home/dwilliams/Desktop/Repos/hbase/target/test-data/1140edc6-7242-40cd-8ed8-05847fb14949/dfscluster_1e40ce89-1986-450b-ba6d-983caa9aeb78/dfs/data/data2 >> >> lein test :only hbase.table-test/create-table >> >> >> ERROR in (create-table) (MiniDFSCluster.java:426) >> Uncaught exception, not in assertion. >> expected: nil >> actual: java.lang.NullPointerException: null >> at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes >> (MiniDFSCluster.java:426) >> org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284) >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster >> (HBaseTestingUtility.java:451) >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster >> (HBaseTestingUtility.java:619) >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster >> (HBaseTestingUtility.java:575) >> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster >> (HBaseTestingUtility.java:562) >> hbase.table_test$test_config.doInvoke (table_test.clj:10) >> clojure.lang.RestFn.invoke (RestFn.java:397) >> hbase.table_test/fn (table_test.clj:19) >> >> On Jul 13, 2013, at 1:11 PM, Ted Yu <[email protected]> wrote: >> >> Do you mind trying the following change to see if the problem persists >> for latest 0.94 release ? >> >> Thanks >> >> diff --git a/project.clj b/project.clj >> index 2554784..0d8be0e 100644 >> --- a/project.clj >> +++ b/project.clj >> @@ -5,8 +5,8 @@ >> :dependencies [ >> [org.clojure/clojure "1.5.1"] >> [org.apache.hadoop/hadoop-core "1.2.0"] >> - [org.apache.hbase/hbase "0.94.6.1"] >> + [org.apache.hbase/hbase "0.94.9"] >> [org.apache.hadoop/hadoop-test "1.2.0"] >> - [org.apache.hbase/hbase "0.94.6.1" :classifier "tests"]] >> + [org.apache.hbase/hbase "0.94.9" :classifier "tests"]] >> :plugins [[lein-marginalia "0.7.1"]]) >> >> >> On Fri, Jul 12, 2013 at 10:22 PM, David Williams < >> [email protected]> wrote: >> >>> Hi Ted, >>> >>> In terms of versions, here are the jars I'm using, which come from Maven >>> Central. >>> >>> org.apache.hadoop/hadoop-core "1.2.0" >>> >>> org.apache.hbase/hbase "0.94.6.1" >>> org.apache.hadoop/hadoop-test "1.2.0" >>> >>> org.apache.hbase/hbase "0.94.6.1" :classifier "tests" >>> >>> >>> The flag ':classifier "tests"' above is a specific instruction to to the >>> Leiningen 2.0 to use the pomegranate library to handle Sonatype Aether and >>> dynamic runtime modification of the classpath, which in this case is needed >>> to import org.apache.hadoop.hbase HBaseTestingUtility. >>> >>> https://github.com/cemerick/pomegranate >>> >>> >>> I just checked on address resolution, on my Mac OSX where the >>> TestingUtility passes: >>> >>> user=> (import 'java.net.InetSocketAddress) >>> java.net.InetSocketAddress >>> user=> (def x (InetSocketAddress. 8000)) >>> #'user/x >>> user=> (.getAddress x) >>> #<Inet4Address 0.0.0.0/0.0.0.0> >>> user=> >>> >>> Then I check on a CentOS vm, the unit tests still fail but the address >>> resolution also worked in the repl and produced the same output as above. >>> >>> >>> >>> >>> On Jul 12, 2013, at 9:30 PM, Ted Yu wrote: >>> >>> I installed lein on Mac and Linux. >>> >>> I tried 'lein test' on both platforms and the test passed on both: >>> >>> lein test hbase.table-test >>> Starting DataNode 0 with dfs.data.dir: >>> /homes/hortonzy/mobius/target/test-data/246828b9-1be9-4949-9bbc-b215b378fb67/dfscluster_9ed0bd88-d309-4fed-9823-3bbf86973ae4/dfs/data/data1,/homes/hortonzy/mobius/target/test-data/246828b9-1be9-4949-9bbc-b215b378fb67/dfscluster_9ed0bd88-d309-4fed-9823-3bbf86973ae4/dfs/data/data2 >>> Cluster is active >>> >>> Ran 11 tests containing 14 assertions. >>> 0 failures, 0 errors. >>> >>> Here is Linux OS: >>> >>> Linux a.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13 >>> 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux >>> >>> Looking at MiniDFSCluster.java, line 426: >>> >>> String ipAddr = dn.getSelfAddr().getAddress().getHostAddress(); >>> >>> It seems dn.getSelfAddr().getAddress() returned null. >>> >>> According to: >>> >>> http://docs.oracle.com/javase/7/docs/api/java/net/InetSocketAddress.html#getAddress() >>> >>> This would mean address resolution problem. >>> >>> Can you check ? >>> >>> Cheers >>> >>> On Fri, Jul 12, 2013 at 7:37 PM, David Williams < >>> [email protected]> wrote: >>> >>>> Hi all, >>>> >>>> I am having an issue starting the a mini cluster for the >>>> HBaseTestingUtility. In short I can on Mac OSX, but cannot on Linux. But >>>> the error is cryptic and I don't know what to do next. >>>> >>>> I submitted a ticket with full details on StackOverflow, >>>> >>>> >>>> http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments >>>> >>>> But when I call .startMiniCluster on an instance of >>>> HBaseTestingUtility , on Linux (CentOS x86_64), I receive this error: >>>> >>>> >>>> >>>> ERROR in (create-table) (MiniDFSCluster.java:426) >>>> Uncaught exception, not in assertion. >>>> expected: nil >>>> actual: java.lang.NullPointerException: null >>>> at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes >>>> (MiniDFSCluster.java:426) >>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init> >>>> (MiniDFSCluster.java:284) >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster >>>> (HBaseTestingUtility.java:444) >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster >>>> (HBaseTestingUtility.java:612) >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster >>>> (HBaseTestingUtility.java:568) >>>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster >>>> (HBaseTestingUtility.java:555) >>>> >>>> I would appreciate help in finding out whats going on and how to set up >>>> my ENV to use the HBaseTestingUtility. >>>> >>>> Thanks >>>> David >>>> >>> >>> >>> >>> >> >> > >
