Catching the NPE exception and printing a more useful message is a good idea, but if there's other possible things that could cause an NPE, the "useful" message would be misleading. Perhaps a better solution might be to explicitly check the umask somehow and print/log a warning or something. Though from a quick Google search, it doesn't look like there's an easy way to check (or change) the umask from Java.
- Robert On Mon, Oct 15, 2012 at 7:51 AM, Jarek Jarcec Cecho <jar...@apache.org>wrote: > Thank you sir, > I appreciate your help! Do you think that it would make sense to also add > try-catch block in the setUp method for NullPointerException and print out > some warning message that hadoop was not correctly initialized and thus > there will be failures in the tests? I believe that it might people that > missed the umask requirement in docs to get it quickly resolved. > > Jarcec > > On Mon, Oct 15, 2012 at 01:05:44AM -0700, Mohammad Islam wrote: > > My previous reply was for another email. Sorry for the confusion. > > > > I found my umask also defaulted to 0022. That's why we never experienced > the issue. > > I agree we should update that Oozie doc. > > Created a JIRA: > > https://issues.apache.org/jira/browse/OOZIE-1021 > > > > > > Regards, > > Mohammad > > > > > > ________________________________ > > From: Jarek Jarcec Cecho <jar...@apache.org> > > To: dev@oozie.apache.org > > Sent: Sunday, October 14, 2012 3:16 PM > > Subject: Re: Oozie tests seems to be requiring umask 022 > > > > We probably miss understand each other sir. I've experienced described > issues during building Oozie on my local box before any deployment for real > life tests. I've justed checkout repository and tried to build it (mvn > package) - and almost all tests failed as I described in my original email. > Therefore I believe that there is no other special log other than test > related logs that I've read and provided appropriate lines, or at least > those lines that I believe are related to the issue. I might send entire > logs if you want me to. > > > > Jarcec > > > > On Sun, Oct 14, 2012 at 03:06:16PM -0700, Mohammad Islam wrote: > > > You have to see the launcher log. > > > 1. First click on the your ooze job. > > > 2. Select the corresponding shell action. it will pop-up a new window. > > > 3. Select the right of "Console URL" which will take you to the hadoop > JT site. > > > 4. Select hadoop map task and click through to find the log. > > > > > > Regards, > > > Mohammad > > > > > > > > > ________________________________ > > > From: Jarek Jarcec Cecho <jar...@apache.org> > > > To: dev@oozie.apache.org > > > Sent: Sunday, October 14, 2012 2:52 PM > > > Subject: Re: Oozie tests seems to be requiring umask 022 > > > > > > Hi Mohamad, > > > I've actually did not do anything special. I've just checked out svn > repository and executed "mvn test". Based on the maven output I'm expecting > that Oozie is compiling against 1.0.1 and 2.0.0-alpha, I'm not sure which > version is used in tests thought. > > > > > > Appropriate maven output that used: > > > [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT .... SUCCESS > [0.236s] > > > [INFO] Apache Oozie Hadoop Distcp 1.0.1.oozie-3.4.0-SNAPSHOT SUCCESS > [0.151s] > > > [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT Test SUCCESS > [0.360s] > > > [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT SUCCESS > [0.427s] > > > [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT Test > SUCCESS [0.411s] > > > [INFO] Apache Oozie Hadoop Distcp 2.0.0-alpha.oozie-3.4.0-SNAPSHOT > SUCCESS [0.137s] > > > > > > Jarcec > > > > > > On Fri, Oct 12, 2012 at 09:55:09PM -0700, Mohammad Islam wrote: > > > > Hi Jarek, > > > > Your test case is running against which hadoop version. > > > > Regards, > > > > Mohammad > > > > > > > > > > > > ________________________________ > > > > From: Jarek Jarcec Cecho <jar...@apache.org> > > > > To: oozie-...@incubator.apache.org > > > > Sent: Friday, October 12, 2012 11:27 AM > > > > Subject: Oozie tests seems to be requiring umask 022 > > > > > > > > Hi Oozie developers, > > > > I've recently worked on OOZIE-1012 and I've noticed that on my box > oozie tests seems to be requiring umask 022 (linux, Ubuntu 12.04), which > actually seems to be implied by HDFS not by Oozie. > > > > > > > > If I run tests with umask 002 (my default), most of the tests will > fail with following message: > > > > > > > > Cannot lock storage build/test/data/dfs/name1. The directory is > already locked. > > > > > > > > I believe that this error message is just a consequence of following > NullPointerException that appears in setUp method of affected test cases: > > > > > > > > java.lang.NullPointerException > > > > at > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) > > > > at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) > > > > at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124) > > > > at > org.apache.oozie.test.XTestCase.setUpEmbeddedHadoop(XTestCase.java:708) > > > > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:281) > > > > at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:58) > > > > at > org.apache.oozie.action.hadoop.ActionExecutorTestCase.setUp(ActionExecutorTestCase.java:62) > > > > > > > > I believe that reason of this NullPointerException is absence of > usable data directories as *-output logs contains following fragments: > > > > > > > > 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in > dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data1, > expected: rwxr-xr-x, while actual: rwxrwxr-x > > > > 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in > dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data2, > expected: rwxr-xr-x, while actual: rwxrwxr-x > > > > 12/10/12 11:16:12 ERROR datanode.DataNode: All directories in > dfs.data.dir are invalid. > > > > > > > > > > > > Please note that this output was generated with umask 002 and > changing it to 022 will fix the issue. > > > > > > > > Does anyone else noticed this behaviour (failures) as well? If so > then it might make sense to document this need on HowToContribute page [1] > and maybe improve the XTestCase.setUpEmbeddedHadoop by catching NPE and > informing developer that we're not able to bootstrap MiniDFSCluster? > > > > > > > > Jarcec > > > > > > > > Links: > > > > 1: http://incubator.apache.org/oozie/HowToContribute.html >
