Hi Greg, Thanks for your questions (copying the dev@ list per our discussion). My replies inline below:
-----Original Message----- From: Greg Earle <ea...@mipl.jpl.nasa.gov> Date: Wednesday, June 19, 2013 10:55 PM To: jpluser <chris.a.mattm...@jpl.nasa.gov> Cc: Greg Earle <ea...@mipl.jpl.nasa.gov>, "Lee, Hyun H (398J)" <hyun....@jpl.nasa.gov> Subject: OODT build issues (Fwd: APPS Third-party (TPS) software component list) >Hi Chris (and Hyun), > >I was asked by Costin Radulescu to install Apache OODT on one of our >Dev systems. It's a Linux box running 64-bit Red Hat RHEL 6.3. > >He had sent me a TPS request (formalized form we use in MIPL) asking >for the "Latest" version, but later on sent an Excel spreadsheet that >specifically mentioned version 0.3. (But I think that was only because >he made up the spreadsheet some months back.) Yep latest version is 0.6-SNAPSHOT (trunk). Should be released in the next week or so, but it's fairly stable right now. I'm going to RM it. > >I first tried using Maven 3.0.1 but it splatted all over the place. 0.3 will do that to you since we didn't support Maven3 until OODT-369 [1] which will ship in 0.6 (and is available now in trunk/0.6-SNAPSHOT). >I fell back to Maven 2.2.1 which seemed to make OODT a lot happier. Heh, yep. > >Anyway, I was able to build 0.3 OK; but then I thought I should try to >build the latest & greatest 0.5. The following message chronicles the >2 major problems I ran into (in copious detail - sorry) with both 0.5 >and 0.4 (which I tried after 0.5 failed). Executive summary: > >(1) Catalog and Archive Service Generic Multi-valued Metadata Container > > A couple of the tests seem to have a race condition where they try to > clean up their mess and remove a temporary metadata .tests directory > before all the files are completely removed (NFS issue; possibly due > to caching/sync), provoking 2 ENOTEMPTY errors. Yep this is an NFS issue -- you can always change this with a java.io.tmpDir property (passed to Maven with mvn -Djava.io.tmpDir=/some/where/not/on/nfs). Issue described in detail here: https://issues.apache.org/jira/browse/OODT-110 > >(2) CAS Protocol SFTP Implementation > > The "TestJschSftpProtocol" test is failing because it can't contact > whatever server it is trying to contact ("Connection refused"). Odd, it should be trying to connect locally. > >If you could give a quick glance-over to this (overly verbose, sorry) >transcript and suggest anything (like resources to consult) that would >help me potentially get past these 2 issues, I'd appreciate it. No worries -- if you get failing tests like the above and they are blocking progress you can always type mvn -Dmaven.skip.tests=true install and then that should get you past. > >In the meantime I'm going to ask Costin if he can get along with using >version 0.3 in the interim. Try 0.6-SNAPSHOT -- should work better :) Also happy to answer any further questions here on list. Cheers! Chris [1] https://issues.apache.org/jira/browse/OODT-369 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >Begin forwarded message: > >> From: Greg Earle <ea...@mipl.jpl.nasa.gov> >> Date: June 19, 2013 10:28:57 PM PDT >> To: Costin Radulescu <crad...@jpl.nasa.gov>, "Levoe, Steven R (398D)" >><steven.r.le...@jpl.nasa.gov> >> Cc: Alex Leung <alex.le...@jpl.nasa.gov>, Greg Earle >><ea...@mipl.jpl.nasa.gov> >> Subject: Re: APPS Third-party (TPS) software component list >> content-type: text/plain; charset=us-ascii >> >> On Jun 19, 2013, at 5:17 PM, Greg Earle wrote: >> >>> I installed the "apache-oodt-0.3-src.tar.gz" file into "external" >>> because I think you said it had been donated(?) to The Apache >>>Foundation >>> and thus was no longer JPL/NASA Proprietary ... right? >>> >>> I compared the Apache version to the old JPL one and the JPL one >>> just has a "oodt.jar" file (with some docs) - but the Apache one >>> looks like it's a zillion ".java" files that need to be built with >>> Maven, which isn't installed yet. >> >> [Removed Janet from the long boring technical discussion. Sorry Janet.] >> >> Update: >> >> OK - so I found a Maven RPM >>(apache-maven-3.0.3-1.el6.sroycode.noarch.rpm) >> and installed it. >> >> The README.txt file in the OODT 0.3 "src" directory says to just run >> "mvn install" to build it. I tried that but it croaked. >> >> Then I looked in the README.txt file again and it says it uses Maven2. >> >> So I installed Maven 2.2.1 instead (into externals, alongside >>apache-oodt - >> no RPM for it that I could find which didn't have a ton of dependencies) >> and this time it built all the way through; albeit with a bunch of >> "WARNING" messages. >> >> It looks like it built a bunch of .jar files, rather than a single >> "oodt.jar" file like back in the pre-Apache, JPL days. To compare: >> >> -- >> [root@miplapps2 v0.3]# pwd >> /usr/local/vicar/external/apache-oodt/v0.3 >> >> [root@miplapps2 v0.3]# ls -1lF */target/*.jar >> -rw-r--r-- 1 cmbld cm 235380 Jun 19 17:57 >>catalog/target/cas-catalog-0.3.jar >> -rw-r--r-- 1 cmbld cm 219372 Jun 19 17:55 >>commons/target/oodt-commons-0.3.jar >> -rw-r--r-- 1 cmbld cm 65682 Jun 19 17:58 >>crawler/target/cas-crawler-0.3.jar >> -rw-r--r-- 1 cmbld cm 463421 Jun 19 17:57 >>filemgr/target/cas-filemgr-0.3.jar >> -rw-r--r-- 1 cmbld cm 65707 Jun 19 17:56 >>metadata/target/cas-metadata-0.3.jar >> -rw-r--r-- 1 cmbld cm 37947 Jun 19 18:00 >>opendapps/target/opendapps-0.3.jar >> -rw-r--r-- 1 cmbld cm 49859 Jun 19 17:59 pge/target/cas-pge-0.3.jar >> -rw-r--r-- 1 cmbld cm 42226 Jun 19 18:00 >>product/target/oodt-product-0.3.jar >> -rw-r--r-- 1 cmbld cm 97752 Jun 19 18:00 >>profile/target/oodt-profile-0.3.jar >> -rw-r--r-- 1 cmbld cm 257273 Jun 19 18:00 >>pushpull/target/cas-pushpull-0.3.jar >> -rw-r--r-- 1 cmbld cm 104657 Jun 19 17:57 >>resource/target/cas-resource-0.3.jar >> -rw-r--r-- 1 cmbld cm 10788 Jun 19 17:56 sso/target/oodt-sso-0.3.jar >> -rw-r--r-- 1 cmbld cm 172360 Jun 19 17:58 >>workflow/target/cas-workflow-0.3.jar >> -rw-r--r-- 1 cmbld cm 57928 Jun 19 18:00 >>xmlps/target/oodt-xmlps-0.3.jar >> -rw-r--r-- 1 cmbld cm 52906 Jun 19 17:56 >>xmlquery/target/oodt-xmlquery-0.3.jar >> >> [root@miplapps2 v1.1.0]# pwd >> /usr/local/vicar/proprietary/oodt/v1.1.0 >> >> [root@miplapps2 v1.1.0]# ls -1lF `find . -name \*.jar -print` >> -rw-rw-r-- 1 1662 dev 262796 Jun 19 2000 ./lib/oodt.jar >> -- >> >> I hope you can find what you're looking for amongst all those JAR files. >> >> -- >> >> BTW Costin, I noticed in your original TPS request, you asked for the >> "Latest" version of OODT. But in your Excel spreadsheet you forwarded >> to me, you specified version 0.3. (The current version is 0.5.) >> >> So I tried to build 0.5 but it failed. Then I downloaded 0.4 and tried >> to build that version; it also failed. >> >> Can we get by with 0.3? >> >> - Greg >> >> P.S. The tediously long analysis of the 0.4/0.5 build failure messages: >> >> -- >> [INFO] >>------------------------------------------------------------------------ >> [INFO] Building Catalog and Archive Service Generic Multi-valued >>Metadata Container >> [INFO] task-segment: [install] >> [INFO] >>------------------------------------------------------------------------ >> [...] >> >> ------------------------------------------------------- >> T E S T S >> ------------------------------------------------------- >> Running org.apache.oodt.cas.metadata.TestMetadata >> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec >> [...] >> Running >>org.apache.oodt.cas.metadata.extractors.TestCopyAndRewriteExtractor >> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.053 >>sec <<< FAILURE! >> [...] >> >> Results : >> >> Tests in error: >> >>testExtractMetadata(org.apache.oodt.cas.metadata.extractors.TestCopyAndRe >>writeExtractor): Cannot delete temporary directory >>/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/metadata6477 >>311802555701610.tests >> >>testReplaceOrigMetFilePath(org.apache.oodt.cas.metadata.extractors.TestCo >>pyAndRewriteExtractor): Cannot delete temporary directory >>/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/metadata6208 >>598354758363243.tests >> >> Tests run: 20, Failures: 0, Errors: 2, Skipped: 0 >> >> [INFO] >>------------------------------------------------------------------------ >> [ERROR] BUILD FAILURE >> [INFO] >>------------------------------------------------------------------------ >> [INFO] There are test failures. >> >> Please refer to >>/pkg/vicar/external/apache-oodt/v0.4/metadata/target/surefire-reports >>for the individual test results. >> -- >> >> I don't know why it can't delete these temporary directories. I am >>running >> as "root"! >> >> The stack traceback of the failed test was >> >> -- >> >>testExtractMetadata(org.apache.oodt.cas.metadata.extractors.TestCopyAndRe >>writeExtractor) Time elapsed: 0.046 sec <<< ERROR! >> java.io.IOException: Cannot delete temporary directory >>/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/metadata3611 >>359663504997822.tests >> at >>org.apache.oodt.cas.metadata.MetadataTestCase.tearDown(MetadataTestCase.j >>ava:75) >> at >>org.apache.oodt.cas.metadata.extractors.TestCopyAndRewriteExtractor.tearD >>own(TestCopyAndRewriteExtractor.java:151) >> [...] >> >> and >> >> >>testReplaceOrigMetFilePath(org.apache.oodt.cas.metadata.extractors.TestCo >>pyAndRewriteExtractor) Time elapsed: 0.025 sec <<< ERROR! >> java.io.IOException: Cannot delete temporary directory >>/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/metadata4674 >>468323809077561.tests >> at >>org.apache.oodt.cas.metadata.MetadataTestCase.tearDown(MetadataTestCase.j >>ava:75) >> at >>org.apache.oodt.cas.metadata.extractors.TestCopyAndRewriteExtractor.tearD >>own(TestCopyAndRewriteExtractor.java:151) >> [...] >> -- >> >> "MetadataTestCase.java:75" is >> >> >>http://oodt.apache.org/components/maven/xref-test/org/apache/oodt/cas/met >>adata/MetadataTestCase.html >> >> -- >> if (!tmpDir.delete()) >> // Nuke the play area >> throw new IOException("Cannot delete temporary directory " + >>tmpDir); >> -- >> >> I ran it under "strace" and I see things like >> >> -- >> 31222 >>unlink("/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/meta >>data7770994790488970829.tests") = -1 EISDIR (Is a directory) >> 31222 >>unlink("/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/meta >>data5397511724788750068.tests/samplemet.xml" <unfinished ...> >> 31222 >>unlink("/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/meta >>data5397511724788750068.tests") = -1 EISDIR (Is a directory) >> 31222 >>rmdir("/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/metad >>ata5397511724788750068.tests") = -1 ENOTEMPTY (Directory not empty) >> 31222 >> >> >>unlink("/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/meta >>data8856046235876187471.tests") = -1 EISDIR (Is a directory) >> 31222 >>rmdir("/pkg/vicar/external/apache-oodt/v0.4/metadata/target/classes/metad >>ata8856046235876187471.tests") = -1 ENOTEMPTY (Directory not empty) >> 31222 >> -- >> >> So for some reason the directory isn't completely empty when it goes to >> delete it in each of the 2 failing tests. I don't really have any ideas >> at this point. I believe it is an NFS issue of some sort, because ... >> >> ... I then tried to build both 0.4 and 0.5 in a local (non-NFS) >>directory, >> under "/var" - and they both failed in a completely different way - an >>SFTP >> test harness that fails due to "Connection refused": >> >> -- >> [...] >> >>------------------------------------------------------------------------- >>------ >> Test set: org.apache.oodt.cas.protocol.sftp.TestJschSftpProtocol >> >>------------------------------------------------------------------------- >>------ >> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.766 >>sec <<< FAILURE! >> testGET(org.apache.oodt.cas.protocol.sftp.TestJschSftpProtocol) Time >>elapsed: 0.01 sec <<< ERROR! >> org.apache.oodt.cas.protocol.exceptions.ProtocolException: Failed to >>connect to host localhost : java.net.ConnectException: Connection refused >> at >>org.apache.oodt.cas.protocol.sftp.JschSftpProtocol.connect(JschSftpProtoc >>ol.java:119) >> at >>org.apache.oodt.cas.protocol.sftp.TestJschSftpProtocol.testGET(TestJschSf >>tpProtocol.java:122) >> Caused by: com.jcraft.jsch.JSchException: java.net.ConnectException: >>Connection refused >> at com.jcraft.jsch.Util.createSocket(Util.java:341) >> at com.jcraft.jsch.Session.connect(Session.java:182) >> at com.jcraft.jsch.Session.connect(Session.java:150) >> at >>org.apache.oodt.cas.protocol.sftp.JschSftpProtocol.connect(JschSftpProtoc >>ol.java:114) >> -- >> >> The fact that neither fails on the ENOTEMPTY error tells me that it was >>an >> NFS problem when I built them out of the VICAR external tree. But in >>any >> case, even if I could solve that one, now this SFTP issue has come >>along. >> >> The reason why 0.3 does not have this problem btw is because the >>"protocol" >> stuff (an API, with ftp/http/imaps/sftp support) is new starting with >>0.4. >> >> At this point I'm inclined to give up on 0.4/0.5 unless absolutely >>needed. >> >> - Greg >> >
