Re: [oodt-dev] pushpull with passive FTP

2014-08-03 Thread Cameron Goodale
Tim and Chris,

In the past I have run into a similar issue and I ended up using something
else (wget or python) to fetch the data.  That being said I also lost the
ability to check with FileManager to see if I had already downloaded and
archived a file.

All this to say I think it would be a good idea to setup an explicit
option/setting which would be active by default, but could be set to
passive for cases like Tim's.  This option becomes increasingly important
as more and more of us are deploying code within VMs.

I think this at least warrants a ticket in JIRA.


-Cameron


On Wed, Jul 23, 2014 at 5:15 PM, Mattmann, Chris A (3980) 
chris.a.mattm...@jpl.nasa.gov wrote:

 Moving this conversation to Apache OODT :)



 ++
 Chris Mattmann, Ph.D.
 Chief Architect
 Instrument Software and Science Data Systems Section (398)
 NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
 Office: 168-519, Mailstop: 168-527
 Email: chris.a.mattm...@nasa.gov
 WWW:  http://sunset.usc.edu/~mattmann/
 ++
 Adjunct Associate Professor, Computer Science Department
 University of Southern California, Los Angeles, CA 90089 USA
 ++






 -Original Message-
 From: Stough, Timothy M   (398F) timothy.m.sto...@jpl.nasa.gov
 Reply-To: OODT Developers oodt-...@list.jpl.nasa.gov
 Date: Wednesday, July 23, 2014 3:52 PM
 To: oodt-dev oodt-...@jpl.nasa.gov
 Subject: [oodt-dev] pushpull with passive FTP

 
 
 
 Hello All,
 
 
 I'm trying to use pushpull with Radix in a virtual machine.  The FTP site
 that I'm talking to gives me a 500 error because, I think, I'm trying to
 use active FTP from within a NAT connected VM.  So, how can I convince
 pushpull to use passive FTP?
 
 
 Thanks,
 Tim
 
 -
 Tim Stough
 NASA/Caltech Jet Propulsion Lab
 Senior System ArchitectAssociate
  Program Manager,
 
  NASA Applied Sciences Program,
 
  Disaster Management for Solid Earth Hazards
 Data Understanding Group (Section 398)
 626-644-6574 (cell)
 818-393-5347 (office)
 -
 
 
 
 




Re: trunk filemgr busted

2014-08-03 Thread Mattmann, Chris A (3980)
Guys I finally figured out what was wrong here! There was a patch
sitting all along to fix it!! ahhh!

It was trunk RADIX that was broken and it was OODT-690, evil bug
where a metadata end tag was missing from the product metadata
tag in GenericFile! Ahhh!!

https://issues.apache.org/jira/browse/OODT-690


Committed it finally. Thanks for checking!

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Lewis John Mcgibbney lewis.mcgibb...@gmail.com
Reply-To: dev@oodt.apache.org dev@oodt.apache.org
Date: Saturday, June 28, 2014 10:57 AM
To: dev@oodt.apache.org dev@oodt.apache.org
Subject: Re: trunk filemgr busted

Hi Chris,

On Sat, Jun 28, 2014 at 8:08 AM, Mattmann, Chris A (3980) 
chris.a.mattm...@jpl.nasa.gov wrote:


 On DARPA XDATA, I built a new RADIX-based system for processing an
 employment dataset using 0.7-radix (trunk). I then created a set of
 policy for the File Manager for employment datasets containing ~4
 product types, etc. I then tried to start file manager. It was up and
 running, but would keep giving me NPEs when trying to get ProductTypes
 (look them up), and then wouldn't ingest, etc.


Can this be related to Rishi's issue here?
https://issues.apache.org/jira/browse/OODT-686


   So there seems to be something in
 0.7 filemgr that wouldn't read my policy correctly and I was wondering
if
 it had to do with the updates you were making (maybe it didn't).


OK lets go deeper.
CHANGES TO FILEMGR/SRC/MAIN/JAVA

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/catalog/DataSourceCatalog.java?r1=1603049r2=1604981diff_
format=h

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/datatransfer/LocalDataTransferer.java?r1=1603049r2=160498
1

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/ingest/CmdLineIngester.java?r1=1603049r2=1604981diff_for
mat=h

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/repository/XMLRepositoryManager.java?r1=1603049r2=1604981

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/structs/exceptions/RepositoryManagerException.java?r1=1603
049r2=1604981diff_format=h

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/system/XmlRpcFileManager.java?r1=1603049r2=1604981

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/system/XmlRpcFileManagerClient.java?r1=1603049r2=1604981
diff_format=h

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/util/GenericFileManagerObjectFactory.java?r1=1603049r2=16
04981diff_format=h

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/util/XmlRpcStructFactory.java?r1=1603049r2=1604981diff_f
ormat=h

http://svn.apache.org/viewvc/oodt/trunk/filemgr/src/main/java/org/apache/o
odt/cas/filemgr/util/XmlStructFactory.java?r1=1603049r2=1604981diff_form
at=h

if you have the time to look at the above diffs, you will see that nothing
has been changed with respect to semantics of the code. All the commit
addressed was javac warnings and some Javadoc substantiation.




 One thing I did wonder though is that does the trunk filemgr still
produce
 a tgz when running mvn install?


I was under the impression that .zip files were the only concrete
distributable source artifacts from OODT.
This would be reflected in what was published in 0.6 release
http://apache.mirrors.pair.com/oodt/
This being said, I still get one generated when I run mvn install

lmcgibbn@LMC-032857 /usr/local/oodt/filemgr(master) $ cd target/
.plxarc   classes/
site/
archive-tmp/  generated-sources/
surefire/
cas-filemgr-0.7-SNAPSHOT-dist.tar.gz  javadoc-bundle-options/
surefire-reports/
cas-filemgr-0.7-SNAPSHOT-dist.zip maven-archiver/
test-classes/
cas-filemgr-0.7-SNAPSHOT.jar  maven-shared-archive-resources/



 It didn't seem to anymore for me and if it
 doesn't (per the test updates) that should be fixed/reverted since we
need
 to produce a tgz on mvn install (so those get published to the Central
 repo).
 Are you seeing this too?


No, I'm not able to reproduce Chris... I think we need to look more
closely
at this to converge on whats wrong here.



New wiki page on understanding metadata flow in pipeline processing

2014-08-03 Thread Mattmann, Chris A (3980)
Hey Guys,

I wrote a new page on the wiki describing my fun in remembering and
in some cases, understanding for the first time, the way metadata flows
around in pipeline processing:

http://s.apache.org/Ljo


Updates are welcomed, and encouraged. Hope this helps some of you!

I'm working on a number of improvements and solutions to the above.
I am also committed to getting File Manager, Workflow Manager, Resource
Manager and CAS-PGE fully working again together (via RADIX and otherwise)
in 0.7. Also getting close to a 0.7 release.

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






Jenkins build became unstable: oodt-trunk » CAS Protocol IMAPS Implementation #1423

2014-08-03 Thread Apache Jenkins Server
See 
https://builds.apache.org/job/oodt-trunk/org.apache.oodt$cas-protocol-imaps/1423/



Jenkins build is still unstable: oodt-trunk » Catalog and Archive File Management Component #1423

2014-08-03 Thread Apache Jenkins Server
See https://builds.apache.org/job/oodt-trunk/org.apache.oodt$cas-filemgr/1423/



Jenkins build became unstable: oodt-trunk » Catalog and Archive Workflow Management Component #1424

2014-08-03 Thread Apache Jenkins Server
See 
https://builds.apache.org/job/oodt-trunk/org.apache.oodt$cas-workflow/1424/



Jenkins build is still unstable: oodt-trunk » CAS Protocol IMAPS Implementation #1424

2014-08-03 Thread Apache Jenkins Server
See 
https://builds.apache.org/job/oodt-trunk/org.apache.oodt$cas-protocol-imaps/1424/



Jenkins build is still unstable: oodt-trunk » Catalog and Archive File Management Component #1424

2014-08-03 Thread Apache Jenkins Server
See https://builds.apache.org/job/oodt-trunk/org.apache.oodt$cas-filemgr/1424/



Jenkins build is still unstable: oodt-trunk » Catalog and Archive Workflow Management Component #1425

2014-08-03 Thread Apache Jenkins Server
See 
https://builds.apache.org/job/oodt-trunk/org.apache.oodt$cas-workflow/1425/



Jenkins build is still unstable: oodt-trunk #1425

2014-08-03 Thread Apache Jenkins Server
See https://builds.apache.org/job/oodt-trunk/changes