Re: OODT File Manager - Airavata Integration Document

[Chris Mattmann]

Great to hear. I’ll check it out.

—
Chris Mattmann
chris.mattm...@gmail.com






-Original Message-
From: DImuthu Upeksha 
Reply-To: 
Date: Monday, August 31, 2015 at 9:03 PM
To: , "d...@airavata.apache.org"

Subject: Re: OODT File Manager - Airavata Integration Document

>Hi Chris,
>
>Thank you for your feedback. This is a POC level implementation done by me
>and there is a lot of room for improvement. It would be great if both
>Airavata and OODT teams can go through the design and decide whether
>Airavata can be integrated with OODT File manager. But my idea is, yes it
>is possible.
>
>Thanks,
>Dimuthu
>
>On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Great work Dimuthu!
>>
>> So this means now Airavata and OODT’s File Manager are integrated?
>> :=)
>>
>> CC/dev@oodt.a.o
>>
>> Cheers,
>> Chris
>>
>> ++
>> Chris Mattmann, Ph.D.
>> Chief Architect
>> Instrument Software and Science Data Systems Section (398)
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 168-519, Mailstop: 168-527
>> Email: chris.a.mattm...@nasa.gov
>> WWW:  http://sunset.usc.edu/~mattmann/
>> ++
>> Adjunct Associate Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++
>>
>>
>>
>>
>>
>> -Original Message-
>> From: DImuthu Upeksha 
>> Reply-To: "d...@airavata.apache.org" 
>> Date: Saturday, August 29, 2015 at 1:27 AM
>> To: "d...@airavata.apache.org" 
>> Subject: OODT File Manager - Airavata Integration Document
>>
>> >Hi all,
>> >
>> >
>> >Please find the document [1] that describes the steps followed in the
>> >attempt of integrating OODT File Manager and AIravata. This setup is
>> >currently working on gw77.
>> >
>> >
>> >[1]
>> >
>> 
>>https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP-
>>M
>> >OFUDw/edit?usp=sharing
>> >
>> >
>> >Thanks
>> >Dimuthu
>> >
>> >
>> >--
>> >Regards
>> >W.Dimuthu Upeksha
>> >Undergraduate
>> >Department of Computer Science And Engineering
>> >University of Moratuwa, Sri Lanka
>> >
>> >
>> >
>>
>>
>
>
>-- 
>Regards
>
>W.Dimuthu Upeksha
>Undergraduate
>Department of Computer Science And Engineering
>
>University of Moratuwa, Sri Lanka

Re: [VOTE] Apache OODT 0.10 RC #1

[Lewis John Mcgibbney]

OK here are the DRAT license violations
https://issues.apache.org/jira/browse/OODT-871
Please note that not all are actual violations. JS is a common 'honest
culprit'. We just need to make sure NOTICE is included and we are good.
Thanks, I'll try and address this tomorrow.
Lewis

On Mon, Aug 31, 2015 at 10:14 PM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> Hi Chris,
> OK here we go.
>
> lmcgibbn@LMC-032857
> /usr/local/oodt_0.10_rc#1/apache-oodt-0.10-src(master) $ mvn -version
> Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06;
> 2015-04-22T04:57:37-07:00)
> Maven home: /usr/local/apache-maven-3.3.3
> Java version: 1.7.0_79, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"
>
> SIGS good
>
> lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1(master) $ gpg --verify
> apache-oodt-0.10-src.zip.asc
> gpg: Signature made Mon Aug 31 10:01:48 2015 PDT using DSA key ID B876884A
> gpg: Good signature from "Chris Mattmann (CODE SIGNING KEY) <
> mattm...@apache.org>"
> gpg: WARNING: This key is not certified with a trusted signature!
> gpg:  There is no indication that the signature belongs to the
> owner.
> Primary key fingerprint: 03CC 5FFA 61AA AD3C 8FF2  5E92 70F0 9CC6 B876 884A
> lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1(master) $ md5
> apache-oodt-0.10-src.zip
> MD5 (apache-oodt-0.10-src.zip) = 4cb579865833e49cb89db9ad9ce9397a
> lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1(master) $ openssl sha1
> apache-oodt-0.10-src.zip
> SHA1(apache-oodt-0.10-src.zip)= b784d0eaf471fd9ac0bcaac8cf712fd18128ce1f
>
> mvn clean install
>
> Results :
>
> Failed tests:
>
> Tests run: 63, Failures: 2, Errors: 0, Skipped: 0
>
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] OODT Core .. SUCCESS [
> 2.690 s]
> [INFO] Common Utilities ... FAILURE [
> 18.423 s]
> [INFO] CAS Command Line Interface . SKIPPED
> [INFO] Process Control System Input Data Package .. SKIPPED
> [INFO] Catalog and Archive Service Generic Multi-valued Metadata Container
> SKIPPED
> [INFO] CAS Protocol ... SKIPPED
> [INFO] CAS Protocol FTP Implementation  SKIPPED
> [INFO] CAS Protocol HTTP Implementation ... SKIPPED
> [INFO] CAS Protocol IMAPS Implementation .. SKIPPED
> [INFO] CAS Protocol SFTP Implementation ... SKIPPED
> [INFO] Query Expression ... SKIPPED
> [INFO] OODT Single Sign On Security Package ... SKIPPED
> [INFO] Catalog and Archive File Management Component .. SKIPPED
> [INFO] OODT CAS Virtual Catalog and Integration Service. .. SKIPPED
> [INFO] Catalog and Archive Resource Management Component .. SKIPPED
> [INFO] Catalog and Archive Workflow Management Component .. SKIPPED
> [INFO] Catalog and Archive Crawling Framework . SKIPPED
> [INFO] CAS Curation Web Services .. SKIPPED
> [INFO] Process Control System Core Package  SKIPPED
> [INFO] OODT Wicket Web Components . SKIPPED
> [INFO] CAS Curation Interface . SKIPPED
> [INFO] CAS PGE Adaptor Framework .. SKIPPED
> [INFO] CAS Installer Maven Mojo ... SKIPPED
> [INFO] OODT :: Archetypes :: OpsUI  SKIPPED
> [INFO] OODT :: Archetypes :: RADiX  SKIPPED
> [INFO] OODT :: Archetypes . SKIPPED
> [INFO] CAS Push-Pull-Framework  SKIPPED
> [INFO] Product Service  SKIPPED
> [INFO] Profile Service  SKIPPED
> [INFO] OODT Web Grid .. SKIPPED
> [INFO] XML-configured, DBMS-based Product and Profile Server SKIPPED
> [INFO] Apache OODT Configurable OPeNDAP Profile Server  SKIPPED
> [INFO] CAS File Manager Browser Web App ... SKIPPED
> [INFO] CAS Product Server Web Application . SKIPPED
> [INFO] CAS Workflow Manager Monitor Web App ... SKIPPED
> [INFO] Catalog and Archive File Management Browser  SKIPPED
> [INFO] Catalog and Archive Workflow Management GUI Editor . SKIPPED
> [INFO] Process Control System Operator Interface Webapp ... SKIPPED
> [INFO] OODT Process Control System JAX-RS service layer ... SKIPPED
> [INFO] Apache OODT  SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 

Re: [VOTE] Apache OODT 0.10 RC #1

[Lewis John Mcgibbney]

Hi Chris,
OK here we go.

lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1/apache-oodt-0.10-src(master)
$ mvn -version
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06;
2015-04-22T04:57:37-07:00)
Maven home: /usr/local/apache-maven-3.3.3
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home:
/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: "mac"

SIGS good

lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1(master) $ gpg --verify
apache-oodt-0.10-src.zip.asc
gpg: Signature made Mon Aug 31 10:01:48 2015 PDT using DSA key ID B876884A
gpg: Good signature from "Chris Mattmann (CODE SIGNING KEY) <
mattm...@apache.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:  There is no indication that the signature belongs to the
owner.
Primary key fingerprint: 03CC 5FFA 61AA AD3C 8FF2  5E92 70F0 9CC6 B876 884A
lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1(master) $ md5
apache-oodt-0.10-src.zip
MD5 (apache-oodt-0.10-src.zip) = 4cb579865833e49cb89db9ad9ce9397a
lmcgibbn@LMC-032857 /usr/local/oodt_0.10_rc#1(master) $ openssl sha1
apache-oodt-0.10-src.zip
SHA1(apache-oodt-0.10-src.zip)= b784d0eaf471fd9ac0bcaac8cf712fd18128ce1f

mvn clean install

Results :

Failed tests:

Tests run: 63, Failures: 2, Errors: 0, Skipped: 0

[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] OODT Core .. SUCCESS [
2.690 s]
[INFO] Common Utilities ... FAILURE [
18.423 s]
[INFO] CAS Command Line Interface . SKIPPED
[INFO] Process Control System Input Data Package .. SKIPPED
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container
SKIPPED
[INFO] CAS Protocol ... SKIPPED
[INFO] CAS Protocol FTP Implementation  SKIPPED
[INFO] CAS Protocol HTTP Implementation ... SKIPPED
[INFO] CAS Protocol IMAPS Implementation .. SKIPPED
[INFO] CAS Protocol SFTP Implementation ... SKIPPED
[INFO] Query Expression ... SKIPPED
[INFO] OODT Single Sign On Security Package ... SKIPPED
[INFO] Catalog and Archive File Management Component .. SKIPPED
[INFO] OODT CAS Virtual Catalog and Integration Service. .. SKIPPED
[INFO] Catalog and Archive Resource Management Component .. SKIPPED
[INFO] Catalog and Archive Workflow Management Component .. SKIPPED
[INFO] Catalog and Archive Crawling Framework . SKIPPED
[INFO] CAS Curation Web Services .. SKIPPED
[INFO] Process Control System Core Package  SKIPPED
[INFO] OODT Wicket Web Components . SKIPPED
[INFO] CAS Curation Interface . SKIPPED
[INFO] CAS PGE Adaptor Framework .. SKIPPED
[INFO] CAS Installer Maven Mojo ... SKIPPED
[INFO] OODT :: Archetypes :: OpsUI  SKIPPED
[INFO] OODT :: Archetypes :: RADiX  SKIPPED
[INFO] OODT :: Archetypes . SKIPPED
[INFO] CAS Push-Pull-Framework  SKIPPED
[INFO] Product Service  SKIPPED
[INFO] Profile Service  SKIPPED
[INFO] OODT Web Grid .. SKIPPED
[INFO] XML-configured, DBMS-based Product and Profile Server SKIPPED
[INFO] Apache OODT Configurable OPeNDAP Profile Server  SKIPPED
[INFO] CAS File Manager Browser Web App ... SKIPPED
[INFO] CAS Product Server Web Application . SKIPPED
[INFO] CAS Workflow Manager Monitor Web App ... SKIPPED
[INFO] Catalog and Archive File Management Browser  SKIPPED
[INFO] Catalog and Archive Workflow Management GUI Editor . SKIPPED
[INFO] Process Control System Operator Interface Webapp ... SKIPPED
[INFO] OODT Process Control System JAX-RS service layer ... SKIPPED
[INFO] Apache OODT  SKIPPED
[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time: 21.928 s
[INFO] Finished at: 2015-08-31T21:19:30-07:00
[INFO] Final Memory: 41M/456M
[INFO]



The failure is here

Running org.apache.oodt.commons.io.DirectorySelectorTest
Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.038 sec
<<< FAILURE!

---
Test set: org.apache.oodt.commons.io.DirectorySelectorTest
--

Re: OODT File Manager - Airavata Integration Document

[DImuthu Upeksha]

Hi Chris,

Thank you for your feedback. This is a POC level implementation done by me
and there is a lot of room for improvement. It would be great if both
Airavata and OODT teams can go through the design and decide whether
Airavata can be integrated with OODT File manager. But my idea is, yes it
is possible.

Thanks,
Dimuthu

On Tue, Sep 1, 2015 at 5:07 AM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Great work Dimuthu!
>
> So this means now Airavata and OODT’s File Manager are integrated?
> :=)
>
> CC/dev@oodt.a.o
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
> -Original Message-
> From: DImuthu Upeksha 
> Reply-To: "d...@airavata.apache.org" 
> Date: Saturday, August 29, 2015 at 1:27 AM
> To: "d...@airavata.apache.org" 
> Subject: OODT File Manager - Airavata Integration Document
>
> >Hi all,
> >
> >
> >Please find the document [1] that describes the steps followed in the
> >attempt of integrating OODT File Manager and AIravata. This setup is
> >currently working on gw77.
> >
> >
> >[1]
> >
> https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP-M
> >OFUDw/edit?usp=sharing
> >
> >
> >Thanks
> >Dimuthu
> >
> >
> >--
> >Regards
> >W.Dimuthu Upeksha
> >Undergraduate
> >Department of Computer Science And Engineering
> >University of Moratuwa, Sri Lanka
> >
> >
> >
>
>


-- 
Regards

W.Dimuthu Upeksha
Undergraduate
Department of Computer Science And Engineering

University of Moratuwa, Sri Lanka

Re: OODT File Manager - Airavata Integration Document

[Mattmann, Chris A (3980)]

Great work Dimuthu!

So this means now Airavata and OODT’s File Manager are integrated?
:=)

CC/dev@oodt.a.o

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++





-Original Message-
From: DImuthu Upeksha 
Reply-To: "d...@airavata.apache.org" 
Date: Saturday, August 29, 2015 at 1:27 AM
To: "d...@airavata.apache.org" 
Subject: OODT File Manager - Airavata Integration Document

>Hi all,
>
>
>Please find the document [1] that describes the steps followed in the
>attempt of integrating OODT File Manager and AIravata. This setup is
>currently working on gw77.
>
>
>[1] 
>https://docs.google.com/document/d/1mr3k_OmgEweeVhJtuMTvofL__v9uPwyWbWrP-M
>OFUDw/edit?usp=sharing
>
>
>Thanks
>Dimuthu
>
>
>-- 
>Regards
>W.Dimuthu Upeksha
>Undergraduate
>Department of Computer Science And Engineering
>University of Moratuwa, Sri Lanka
>
>
>

Re: svn commit: r963740 - /websites/production/oodt/content/

[Mattmann, Chris A (3980)]

Thanks for this, Lewis..

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++





-Original Message-
From: "lewi...@apache.org" 
Reply-To: "dev@oodt.apache.org" 
Date: Monday, August 31, 2015 at 1:53 PM
To: "comm...@oodt.apache.org" 
Subject: svn commit: r963740 - /websites/production/oodt/content/

>Author: lewismc
>Date: Mon Aug 31 20:53:05 2015
>New Revision: 963740
>
>Log:
>Update to lua script
>
>Added:
>websites/production/oodt/content/
>  - copied from r963739, websites/staging/oodt/trunk/content/
>

RE: catalog queries in pge config files and filemgr-client problem

[Mallder, Valerie]

No problem!


Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -Original Message-
> From: Lewis John Mcgibbney [mailto:lewis.mcgibb...@gmail.com]
> Sent: Monday, August 31, 2015 6:02 PM
> To: dev@oodt.apache.org
> Subject: Re: catalog queries in pge config files and filemgr-client problem
> 
> Grand
> Apologies for incorrect advice on logging Level. Log4j Vs.
> java.util.Logging I think!
> 
> On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie < 
> valerie.mall...@jhuapl.edu>
> wrote:
> 
> > Hi Lewis,
> >
> > Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> > recognized level. But, I used FINEST and added more log message and
> > eventually found that I had run into the issue that has already been
> > documented in OODT-854 which is supposed to be fixed in 0.10.  It has
> > to do with where I was putting my policy files for the filemgr.  It
> > sounds like the fix for OODT-854 will solve my problem.  Once I put
> > all of my policy files in FILEMGR_HOME/policy/core, the catalog query 
> > started
> working.
> >
> > Thanks,
> > Val
> >
> >
> >
> >
> > > -Original Message-
> > > From: Lewis John Mcgibbney [mailto:lewis.mcgibb...@gmail.com]
> > > Sent: Thursday, August 27, 2015 8:07 PM
> > > To: dev@oodt.apache.org
> > > Subject: Re: catalog queries in pge config files and filemgr-client
> > problem
> > >
> > > Hi Val,
> > > Wise idea to start debugging this from the filemgr-client.
> > > Can you please try changing logging levels in the following file to
> > > DEBUG
> > >
> > >
> > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/l
> > ogging.pro
> > > perties#L40-L74
> > >
> > > If you try your queries again then hopefully you should see some
> > > patterns/indications within FileMgr logs to indicate what is going on.
> > >
> > >
> > > On Wednesday, August 26, 2015, Mallder, Valerie <
> > valerie.mall...@jhuapl.edu>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > So, I'm phasing off of the New Horizons project for now and I am
> > > > back to my science data pipeline work.  I am in the process of
> > > > trying to pick up where I left off a few months ago - but with a
> > > > twist J  I graduated from RADiX and now I am trying to get my
> > > > pipeline system set up to run with the 'regular' OODT.  I have
> > > > made extremely good progress so far. I have pretty much replicated
> > > > all of the functionality of filemgr and workflow manager that I
> > > > had been using with RADiX, with the exception of one thing that
> > > > doesn't seem to be working the way it used too.  And I'm hoping
> > > > someone out there can
> > give me
> > > some ideas on how to fix it.
> > > >
> > > > I have a catalog query in one of my PGE config files. I copied the
> > > > syntax for the query statement directly from one of Chris' PGE
> > > > config files
> > > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > > >
> > > > Here is Chris' statement:
> > > >
> > > >  > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM RatLog}"/>
> > > >
> > > > Here is my statement:
> > > >
> > > >   > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM ScienceFile}"/>
> > > >
> > > > And here are the results from the query:
> > > >
> > > >
> > > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > > enam
> > > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > > ilen
> > > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > > $Fil
> > > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > > n/$F
> > > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > > ion/
> > > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > > atio
> > > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > > ocat
> > > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
> > > > eLoc
> > > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
> > > > ileL ocation/$Filename,$FileLocation/$Filename,
> > > >
> > > > And those results are not exactly what I wanted J
> > > >
> > > > So I ran some experiments with ./filemgr-client on the command
> > > > line to make sure the files I expected to be in the catalog were
> > > > in fact in the catalog.
> > > >
> > > > The following operation worked just as expected.  The expected
> > > > number of products were listed.
> > > > ./filemgr-client --url http://localhost:9016 --operation
> > > > --getFirstPage --productTypeName ScienceFile
> > > >
> > > > However, this query gave some unexpected results. And this may be
> > > > related the problem with the query in my PGE config fil

Re: catalog queries in pge config files and filemgr-client problem

[Lewis John Mcgibbney]

Grand
Apologies for incorrect advice on logging Level. Log4j Vs.
java.util.Logging I think!

On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie <
valerie.mall...@jhuapl.edu> wrote:

> Hi Lewis,
>
> Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> recognized level. But, I used FINEST and added more log message and
> eventually found that I had run into the issue that has already been
> documented in OODT-854 which is supposed to be fixed in 0.10.  It has to do
> with where I was putting my policy files for the filemgr.  It sounds like
> the fix for OODT-854 will solve my problem.  Once I put all of my policy
> files in FILEMGR_HOME/policy/core, the catalog query started working.
>
> Thanks,
> Val
>
>
>
>
> > -Original Message-
> > From: Lewis John Mcgibbney [mailto:lewis.mcgibb...@gmail.com]
> > Sent: Thursday, August 27, 2015 8:07 PM
> > To: dev@oodt.apache.org
> > Subject: Re: catalog queries in pge config files and filemgr-client
> problem
> >
> > Hi Val,
> > Wise idea to start debugging this from the filemgr-client.
> > Can you please try changing logging levels in the following file to DEBUG
> >
> >
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/logging.pro
> > perties#L40-L74
> >
> > If you try your queries again then hopefully you should see some
> > patterns/indications within FileMgr logs to indicate what is going on.
> >
> >
> > On Wednesday, August 26, 2015, Mallder, Valerie <
> valerie.mall...@jhuapl.edu>
> > wrote:
> >
> > > Hi All,
> > >
> > > So, I'm phasing off of the New Horizons project for now and I am back
> > > to my science data pipeline work.  I am in the process of trying to
> > > pick up where I left off a few months ago - but with a twist J  I
> > > graduated from RADiX and now I am trying to get my pipeline system set
> > > up to run with the 'regular' OODT.  I have made extremely good
> > > progress so far. I have pretty much replicated all of the
> > > functionality of filemgr and workflow manager that I had been using
> > > with RADiX, with the exception of one thing that doesn't seem to be
> > > working the way it used too.  And I'm hoping someone out there can
> give me
> > some ideas on how to fix it.
> > >
> > > I have a catalog query in one of my PGE config files. I copied the
> > > syntax for the query statement directly from one of Chris' PGE config
> > > files
> > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > >
> > > Here is Chris' statement:
> > >
> > >  > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > > RatLog}"/>
> > >
> > > Here is my statement:
> > >
> > >   > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > > ScienceFile}"/>
> > >
> > > And here are the results from the query:
> > >
> > >
> > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filenam
> > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filen
> > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > ocation/$Filename,$FileLocation/$Filename,
> > >
> > > And those results are not exactly what I wanted J
> > >
> > > So I ran some experiments with ./filemgr-client on the command line to
> > > make sure the files I expected to be in the catalog were in fact in
> > > the catalog.
> > >
> > > The following operation worked just as expected.  The expected number
> > > of products were listed.
> > > ./filemgr-client --url http://localhost:9016 --operation
> > > --getFirstPage --productTypeName ScienceFile
> > >
> > > However, this query gave some unexpected results. And this may be
> > > related the problem with the query in my PGE config file, but I am not
> sure.
> > > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > > -query "SELECT FileLocation,Filename FROM ScienceFile"
> > >
> > > That query only outputs several blank lines to the screen and nothing
> else.
> > >
> > > I am not very familiar with all of the catalog related code yet so I
> > > am not sure where to start looking for the problem.  If you know this
> > > area of the code well and have some debugging ideas, please let me
> know.
> > >
> > > Thanks so much!
> > >
> > > Val
> > >
> > >
>



-- 
*Lewis*

RE: (to Michael Starch) Problems Using Serializable Metadata after Loading Validation Layer

[Mallder, Valerie]

Hi Michael,

Thanks for the response. I do have FILEMGR_HOME set in my environment, but this 
problem only happens when I am running my system without the resource manager 
and batch_stub.  When I run my system with just filemgr and workflow manager, I 
do not get this problem, i.e. the pge_metadata file is written out just fine. I 
do export my FILEMGR_HOME variable though and I start the resource manager and 
batch_stub from the same script that FILEMGR_HOME is exported from. But I'm 
wondering if either of them are not seeing it for some reason. I will do some 
investigating in that area.

Thanks,
Val
 


Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -Original Message-
> From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of Michael
> Starch
> Sent: Monday, August 31, 2015 5:47 PM
> To: dev@oodt.apache.org
> Subject: RE: (to Michael Starch) Problems Using Serializable Metadata after
> Loading Validation Layer
> 
> Valerie,
> 
> My issue was actually using "System.setProperties(Properties properties)"
> in my code as this clobbers all existing system properties in your current 
> JVM.
> The solution was to set each specific property I needed using "System.
> setPropertie(String name, String value)" as this doesn't clobber other values.
> 
> If you are not calling these things programmaticly, the it is a different 
> issue. I will
> happily look at any errors If you supply them but a quick
> note: FILEMGR_HOME may not be set is this environment variable defined in
> your?
> 
> Michael
> On Sep 1, 2015 4:23 AM, "Mallder, Valerie" 
> wrote:
> 
> > Hi Michael,
> >
> > Could you explain your fix to this issue with more detail?  I am
> > having the same problem. The default filemgr.properties file sets
> > these two values to directories that do not exist (so I have to set them to
> 'something'
> > valid). Here are the default settings:
> >
> > # XML repository manager configuration
> > org.apache.oodt.cas.filemgr.repositorymgr.dirs=file:///dir1,file:///di
> > r2
> >
> > # XML validation layer configuration
> > org.apache.oodt.cas.filemgr.validation.dirs=file:///dir1,file:///dir2
> >
> >
> > And this is what I set them to:
> >
> > # XML repository manager configuration
> >
> > org.apache.oodt.cas.filemgr.repositorymgr.dirs=file://[FILEMGR_HOME]/p
> > olicy/core
> >
> > # XML validation layer configuration
> >
> > org.apache.oodt.cas.filemgr.validation.dirs=file://[FILEMGR_HOME]/poli
> > cy/core
> >
> > And I still get the error.  Could you explain more about how I can
> > work around this issue?
> >
> > Thanks,
> > Valerie
> >
> >
> > > -Original Message-
> > > From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of
> > Michael
> > > Starch
> > > Sent: Friday, July 31, 2015 12:20 PM
> > > To: dev@oodt.apache.org
> > > Subject: Re: Problems Using Serializable Metadata after Loading
> > Validation Layer
> > >
> > > All,
> > >
> > > I found the issue.
> > >
> > > Using "System.setProperties()" and filling it from properties read
> > > from filemanager.properties clears out other properties setup by the
> > > system
> > which was
> > > needed in the XML calls for SerializableMetadata. I did the above
> > > call
> > to setup
> > > properties needed by the XMLValidationLayer
> > >
> > > To fix set only the properties you need individually.  This adds to
> > > the
> > System
> > > properties, not erasing them.
> > >
> > > System.setProperty("org.apache.oodt.cas.filemgr.repositorymgr.dirs",
> > ...);
> > >
> > > System.setProperty("org.apache.oodt.cas.filemgr.validation.dirs",
> > > ...);
> > >
> > > -Michael
> > >
> > >
> > > On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch 
> > wrote:
> > >
> > > > Here is the stack trace, but this only happens after a completely
> > > > unrelated peice of the process load the XML Validation Layer.
> > > >
> > > > -Michael
> > > >
> > > > java.lang.NullPointerException
> > > > at
> > > >
> > com.sun.org.apache.xml.internal.serializer.ToStream.(ToStream.ja
> > va:143)
> > > > at
> > > >
> > >
> >
> com.sun.org.apache.xml.internal.serializer.ToXMLStream.(ToXMLStream.java:
> > > 67)
> > > > at
> > > >
> > >
> > com.sun.org.apache.xml.internal.serializer.ToUnknownStream.(ToUn
> > knownStr
> > > eam.java:143)
> > > > at
> > > >
> > >
> > com.sun.org.apache.xalan.internal.xsltc.runtime.output.TransletOutputH
> > andlerFacto
> > > ry.getSerializationHandler(TransletOutputHandlerFactory.java:160)
> > > > at
> > > >
> > >
> > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutput
> > Handler(Tra
> > > nsformerImpl.java:461)
> > > > at
> > > >
> > >
> > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform
> > (Transforme
> > > rImpl.java:344)
> > > > at
> > > > org.apache.oodt.cas.metadata.SerializableMetadata.writeMetadataToX
> > > > mlSt
> > > > ream(SerializableMetadata.java:157)
> > > >
> > > >
> > > > On Thu, Jul 30, 20

RE: (to Michael Starch) Problems Using Serializable Metadata after Loading Validation Layer

[Michael Starch]

Valerie,

My issue was actually using "System.setProperties(Properties properties)"
in my code as this clobbers all existing system properties in your current
JVM.  The solution was to set each specific property I needed using
"System. setPropertie(String name, String value)" as this doesn't clobber
other values.

If you are not calling these things programmaticly, the it is a different
issue. I will happily look at any errors If you supply them but a quick
note: FILEMGR_HOME may not be set is this environment variable defined in
your?

Michael
On Sep 1, 2015 4:23 AM, "Mallder, Valerie" 
wrote:

> Hi Michael,
>
> Could you explain your fix to this issue with more detail?  I am having
> the same problem. The default filemgr.properties file sets these two values
> to directories that do not exist (so I have to set them to 'something'
> valid). Here are the default settings:
>
> # XML repository manager configuration
> org.apache.oodt.cas.filemgr.repositorymgr.dirs=file:///dir1,file:///dir2
>
> # XML validation layer configuration
> org.apache.oodt.cas.filemgr.validation.dirs=file:///dir1,file:///dir2
>
>
> And this is what I set them to:
>
> # XML repository manager configuration
>
> org.apache.oodt.cas.filemgr.repositorymgr.dirs=file://[FILEMGR_HOME]/policy/core
>
> # XML validation layer configuration
>
> org.apache.oodt.cas.filemgr.validation.dirs=file://[FILEMGR_HOME]/policy/core
>
> And I still get the error.  Could you explain more about how I can work
> around this issue?
>
> Thanks,
> Valerie
>
>
> > -Original Message-
> > From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of
> Michael
> > Starch
> > Sent: Friday, July 31, 2015 12:20 PM
> > To: dev@oodt.apache.org
> > Subject: Re: Problems Using Serializable Metadata after Loading
> Validation Layer
> >
> > All,
> >
> > I found the issue.
> >
> > Using "System.setProperties()" and filling it from properties read from
> > filemanager.properties clears out other properties setup by the system
> which was
> > needed in the XML calls for SerializableMetadata. I did the above call
> to setup
> > properties needed by the XMLValidationLayer
> >
> > To fix set only the properties you need individually.  This adds to the
> System
> > properties, not erasing them.
> >
> > System.setProperty("org.apache.oodt.cas.filemgr.repositorymgr.dirs",
> ...);
> >
> > System.setProperty("org.apache.oodt.cas.filemgr.validation.dirs", ...);
> >
> > -Michael
> >
> >
> > On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch 
> wrote:
> >
> > > Here is the stack trace, but this only happens after a completely
> > > unrelated peice of the process load the XML Validation Layer.
> > >
> > > -Michael
> > >
> > > java.lang.NullPointerException
> > > at
> > >
> com.sun.org.apache.xml.internal.serializer.ToStream.(ToStream.java:143)
> > > at
> > >
> >
> com.sun.org.apache.xml.internal.serializer.ToXMLStream.(ToXMLStream.java:
> > 67)
> > > at
> > >
> >
> com.sun.org.apache.xml.internal.serializer.ToUnknownStream.(ToUnknownStr
> > eam.java:143)
> > > at
> > >
> >
> com.sun.org.apache.xalan.internal.xsltc.runtime.output.TransletOutputHandlerFacto
> > ry.getSerializationHandler(TransletOutputHandlerFactory.java:160)
> > > at
> > >
> >
> com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHandler(Tra
> > nsformerImpl.java:461)
> > > at
> > >
> >
> com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Transforme
> > rImpl.java:344)
> > > at
> > > org.apache.oodt.cas.metadata.SerializableMetadata.writeMetadataToXmlSt
> > > ream(SerializableMetadata.java:157)
> > >
> > >
> > > On Thu, Jul 30, 2015 at 4:16 PM, Chris Mattmann
> > > 
> > > wrote:
> > >
> > >> Mike can you give some specific line numbers? I can help look
> > >>
> > >> —
> > >> Chris Mattmann
> > >> chris.mattm...@gmail.com
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -Original Message-
> > >> From:  on behalf of Michael Starch <
> > >> starc...@umich.edu>
> > >> Reply-To: 
> > >> Date: Thursday, July 30, 2015 at 4:03 PM
> > >> To: 
> > >> Subject: Problems Using Serializable Metadata after Loading
> > >> Validation Layer
> > >>
> > >> >All,
> > >> >
> > >> >I am getting a NullPointerException deep in the XML library if I try
> > >> >to use the SerializableMetadata's write to xml function after I load
> > >> >in the XML Validation Layer from the filemanager. However, if I
> > >> >remove the call to load in the XML Validation Layer, everything
> > >> >works fine.  Any ideas as to what might cause this issue?
> > >> >
> > >> >Thanks,
> > >> >
> > >> >Michael
> > >>
> > >>
> > >>
> > >
>

RE: catalog queries in pge config files and filemgr-client problem

[Mallder, Valerie]

Hi Lewis,

Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a 
recognized level. But, I used FINEST and added more log message and eventually 
found that I had run into the issue that has already been documented in 
OODT-854 which is supposed to be fixed in 0.10.  It has to do with where I was 
putting my policy files for the filemgr.  It sounds like the fix for OODT-854 
will solve my problem.  Once I put all of my policy files in 
FILEMGR_HOME/policy/core, the catalog query started working. 

Thanks,
Val




> -Original Message-
> From: Lewis John Mcgibbney [mailto:lewis.mcgibb...@gmail.com]
> Sent: Thursday, August 27, 2015 8:07 PM
> To: dev@oodt.apache.org
> Subject: Re: catalog queries in pge config files and filemgr-client problem
> 
> Hi Val,
> Wise idea to start debugging this from the filemgr-client.
> Can you please try changing logging levels in the following file to DEBUG
> 
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/logging.pro
> perties#L40-L74
> 
> If you try your queries again then hopefully you should see some
> patterns/indications within FileMgr logs to indicate what is going on.
> 
> 
> On Wednesday, August 26, 2015, Mallder, Valerie 
> wrote:
> 
> > Hi All,
> >
> > So, I'm phasing off of the New Horizons project for now and I am back
> > to my science data pipeline work.  I am in the process of trying to
> > pick up where I left off a few months ago - but with a twist J  I
> > graduated from RADiX and now I am trying to get my pipeline system set
> > up to run with the 'regular' OODT.  I have made extremely good
> > progress so far. I have pretty much replicated all of the
> > functionality of filemgr and workflow manager that I had been using
> > with RADiX, with the exception of one thing that doesn't seem to be
> > working the way it used too.  And I'm hoping someone out there can give me
> some ideas on how to fix it.
> >
> > I have a catalog query in one of my PGE config files. I copied the
> > syntax for the query statement directly from one of Chris' PGE config
> > files
> > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> >
> > Here is Chris' statement:
> >
> >  > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > RatLog}"/>
> >
> > Here is my statement:
> >
> >   > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > ScienceFile}"/>
> >
> > And here are the results from the query:
> >
> >
> > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filenam
> > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filen
> > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > ocation/$Filename,$FileLocation/$Filename,
> >
> > And those results are not exactly what I wanted J
> >
> > So I ran some experiments with ./filemgr-client on the command line to
> > make sure the files I expected to be in the catalog were in fact in
> > the catalog.
> >
> > The following operation worked just as expected.  The expected number
> > of products were listed.
> > ./filemgr-client --url http://localhost:9016 --operation
> > --getFirstPage --productTypeName ScienceFile
> >
> > However, this query gave some unexpected results. And this may be
> > related the problem with the query in my PGE config file, but I am not sure.
> > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > -query "SELECT FileLocation,Filename FROM ScienceFile"
> >
> > That query only outputs several blank lines to the screen and nothing else.
> >
> > I am not very familiar with all of the catalog related code yet so I
> > am not sure where to start looking for the problem.  If you know this
> > area of the code well and have some debugging ideas, please let me know.
> >
> > Thanks so much!
> >
> > Val
> >
> >

Re: (to Michael Starch) Problems Using Serializable Metadata after Loading Validation Layer

[Mattmann, Chris A (3980)]

Hey Val,

I think putting the docs in etc/filemgr.properties (or in
src/main/resources/filemgr.properties in the source tree) is
where it should be documented.

Thanks mucho.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++





-Original Message-
From: "Mallder, Valerie" 
Reply-To: "dev@oodt.apache.org" 
Date: Monday, August 31, 2015 at 1:52 PM
To: "dev@oodt.apache.org" 
Subject: RE: (to Michael Starch) Problems Using Serializable Metadata
after Loading Validation Layer

>I can do it. But I want to wait until I get past my current problem just
>in case there are any other issues worth opening.  I will put this one on
>my list.  I assume you want want them set to exactly what I have shown
>below?  And is there a place I should document that FILEMGR_HOME is going
>to be a required variable?
>
>
>
>Valerie A. Mallder
>New Horizons Deputy Mission System Engineer
>Johns Hopkins University/Applied Physics Laboratory
>
>
>> -Original Message-
>> From: Chris Mattmann [mailto:chris.mattm...@gmail.com]
>> Sent: Monday, August 31, 2015 4:47 PM
>> To: dev@oodt.apache.org
>> Subject: Re: (to Michael Starch) Problems Using Serializable Metadata
>>after
>> Loading Validation Layer
>> 
>> Hmm, there is no reason that we shouldn’t simply just say FILEMGR_HOME
>>is a
>> required variable, and make the default configuration more sensible.
>> 
>> Val, Mike, anyone willing to open an issue for 0.11 for this?
>> 
>> —
>> Chris Mattmann
>> chris.mattm...@gmail.com
>> 
>> 
>> 
>> 
>> 
>> 
>> -Original Message-
>> From: "Mallder, Valerie" 
>> Reply-To: 
>> Date: Monday, August 31, 2015 at 1:22 PM
>> To: "dev@oodt.apache.org" 
>> Subject: RE: (to Michael Starch) Problems Using Serializable Metadata
>>after
>> Loading Validation Layer
>> 
>> >Hi Michael,
>> >
>> >Could you explain your fix to this issue with more detail?  I am having
>> >the same problem. The default filemgr.properties file sets these two
>> >values to directories that do not exist (so I have to set them to
>> >'something' valid). Here are the default settings:
>> >
>> ># XML repository manager configuration
>> >org.apache.oodt.cas.filemgr.repositorymgr.dirs=file:///dir1,file:///dir
>> >2
>> >
>> ># XML validation layer configuration
>> >org.apache.oodt.cas.filemgr.validation.dirs=file:///dir1,file:///dir2
>> >
>> >
>> >And this is what I set them to:
>> >
>> ># XML repository manager configuration
>> >org.apache.oodt.cas.filemgr.repositorymgr.dirs=file://[FILEMGR_HOME]/po
>> >lic
>> >y/core
>> >
>> ># XML validation layer configuration
>> >org.apache.oodt.cas.filemgr.validation.dirs=file://[FILEMGR_HOME]/polic
>> >y/c
>> >ore
>> >
>> >And I still get the error.  Could you explain more about how I can work
>> >around this issue?
>> >
>> >Thanks,
>> >Valerie
>> >
>> >
>> >> -Original Message-
>> >> From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of
>> >>Michael  Starch
>> >> Sent: Friday, July 31, 2015 12:20 PM
>> >> To: dev@oodt.apache.org
>> >> Subject: Re: Problems Using Serializable Metadata after Loading
>> >>Validation Layer
>> >>
>> >> All,
>> >>
>> >> I found the issue.
>> >>
>> >> Using "System.setProperties()" and filling it from properties read
>> >>from  filemanager.properties clears out other properties setup by the
>> >>system which was  needed in the XML calls for SerializableMetadata. I
>> >>did the above call to setup  properties needed by the
>> >>XMLValidationLayer
>> >>
>> >> To fix set only the properties you need individually.  This adds to
>> >>the System  properties, not erasing them.
>> >>
>> >> System.setProperty("org.apache.oodt.cas.filemgr.repositorymgr.dirs",
>> >>...);
>> >>
>> >> System.setProperty("org.apache.oodt.cas.filemgr.validation.dirs",
>> >> ...);
>> >>
>> >> -Michael
>> >>
>> >>
>> >> On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch 
>> >>wrote:
>> >>
>> >> > Here is the stack trace, but this only happens after a completely
>> >> > unrelated peice of the process load the XML Validation Layer.
>> >> >
>> >> > -Michael
>> >> >
>> >> > java.lang.NullPointerException
>> >> > at
>> >> >
>> 
com.sun.org.apache.xml.internal.serializer.ToStream.(ToStream.jav
a:
>> >>143)
>> >> > at
>> >> >
>> >>
>> >>com.sun.org.apache.xml.internal.serializer.ToXMLStream.(ToXMLStr
>> >>eam
>> >>.java:
>> >> 67)
>> >> > at
>> >> >
>> >>
>> >>com.sun.org.apache.xml.internal.serializer.ToUnknownStream.(ToUn
>> >>kno
>> >>wnStr
>> >> eam.java:143)
>> >> > 

RE: (to Michael Starch) Problems Using Serializable Metadata after Loading Validation Layer

[Mallder, Valerie]

I can do it. But I want to wait until I get past my current problem just in 
case there are any other issues worth opening.  I will put this one on my list. 
 I assume you want want them set to exactly what I have shown below?  And is 
there a place I should document that FILEMGR_HOME is going to be a required 
variable?



Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -Original Message-
> From: Chris Mattmann [mailto:chris.mattm...@gmail.com]
> Sent: Monday, August 31, 2015 4:47 PM
> To: dev@oodt.apache.org
> Subject: Re: (to Michael Starch) Problems Using Serializable Metadata after
> Loading Validation Layer
> 
> Hmm, there is no reason that we shouldn’t simply just say FILEMGR_HOME is a
> required variable, and make the default configuration more sensible.
> 
> Val, Mike, anyone willing to open an issue for 0.11 for this?
> 
> —
> Chris Mattmann
> chris.mattm...@gmail.com
> 
> 
> 
> 
> 
> 
> -Original Message-
> From: "Mallder, Valerie" 
> Reply-To: 
> Date: Monday, August 31, 2015 at 1:22 PM
> To: "dev@oodt.apache.org" 
> Subject: RE: (to Michael Starch) Problems Using Serializable Metadata after
> Loading Validation Layer
> 
> >Hi Michael,
> >
> >Could you explain your fix to this issue with more detail?  I am having
> >the same problem. The default filemgr.properties file sets these two
> >values to directories that do not exist (so I have to set them to
> >'something' valid). Here are the default settings:
> >
> ># XML repository manager configuration
> >org.apache.oodt.cas.filemgr.repositorymgr.dirs=file:///dir1,file:///dir
> >2
> >
> ># XML validation layer configuration
> >org.apache.oodt.cas.filemgr.validation.dirs=file:///dir1,file:///dir2
> >
> >
> >And this is what I set them to:
> >
> ># XML repository manager configuration
> >org.apache.oodt.cas.filemgr.repositorymgr.dirs=file://[FILEMGR_HOME]/po
> >lic
> >y/core
> >
> ># XML validation layer configuration
> >org.apache.oodt.cas.filemgr.validation.dirs=file://[FILEMGR_HOME]/polic
> >y/c
> >ore
> >
> >And I still get the error.  Could you explain more about how I can work
> >around this issue?
> >
> >Thanks,
> >Valerie
> >
> >
> >> -Original Message-
> >> From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of
> >>Michael  Starch
> >> Sent: Friday, July 31, 2015 12:20 PM
> >> To: dev@oodt.apache.org
> >> Subject: Re: Problems Using Serializable Metadata after Loading
> >>Validation Layer
> >>
> >> All,
> >>
> >> I found the issue.
> >>
> >> Using "System.setProperties()" and filling it from properties read
> >>from  filemanager.properties clears out other properties setup by the
> >>system which was  needed in the XML calls for SerializableMetadata. I
> >>did the above call to setup  properties needed by the
> >>XMLValidationLayer
> >>
> >> To fix set only the properties you need individually.  This adds to
> >>the System  properties, not erasing them.
> >>
> >> System.setProperty("org.apache.oodt.cas.filemgr.repositorymgr.dirs",
> >>...);
> >>
> >> System.setProperty("org.apache.oodt.cas.filemgr.validation.dirs",
> >> ...);
> >>
> >> -Michael
> >>
> >>
> >> On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch 
> >>wrote:
> >>
> >> > Here is the stack trace, but this only happens after a completely
> >> > unrelated peice of the process load the XML Validation Layer.
> >> >
> >> > -Michael
> >> >
> >> > java.lang.NullPointerException
> >> > at
> >> >
> >>com.sun.org.apache.xml.internal.serializer.ToStream.(ToStream.java:
> >>143)
> >> > at
> >> >
> >>
> >>com.sun.org.apache.xml.internal.serializer.ToXMLStream.(ToXMLStr
> >>eam
> >>.java:
> >> 67)
> >> > at
> >> >
> >>
> >>com.sun.org.apache.xml.internal.serializer.ToUnknownStream.(ToUn
> >>kno
> >>wnStr
> >> eam.java:143)
> >> > at
> >> >
> >>
> >>com.sun.org.apache.xalan.internal.xsltc.runtime.output.TransletOutputH
> >>and
> >>lerFacto
> >> ry.getSerializationHandler(TransletOutputHandlerFactory.java:160)
> >> > at
> >> >
> >>
> >>com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutput
> >>Han
> >>dler(Tra
> >> nsformerImpl.java:461)
> >> > at
> >> >
> >>
> >>com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform
> >>(Tr
> >>ansforme
> >> rImpl.java:344)
> >> > at
> >> > org.apache.oodt.cas.metadata.SerializableMetadata.writeMetadataToXm
> >> > lSt
> >> > ream(SerializableMetadata.java:157)
> >> >
> >> >
> >> > On Thu, Jul 30, 2015 at 4:16 PM, Chris Mattmann
> >> > 
> >> > wrote:
> >> >
> >> >> Mike can you give some specific line numbers? I can help look
> >> >>
> >> >> —
> >> >> Chris Mattmann
> >> >> chris.mattm...@gmail.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> -Original Message-
> >> >> From:  on behalf of Michael Starch <
> >> >> starc...@umich.edu>
> >> >> Reply-To: 
> >> >> Date: Thursday, July 30, 2015 at 4:03 PM
> >> >> To: 
> >> >> Subject: Problems Using Serializable Metadata after Loading
> >>

Re: (to Michael Starch) Problems Using Serializable Metadata after Loading Validation Layer

[Chris Mattmann]

Hmm, there is no reason that we shouldn’t simply just say FILEMGR_HOME is
a required
variable, and make the default configuration more sensible.

Val, Mike, anyone willing to open an issue for 0.11 for this?

—
Chris Mattmann
chris.mattm...@gmail.com






-Original Message-
From: "Mallder, Valerie" 
Reply-To: 
Date: Monday, August 31, 2015 at 1:22 PM
To: "dev@oodt.apache.org" 
Subject: RE: (to Michael Starch) Problems Using Serializable Metadata
after Loading Validation Layer

>Hi Michael,
>
>Could you explain your fix to this issue with more detail?  I am having
>the same problem. The default filemgr.properties file sets these two
>values to directories that do not exist (so I have to set them to
>'something' valid). Here are the default settings:
>
># XML repository manager configuration
>org.apache.oodt.cas.filemgr.repositorymgr.dirs=file:///dir1,file:///dir2
>
># XML validation layer configuration
>org.apache.oodt.cas.filemgr.validation.dirs=file:///dir1,file:///dir2
>
>
>And this is what I set them to:
>
># XML repository manager configuration
>org.apache.oodt.cas.filemgr.repositorymgr.dirs=file://[FILEMGR_HOME]/polic
>y/core
>
># XML validation layer configuration
>org.apache.oodt.cas.filemgr.validation.dirs=file://[FILEMGR_HOME]/policy/c
>ore
>
>And I still get the error.  Could you explain more about how I can work
>around this issue?
>
>Thanks,
>Valerie
>
>
>> -Original Message-
>> From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of
>>Michael
>> Starch
>> Sent: Friday, July 31, 2015 12:20 PM
>> To: dev@oodt.apache.org
>> Subject: Re: Problems Using Serializable Metadata after Loading
>>Validation Layer
>> 
>> All,
>> 
>> I found the issue.
>> 
>> Using "System.setProperties()" and filling it from properties read from
>> filemanager.properties clears out other properties setup by the system
>>which was
>> needed in the XML calls for SerializableMetadata. I did the above call
>>to setup
>> properties needed by the XMLValidationLayer
>> 
>> To fix set only the properties you need individually.  This adds to the
>>System
>> properties, not erasing them.
>> 
>> System.setProperty("org.apache.oodt.cas.filemgr.repositorymgr.dirs",
>>...);
>> 
>> System.setProperty("org.apache.oodt.cas.filemgr.validation.dirs", ...);
>> 
>> -Michael
>> 
>> 
>> On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch 
>>wrote:
>> 
>> > Here is the stack trace, but this only happens after a completely
>> > unrelated peice of the process load the XML Validation Layer.
>> >
>> > -Michael
>> >
>> > java.lang.NullPointerException
>> > at
>> > 
>>com.sun.org.apache.xml.internal.serializer.ToStream.(ToStream.java:
>>143)
>> > at
>> >
>> 
>>com.sun.org.apache.xml.internal.serializer.ToXMLStream.(ToXMLStream
>>.java:
>> 67)
>> > at
>> >
>> 
>>com.sun.org.apache.xml.internal.serializer.ToUnknownStream.(ToUnkno
>>wnStr
>> eam.java:143)
>> > at
>> >
>> 
>>com.sun.org.apache.xalan.internal.xsltc.runtime.output.TransletOutputHand
>>lerFacto
>> ry.getSerializationHandler(TransletOutputHandlerFactory.java:160)
>> > at
>> >
>> 
>>com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHan
>>dler(Tra
>> nsformerImpl.java:461)
>> > at
>> >
>> 
>>com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Tr
>>ansforme
>> rImpl.java:344)
>> > at
>> > org.apache.oodt.cas.metadata.SerializableMetadata.writeMetadataToXmlSt
>> > ream(SerializableMetadata.java:157)
>> >
>> >
>> > On Thu, Jul 30, 2015 at 4:16 PM, Chris Mattmann
>> > 
>> > wrote:
>> >
>> >> Mike can you give some specific line numbers? I can help look
>> >>
>> >> —
>> >> Chris Mattmann
>> >> chris.mattm...@gmail.com
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -Original Message-
>> >> From:  on behalf of Michael Starch <
>> >> starc...@umich.edu>
>> >> Reply-To: 
>> >> Date: Thursday, July 30, 2015 at 4:03 PM
>> >> To: 
>> >> Subject: Problems Using Serializable Metadata after Loading
>> >> Validation Layer
>> >>
>> >> >All,
>> >> >
>> >> >I am getting a NullPointerException deep in the XML library if I try
>> >> >to use the SerializableMetadata's write to xml function after I load
>> >> >in the XML Validation Layer from the filemanager. However, if I
>> >> >remove the call to load in the XML Validation Layer, everything
>> >> >works fine.  Any ideas as to what might cause this issue?
>> >> >
>> >> >Thanks,
>> >> >
>> >> >Michael
>> >>
>> >>
>> >>
>> >

Re: Strategy for Closing off GSoC 2015 Work

[Tom Barber]

Let me know when you've got stuff sorted Lewis, I'm happy to be a
XMLRPC-less File Manager guinea pig.

Tom

On Mon, Aug 31, 2015 at 9:09 PM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> ACK
> Will do this shortly Chris.
> Thanks
>
>
> On Mon, Aug 31, 2015 at 12:58 PM, Mattmann, Chris A (3980) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > +1 from me to push this up into a branch.
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattm...@nasa.gov
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
> >
> >
> >
> >
> > -Original Message-
> > From: Lewis John Mcgibbney 
> > Reply-To: "dev@oodt.apache.org" 
> > Date: Thursday, August 27, 2015 at 4:35 PM
> > To: "dev@oodt.apache.org" 
> > Subject: Re: Strategy for Closing off GSoC 2015 Work
> >
> > >I'm thinking that we branch trunk and integrate the code (once we rebase
> > >properly and compile a list of TODO's) into the branch.
> > >I'll be happy to ensure that the branch stays up to date with trunk.
> > >This way we can test, stabilize and build confidence in the new remote
> > >procedure implementations for various modules.
> > >We also still need to build it in to radix so that is a major step as
> > >well.
> > >Any objections to branching trunk post 0.10 release?
> > >
> > >On Tuesday, August 25, 2015, Tom Barber 
> wrote:
> > >
> > >> Yeah I'm well up for getting this all complete xmlrpc in osgi is
> giving
> > >>me
> > >> a rash.
> > >>
> > >> The stuff I'm developing I'll happily skip xmlrpc direct to avro if
> it's
> > >> functional.
> > >>
> > >> Tom
> > >> On 25 Aug 2015 07:58, "Chris Mattmann"  > >> > wrote:
> > >>
> > >> > Agreed.
> > >> >
> > >> > I am committed to getting this part of the master code
> > >> > base in Apache OODT. You’ve done amazing, Radu.
> > >> >
> > >> > Cheers,
> > >> > Chris
> > >> >
> > >> > —
> > >> > Chris Mattmann
> > >> > chris.mattm...@gmail.com 
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > -Original Message-
> > >> > From: Lewis John Mcgibbney  >
> > >> > Reply-To: >
> > >> > Date: Monday, August 24, 2015 at 10:28 PM
> > >> > To: "dev@oodt.apache.org "  > >> >, Radu Manole
> > >> > >
> > >> > Subject: Strategy for Closing off GSoC 2015 Work
> > >> >
> > >> > >Hi Folks,
> > >> > >As Radu's project is coming to an end, it is wise for us to start
> > >> thinking
> > >> > >about a strategy for closing off his work. This will involve one or
> > >>more
> > >> > >of
> > >> > >the following and potentially some other tasks.
> > >> > > 1) Identifying final TODO's across Review Board, Jira,
> Documentation
> > >> and
> > >> > >Tests.
> > >> > > 2) Having the community test this out. Potentially by creating a
> > >>branch
> > >> > >with the work merged into it accompanied with guidelines for how to
> > >>use
> > >> > >the
> > >> > >work and test out the functionality.
> > >> > > 3) Thinking about merging the code into what will soon become OODT
> > >> master
> > >> > >branch.
> > >> > >Any other ideas on top of this?
> > >> > >@Radu,
> > >> > >I am going through your various issues and will work with Chris and
> > >>team
> > >> > >to
> > >> > >make sure that OODT trunk does not diverge too much from your work.
> > >> > >Thank you for your work over the summer. This has been a
> challenging
> > >> task
> > >> > >but one which you've done very well to schedule and address.
> > >> > >On behalf of the OODT community, thank you.
> > >> > >Lewis
> > >> > >
> > >> > >--
> > >> > >*Lewis*
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> > >--
> > >*Lewis*
> >
> >
>
>
> --
> *Lewis*
>

RE: (to Michael Starch) Problems Using Serializable Metadata after Loading Validation Layer

[Mallder, Valerie]

Hi Michael,

Could you explain your fix to this issue with more detail?  I am having the 
same problem. The default filemgr.properties file sets these two values to 
directories that do not exist (so I have to set them to 'something' valid). 
Here are the default settings:  

# XML repository manager configuration
org.apache.oodt.cas.filemgr.repositorymgr.dirs=file:///dir1,file:///dir2

# XML validation layer configuration
org.apache.oodt.cas.filemgr.validation.dirs=file:///dir1,file:///dir2


And this is what I set them to:

# XML repository manager configuration
org.apache.oodt.cas.filemgr.repositorymgr.dirs=file://[FILEMGR_HOME]/policy/core

# XML validation layer configuration
org.apache.oodt.cas.filemgr.validation.dirs=file://[FILEMGR_HOME]/policy/core

And I still get the error.  Could you explain more about how I can work around 
this issue?

Thanks,
Valerie


> -Original Message-
> From: mdsta...@gmail.com [mailto:mdsta...@gmail.com] On Behalf Of Michael
> Starch
> Sent: Friday, July 31, 2015 12:20 PM
> To: dev@oodt.apache.org
> Subject: Re: Problems Using Serializable Metadata after Loading Validation 
> Layer
> 
> All,
> 
> I found the issue.
> 
> Using "System.setProperties()" and filling it from properties read from
> filemanager.properties clears out other properties setup by the system which 
> was
> needed in the XML calls for SerializableMetadata. I did the above call to 
> setup
> properties needed by the XMLValidationLayer
> 
> To fix set only the properties you need individually.  This adds to the System
> properties, not erasing them.
> 
> System.setProperty("org.apache.oodt.cas.filemgr.repositorymgr.dirs", ...);
> 
> System.setProperty("org.apache.oodt.cas.filemgr.validation.dirs", ...);
> 
> -Michael
> 
> 
> On Thu, Jul 30, 2015 at 4:20 PM, Michael Starch  wrote:
> 
> > Here is the stack trace, but this only happens after a completely
> > unrelated peice of the process load the XML Validation Layer.
> >
> > -Michael
> >
> > java.lang.NullPointerException
> > at
> > com.sun.org.apache.xml.internal.serializer.ToStream.(ToStream.java:143)
> > at
> >
> com.sun.org.apache.xml.internal.serializer.ToXMLStream.(ToXMLStream.java:
> 67)
> > at
> >
> com.sun.org.apache.xml.internal.serializer.ToUnknownStream.(ToUnknownStr
> eam.java:143)
> > at
> >
> com.sun.org.apache.xalan.internal.xsltc.runtime.output.TransletOutputHandlerFacto
> ry.getSerializationHandler(TransletOutputHandlerFactory.java:160)
> > at
> >
> com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getOutputHandler(Tra
> nsformerImpl.java:461)
> > at
> >
> com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Transforme
> rImpl.java:344)
> > at
> > org.apache.oodt.cas.metadata.SerializableMetadata.writeMetadataToXmlSt
> > ream(SerializableMetadata.java:157)
> >
> >
> > On Thu, Jul 30, 2015 at 4:16 PM, Chris Mattmann
> > 
> > wrote:
> >
> >> Mike can you give some specific line numbers? I can help look
> >>
> >> —
> >> Chris Mattmann
> >> chris.mattm...@gmail.com
> >>
> >>
> >>
> >>
> >>
> >>
> >> -Original Message-
> >> From:  on behalf of Michael Starch <
> >> starc...@umich.edu>
> >> Reply-To: 
> >> Date: Thursday, July 30, 2015 at 4:03 PM
> >> To: 
> >> Subject: Problems Using Serializable Metadata after Loading
> >> Validation Layer
> >>
> >> >All,
> >> >
> >> >I am getting a NullPointerException deep in the XML library if I try
> >> >to use the SerializableMetadata's write to xml function after I load
> >> >in the XML Validation Layer from the filemanager. However, if I
> >> >remove the call to load in the XML Validation Layer, everything
> >> >works fine.  Any ideas as to what might cause this issue?
> >> >
> >> >Thanks,
> >> >
> >> >Michael
> >>
> >>
> >>
> >

Re: Strategy for Closing off GSoC 2015 Work

[Lewis John Mcgibbney]

ACK
Will do this shortly Chris.
Thanks


On Mon, Aug 31, 2015 at 12:58 PM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> +1 from me to push this up into a branch.
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
> -Original Message-
> From: Lewis John Mcgibbney 
> Reply-To: "dev@oodt.apache.org" 
> Date: Thursday, August 27, 2015 at 4:35 PM
> To: "dev@oodt.apache.org" 
> Subject: Re: Strategy for Closing off GSoC 2015 Work
>
> >I'm thinking that we branch trunk and integrate the code (once we rebase
> >properly and compile a list of TODO's) into the branch.
> >I'll be happy to ensure that the branch stays up to date with trunk.
> >This way we can test, stabilize and build confidence in the new remote
> >procedure implementations for various modules.
> >We also still need to build it in to radix so that is a major step as
> >well.
> >Any objections to branching trunk post 0.10 release?
> >
> >On Tuesday, August 25, 2015, Tom Barber  wrote:
> >
> >> Yeah I'm well up for getting this all complete xmlrpc in osgi is giving
> >>me
> >> a rash.
> >>
> >> The stuff I'm developing I'll happily skip xmlrpc direct to avro if it's
> >> functional.
> >>
> >> Tom
> >> On 25 Aug 2015 07:58, "Chris Mattmann"  >> > wrote:
> >>
> >> > Agreed.
> >> >
> >> > I am committed to getting this part of the master code
> >> > base in Apache OODT. You’ve done amazing, Radu.
> >> >
> >> > Cheers,
> >> > Chris
> >> >
> >> > —
> >> > Chris Mattmann
> >> > chris.mattm...@gmail.com 
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > -Original Message-
> >> > From: Lewis John Mcgibbney >
> >> > Reply-To: >
> >> > Date: Monday, August 24, 2015 at 10:28 PM
> >> > To: "dev@oodt.apache.org "  >> >, Radu Manole
> >> > >
> >> > Subject: Strategy for Closing off GSoC 2015 Work
> >> >
> >> > >Hi Folks,
> >> > >As Radu's project is coming to an end, it is wise for us to start
> >> thinking
> >> > >about a strategy for closing off his work. This will involve one or
> >>more
> >> > >of
> >> > >the following and potentially some other tasks.
> >> > > 1) Identifying final TODO's across Review Board, Jira, Documentation
> >> and
> >> > >Tests.
> >> > > 2) Having the community test this out. Potentially by creating a
> >>branch
> >> > >with the work merged into it accompanied with guidelines for how to
> >>use
> >> > >the
> >> > >work and test out the functionality.
> >> > > 3) Thinking about merging the code into what will soon become OODT
> >> master
> >> > >branch.
> >> > >Any other ideas on top of this?
> >> > >@Radu,
> >> > >I am going through your various issues and will work with Chris and
> >>team
> >> > >to
> >> > >make sure that OODT trunk does not diverge too much from your work.
> >> > >Thank you for your work over the summer. This has been a challenging
> >> task
> >> > >but one which you've done very well to schedule and address.
> >> > >On behalf of the OODT community, thank you.
> >> > >Lewis
> >> > >
> >> > >--
> >> > >*Lewis*
> >> >
> >> >
> >> >
> >>
> >
> >
> >--
> >*Lewis*
>
>


-- 
*Lewis*

Re: catalog queries in pge config files and filemgr-client problem

[Mattmann, Chris A (3980)]

Val, I would inspect your file manager catalog. Can you use the OPSUI
or some other browser (Lucene) to see the metadata that is recorded
for your files? I think this is part of the problem - from the query
the only reason that $FileLocation/$Filename wouldn’t be resolved in
your queries is that they aren’t recorded as metadata fields for your
products.

Can you check?

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++





-Original Message-
From: Lewis John Mcgibbney 
Reply-To: "dev@oodt.apache.org" 
Date: Thursday, August 27, 2015 at 5:07 PM
To: "dev@oodt.apache.org" 
Subject: Re: catalog queries in pge config files and filemgr-client problem

>Hi Val,
>Wise idea to start debugging this from the filemgr-client.
>Can you please try changing logging levels in the following file to DEBUG
>
>https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/loggi
>ng.properties#L40-L74
>
>If you try your queries again then hopefully you should see some
>patterns/indications within FileMgr logs to indicate what is going on.
>
>
>On Wednesday, August 26, 2015, Mallder, Valerie
>
>wrote:
>
>> Hi All,
>>
>> So, I'm phasing off of the New Horizons project for now and I am back to
>> my science data pipeline work.  I am in the process of trying to pick up
>> where I left off a few months ago - but with a twist J  I graduated from
>> RADiX and now I am trying to get my pipeline system set up to run with
>>the
>> 'regular' OODT.  I have made extremely good progress so far. I have
>>pretty
>> much replicated all of the functionality of filemgr and workflow manager
>> that I had been using with RADiX, with the exception of one thing that
>> doesn't seem to be working the way it used too.  And I'm hoping someone
>>out
>> there can give me some ideas on how to fix it.
>>
>> I have a catalog query in one of my PGE config files. I copied the
>>syntax
>> for the query statement directly from one of Chris' PGE config files
>> (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
>>
>> Here is Chris' statement:
>>
>> > 
>>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim
>>e'){SELECT
>> FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}"/>
>>
>> Here is my statement:
>>
>>  > 
>>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim
>>e'){SELECT
>> FileLocation,Filename,CAS.ProductReceivedTime FROM ScienceFile}"/>
>>
>> And here are the results from the query:
>>
>>
>> 
>>$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$
>>FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
>>ileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fi
>>leLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
>>eLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$File
>>Location/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
>>ocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLo
>>cation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
>>ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoca
>>tion/$Filename,
>>
>> And those results are not exactly what I wanted J
>>
>> So I ran some experiments with ./filemgr-client on the command line to
>> make sure the files I expected to be in the catalog were in fact in the
>> catalog.
>>
>> The following operation worked just as expected.  The expected number of
>> products were listed.
>> ./filemgr-client --url http://localhost:9016 --operation --getFirstPage
>> --productTypeName ScienceFile
>>
>> However, this query gave some unexpected results. And this may be
>>related
>> the problem with the query in my PGE config file, but I am not sure.
>> ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
>>-query
>> "SELECT FileLocation,Filename FROM ScienceFile"
>>
>> That query only outputs several blank lines to the screen and nothing
>>else.
>>
>> I am not very familiar with all of the catalog related code yet so I am
>> not sure where to start looking for the problem.  If you know this area
>>of
>> the code well and have some debugging ideas, please let me know.
>>
>> Thanks so much!
>>
>> Val
>>
>>

Re: Strategy for Closing off GSoC 2015 Work

[Mattmann, Chris A (3980)]

+1 from me to push this up into a branch.

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++





-Original Message-
From: Lewis John Mcgibbney 
Reply-To: "dev@oodt.apache.org" 
Date: Thursday, August 27, 2015 at 4:35 PM
To: "dev@oodt.apache.org" 
Subject: Re: Strategy for Closing off GSoC 2015 Work

>I'm thinking that we branch trunk and integrate the code (once we rebase
>properly and compile a list of TODO's) into the branch.
>I'll be happy to ensure that the branch stays up to date with trunk.
>This way we can test, stabilize and build confidence in the new remote
>procedure implementations for various modules.
>We also still need to build it in to radix so that is a major step as
>well.
>Any objections to branching trunk post 0.10 release?
>
>On Tuesday, August 25, 2015, Tom Barber  wrote:
>
>> Yeah I'm well up for getting this all complete xmlrpc in osgi is giving
>>me
>> a rash.
>>
>> The stuff I'm developing I'll happily skip xmlrpc direct to avro if it's
>> functional.
>>
>> Tom
>> On 25 Aug 2015 07:58, "Chris Mattmann" > > wrote:
>>
>> > Agreed.
>> >
>> > I am committed to getting this part of the master code
>> > base in Apache OODT. You’ve done amazing, Radu.
>> >
>> > Cheers,
>> > Chris
>> >
>> > —
>> > Chris Mattmann
>> > chris.mattm...@gmail.com 
>> >
>> >
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Lewis John Mcgibbney >
>> > Reply-To: >
>> > Date: Monday, August 24, 2015 at 10:28 PM
>> > To: "dev@oodt.apache.org " > >, Radu Manole
>> > >
>> > Subject: Strategy for Closing off GSoC 2015 Work
>> >
>> > >Hi Folks,
>> > >As Radu's project is coming to an end, it is wise for us to start
>> thinking
>> > >about a strategy for closing off his work. This will involve one or
>>more
>> > >of
>> > >the following and potentially some other tasks.
>> > > 1) Identifying final TODO's across Review Board, Jira, Documentation
>> and
>> > >Tests.
>> > > 2) Having the community test this out. Potentially by creating a
>>branch
>> > >with the work merged into it accompanied with guidelines for how to
>>use
>> > >the
>> > >work and test out the functionality.
>> > > 3) Thinking about merging the code into what will soon become OODT
>> master
>> > >branch.
>> > >Any other ideas on top of this?
>> > >@Radu,
>> > >I am going through your various issues and will work with Chris and
>>team
>> > >to
>> > >make sure that OODT trunk does not diverge too much from your work.
>> > >Thank you for your work over the summer. This has been a challenging
>> task
>> > >but one which you've done very well to schedule and address.
>> > >On behalf of the OODT community, thank you.
>> > >Lewis
>> > >
>> > >--
>> > >*Lewis*
>> >
>> >
>> >
>>
>
>
>-- 
>*Lewis*

Re: Ubuntu build issues with corrupt Jars/Poms for Apache OODT

[Tom Barber]

Ah never dawned on me it could be slave related! It was doing my head in.
Thanks Chris.
On 31 Aug 2015 19:02, "Mattmann, Chris A (3980)" <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hey Folks,
>
> I’ve noticed the past few builds for Apache OODT are running into
> some corrupt HP Jena jar files on the build slaves. In particular,
> see:
>
> https://builds.apache.org/job/oodt-trunk/2008/org.apache.oodt$opendapps/con
> sole
>
> [WARNING] error reading
> /home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jen
> a/iri/0.8/iri-0.8.jar; error in opening zip file
> [WARNING] error reading
> /home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jen
> a/iri/0.8/iri-0.8.jar; error in opening zip file
> [WARNING] POM for 'com.hp.hpl.jena:iri:pom:0.8:compile' is invalid.
> Its dependencies (if any) will NOT be available to the current build.
> [WARNING] POM for 'com.ibm.icu:icu4j:pom:3.4.4:compile' is invalid.
> Its dependencies (if any) will NOT be available to the current build.
> [WARNING] POM for 'xerces:xercesImpl:pom:2.7.1:compile' is invalid.
> Its dependencies (if any) will NOT be available to the current build.
> [WARNING] POM for 'log4j:log4j:pom:1.2.13:runtime' is invalid.
>
>
> If these jars could be fixed on the ubuntu slaves, I’d appreciate it.
> Looks like this ran on ubuntu1:
> https://builds.apache.org/job/oodt-trunk/2008/
>
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>

Ubuntu build issues with corrupt Jars/Poms for Apache OODT

[Mattmann, Chris A (3980)]

Hey Folks,

I’ve noticed the past few builds for Apache OODT are running into
some corrupt HP Jena jar files on the build slaves. In particular,
see:

https://builds.apache.org/job/oodt-trunk/2008/org.apache.oodt$opendapps/con
sole

[WARNING] error reading
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jen
a/iri/0.8/iri-0.8.jar; error in opening zip file
[WARNING] error reading
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jen
a/iri/0.8/iri-0.8.jar; error in opening zip file
[WARNING] POM for 'com.hp.hpl.jena:iri:pom:0.8:compile' is invalid.
Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'com.ibm.icu:icu4j:pom:3.4.4:compile' is invalid.
Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'xerces:xercesImpl:pom:2.7.1:compile' is invalid.
Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'log4j:log4j:pom:1.2.13:runtime' is invalid.


If these jars could be fixed on the ubuntu slaves, I’d appreciate it.
Looks like this ran on ubuntu1:
https://builds.apache.org/job/oodt-trunk/2008/


Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Jenkins build is back to stable : oodt-trunk » CAS PGE Adaptor Framework #2008

[Apache Jenkins Server]

See 

Build failed in Jenkins: oodt-trunk #2008

[Apache Jenkins Server]

See 

--
[...truncated 9182 lines...]
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 124151 bytes
Compression is 0.0%
Took 55 ms
[JENKINS] Archiving  
to org.apache.oodt/oodt-xmlps/0.11-SNAPSHOT/oodt-xmlps-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/oodt-xmlps/0.11-SNAPSHOT/oodt-xmlps-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » XML-configured, DBMS-based 
Product and Profile Server #2007
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 65566 bytes
Compression is 0.0%
Took 38 ms
[JENKINS] Archiving 

 to org.apache.oodt/oodt-core/0.11-SNAPSHOT/oodt-core-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/oodt-core/0.11-SNAPSHOT/oodt-core-0.11-SNAPSHOT-site.xml

 is not inside  will 
archive in a separate pass
Sending artifact delta relative to oodt-trunk » OODT Core #2007
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 1025 bytes
Compression is 0.0%
Took 25 ms
Sending artifact delta relative to oodt-trunk » OODT Core #2007
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 18909 bytes
Compression is 0.0%
Took 27 ms
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-protocol-http/0.11-SNAPSHOT/cas-protocol-http-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-protocol-http/0.11-SNAPSHOT/cas-protocol-http-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » CAS Protocol HTTP 
Implementation #2007
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 23174 bytes
Compression is 0.0%
Took 28 ms
[JENKINS] Archiving 
 to 
org.apache.oodt/curator-services/0.11-SNAPSHOT/curator-services-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/curator-services/0.11-SNAPSHOT/curator-services-0.11-SNAPSHOT.war
Sending artifact delta relative to oodt-trunk » CAS Curation Web Services #2007
Archived 2 artifacts
Archive block size is 32768
Received 1571 blocks and 4085936 bytes
Compression is 92.6%
Took 2.8 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT.jar
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT-dist.tar.gz
[JENKINS] Archiving 

 to org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT-dist.zip
Sending artifact delta relative to oodt-trunk » Catalog and Archive Crawling 
Framework #2007
Archived 4 artifacts
Archive block size is 32768
Received 1462 blocks and 56458107 bytes
Compression is 45.9%
Took 17 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/oodt-webapp-components/0.11-SNAPSHOT/oodt-webapp-components-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/oodt-webapp-components/0.11-SNAPSHOT/oodt-webapp-components-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » OODT Wicket Web Components #2007
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 329949 bytes
Compression is 0.0%
Took 0.1 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-protocol-api/0.11-SNAPSHOT/cas-protocol-api-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-protocol-api/0.11-SNAPSHOT/cas-protocol-api-0.11-SNAPSHOT.jar
[JENKINS] Archiving 

Build failed in Jenkins: oodt-trunk » Apache OODT Configurable OPeNDAP Profile Server #2008

[Apache Jenkins Server]

See 

--
[INFO] 
[INFO] Building Apache OODT Configurable OPeNDAP Profile Server
[INFO]task-segment: [install]
[INFO] 
[INFO] [remote-resources:process {execution: default}]
Downloading: http://repo1.maven.org/maven2/edu/ucar/netcdf/4.2/netcdf-4.2.pom
4K downloaded  (netcdf-4.2.pom)
Downloading: http://repo1.maven.org/maven2/edu/ucar/netcdf/4.2/netcdf-4.2.jar
Downloading: 
http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar
21K downloaded  (slf4j-api-1.5.6.jar)
11196K downloaded  (netcdf-4.2.jar)
[WARNING] Invalid project model for artifact [iri:com.hp.hpl.jena:0.8]. It will 
be ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [iri:com.hp.hpl.jena:0.8]. It will 
be ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [log4j:log4j:1.2.13]. It will be 
ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [icu4j:com.ibm.icu:3.4.4]. It will 
be ignored by the remote resources Mojo.
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 23 source files to 

[WARNING] error reading 
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jena/iri/0.8/iri-0.8.jar;
 error in opening zip file
[WARNING] error reading 
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jena/iri/0.8/iri-0.8.jar;
 error in opening zip file
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] No sources to compile
[INFO] [surefire:test {execution: default-test}]
[JENKINS] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: 

[INFO] Preparing javadoc:javadoc
[WARNING] Removing: javadoc from forked lifecycle, to prevent recursive 
invocation.
[INFO] No goals needed for project - skipping
[WARNING] DEPRECATED [aggregate]: since 2.5. Use the goals 
javadoc:aggregate and javadoc:test-aggregate instead.
[INFO] [javadoc:javadoc {execution: attach-javadocs}]
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-io/2.0.9/plexus-io-2.0.9.pom
2K downloaded  (plexus-io-2.0.9.pom)
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/3.0.10/plexus-utils-3.0.10.pom
3K downloaded  (plexus-utils-3.0.10.pom)
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-io/2.0.9/plexus-io-2.0.9.jar
57K downloaded  (plexus-io-2.0.9.jar)
[INFO] [assembly:single {execution: make-assembly}]
[WARNING] POM for 'com.hp.hpl.jena:iri:pom:0.8:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'com.ibm.icu:icu4j:pom:3.4.4:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'xerces:xercesImpl:pom:2.7.1:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'log4j:log4j:pom:1.2.13:runtime' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[INFO] 
[ERROR] BUILD ERROR
[INFO] 
[INFO] Failed to create assembly: Error creating assembly archive 
jar-with-dependencies: error in opening zip file

[INFO] 
[INFO] For more information, run Maven with the -e switch
[INFO] 
[INFO] Total time: 6 minutes 47 seconds
[INFO] Finished at: Mon Aug 31 17:57:12 UTC 2015
[INFO] Fina

Build failed in Jenkins: oodt-trunk #2007

[Apache Jenkins Server]

See 

--
[...truncated 9182 lines...]
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 124150 bytes
Compression is 0.0%
Took 62 ms
[JENKINS] Archiving  
to org.apache.oodt/oodt-xmlps/0.11-SNAPSHOT/oodt-xmlps-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/oodt-xmlps/0.11-SNAPSHOT/oodt-xmlps-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » XML-configured, DBMS-based 
Product and Profile Server #2006
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 65566 bytes
Compression is 0.0%
Took 37 ms
[JENKINS] Archiving 

 to org.apache.oodt/oodt-core/0.11-SNAPSHOT/oodt-core-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/oodt-core/0.11-SNAPSHOT/oodt-core-0.11-SNAPSHOT-site.xml

 is not inside  will 
archive in a separate pass
Sending artifact delta relative to oodt-trunk » OODT Core #2006
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 1025 bytes
Compression is 0.0%
Took 30 ms
Sending artifact delta relative to oodt-trunk » OODT Core #2006
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 18909 bytes
Compression is 0.0%
Took 27 ms
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-protocol-http/0.11-SNAPSHOT/cas-protocol-http-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-protocol-http/0.11-SNAPSHOT/cas-protocol-http-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » CAS Protocol HTTP 
Implementation #2006
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 23174 bytes
Compression is 0.0%
Took 42 ms
[JENKINS] Archiving 
 to 
org.apache.oodt/curator-services/0.11-SNAPSHOT/curator-services-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/curator-services/0.11-SNAPSHOT/curator-services-0.11-SNAPSHOT.war
Sending artifact delta relative to oodt-trunk » CAS Curation Web Services #2006
Archived 2 artifacts
Archive block size is 32768
Received 1567 blocks and 4217115 bytes
Compression is 92.4%
Took 3.2 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT.jar
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT-dist.tar.gz
[JENKINS] Archiving 

 to org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT-dist.zip
Sending artifact delta relative to oodt-trunk » Catalog and Archive Crawling 
Framework #2006
Archived 4 artifacts
Archive block size is 32768
Received 1461 blocks and 56490397 bytes
Compression is 45.9%
Took 17 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/oodt-webapp-components/0.11-SNAPSHOT/oodt-webapp-components-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/oodt-webapp-components/0.11-SNAPSHOT/oodt-webapp-components-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » OODT Wicket Web Components #2006
Archived 2 artifacts
Archive block size is 32768
Received 1 blocks and 297181 bytes
Compression is 9.9%
Took 0.33 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-protocol-api/0.11-SNAPSHOT/cas-protocol-api-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-protocol-api/0.11-SNAPSHOT/cas-protocol-api-0.11-SNAPSHOT.jar
[JENKINS] Archiving 

Build failed in Jenkins: oodt-trunk » Apache OODT Configurable OPeNDAP Profile Server #2007

[Apache Jenkins Server]

See 

--
[INFO] 
[INFO] Building Apache OODT Configurable OPeNDAP Profile Server
[INFO]task-segment: [install]
[INFO] 
[INFO] [remote-resources:process {execution: default}]
Downloading: http://repo1.maven.org/maven2/edu/ucar/netcdf/4.2/netcdf-4.2.pom
4K downloaded  (netcdf-4.2.pom)
Downloading: http://repo1.maven.org/maven2/edu/ucar/netcdf/4.2/netcdf-4.2.jar
Downloading: 
http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar
21K downloaded  (slf4j-api-1.5.6.jar)
11196K downloaded  (netcdf-4.2.jar)
[WARNING] Invalid project model for artifact [iri:com.hp.hpl.jena:0.8]. It will 
be ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [iri:com.hp.hpl.jena:0.8]. It will 
be ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [log4j:log4j:1.2.13]. It will be 
ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [icu4j:com.ibm.icu:3.4.4]. It will 
be ignored by the remote resources Mojo.
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 23 source files to 

[WARNING] error reading 
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jena/iri/0.8/iri-0.8.jar;
 error in opening zip file
[WARNING] error reading 
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jena/iri/0.8/iri-0.8.jar;
 error in opening zip file
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] No sources to compile
[INFO] [surefire:test {execution: default-test}]
[JENKINS] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: 

[INFO] Preparing javadoc:javadoc
[WARNING] Removing: javadoc from forked lifecycle, to prevent recursive 
invocation.
[INFO] No goals needed for project - skipping
[WARNING] DEPRECATED [aggregate]: since 2.5. Use the goals 
javadoc:aggregate and javadoc:test-aggregate instead.
[INFO] [javadoc:javadoc {execution: attach-javadocs}]
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-io/2.0.9/plexus-io-2.0.9.pom
2K downloaded  (plexus-io-2.0.9.pom)
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/3.0.10/plexus-utils-3.0.10.pom
3K downloaded  (plexus-utils-3.0.10.pom)
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-io/2.0.9/plexus-io-2.0.9.jar
57K downloaded  (plexus-io-2.0.9.jar)
[INFO] [assembly:single {execution: make-assembly}]
[WARNING] POM for 'com.hp.hpl.jena:iri:pom:0.8:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'com.ibm.icu:icu4j:pom:3.4.4:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'xerces:xercesImpl:pom:2.7.1:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'log4j:log4j:pom:1.2.13:runtime' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[INFO] 
[ERROR] BUILD ERROR
[INFO] 
[INFO] Failed to create assembly: Error creating assembly archive 
jar-with-dependencies: error in opening zip file

[INFO] 
[INFO] For more information, run Maven with the -e switch
[INFO] 
[INFO] Total time: 7 minutes 38 seconds
[INFO] Finished at: Mon Aug 31 17:44:40 UTC 2015
[INFO] Fina

Jenkins build became unstable: oodt-trunk » CAS PGE Adaptor Framework #2007

[Apache Jenkins Server]

See 

Re: Review Request 37701: Full Avro RPC impementation in crawler module.

[Chris Mattmann]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37701/#review97093
---

Ship it!


Ship It!

- Chris Mattmann


On Aug. 27, 2015, 12:11 p.m., Radu Manole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37701/
> ---
> 
> (Updated Aug. 27, 2015, 12:11 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> Modified the DaemonCrawler to use Avro RPC login instead of XML-RPC.
> 
> 
> Diffs
> -
> 
>   trunk/crawler/pom.xml 1693501 
>   trunk/crawler/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/crawler/src/main/bin/crawlctl 1693501 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/cli/action/CrawlerLauncherCliAction.java
>  1693501 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroCrawlDaemonController.java
>  PRE-CREATION 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/AvroRpcCrawlerDaemon.java
>  PRE-CREATION 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemon.java 
> 1693501 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/CrawlDaemonController.java
>  1693501 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlDaemonController.java
>  PRE-CREATION 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/daemon/XmlRpcCrawlerDaemon.java
>  PRE-CREATION 
>   
> trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/DummyDaemonCrawler.java
>  PRE-CREATION 
>   
> trunk/crawler/src/test/java/org/apache/oodt/cas/crawl/daemon/TestDaemonCrawler.java
>  PRE-CREATION 
>   
> trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/crawler/src/main/resources/bin/crawlctl
>  1693501 
>   
> trunk/pcs/core/src/main/java/org/apache/oodt/pcs/tools/PCSHealthMonitor.java 
> 1693501 
> 
> Diff: https://reviews.apache.org/r/37701/diff/
> 
> 
> Testing
> ---
> 
> No test was found for xmlrpc DaemonCrawler logic, so I made a 
> DummyAvroDaemonCrawler class to test only the rpc aspect of daemon crawler. 
> and created a class TestDaemonCrawler where I tested the connection and the 
> communication between AvroDaemonCrawler and AvroDaemonCrawlerController.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>

[VOTE] Apache OODT 0.10 RC #1

[Chris Mattmann]

Hi Folks,

I have posted a 1st release candidate for the Apache OODT 0.10
release.  The source code is at:

https://dist.apache.org/repos/dist/dev/oodt/

For more detailed information, see the included CHANGES.txt file
for details on release contents and latest changes. The release was
made using the OODT release process, documented on the Wiki here:

https://cwiki.apache.org/confluence/display/OODT/Release+Process

The release was made from the OODT 0.10 tag (r1700276) at:

http://svn.apache.org/repos/asf/oodt/tags/0.10-rc1/

A staged Maven repository is available at 2 locations
due to me pausing the release and then restarting it:

1st repo:
https://repository.apache.org/content/repositories/orgapacheoodt-1005/

2nd repo:
https://repository.apache.org/content/repositories/orgapacheoodt-1006/


Please vote on releasing these packages as Apache OODT 0.10. The
vote is open for at least the next 72 hours.

Only votes from OODT PMC are binding, but folks are welcome to check
the release candidate and voice their approval or disapproval. The
vote passes if at least three binding +1 votes are cast.

[ ] +1 Release the packages as Apache OODT 0.10

[ ] -1 Do not release the packages because...

Thanks!

Cheers, 
Chris

P.S. Here is my +1.

Build failed in Jenkins: oodt-trunk » Apache OODT Configurable OPeNDAP Profile Server #2006

[Apache Jenkins Server]

See 


Changes:

[mattmann] [maven-release-plugin] prepare for next development iteration

[mattmann] [maven-release-plugin] prepare release 0.10-rc1

--
[INFO] 
[INFO] Building Apache OODT Configurable OPeNDAP Profile Server
[INFO]task-segment: [install]
[INFO] 
[INFO] [remote-resources:process {execution: default}]
Downloading: http://repo1.maven.org/maven2/edu/ucar/netcdf/4.2/netcdf-4.2.pom
4K downloaded  (netcdf-4.2.pom)
Downloading: http://repo1.maven.org/maven2/edu/ucar/netcdf/4.2/netcdf-4.2.jar
Downloading: 
http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.5.6/slf4j-api-1.5.6.jar
21K downloaded  (slf4j-api-1.5.6.jar)
11196K downloaded  (netcdf-4.2.jar)
[WARNING] Invalid project model for artifact [iri:com.hp.hpl.jena:0.8]. It will 
be ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [iri:com.hp.hpl.jena:0.8]. It will 
be ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [log4j:log4j:1.2.13]. It will be 
ignored by the remote resources Mojo.
[WARNING] Invalid project model for artifact [icu4j:com.ibm.icu:3.4.4]. It will 
be ignored by the remote resources Mojo.
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 23 source files to 

[WARNING] error reading 
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jena/iri/0.8/iri-0.8.jar;
 error in opening zip file
[WARNING] error reading 
/home/jenkins/jenkins-slave/workspace/oodt-trunk/.repository/com/hp/hpl/jena/iri/0.8/iri-0.8.jar;
 error in opening zip file
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] No sources to compile
[INFO] [surefire:test {execution: default-test}]
[JENKINS] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: 

[INFO] Preparing javadoc:javadoc
[WARNING] Removing: javadoc from forked lifecycle, to prevent recursive 
invocation.
[INFO] No goals needed for project - skipping
[WARNING] DEPRECATED [aggregate]: since 2.5. Use the goals 
javadoc:aggregate and javadoc:test-aggregate instead.
[INFO] [javadoc:javadoc {execution: attach-javadocs}]
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-io/2.0.9/plexus-io-2.0.9.pom
2K downloaded  (plexus-io-2.0.9.pom)
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/3.0.10/plexus-utils-3.0.10.pom
3K downloaded  (plexus-utils-3.0.10.pom)
Downloading: 
http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-io/2.0.9/plexus-io-2.0.9.jar
57K downloaded  (plexus-io-2.0.9.jar)
[INFO] [assembly:single {execution: make-assembly}]
[WARNING] POM for 'com.hp.hpl.jena:iri:pom:0.8:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'com.ibm.icu:icu4j:pom:3.4.4:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'xerces:xercesImpl:pom:2.7.1:compile' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'log4j:log4j:pom:1.2.13:runtime' is invalid.

Its dependencies (if any) will NOT be available to the current build.
[INFO] 
[ERROR] BUILD ERROR
[INFO] 
[INFO] Failed to create assembly: Error creating assembly archive 
jar-with-dependencies: error in opening zip file

[INFO] 
[INFO] For more information, run Maven with the -e switch
[INFO] -

Build failed in Jenkins: oodt-trunk #2006

[Apache Jenkins Server]

See 

Changes:

[mattmann] [maven-release-plugin] prepare for next development iteration

[mattmann] [maven-release-plugin] prepare release 0.10-rc1

--
[...truncated 9179 lines...]
Sending artifact delta relative to oodt-trunk » Catalog and Archive Service 
Generic Multi-valued Metadata Container #2005
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 124149 bytes
Compression is 0.0%
Took 0.13 sec
[JENKINS] Archiving  
to org.apache.oodt/oodt-xmlps/0.11-SNAPSHOT/oodt-xmlps-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/oodt-xmlps/0.11-SNAPSHOT/oodt-xmlps-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » XML-configured, DBMS-based 
Product and Profile Server #2005
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 65565 bytes
Compression is 0.0%
Took 40 ms
[JENKINS] Archiving 

 to org.apache.oodt/oodt-core/0.11-SNAPSHOT/oodt-core-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/oodt-core/0.11-SNAPSHOT/oodt-core-0.11-SNAPSHOT-site.xml

 is not inside  will 
archive in a separate pass
Sending artifact delta relative to oodt-trunk » OODT Core #2005
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 1025 bytes
Compression is 0.0%
Took 28 ms
Sending artifact delta relative to oodt-trunk » OODT Core #2005
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 18909 bytes
Compression is 0.0%
Took 0.13 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-protocol-http/0.11-SNAPSHOT/cas-protocol-http-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-protocol-http/0.11-SNAPSHOT/cas-protocol-http-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » CAS Protocol HTTP 
Implementation #2005
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 23174 bytes
Compression is 0.0%
Took 0.1 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/curator-services/0.11-SNAPSHOT/curator-services-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/curator-services/0.11-SNAPSHOT/curator-services-0.11-SNAPSHOT.war
Sending artifact delta relative to oodt-trunk » CAS Curation Web Services #2005
Archived 2 artifacts
Archive block size is 32768
Received 1568 blocks and 4184047 bytes
Compression is 92.5%
Took 4.9 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT.jar
[JENKINS] Archiving 

 to 
org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT-dist.tar.gz
[JENKINS] Archiving 

 to org.apache.oodt/cas-crawler/0.11-SNAPSHOT/cas-crawler-0.11-SNAPSHOT-dist.zip
Sending artifact delta relative to oodt-trunk » Catalog and Archive Crawling 
Framework #2005
Archived 4 artifacts
Archive block size is 32768
Received 1461 blocks and 56490694 bytes
Compression is 45.9%
Took 22 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/oodt-webapp-components/0.11-SNAPSHOT/oodt-webapp-components-0.11-SNAPSHOT.pom
[JENKINS] Archiving 

 to 
org.apache.oodt/oodt-webapp-components/0.11-SNAPSHOT/oodt-webapp-components-0.11-SNAPSHOT.jar
Sending artifact delta relative to oodt-trunk » OODT Wicket Web Components #2005
Archived 2 artifacts
Archive block size is 32768
Received 1 blocks and 297182 bytes
Compression is 9.9%
Took 0.1 sec
[JENKINS] Archiving 
 to 
org.apache.oodt/cas-protocol-api/0.11-SNAPSHOT/cas-protocol-ap