Re: RADiX filemgr-client script cannot delete product by id or name

[MengYing Wang]

Dear Prof. Mattmann,

Now the ./query-tool sql query does work too. -_-!

$ ./query-tool --url http://localhost:9000 --sql -query "SELECT * FROM
GenericFile"

org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to
perform complex query : null

at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)

at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)

at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)

at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)

at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)

at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)

at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:1003)

at
org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:252)

at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242)

Exception in thread "main"
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
java.lang.Exception:
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to
perform complex query : null

at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:1009)

at
org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:252)

at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242)

Best,

Mengying (Angela) Wang

On Sat, Nov 1, 2014 at 1:09 PM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Can you please remove your FILEMGR_HOME/catalog (or your
> RADIX_HOME/data/catalog)
> directory, restart file manager, and then try again?
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
>
> -Original Message-
> From: MengYing Wang 
> Date: Saturday, November 1, 2014 at 1:03 PM
> To: "dev@oodt.apache.org" 
> Cc: Chris Mattmann 
> Subject: Re: RADiX filemgr-client script cannot delete product by id or
> name
>
> >Dear Prof. Mattmann,
> >
> >
> >Thank you for your help! Actually, I have set the FILEMGR_URL. Below are
> >the variables that I have set:
> >
> >
> >export OODT_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7"
> >export
> >CRAWLER_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/crawler"
> >export
> >FILEMGR_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr"
> >export
> >WORKFLOW_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/workflow"
> >export PGE_ROOT="/Users/AngelaWang/Documents/programs/oodt-0.7/pge"
> >export WORKFLOW_URL="http://localhost:9200";
> >export FILEMGR_URL="http://localhost:9000";
> >
> >
> >
> >Best,
> >Mengying (Angela) Wang
> >
> >
> >On Sat, Nov 1, 2014 at 10:46 AM, Mattmann, Chris A (3980)
> > wrote:
> >
> >This is an environment variable issue - your FILEMGR_URL is not set.
> >Can you check it Angela?
> >
> >Cheers,
> >Chris
> >
> >
> >++
> >Chris Mattmann, Ph.D.
> >Chief Architect
> >Instrument Software and Science Data Systems Section (398)
> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >Office: 168-519, Mailstop: 168-527
> >Email: chris.a.mattm...@nasa.gov
> >WWW:  http://sunset.usc.edu/~mattmann/
> >++
> >Adjunct Associate Professor, Computer Science Department
> >University of Southern California, Los Angeles, CA 90089 USA
> >++
> >
> >
> >
> >
> >
> >
> >-Original Message-
> >From: MengYing Wang 
> >Reply-To: "dev@oodt.apache.org" 
> >Date: Thursday, October 30, 2014 at 3:02 PM
> >To: "dev@oodt.apache.org" 
> >Subject: RADiX filemgr-client script cannot delete product by id or name
> >
> >>Hey everyone,
> >>
> >>I am trying to delete some product in the OODT File Manager with its
> >>product id or name. However, the system raises
> >>"java.lang.IllegalArgumentException: URI is not absolute" error. Please
> >>help me! Thank you!
> >>
> >>My commands are:
> >>
> >>./filemgr-client --op --dbyid --productId
> >>2b0e7582-6077-11e4-ac24-8d456d2e5b9b --url
> >http://localhost:9000 
> >>
> >>./filemgr-client -op --deleteProductByName --productName
> >>concatenatedOutputFile-2014-10-30T20:56:02.189Z.tx

Re: RADiX filemgr-client script cannot delete product by id or name

[Mattmann, Chris A (3980)]

Yes, you have to reingest your data after restarting your file manager
after deleting
the catalog. Then try the query tool

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: MengYing Wang 
Date: Saturday, November 1, 2014 at 1:47 PM
To: Chris Mattmann 
Cc: "dev@oodt.apache.org" , Chris Mattmann

Subject: Re: RADiX filemgr-client script cannot delete product by id or
name

>Dear Prof. Mattmann,
>
>
>Now the ./query-tool sql query does work too. -_-!
>$ ./query-tool --url http://localhost:9000 --sql -query "SELECT * FROM
>GenericFile"
>org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>to perform complex query : null
>at 
>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClie
>ntResponseProcessor.java:104)
>at 
>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClien
>tResponseProcessor.java:71)
>at 
>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>at 
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(Xm
>lRpcFileManagerClient.java:1003)
>at 
>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java
>:252)
>at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242)
>Exception in thread "main"
>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>java.lang.Exception:
>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>to perform complex query : null
>at 
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(Xm
>lRpcFileManagerClient.java:1009)
>at 
>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java
>:252)
>at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242)
>Best,
>Mengying (Angela) Wang
>
>
>
>On Sat, Nov 1, 2014 at 1:09 PM, Mattmann, Chris A (3980)
> wrote:
>
>Can you please remove your FILEMGR_HOME/catalog (or your
>RADIX_HOME/data/catalog)
>directory, restart file manager, and then try again?
>
>++
>Chris Mattmann, Ph.D.
>Chief Architect
>Instrument Software and Science Data Systems Section (398)
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 168-519, Mailstop: 168-527
>Email: chris.a.mattm...@nasa.gov
>WWW:  http://sunset.usc.edu/~mattmann/
>++
>Adjunct Associate Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++
>
>
>
>
>
>
>-Original Message-
>From: MengYing Wang 
>Date: Saturday, November 1, 2014 at 1:03 PM
>To: "dev@oodt.apache.org" 
>Cc: Chris Mattmann 
>Subject: Re: RADiX filemgr-client script cannot delete product by id or
>name
>
>>Dear Prof. Mattmann,
>>
>>
>>Thank you for your help! Actually, I have set the FILEMGR_URL. Below are
>>the variables that I have set:
>>
>>
>>export OODT_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7"
>>export
>>CRAWLER_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/crawler"
>>export
>>FILEMGR_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr"
>>export
>>WORKFLOW_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/workflow"
>>export PGE_ROOT="/Users/AngelaWang/Documents/programs/oodt-0.7/pge"
>>export WORKFLOW_URL="http://localhost:9200";
>>export FILEMGR_URL="http://localhost:9000";
>>
>>
>>
>>Best,
>>Mengying (Angela) Wang
>>
>>
>>On Sat, Nov 1, 2014 at 10:46 AM, Mattmann, Chris A (3980)
>> wrote:
>>
>>This is an environment variable issue - your FILEMGR_URL is not set.
>>Can you check it Angela?
>>
>>Cheers,
>>Chris
>>
>>
>>++
>>Chris Mattmann, Ph.D.
>>Chief Architect
>>Instrument Software and Science Data Systems Section (398)
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 168-519, Mailstop: 168-527
>>Email: chris.a.mattm...@nasa.gov
>>WWW:  http://sunset.usc.edu/~mattmann/
>>++
>>Adjunct Associate Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 900

Re: Questions about OODT

[Christian Alan Mattmann]

Hi Kang,

Thanks for your questions. They are answered below:



Chris Mattmann, Ph.D.
Adjunct Associate Professor, Computer Science Department
University of Southern California
Los Angeles, CA 90089 USA
Email: mattm...@usc.edu
WWW: http://sunset.usc.edu/~mattmann/





-Original Message-
From: Kang Wang 
Date: Wednesday, October 29, 2014 at 12:16 AM
To: Chris Mattmann 
Subject: Questions about OODT

>Hi, pf Chris
>
>
>I have two questions about using OODT.
>
>
>1. What is product in OODT?
>Is a single json file is a product or whole dataset is a product?
>I don't know for --productStructure we should use Flat file or
>directory.

For the purposes of this assignment, consider a ³product² to be a JSON
file,
a single instance of it. Just use the Flat structure.

>
>
>2. Should we create .met metadata file?
>I means in job posting josn file, it has already contained job
>posting metadata, 
>why we need to establish metadata file again?
>Does metadata file is for mapping metadata of job posting json to
>elements defined in product-type?
>If we need to create .met file, how to customize metadata file? I
>just see the following
>snippet about .met file.
>blah.txt.met
> xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";
>>
>
>

Yep you need this .met file since OODT uses that to index it in the File
Manager.

Cheers,
Chris

>
>
>
>
>
>Best Regards,
>Kang
>
>
>

Re: Why is pushpull not always working?

[Cameron Goodale]

Hey Lewis,

Last time I dug into this I was running toward the "Active vs. Passive" FTP
settings, but I am not sure that is the root cause.  One option or fix
would be to add in a timeout parameter, where if the pushpull daemon
doesn't get a reply in a set amount of time, it should stop and restart.
Every time I have seen pushpull stall or hang, a simple stop and restart
gets it working again.

I know Ross Laidlaw is using pushpull on a his Near Real Time snow
processing pipeline, so I am sure he will be very interested in your work
Lewis.

Thanks,


Cameron

On Sat, Nov 1, 2014 at 10:58 AM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> Thanks Chris. OK doke. i am going to debug this today and find out what is
> happening.
> Thanks for heads up.
> Lewis
>
> On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann 
> wrote:
>
> > Thanks Lewis - in my mind it boils down somewhat to
> > the protocols that are implemented and the way they
> > behave with FTP servers and so forth. Ultimately, we
> > should be using the ones that are in PushPull plugins,
> > but they are virally licensed and I haven¹t found a
> > better alternative.
> >
> > Another @bfoster update here would be great, Brian?
> >
> > Cheers,
> > Chris
> >
> >
> > 
> > Chris Mattmann
> > chris.mattm...@gmail.com
> >
> >
> >
> >
> > -Original Message-
> > From: Lewis John Mcgibbney 
> > Reply-To: 
> > Date: Saturday, November 1, 2014 at 12:56 AM
> > To: "dev@oodt.apache.org" 
> > Subject: Why is pushpull not always working?
> >
> > >Hi Folks,
> > >I really couldn't think of a better title for this thread so I apologize
> > >in
> > >advance.
> > >I recently set up pushpull with FTP to crawl remote ftp site [0] which
> is
> > >publicly available.
> > >I am getting the following
> > >
> > >INFO: Launching Daemons . . .
> > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> > >registerRMIServer
> > >INFO: Created Daemon ID = 90121 on RMI registry port 9012
> > >Nov 01, 2014 12:46:01 AM
> > >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
> > >launchDaemons
> > >INFO: Creating Daemon with ID = 1
> > >Nov 01, 2014 12:46:01 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:48:00 PDT 2014
> > >Nov 01, 2014 12:48:00 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:51:00 PDT 2014
> > >Nov 01, 2014 12:50:59 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:54:00 PDT 2014
> > >Nov 01, 2014 12:53:59 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:57:00 PDT 2014
> > >
> > >No errors... no pulling... no nothing!
> > >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
> > >plugins page [2].
> > >My question is simple... why is pushpull known NOT to work on occasion?
> I
> > >am going to debug this tomorrow but I thought I would ask this
> > >none-the-less.
> > >Thanks
> > >Lewis
> > >
> > >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
> > >[1]
> > >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
> >
> >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
> > >ningthe./pushpullscript.WhatshouldIdo
> > >?
> > >[2]
> > >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
> > >
> > >
> > >--
> > >*Lewis*
> >
> >
> >
>
>
> --
> *Lewis*
>



-- 

Sent from a Tin Can attached to a String

Re: RADiX filemgr-client script cannot delete product by id or name

[Mattmann, Chris A (3980)]

Can you please remove your FILEMGR_HOME/catalog (or your
RADIX_HOME/data/catalog)
directory, restart file manager, and then try again?

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: MengYing Wang 
Date: Saturday, November 1, 2014 at 1:03 PM
To: "dev@oodt.apache.org" 
Cc: Chris Mattmann 
Subject: Re: RADiX filemgr-client script cannot delete product by id or
name

>Dear Prof. Mattmann,
>
>
>Thank you for your help! Actually, I have set the FILEMGR_URL. Below are
>the variables that I have set:
>
>
>export OODT_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7"
>export 
>CRAWLER_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/crawler"
>export 
>FILEMGR_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr"
>export 
>WORKFLOW_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/workflow"
>export PGE_ROOT="/Users/AngelaWang/Documents/programs/oodt-0.7/pge"
>export WORKFLOW_URL="http://localhost:9200";
>export FILEMGR_URL="http://localhost:9000";
>
>
>
>Best,
>Mengying (Angela) Wang
>
>
>On Sat, Nov 1, 2014 at 10:46 AM, Mattmann, Chris A (3980)
> wrote:
>
>This is an environment variable issue - your FILEMGR_URL is not set.
>Can you check it Angela?
>
>Cheers,
>Chris
>
>
>++
>Chris Mattmann, Ph.D.
>Chief Architect
>Instrument Software and Science Data Systems Section (398)
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 168-519, Mailstop: 168-527
>Email: chris.a.mattm...@nasa.gov
>WWW:  http://sunset.usc.edu/~mattmann/
>++
>Adjunct Associate Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++
>
>
>
>
>
>
>-Original Message-
>From: MengYing Wang 
>Reply-To: "dev@oodt.apache.org" 
>Date: Thursday, October 30, 2014 at 3:02 PM
>To: "dev@oodt.apache.org" 
>Subject: RADiX filemgr-client script cannot delete product by id or name
>
>>Hey everyone,
>>
>>I am trying to delete some product in the OODT File Manager with its
>>product id or name. However, the system raises
>>"java.lang.IllegalArgumentException: URI is not absolute" error. Please
>>help me! Thank you!
>>
>>My commands are:
>>
>>./filemgr-client --op --dbyid --productId
>>2b0e7582-6077-11e4-ac24-8d456d2e5b9b --url
>http://localhost:9000 
>>
>>./filemgr-client -op --deleteProductByName --productName
>>concatenatedOutputFile-2014-10-30T20:56:02.189Z.txt --url
>>http://localhost:9000
>>
>>The logs of the file manager is:
>>..
>>
>>Oct 30, 2014 2:56:21 PM
>>org.springframework.beans.factory.support.AbstractBeanFactory doGetBean
>>
>>FINE: Returning cached instance of singleton bean 'sqlQuery'
>>
>>Oct 30, 2014 2:56:21 PM
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>>
>>INFO: Loading File Manager Configuration Properties from:
>>[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.proper
>>t
>>ies]
>>
>>Oct 30, 2014 2:56:21 PM
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>>
>>INFO: Loading File Manager Configuration Properties from:
>>[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.proper
>>t
>>ies]
>>
>
>
>>ERROR: Failed to delete product
>>'org.apache.oodt.cas.filemgr.structs.Product@5433d17' :
>>java.lang.Exception: java.lang.IllegalArgumentException: URI is not
>>absolute
>>
>>--
>>Best,
>>Mengying (Angela) Wang
>
>
>
>
>
>
>
>
>
>
>-- 
>Best,
>Mengying (Angela) Wang
>
>
>

Re: RADiX filemgr-client script cannot delete product by id or name

[MengYing Wang]

Dear Prof. Mattmann,

Thank you for your help! Actually, I have set the FILEMGR_URL. Below are
the variables that I have set:

export OODT_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7"
export CRAWLER_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/crawler"
export FILEMGR_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr"
export
WORKFLOW_HOME="/Users/AngelaWang/Documents/programs/oodt-0.7/workflow"
export PGE_ROOT="/Users/AngelaWang/Documents/programs/oodt-0.7/pge"
export WORKFLOW_URL="http://localhost:9200";
export FILEMGR_URL="http://localhost:9000";

Best,
Mengying (Angela) Wang

On Sat, Nov 1, 2014 at 10:46 AM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> This is an environment variable issue - your FILEMGR_URL is not set.
> Can you check it Angela?
>
> Cheers,
> Chris
>
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
>
> -Original Message-
> From: MengYing Wang 
> Reply-To: "dev@oodt.apache.org" 
> Date: Thursday, October 30, 2014 at 3:02 PM
> To: "dev@oodt.apache.org" 
> Subject: RADiX filemgr-client script cannot delete product by id or name
>
> >Hey everyone,
> >
> >I am trying to delete some product in the OODT File Manager with its
> >product id or name. However, the system raises
> >"java.lang.IllegalArgumentException: URI is not absolute" error. Please
> >help me! Thank you!
> >
> >My commands are:
> >
> >./filemgr-client --op --dbyid --productId
> >2b0e7582-6077-11e4-ac24-8d456d2e5b9b --url http://localhost:9000
> >
> >./filemgr-client -op --deleteProductByName --productName
> >concatenatedOutputFile-2014-10-30T20:56:02.189Z.txt --url
> >http://localhost:9000
> >
> >The logs of the file manager is:
> >..
> >
> >Oct 30, 2014 2:56:21 PM
> >org.springframework.beans.factory.support.AbstractBeanFactory doGetBean
> >
> >FINE: Returning cached instance of singleton bean 'sqlQuery'
> >
> >Oct 30, 2014 2:56:21 PM
> >org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
> >
> >INFO: Loading File Manager Configuration Properties from:
> >[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.propert
> >ies]
> >
> >Oct 30, 2014 2:56:21 PM
> >org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
> >
> >INFO: Loading File Manager Configuration Properties from:
> >[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.propert
> >ies]
> >
> >ERROR: Failed to delete product
> >'org.apache.oodt.cas.filemgr.structs.Product@5433d17' :
> >java.lang.Exception: java.lang.IllegalArgumentException: URI is not
> >absolute
> >
> >--
> >Best,
> >Mengying (Angela) Wang
>
>


-- 
Best,
Mengying (Angela) Wang

Re: FileManager Installation Problem

[Mattmann, Chris A (3980)]

Hi Zichen,

Sorry for any issues you are having here. I think to understand
what¹s going on, you need to understand what RADIX is doing:

1. RADIX is constructing a ³typical OODT project² that combines
filemgr, workflow, resmgr, etc. into one build directory. The
resultant project from RADIX isn¹t itself an OODT deployment.

2. When you ³build² with Maven your resultant project from #1
you are actually constructing the OODT ³tar ball² deployment
dms-0.1-dist.tar.gz. _That_ is your OODT deployment tar ball
that you will run ODOT out of in 3.

3. take the tar ball, unpack it (I usually do it to /usr/local/oodt,
but You can do it to e.g., $HOME/oodt-deploy too, etc.). The stuff
inside is your $OODT_HOME, etc.

You need to make sure some of the required RADIX env vars are set.
Try that out and let me know what you think.

Thanks!

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Zichen Nie 
Date: Sunday, October 26, 2014 at 11:28 PM
To: Chris Mattmann 
Subject: FileManager Installation Problem

>Dear Professor,
>
>
>I tried to run the instruction on page
>https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>However, despite the difference that the Target name is
>oodt-filemgr-0.1-bin.tar.gz
>rather than cas-filemgr-0.4-SNAPSHOT-dist.tar.gz, which means my file
>manager folder is filemgr not cas-filemgr, there are some other commands
>that I am not able to run correctly.
> 
>In the filemgr/bin folder, I can start the file manager by command
>./filemgr start, but when I enter ./filemgr-client --help and
>./query_tool, the terminal give me completely different from the
>instruction. For command ./filemgr-client, it gives me a very
> long log-like record and followed by short/long parameter and its
>description table. For command ./query_tool, it gives me two error, one
>is cannot find path, the other is command not found. I am really
>confused. When I open the query_tool file, I saw:
>if [ -z "$FILEMGR_HOME" ]; then
>  FILEMGR_HOME="$OODT_HOME"/components/filemgr
>  export FILEMGR_HOME
>But I do not have components folder in OODT directory.
>I also found 
>"$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>  -Djava.endorsed.dirs=../lib \
>  org.apache.oodt.cas.filemgr.tools.QueryTool "$@"
>
>This is where terminal said command not found. Since my target has no
>"cas" in it, its oodt-filemgr-0.1-bin.tar.gz, I guess this is the reason
>why it cannot run. Anyway, I am not sure about this since this file is
>extracted from  oodt-filemgr-0.1-bin.tar.gz.
> I am so confused.
>
>
>Is there any suggestions that could help me with my problem? Thanks in
>advance.
>
>
>Best,
>Zichen
>

FW: FileManager Installation Problem

[Mattmann, Chris A (3980)]

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Zichen Nie 
Date: Monday, October 27, 2014 at 12:17 PM
To: Chris Mattmann 
Subject: Re: FileManager Installation Problem

>Dear Professor,
>
>
>Sorry to bother you again. I checked your Github code modification and
>now my ./query_tool works well again. I was at class when you said there
>would be some changes in OODT, but I did not realize this is exactly the
>problem. 
>
>
>Thanks again !
>
>
>Best,
>Zichen
>
>
>2014-10-26 23:28 GMT-07:00 Zichen Nie :
>
>Dear Professor,
>
>
>I tried to run the instruction on page
>https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>However, despite the difference that the Target name is
>oodt-filemgr-0.1-bin.tar.gz
>rather than cas-filemgr-0.4-SNAPSHOT-dist.tar.gz, which means my file
>manager folder is filemgr not cas-filemgr, there are some other commands
>that I am not able to run correctly.
> 
>In the filemgr/bin folder, I can start the file manager by command
>./filemgr start, but when I enter ./filemgr-client --help and
>./query_tool, the terminal give me completely different from the
>instruction. For command ./filemgr-client, it gives me a very
> long log-like record and followed by short/long parameter and its
>description table. For command ./query_tool, it gives me two error, one
>is cannot find path, the other is command not found. I am really
>confused. When I open the query_tool file, I saw:
>if [ -z "$FILEMGR_HOME" ]; then
>  FILEMGR_HOME="$OODT_HOME"/components/filemgr
>  export FILEMGR_HOME
>But I do not have components folder in OODT directory.
>I also found 
>"$_RUNJAVA" $JAVA_OPTS $OODT_OPTS \
>  -Djava.endorsed.dirs=../lib \
>  org.apache.oodt.cas.filemgr.tools.QueryTool "$@"
>
>This is where terminal said command not found. Since my target has no
>"cas" in it, its oodt-filemgr-0.1-bin.tar.gz, I guess this is the reason
>why it cannot run. Anyway, I am not sure about this since this file is
>extracted from  oodt-filemgr-0.1-bin.tar.gz.
> I am so confused.
>
>
>Is there any suggestions that could help me with my problem? Thanks in
>advance.
>
>
>Best,
>Zichen
>
>
>
>
>
>

Re: Getting MojoException on compiling resource manager from maven 3.2

[Mattmann, Chris A (3980)]

Thanks Sunil,

can you suggest what tests are failing? What does
resource/target/surefire-reports/*.txt
have in them?

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Sunil Iyer Kolar Suresh Kumar 
Date: Saturday, October 25, 2014 at 5:39 PM
To: Chris Mattmann 
Subject: Getting MojoException on compiling resource manager from maven 3.2

>Dear Professor,
>
>
>I have finished installing File manager and workflow manager in OODT but
>I am badly stuck in installing Resource Manager.
>I am constantly getting this error :
>
>I am not able to resolve this error from morning. Could you please give a
>hint what might be going wrong?
> 
>
>[INFO] 
>
>[INFO] BUILD FAILURE
>[INFO] 
>
>[INFO] Total time: 13.475 s
>[INFO] Finished at: 2014-10-25T17:31:53-07:00
>[INFO] Final Memory: 10M/29M
>[INFO] 
>
>[ERROR] Failed to execute goal
>org.apache.maven.plugins:maven-surefire-plugin:2.4:test (default-test) on
>project cas-resource: There are test failures.
>[ERROR] 
>[ERROR] Please refer to /usr/local/src/resource/target/surefire-reports
>for the individual test results.
>[ERROR] -> [Help 1]
>[ERROR] 
>[ERROR] To see the full stack trace of the errors, re-run Maven with the
>-e switch.
>[ERROR] Re-run Maven using the -X switch to enable full debug logging.
>[ERROR] 
>[ERROR] For more information about the errors and possible solutions,
>please read the following articles:
>[ERROR] [Help 1] 
>http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>
>
>
>Thanks,
>
>Sunil iyer
>
>

Re: Passing WorkflowInstId to extern extractor

[Mattmann, Chris A (3980)]

Hi Val,

Sorry for the delay in answering this. The key way to make sure that
the workflow id and other properties like JobId are passed to the met
file would be to use the PcsMetListFileWriter and add it as an additional
writer for your files. All met writers that match the same files using
the regExp tag will be called. An example of using the PcsMetFileListWriter
is here in DRAT:

https://github.com/chrismattmann/drat/blob/master/pge/src/main/resources/co
nfig/PgeConfig_Rat.xml

https://github.com/chrismattmann/drat/blob/master/pge/src/main/resources/co
nfig/metout/rat_log_metout.xml


Here is some info on the philosophy behind the flow of metadata in
CAS-PGE that I recently penned on the wiki:

https://cwiki.apache.org/confluence/display/OODT/Understanding+CAS-PGE+Meta
data+Precendence

https://cwiki.apache.org/confluence/display/OODT/Understanding+the+flow+of+
Metadata+during+PGE+based+Processing


Let me know if that helps!

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: , Valerie 
Reply-To: "dev@oodt.apache.org" 
Date: Monday, October 27, 2014 at 12:57 PM
To: "dev@oodt.apache.org" 
Subject: Passing WorkflowInstId to extern extractor

>Hello Everyone,
>
>I would like to query the file manager for all files ingested by a
>particular WorkflowInstId, or JobId, so I am trying to figure out how to
>get the WorkflowInstId or JobId written to each product's met file.  My
>met writer is an ExternExtractorMetWriter  so I would like to pass the
>WorkflowInstId or JobId as a parameter to the extractor. I tried
>specifying the parameter [WorkflowInstId] (also tried JobId) to the
>extern extractor config file like shown below, but this did not work.
>Does anyone have any ideas on how I could do this?
>
>http://oodt.jpl.nasa.gov/1.0/cas";>
>  
>envReplace="true">[OODT_HOME]/extensions/extractors/level0ExternMetExtract
>or.pl
>
>   
>   [WorkflowInstId]
>
>   
>
>
>Thanks!!
>Val
>
>
>
>
>Valerie A. Mallder
>
>New Horizons Deputy Mission System Engineer
>The Johns Hopkins University/Applied Physics Laboratory
>11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
>240-228-7846 (Office) 410-504-2233 (Blackberry)
>

Re: Why is pushpull not always working?

[Lewis John Mcgibbney]

Thanks Chris. OK doke. i am going to debug this today and find out what is
happening.
Thanks for heads up.
Lewis

On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann 
wrote:

> Thanks Lewis - in my mind it boils down somewhat to
> the protocols that are implemented and the way they
> behave with FTP servers and so forth. Ultimately, we
> should be using the ones that are in PushPull plugins,
> but they are virally licensed and I haven¹t found a
> better alternative.
>
> Another @bfoster update here would be great, Brian?
>
> Cheers,
> Chris
>
>
> 
> Chris Mattmann
> chris.mattm...@gmail.com
>
>
>
>
> -Original Message-
> From: Lewis John Mcgibbney 
> Reply-To: 
> Date: Saturday, November 1, 2014 at 12:56 AM
> To: "dev@oodt.apache.org" 
> Subject: Why is pushpull not always working?
>
> >Hi Folks,
> >I really couldn't think of a better title for this thread so I apologize
> >in
> >advance.
> >I recently set up pushpull with FTP to crawl remote ftp site [0] which is
> >publicly available.
> >I am getting the following
> >
> >INFO: Launching Daemons . . .
> >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> >registerRMIServer
> >INFO: Created Daemon ID = 90121 on RMI registry port 9012
> >Nov 01, 2014 12:46:01 AM
> >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
> >launchDaemons
> >INFO: Creating Daemon with ID = 1
> >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:48:00 PDT 2014
> >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:51:00 PDT 2014
> >Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:54:00 PDT 2014
> >Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 90121 was given permission to run
> >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
> >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> >until Sat Nov 01 00:57:00 PDT 2014
> >
> >No errors... no pulling... no nothing!
> >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
> >plugins page [2].
> >My question is simple... why is pushpull known NOT to work on occasion? I
> >am going to debug this tomorrow but I thought I would ask this
> >none-the-less.
> >Thanks
> >Lewis
> >
> >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
> >[1]
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
> >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
> >ningthe./pushpullscript.WhatshouldIdo
> >?
> >[2]
> >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
> >
> >
> >--
> >*Lewis*
>
>
>


-- 
*Lewis*

Re: RADiX OPSUI Webapp Page Expired Error

[Lewis John Mcgibbney]

ACK. I get the same.
I'll monitor this issue and try to chip in a fix as well

On Sat, Nov 1, 2014 at 10:27 AM, Chris Mattmann 
wrote:

> Guys also note I opened an issue for this a while
> back:
>
> https://issues.apache.org/jira/browse/OODT-751
>
>
> 
> Chris Mattmann
> chris.mattm...@gmail.com
>
>
>
>
> -Original Message-
> From: "Verma, Rishi (398M)" 
> Reply-To: 
> Date: Friday, October 31, 2014 at 2:20 PM
> To: MengYing Wang 
> Cc: "dev@oodt.apache.org" , Chris Mattmann
> 
> Subject: Re: RADiX OPSUI Webapp Page Expired Error
>
> >Hi Mengying,
> >
> >The ³fix code² is still there [1]. However, it seems this particular
> >solution is not working anymore in 0.7.
> >
> >Could you try removing the fix and observing the results? This was a
> >temporary fix, and a more fundamental one may be necessary. Check out the
> >opsui code [2] to play around with potential solutions. Will look into it
> >as well.
> >
> >Thanks much,
> >Rishi
> >
> >[1]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apa
> >che/oodt/pcs/opsui/OPSUIWebPage.java
> >[2]
> >
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apa
> >che/oodt/pcs/opsui/
> >
> >On Oct 30, 2014, at 3:23 PM, MengYing Wang
> >mailto:mengyingwa...@gmail.com>> wrote:
> >
> >Hey Rishi,
> >
> >I see you have fixed this error in the OODT 0.6
> >https://issues.apache.org/jira/browse/OODT-596. However, currently I am
> >using the OODT 0.7 with RADiX, and keep receiving the error. Could you
> >tell me what should I do to fix it, or is there any patch file that I
> >could apply? Many thanks!
> >
> >--
> >Best,
> >Mengying (Angela) Wang
> >
> >---
> >Rishi Verma
> >NASA Jet Propulsion Laboratory
> >California Institute of Technology
> >4800 Oak Grove Drive, M/S 158-248
> >Pasadena, CA 91109
> >Tel: 1-818-393-5826
> >
>
>
>


-- 
*Lewis*

Re: re： Question about OODT file manager

[Mattmann, Chris A (3980)]

Dear Luke, just confirming, we solved this in class right? It had
to do with the batch stub not being turned on.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Luke 
Date: Tuesday, October 28, 2014 at 12:52 PM
To: Chris Mattmann , "dev@oodt.apache.org"

Cc: Chris Mattmann , "zhouj...@usc.edu"
, "xiaoy...@usc.edu" , 'Zichuan Wang'

Subject: RE: re： Question about OODT file manager

>Dear Professor Mattamnn,
>Thanks a lot Professor Mattmann for the kind help, it is appreciated,
>sorry for getting back to you with my appreciation, I have been
>conducting tests with OODT based on your advice, but unfortunately I am
>having another problem
>
>I am following the steps
>(https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example
>) to get a sense of how to get workflow to work.
>The problem is that the File-Concatenator-PGE (by running the wmgr-client
>command-line) does not seems to be invoked or executed, but I am seeing
>the tasks are getting stacked up in the workflow manager with status
>either "RSUBMIT" or "QUEUED", but they are not getting executed, PFA:
>workflow_monitor.jpg, please note, by default the workflow min pool size
>is 6; so here comes another problem, i have 6 submitted tasks with status
>RSUBMIT, but any new incoming tasks will be forwarded to the waiting
>QUEUE with status "QUEUED"...please refer to the workflow_monitor.jpg for
>details, where I have 3 QUEUED workflow task and 6 RSUMBITE tasks.
>
>Question 1): not sure why the workflow is not being executed, and hanging
>at the state of "RSUBMIT", after enabling the log level, I am seeing the
>following entry in the log, not sure if this has anything to do with the
>"hanging" problem where workflow is not getting executed and hanging at
>state of "RSUBMIT".
>   Oct 28, 2014 3:35:07 AM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>safeCheckJobComplete
>   WARNING: Exception checking completion status for job:
>[2014-10-28T01:59:32.813-07:00]: Messsage: java.lang.Exception:
>java.lang.NullPointerException
>
>Question 2): I think currently on my side any new incoming workflow task
>I am sending with the following command is being directed to the waiting
>"QUEUE" because of the min pool size (i.e. 6) (I can increase this to a
>larger number though),
>   ./wmgr-client --url http://localhost:9200 --operation 
> --sendEvent
>--eventName fileconcatenator-pge --metaData --key RunID testNumber1
>   If possible, I would like to please know if there is a way we can purge
>the queue and get rid of those workflow tasks either in "RSUMBIT" and
>"QUEUED" I have already sent, please kindly help.
>
>Very sorry for troubling you with this, to be honest I find OODT a bit
>challenging to grasp within a short time frame, probably because there is
>no book like OODT in action like Solr and what I am doing is just
>trial and error blended with guess, but I don’t want to make a blind
>guess, it will be appreciated if you can please also shed some lights on
>where I can get more information logging or other way where I can
>troubleshoot. I think it might be worth tracking what is happening when
>workflow reach the status "RSUBMIT" and how to get a specific logging
>info specific to it...
>
>Again your advice and kind help will be appreciated usual.
>
>
>Thanks
>Luke
>
>> -Original Message-
>> From: Mattmann, Chris A (3980) [mailto:chris.a.mattm...@jpl.nasa.gov]
>> Sent: 2014年10月26日 22:18
>> To: Luke; 'Zichuan Wang'
>> Cc: 'Christian Alan Mattmann'; zhouj...@usc.edu; xiaoy...@usc.edu;
>> dev@oodt.apache.org
>> Subject: Re: re： Question about OODT file manager
>> 
>> Hi Luke,
>> 
>> Thanks and sorry it’s taken me a while to reply. Here are some details
>>below:
>> 
>> 
>> -Original Message-
>> From: Luke 
>> Date: Sunday, October 26, 2014 at 6:19 PM
>> To: Chris Mattmann , 'Zichuan Wang'
>> 
>> Cc: Chris Mattmann , "zhouj...@usc.edu"
>> , "xiaoy...@usc.edu" ,
>> "dev@oodt.apache.org" 
>> Subject: RE: re： Question about OODT file manager
>> 
>> >Hi Professor Mattmann and OODT DEV,
>> >
>> >Sorry to trouble you with this email, our team has been struggling in
>> >the oodt to send json files to solr.
>> >One of the difficulties is still getting OODT workflow to call the
>> >poster.py in etllib.
>> 
>> Sorry that you’re having difficulty let me try and help.
>> 
>> >
>> >I am not sure if my understanding 

RADiX CAS-Crawler cannot trigger workflow event

[MengYing Wang]

Hi Everyone,

I am trying to trigger some workflow evert using the crawler. However, it
raises a "no protocol" error:

INFO: Performing action (id = TriggerPostIngestWorkflow : description =
Triggers workflow event with the name [ProductType]Ingest)
Oct 30, 2014 4:41:45 PM org.apache.oodt.cas.crawl.ProductCrawler
performProductCrawlerActions
WARNING: Failed to perform crawler action : Failed to update workflow
manager : no protocol: null
org.apache.oodt.cas.crawl.structs.exceptions.CrawlerActionException: Failed
to update workflow manager : no protocol: null
at
org.apache.oodt.cas.crawl.action.WorkflowMgrStatusUpdate.performAction(WorkflowMgrStatusUpdate.java:59)
at
org.apache.oodt.cas.crawl.ProductCrawler.performProductCrawlerActions(ProductCrawler.java:357)
at
org.apache.oodt.cas.crawl.ProductCrawler.performPostIngestOnSuccessActions(ProductCrawler.java:331)
at
org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:194)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:108)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:75)
at
org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)

It seems some thing wrong with my crawler. Should I do some configuration
to my crawler? And how? Thank you for your time and help!

-- 
Best,
Mengying (Angela) Wang

Re: RADiX filemgr-client script cannot delete product by id or name

[Mattmann, Chris A (3980)]

This is an environment variable issue - your FILEMGR_URL is not set.
Can you check it Angela?

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: MengYing Wang 
Reply-To: "dev@oodt.apache.org" 
Date: Thursday, October 30, 2014 at 3:02 PM
To: "dev@oodt.apache.org" 
Subject: RADiX filemgr-client script cannot delete product by id or name

>Hey everyone,
>
>I am trying to delete some product in the OODT File Manager with its
>product id or name. However, the system raises
>"java.lang.IllegalArgumentException: URI is not absolute" error. Please
>help me! Thank you!
>
>My commands are:
>
>./filemgr-client --op --dbyid --productId
>2b0e7582-6077-11e4-ac24-8d456d2e5b9b --url http://localhost:9000
>
>./filemgr-client -op --deleteProductByName --productName
>concatenatedOutputFile-2014-10-30T20:56:02.189Z.txt --url
>http://localhost:9000
>
>The logs of the file manager is:
>..
>
>Oct 30, 2014 2:56:21 PM
>org.springframework.beans.factory.support.AbstractBeanFactory doGetBean
>
>FINE: Returning cached instance of singleton bean 'sqlQuery'
>
>Oct 30, 2014 2:56:21 PM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>
>INFO: Loading File Manager Configuration Properties from:
>[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.propert
>ies]
>
>Oct 30, 2014 2:56:21 PM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>
>INFO: Loading File Manager Configuration Properties from:
>[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.propert
>ies]
>
>ERROR: Failed to delete product
>'org.apache.oodt.cas.filemgr.structs.Product@5433d17' :
>java.lang.Exception: java.lang.IllegalArgumentException: URI is not
>absolute
>
>-- 
>Best,
>Mengying (Angela) Wang

Re: Correction of Instruction On CAS-PGE Learn by Example

[Mattmann, Chris A (3980)]

Thank you Zichen!

These are great updates. Can you register on the OODT Apache Wiki and
then tell me your username? I¹ll then grant you permissions to update
the wiki page (or someone else here can). Then you can make these
updates on the wiki page which will help others when they encounter
this.

Thanks!

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Zichen Nie 
Date: Friday, October 31, 2014 at 10:41 AM
To: "dev@oodt.apache.org" , Chris Mattmann

Subject: Correction of Instruction On CAS-PGE Learn by Example

>Hi, I would like to add more instructions on how to compile
>FileConcatenatorPGE project on page
>https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example
>
>
>In the current instruction, step 4,Build and deploy FileConcatenatorPGE
>seems really simple,
>cd /usr/local/src/fileconcatenator-pge
>mvn package
>mv target/fileconcatenator-pge-*.jar $WORKFLOW_HOME/lib
>
>
>however, in practice, the maven fails to build the project. Why? This is
>because in pom.xml file, it specifies oodt to be 0.3-SNAPSHOT.
>
>
>For those who are using oodt 0.7, there are two ways to fix the problem.
>
>
>One way is really simple, change the oodt version to 0.3 in pom.xml, and
>mvn install again. This will work perfectly compatible with oodt 0.7.
> 
>The other solution is to change oodt version to 0.7 and follow the steps
>below:
>1. Add a PgeTaskMetadataKeys.java file in the fileconcatenator home:
>$FILECONCATENATOR_HOME/src/main/java/org/apache/oodt/pge/examples/fileconc
>atenator.
>The PgeTaskMetadataKey.java is one of the import source for
>FileConcatenatorPGETask.java file in the same folder. Without the import
>source, compiler would fail to compile the
>FileConcatenatorPGETask.java.
>
>
>2. Open the FileConcatenatorPGETask.java, add a "thows Exception" for the
>overriden function
>updataStatus(String status).
>
>
>3. Save the above changes and use mvn package or mvn install to compile
>again, this time the build will be successful. Cheers!
>
>
>By the way, the PgeTaskMetadataKeys.java can be found at:
>http://grepcode.com/file/repo1.maven.org/maven2/org.apache.oodt/cas-pge/0.
>3/org/apache/oodt/cas/pge/metadata/PgeTaskMetadataKeys.java
>

Correction of Instruction On CAS-PGE Learn by Example

[Zichen Nie]

Hi, I would like to add more instructions on how to compile
*FileConcatenatorPGE
project* on page
*https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example
*

In the current instruction, step 4,*Build and deploy FileConcatenatorPGE*
seems really simple,
*cd /usr/local/src/fileconcatenator-pge*
*mvn package*
*mv target/fileconcatenator-pge-*.jar $WORKFLOW_HOME/lib*

however, in practice, the maven fails to build the project. Why? This is
because in pom.xml file, it specifies oodt to be 0.3-SNAPSHOT.

For those who are using oodt 0.7, there are two ways to fix the problem.

One way is really simple, change the oodt version to 0.3 in pom.xml, and *mvn
install* again. This will work perfectly compatible with oodt 0.7.

The other solution is to change oodt version to 0.7 and follow the steps
below:
1. Add a *PgeTaskMetadataKeys.java* file in the fileconcatenator home:
*$FILECONCATENATOR_HOME/src/main/java/org/apache/oodt/pge/examples/fileconcatenator.*
The *PgeTaskMetadataKey.java *is one of the import source for
*FileConcatenatorPGETask.java* file in the same folder. Without the import
source, compiler would fail to compile the *FileConcatenatorPGETask.java*.

2. Open the *FileConcatenatorPGETask.java*, add a "*thows Exception*" for
the overriden function *updataStatus(String status).*

3. Save the above changes and use *mvn package* or *mvn install* to compile
again, this time the build will be successful. Cheers!

By the way, the *PgeTaskMetadataKeys.java* can be found at:
*http://grepcode.com/file/repo1.maven.org/maven2/org.apache.oodt/cas-pge/0.3/org/apache/oodt/cas/pge/metadata/PgeTaskMetadataKeys.java
*

Re: RADiX CAS-Crawler cannot trigger workflow event

[Mattmann, Chris A (3980)]

Hi Angela,

This is an issue with you not having your WORKFLOW_URL environment
variable set. Can you make sure it¹s set before you start the workflow
manager and RADIX? Please make sure it¹s getting passed down..

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: MengYing Wang 
Date: Thursday, October 30, 2014 at 9:30 PM
To: "dev@oodt.apache.org" 
Cc: Chris Mattmann 
Subject: RADiX CAS-Crawler cannot trigger workflow event

>Hi Everyone, 
>
>
>I am trying to trigger some workflow evert using the crawler. However, it
>raises a "no protocol" error:
>
>
>INFO: Performing action (id = TriggerPostIngestWorkflow : description =
>Triggers workflow event with the name [ProductType]Ingest)
>Oct 30, 2014 4:41:45 PM org.apache.oodt.cas.crawl.ProductCrawler
>performProductCrawlerActions
>WARNING: Failed to perform crawler action : Failed to update workflow
>manager : no protocol: null
>org.apache.oodt.cas.crawl.structs.exceptions.CrawlerActionException:
>Failed to update workflow manager : no protocol: null
>at 
>org.apache.oodt.cas.crawl.action.WorkflowMgrStatusUpdate.performAction(Wor
>kflowMgrStatusUpdate.java:59)
>at 
>org.apache.oodt.cas.crawl.ProductCrawler.performProductCrawlerActions(Prod
>uctCrawler.java:357)
>at 
>org.apache.oodt.cas.crawl.ProductCrawler.performPostIngestOnSuccessActions
>(ProductCrawler.java:331)
>at 
>org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:19
>4)
>at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:108)
>at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:75)
>at 
>org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(Craw
>lerLauncherCliAction.java:58)
>at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>at 
>org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
>
>
>It seems some thing wrong with my crawler. Should I do some configuration
>to my crawler? And how? Thank you for your time and help!
>
>
>-- 
>Best,
>Mengying (Angela) Wang
>
>
>
>

null

[Yawei Huang]

RADiX OPSUI Webapp Page Expired Error

[MengYing Wang]

Hey Rishi,

I see you have fixed this error in the OODT 0.6
https://issues.apache.org/jira/browse/OODT-596. However, currently I am
using the OODT 0.7 with RADiX, and keep receiving the error. Could you tell
me what should I do to fix it, or is there any patch file that I could
apply? Many thanks!

-- 
Best,
Mengying (Angela) Wang

Re: 回复： Question about OODT file manager

[Mattmann, Chris A (3980)]

Hi Zichuan,

Please see:

http://oodt.apache.org/components/maven/crawler/user/


And also see the UpdateWorkflowStatusToIngest crawler action.

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Zichuan Wang 
Date: Tuesday, October 28, 2014 at 9:37 PM
To: Luke 
Cc: Chris Mattmann , "dev@oodt.apache.org"
, Chris Mattmann ,
"zhouj...@usc.edu" , "xiaoy...@usc.edu"

Subject: 回复： Question about OODT file manager

>Dear Professor, 
>
>
>We are stuck in OODT. The most critical problem we have now is
>
>
>“How to make crawler work with workflow”?
>
>
>-- 
>Zichuan Wang
>University of Southern California, Department of Computer Science
>
>
>
>在 2014年10月28日 星期二，下午12:52，Luke 写道：
>
>Dear Professor Mattamnn,
>Thanks a lot Professor Mattmann for the kind help, it is appreciated,
>sorry for getting back to you with my appreciation, I have been
>conducting tests with OODT based on your advice, but unfortunately I am
>having another problem
>
>
>I am following the steps
>(https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example
>) to get a sense of how to get workflow to work.
>The problem is that the File-Concatenator-PGE (by running the wmgr-client
>command-line) does not seems to be invoked or executed, but I am seeing
>the tasks are getting stacked up in the workflow manager with status
>either "RSUBMIT" or "QUEUED", but they
> are not getting executed, PFA: workflow_monitor.jpg, please note, by
>default the workflow min pool size is 6; so here comes another problem, i
>have 6 submitted tasks with status RSUBMIT, but any new incoming tasks
>will be forwarded to the waiting QUEUE with
> status "QUEUED"...please refer to the workflow_monitor.jpg for details,
>where I have 3 QUEUED workflow task and 6 RSUMBITE tasks.
>
>
>
>Question 1): not sure why the workflow is not being executed, and hanging
>at the state of "RSUBMIT", after enabling the log level, I am seeing the
>following entry in the log, not sure if this has anything to do with the
>"hanging" problem where workflow
> is not getting executed and hanging at state of "RSUBMIT".
>Oct 28, 2014 3:35:07 AM
>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>safeCheckJobComplete
>WARNING: Exception checking completion status for job:
>[2014-10-28T01:59:32.813-07:00]: Messsage: java.lang.Exception:
>java.lang.NullPointerException
>
>
>Question 2): I think currently on my side any new incoming workflow task
>I am sending with the following command is being directed to the waiting
>"QUEUE" because of the min pool size (i.e. 6) (I can increase this to a
>larger number though),
>
>./wmgr-client --url http://localhost:9200 --operation --sendEvent
>--eventName fileconcatenator-pge --metaData --key RunID testNumber1
>If possible, I would like to please know if there is a way we can purge
>the queue and get rid of those workflow tasks either in "RSUMBIT" and
>"QUEUED" I have already sent, please kindly help.
>
>
>Very sorry for troubling you with this, to be honest I find OODT a bit
>challenging to grasp within a short time frame, probably because there is
>no book like OODT in action like Solr and what I am doing is just
>trial and error blended with guess, but
> I don’t want to make a blind guess, it will be appreciated if you can
>please also shed some lights on where I can get more information logging
>or other way where I can troubleshoot. I think it might be worth tracking
>what is happening when workflow reach the
> status "RSUBMIT" and how to get a specific logging info specific to it...
>
>
>Again your advice and kind help will be appreciated usual.
>
>
>
>
>Thanks
>Luke
>
>
>
>-Original Message-
>From: Mattmann, Chris A (3980) [mailto:chris.a.mattm...@jpl.nasa.gov]
>Sent: 2014年10月26日 22:18
>To: Luke; 'Zichuan Wang'
>Cc: 'Christian Alan Mattmann'; zhouj...@usc.edu;
>xiaoy...@usc.edu;
>dev@oodt.apache.org
>Subject: Re: re： Question about OODT file manager
>
>
>Hi Luke,
>
>
>Thanks and sorry it’s taken me a while to reply. Here are some details
>below:
>
>
>
>
>-Original Message-
>From: Luke 
>Date: Sunday, October 26, 2014 at 6:19 PM
>To: Chris Mattmann , 'Zichuan Wang'
>
>Cc: Chris Mattmann , "zhouj...@usc.edu"
>, "xiaoy...@usc.edu" ,
>"dev@oodt.apache.org" 
>Subject: RE: re： Question about OODT file manager
>
>
>
>Hi Professor Mattmann and OODT DEV,
>
>
>Sorry to trouble you with this email, our team has been struggling in
>the oodt to sen

PGE part of RADIX?

[Mattmann, Chris A (3980)]

Hey Guys (specifically Rishi and Shakeh):

See this issue:

https://issues.apache.org/jira/browse/OODT-767


I thought we took care of this in XDATA over the summer?

Cheers,
Chris



++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Re: [File Manager] Users could ingest the same file multiple times, but error occurs if they delete the file more than once

[Mattmann, Chris A (3980)]

Thanks Angela. I think that¹s the intended behavior. Since you
are deleting the product by ID (which is unique), deleting the
same file (with the same name) more than 1x fails (even though they
have different product IDs). Does that make sense?

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Mengying Wang 
Reply-To: "dev@oodt.apache.org" 
Date: Tuesday, October 28, 2014 at 10:11 AM
To: "dev@oodt.apache.org" 
Subject: [File Manager] Users could ingest the same file multiple times,
but error occurs if they delete the file more than once

>Hi Everyone,
>
>Users could use the filemgr-client script to ingest the same file multiple
>times, which leads to several records in the query_tool result. However,
>if
>you delete the same file more than once, then error occurs. Below is the
>scenario:
>
>Step 1: Ingest the test.txt file twice.
>$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>GenericFile"
>Oct 28, 2014 12:23:07 AM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>INFO: Query returned 2 results
>/Users/AngelaWang/files/test.txt,text/plain,text,plain,Flat,9caff30e-5ae3-
>11e4-ada3-4d034e11a919,GenericFile,2014-10-23T11:37:16.085-07:00,test.txt,
>test.txt
>/Users/AngelaWang/files/test.txt,text/plain,text,plain,Flat,77e80905-476d-
>11e4-83cb-fd04a1088b13,GenericFile,2014-09-28T17:13:41.669-07:00,test.txt,
>test.txt
>
>Step 2: Succeed in deleting the test.txt file for the first time
>
>$ ./filemgr-client --op --dbyid --productId
>9caff30e-5ae3-11e4-ada3-4d034e11a919 --url http://localhost:9000
>
>Oct 28, 2014 12:28:25 AM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>
>INFO: Loading File Manager Configuration Properties from:
>[../etc/filemgr.properties]
>
>Oct 28, 2014 12:28:25 AM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>
>INFO: Loading File Manager Configuration Properties from:
>[../etc/filemgr.properties]
>
>Successfully deleted product 'test.txt'
>
>Step 3: Failed in deleting the test.txt file for the second time.
>$ ./filemgr-client --op --dbyid --productId
>77e80905-476d-11e4-83cb-fd04a1088b13 --url http://localhost:9000
>Oct 28, 2014 12:31:50 AM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>INFO: Loading File Manager Configuration Properties from:
>[../etc/filemgr.properties]
>Oct 28, 2014 12:31:50 AM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>INFO: Loading File Manager Configuration Properties from:
>[../etc/filemgr.properties]
>ERROR: Failed to delete product
>'org.apache.oodt.cas.filemgr.structs.Product@2226a790' : Failed to delete
>file 'file:/Users/AngelaWang/files/test.txt/test.txt'
>
>You have to manually create the test.txt file under the archive directory,
>then you could successfully delete the test.txt file once again.
>
>Thanks.
>Mengying (Angela) Wang

[Problem Solved] Could not mvn install OODT from the trunk

[MengYing Wang]

Hey,

After the Jenkins build is back to stable : oodt-trunk #1553, I have tried
again. And now, it works. Many thanks.

Best,
Mengying (Angela) Wang

On Wed, Oct 29, 2014 at 4:31 PM, MengYing Wang 
wrote:

> Hey,
>
> $ mvn -version
>
> Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700)
>
> Java version: 1.7.0_51
>
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre
>
> Default locale: en_US, platform encoding: UTF-8
>
> OS name: "mac os x" version: "10.9.2" arch: "x86_64" Family: "mac"
>
>
> Yes. The error is repeatable on my pc. I have tried several times, all
> failed with the same error.
>
>
> Best,
>
> Mengying (Angela) Wang
>
> On Wed, Oct 29, 2014 at 4:06 PM, Michael Starch 
> wrote:
>
>> MengYing,
>>
>> I cannot duplicate it on Mac OS X 10.9.5, Java 7 nor CentOS with Java 7.
>>
>> Is the error repeatable on your end?
>>
>> Michael
>> On Oct 29, 2014 4:01 PM, "MengYing Wang"  wrote:
>>
>> > Hi Michael,
>> >
>> > $ java -version
>> > java version "1.7.0_51"
>> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>> > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode).
>> >
>> > Thanks.
>> >
>> > Best,
>> > Mengying Wang
>> >
>> > On Wed, Oct 29, 2014 at 6:52 AM, Michael Starch 
>> > wrote:
>> >
>> > > Mengying,
>> > >
>> > > I just tested this on Mac 1.9.5 using Java 7 and had no issues.Can
>> > you
>> > > tell me your java version?  In the meantime I will test with Java 6
>> as I
>> > > suspect that this might be an issue.
>> > >
>> > > --Michael
>> > > Hi Everyone,
>> > >
>> > > This is my command sequence and some log:
>> > >
>> > > $svn co http://svn.apache.org/repos/asf/oodt/trunk/ oodt_trunk
>> > >
>> > > $cd oodt_trunk/
>> > >
>> > > $mvn clean install
>> > >
>> > > ..
>> > >
>> > > ---
>> > >
>> > >  T E S T S
>> > >
>> > > ---
>> > >
>> > > Running org.apache.oodt.cas.resource.mux.TestQueueMuxMonitor
>> > >
>> > > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.277
>> sec
>> > > <<< FAILURE!
>> > >
>> > > Running org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository
>> > >
>> > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.182
>> sec
>> > >
>> > > Running
>> org.apache.oodt.cas.resource.monitor.TestGangliaResourceMonitor
>> > >
>> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.41
>> sec
>> > >
>> > > Running org.apache.oodt.cas.resource.cli.TestResourceCli
>> > >
>> > > Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.848
>> > sec
>> > >
>> > > Running org.apache.oodt.cas.resource.jobqueue.TestJobStack
>> > >
>> > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063
>> sec
>> > >
>> > > Running org.apache.oodt.cas.resource.system.TestXmlRpcResourceManager
>> > >
>> > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.346
>> sec
>> > >
>> > > Running org.apache.oodt.cas.resource.monitor.TestGangliaXMLParser
>> > >
>> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.327
>> sec
>> > >
>> > > Running org.apache.oodt.cas.resource.mux.TestQueueMuxBatchmgr
>> > >
>> > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057
>> sec
>> > >
>> > > Running org.apache.oodt.cas.resource.monitor.TestAssignmentMonitor
>> > >
>> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.248
>> sec
>> > >
>> > > Running org.apache.oodt.cas.resource.util.TestUlimit
>> > >
>> > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046
>> sec
>> > >
>> > >
>> > > Results :
>> > >
>> > >
>> > > Failed tests:
>> > >
>> > >
>> > > Tests run: 48, Failures: 1, Errors: 0, Skipped: 0
>> > >
>> > >
>> > > [INFO]
>> > >
>> 
>> > >
>> > > [ERROR] BUILD FAILURE
>> > >
>> > > [INFO]
>> > >
>> 
>> > >
>> > > [INFO] There are test failures.
>> > >
>> > > Please refer to
>> > >
>> /Users/AngelaWang/Downloads/oodt_trunk/resource/target/surefire-reports
>> > for
>> > > the individual test results.
>> > > ..
>> > >
>> > > Attached is the detailed test result for the failed
>> TestQueueMuxMonitor
>> > > test. I am using Mac OS X 10.9.2. Thank you for your help!
>> > >
>> > > --
>> > > Best,
>> > > Mengying (Angela) Wang
>> > >
>> >
>> >
>> >
>> > --
>> > Best,
>> > Mengying (Angela) Wang
>> >
>>
>
>
>
> --
> Best,
> Mengying (Angela) Wang
>



-- 
Best,
Mengying (Angela) Wang

回复： Question about OODT file manager

[Zichuan Wang]

Dear Professor,  

We are stuck in OODT. The most critical problem we have now is

“How to make crawler work with workflow”?  

--  
Zichuan Wang
University of Southern California, Department of Computer Science


在 2014年10月28日 星期二，下午12:52，Luke 写道：

> Dear Professor Mattamnn,
> Thanks a lot Professor Mattmann for the kind help, it is appreciated, sorry 
> for getting back to you with my appreciation, I have been conducting tests 
> with OODT based on your advice, but unfortunately I am having another 
> problem
>  
> I am following the steps 
> (https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example) 
> to get a sense of how to get workflow to work.
> The problem is that the File-Concatenator-PGE (by running the wmgr-client 
> command-line) does not seems to be invoked or executed, but I am seeing the 
> tasks are getting stacked up in the workflow manager with status either 
> "RSUBMIT" or "QUEUED", but they are not getting executed, PFA: 
> workflow_monitor.jpg, please note, by default the workflow min pool size is 
> 6; so here comes another problem, i have 6 submitted tasks with status 
> RSUBMIT, but any new incoming tasks will be forwarded to the waiting QUEUE 
> with status "QUEUED"...please refer to the workflow_monitor.jpg for details, 
> where I have 3 QUEUED workflow task and 6 RSUMBITE tasks.  
>  
> Question 1): not sure why the workflow is not being executed, and hanging at 
> the state of "RSUBMIT", after enabling the log level, I am seeing the 
> following entry in the log, not sure if this has anything to do with the 
> "hanging" problem where workflow is not getting executed and hanging at state 
> of "RSUBMIT".
> Oct 28, 2014 3:35:07 AM 
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread 
> safeCheckJobComplete
> WARNING: Exception checking completion status for job: 
> [2014-10-28T01:59:32.813-07:00]: Messsage: java.lang.Exception: 
> java.lang.NullPointerException
>  
> Question 2): I think currently on my side any new incoming workflow task I am 
> sending with the following command is being directed to the waiting "QUEUE" 
> because of the min pool size (i.e. 6) (I can increase this to a larger number 
> though),  
> ./wmgr-client --url http://localhost:9200 --operation --sendEvent --eventName 
> fileconcatenator-pge --metaData --key RunID testNumber1
> If possible, I would like to please know if there is a way we can purge the 
> queue and get rid of those workflow tasks either in "RSUMBIT" and "QUEUED" I 
> have already sent, please kindly help.
>  
> Very sorry for troubling you with this, to be honest I find OODT a bit 
> challenging to grasp within a short time frame, probably because there is no 
> book like OODT in action like Solr and what I am doing is just trial and 
> error blended with guess, but I don’t want to make a blind guess, it will be 
> appreciated if you can please also shed some lights on where I can get more 
> information logging or other way where I can troubleshoot. I think it might 
> be worth tracking what is happening when workflow reach the status "RSUBMIT" 
> and how to get a specific logging info specific to it...
>  
> Again your advice and kind help will be appreciated usual.
>  
>  
> Thanks
> Luke
>  
> > -Original Message-
> > From: Mattmann, Chris A (3980) [mailto:chris.a.mattm...@jpl.nasa.gov]
> > Sent: 2014年10月26日 22:18
> > To: Luke; 'Zichuan Wang'
> > Cc: 'Christian Alan Mattmann'; zhouj...@usc.edu (mailto:zhouj...@usc.edu); 
> > xiaoy...@usc.edu (mailto:xiaoy...@usc.edu);
> > dev@oodt.apache.org (mailto:dev@oodt.apache.org)
> > Subject: Re: re： Question about OODT file manager
> >  
> > Hi Luke,
> >  
> > Thanks and sorry it’s taken me a while to reply. Here are some details 
> > below:
> >  
> >  
> > -Original Message-
> > From: Luke mailto:shuai...@usc.edu)>
> > Date: Sunday, October 26, 2014 at 6:19 PM
> > To: Chris Mattmann  > (mailto:chris.a.mattm...@jpl.nasa.gov)>, 'Zichuan Wang'
> > mailto:zichu...@usc.edu)>
> > Cc: Chris Mattmann mailto:mattm...@usc.edu)>, 
> > "zhouj...@usc.edu (mailto:zhouj...@usc.edu)"
> > mailto:zhouj...@usc.edu)>, "xiaoy...@usc.edu 
> > (mailto:xiaoy...@usc.edu)" mailto:xiaoy...@usc.edu)>,
> > "dev@oodt.apache.org (mailto:dev@oodt.apache.org)"  > (mailto:dev@oodt.apache.org)>
> > Subject: RE: re： Question about OODT file manager
> >  
> > > Hi Professor Mattmann and OODT DEV,
> > >  
> > > Sorry to trouble you with this email, our team has been struggling in
> > > the oodt to send json files to solr.
> > > One of the difficulties is still getting OODT workflow to call the
> > > poster.py in etllib.
> > >  
> >  
> >  
> > Sorry that you’re having difficulty let me try and help.
> >  
> > >  
> > > I am not sure if my understanding is correct with OODT requirement, I
> > > hope you can please kindly advice and help with our confusion.
> > >  
> > > a set of goals in my mind with OODT is as follows, please kindly
> > > confirm and clarify:
> > >  
> >

RADiX filemgr-client script cannot delete product by id or name

[MengYing Wang]

Hey everyone,

I am trying to delete some product in the OODT File Manager with its
product id or name. However, the system raises
"java.lang.IllegalArgumentException: URI is not absolute" error. Please
help me! Thank you!

My commands are:

./filemgr-client --op --dbyid --productId
2b0e7582-6077-11e4-ac24-8d456d2e5b9b --url http://localhost:9000

./filemgr-client -op --deleteProductByName --productName
concatenatedOutputFile-2014-10-30T20:56:02.189Z.txt --url
http://localhost:9000

The logs of the file manager is:
..

Oct 30, 2014 2:56:21 PM
org.springframework.beans.factory.support.AbstractBeanFactory doGetBean

FINE: Returning cached instance of singleton bean 'sqlQuery'

Oct 30, 2014 2:56:21 PM
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 

INFO: Loading File Manager Configuration Properties from:
[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.properties]

Oct 30, 2014 2:56:21 PM
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 

INFO: Loading File Manager Configuration Properties from:
[/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr/etc/filemgr.properties]

ERROR: Failed to delete product
'org.apache.oodt.cas.filemgr.structs.Product@5433d17' :
java.lang.Exception: java.lang.IllegalArgumentException: URI is not absolute

-- 
Best,
Mengying (Angela) Wang

Re: Could not mvn install OODT from the trunk

[MengYing Wang]

Hey,

$ mvn -version

Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700)

Java version: 1.7.0_51

Java home:
/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "mac os x" version: "10.9.2" arch: "x86_64" Family: "mac"


Yes. The error is repeatable on my pc. I have tried several times, all
failed with the same error.


Best,

Mengying (Angela) Wang

On Wed, Oct 29, 2014 at 4:06 PM, Michael Starch  wrote:

> MengYing,
>
> I cannot duplicate it on Mac OS X 10.9.5, Java 7 nor CentOS with Java 7.
>
> Is the error repeatable on your end?
>
> Michael
> On Oct 29, 2014 4:01 PM, "MengYing Wang"  wrote:
>
> > Hi Michael,
> >
> > $ java -version
> > java version "1.7.0_51"
> > Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode).
> >
> > Thanks.
> >
> > Best,
> > Mengying Wang
> >
> > On Wed, Oct 29, 2014 at 6:52 AM, Michael Starch 
> > wrote:
> >
> > > Mengying,
> > >
> > > I just tested this on Mac 1.9.5 using Java 7 and had no issues.Can
> > you
> > > tell me your java version?  In the meantime I will test with Java 6 as
> I
> > > suspect that this might be an issue.
> > >
> > > --Michael
> > > Hi Everyone,
> > >
> > > This is my command sequence and some log:
> > >
> > > $svn co http://svn.apache.org/repos/asf/oodt/trunk/ oodt_trunk
> > >
> > > $cd oodt_trunk/
> > >
> > > $mvn clean install
> > >
> > > ..
> > >
> > > ---
> > >
> > >  T E S T S
> > >
> > > ---
> > >
> > > Running org.apache.oodt.cas.resource.mux.TestQueueMuxMonitor
> > >
> > > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.277
> sec
> > > <<< FAILURE!
> > >
> > > Running org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository
> > >
> > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.182
> sec
> > >
> > > Running org.apache.oodt.cas.resource.monitor.TestGangliaResourceMonitor
> > >
> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.41
> sec
> > >
> > > Running org.apache.oodt.cas.resource.cli.TestResourceCli
> > >
> > > Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.848
> > sec
> > >
> > > Running org.apache.oodt.cas.resource.jobqueue.TestJobStack
> > >
> > > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063
> sec
> > >
> > > Running org.apache.oodt.cas.resource.system.TestXmlRpcResourceManager
> > >
> > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.346
> sec
> > >
> > > Running org.apache.oodt.cas.resource.monitor.TestGangliaXMLParser
> > >
> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.327
> sec
> > >
> > > Running org.apache.oodt.cas.resource.mux.TestQueueMuxBatchmgr
> > >
> > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057
> sec
> > >
> > > Running org.apache.oodt.cas.resource.monitor.TestAssignmentMonitor
> > >
> > > Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.248
> sec
> > >
> > > Running org.apache.oodt.cas.resource.util.TestUlimit
> > >
> > > Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046
> sec
> > >
> > >
> > > Results :
> > >
> > >
> > > Failed tests:
> > >
> > >
> > > Tests run: 48, Failures: 1, Errors: 0, Skipped: 0
> > >
> > >
> > > [INFO]
> > >
> 
> > >
> > > [ERROR] BUILD FAILURE
> > >
> > > [INFO]
> > >
> 
> > >
> > > [INFO] There are test failures.
> > >
> > > Please refer to
> > > /Users/AngelaWang/Downloads/oodt_trunk/resource/target/surefire-reports
> > for
> > > the individual test results.
> > > ..
> > >
> > > Attached is the detailed test result for the failed TestQueueMuxMonitor
> > > test. I am using Mac OS X 10.9.2. Thank you for your help!
> > >
> > > --
> > > Best,
> > > Mengying (Angela) Wang
> > >
> >
> >
> >
> > --
> > Best,
> > Mengying (Angela) Wang
> >
>



-- 
Best,
Mengying (Angela) Wang

Profile with id: 'fm-solr-catalog' has not been activated error

[MengYing Wang]

-- Forwarded message --
From: MengYing Wang 
Date: Thu, Oct 23, 2014 at 10:00 PM
Subject: Re: Directed Research Weekly Report from 2014/09/29 - 2014/10/05
To: "Verma, Rishi (398M)" 
Cc: Christian Alan Mattmann , "Mcgibbney, Lewis J (398M)"
, "Bryant, Ann C (398G-Affiliate)" <
anniebry...@gmail.com>, "Ramirez, Paul M (398M)" <
paul.m.rami...@jpl.nasa.gov>, "Mattmann, Chris A (3980)" <
chris.a.mattm...@jpl.nasa.gov>, Tyler Palsulich , "
u...@oodt.apache.org" 


Dear Rishi,

I followed the new steps to use the OODT RADiX. Unfortunately, I got the
same "Profile with id: 'fm-solr-catalog' has not been activated" error.
Below are my commands, and some terminal output. Please check it to see if
I have made some mistakes, or is it possible that something wrong with the
source code? Really appreciate for your help!

Step 1: $svn co http://svn.apache.org/repos/asf/oodt/trunk/ oodt_radix
Aoodt_radix/curator
Aoodt_radix/curator/pom.xml
Aoodt_radix/curator/src
Aoodt_radix/curator/src/test
..
Checked out revision 1633738.

Step 2: $cd oodt_radix/

Step 3: $mvn clean install
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO]   OODT Core
[INFO]   Common Utilities
[INFO]   CAS Command Line Interface
..
[INFO] BUILD SUCCESSFUL
[INFO]

[INFO] Total time: 5 minutes 29 seconds
[INFO] Finished at: Wed Oct 22 20:40:38 PDT 2014
[INFO] Final Memory: 133M/254M

Step 4: $mvn archetype:generate
[INFO] Scanning for projects...
[INFO] Reactor build order:
[INFO]   OODT Core
[INFO]   Common Utilities
..
[INFO] project created from Old (1.x) Archetype in dir:
/Users/AngelaWang/Downloads/oodt_radix/radix-archetype
[INFO]

[INFO] BUILD SUCCESSFUL
[INFO]

[INFO] Total time: 1 minute 40 seconds
[INFO] Finished at: Wed Oct 22 20:52:46 PDT 2014
[INFO] Final Memory: 36M/84M
[INFO]


Step 5: $cd radix-archetype/
Step 6: $mvn clean package -Pfm-solr-catalog
[INFO] Scanning for projects...
[WARNING]
Profile with id: 'fm-solr-catalog' has not been activated.

[INFO]

[INFO] Building radix-archetype
[INFO]task-segment: [clean, package]
[INFO]

.

Best,
Mengying (Angela) Wang

On Sat, Oct 18, 2014 at 3:49 PM, Verma, Rishi (398M) <
rishi.ve...@jpl.nasa.gov> wrote:

>  Hi MengYing,
>
>  Your CMD1 should not have the ‘-Pfm-solr-catalog’ argument. The reason
> is because that command *generates* a new project for you, whereas, the
> ‘-Pfm-solr-catalog’ should only be used to *build* the project once it
> has already been generated. You might want to read up a bit on Maven
> archetypes, which is what OODT RADiX is.
> http://maven.apache.org/guides/introduction/introduction-to-archetypes.html
>
>  Let me explain in this way, here’s the steps to using OODT RADiX:
> 1. Get a hold of the latest OODT RADiX Maven Archetype (you might have
> already done this if you have the full OODT source)
> i.e. download the full OODT source and invoke ‘mvn install’ so that
> you can use the latest RADiX archetype
> http://svn.apache.org/repos/asf/oodt/trunk/
> 2. Use the OODT RADiX Maven Archetype to *generate* a new OODT project
> source folder structure for you (this is the source for your new project!)
> i.e. invoke the command:
> > mvn archetype:generate
> (select RADiX from the list of archetypes you see, and follow the prompts)
> 3. Change into the newly generated directory from above, and *build* a
> tar-ball distribution of OODT that you can run from the source folder
> structure you generated earlier
> > mvn clean package -Pfm-solr-catalog
> 4. Take the build tar-ball distribution, and extract it somewhere else for
> launching OODT
> > tar zxf distribution/target/oodt-*.jar -C /usr/local/my-oodt-project
> 5. Run OODT
> > cd /usr/local/my-oodt-project/bin
> > ./oodt start
>
>  That’s the typical workflow for using RADiX. So the key here is, only
> use the ‘-Pfm-solr-catalog’ argument when *building* OODT, not when
> *generating* the folder structure.
>
>   *If you’re starting from scratch:*
> 1. Use Vagrant Virtual Machine technology to get a pre-built OODT
> deployment connected to Solr in one command:
> https://cwiki.apache.org/confluence/display/OODT/Vagrant+Powered+OODT
>
>  [ I didn't try this approach ]
>
>
>  You should try this! Because all five steps above are automated for
> you via the Vagrant machine.
>
>  Thanks,
> rishi
>
>  On Oct 17, 2014, at 11:29 AM, MengYing Wang 
> wrote:
>
>  Dear Rishi,
>
>  Actually, in the first command of the tutorial
> 

Re: Error occurs when using the query_tool script of the file manager for a new product type

[Mattmann, Chris A (3980)]

Thanks Angela,

Yes you are using the wrong syntax for Lucene. The syntax should be:

³Fieldname:FieldValue²

Please see the syntax here:

http://lucene.apache.org/core/2_9_4/queryparsersyntax.html


Cheers,
Chris

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: MengYing Wang 
Reply-To: "dev@oodt.apache.org" 
Date: Tuesday, October 28, 2014 at 4:17 PM
To: "dev@oodt.apache.org" 
Subject: Re: Error occurs when using the query_tool script of the file
manager for a new product type

>Hi Prof Chris,
>
>Done with Jira: https://issues.apache.org/jira/browse/OODT-766. This
>happens with the SQL Query only, not with the Lucene. At least no
>exception
>is thrown out in Lucene queries.
>
>By the way, maybe I am using the wrong syntax, but it seems the Lucene is
>much difficult to use than Sql. For example:
>
>Suppose I have 3 files with the GenericFile type:
>
>$ ./query_tool --url http://localhost:9000 --sql -query "SELECT
>CAS.ProductReceivedTime,CAS.ProductName,CAS.ProductId,ProductType,ProductT
>ypeId,ProductTypeName,ProductStructure,Filename,FileLocation,MimeType
>FROM GenericFile" -sortBy 'CAS.ProductReceivedTime' -outputFormat
>'$CAS.ProductReceivedTime,$CAS.ProductName,$CAS.ProductId,$ProductType,$Pr
>oductTypeId,$ProductTypeName,$ProductStructure,$Filename,$FileLocation,$Mi
>meType'
>
>Oct 28, 2014 3:51:31 PM
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>
>INFO: Query returned 3 results
>
>2014-09-15T12:41:31.241-07:00,blah.txt,4ad79161-3d10-11e4-b2be-cf9a90c3077
>d,GenericFile,GenericFileId,GenericFileName,Flat,blah.txt,/Users/AngelaWan
>g/files/blah.txt,text/plain,text,plain
>
>2014-10-13T22:04:28.703-07:00,hello.txt,93586611-535f-11e4-9209-4d1d47528f
>7d,GenericFile,GenericFileId,GenericFileName,Flat,hello.txt,/Users/AngelaW
>ang/files/hello.txt,text/plain,text,plain
>
>2014-10-27T23:31:54.319-07:00,concatenatedOutputFile-2014-10-28T06:31:53.5
>97Z.txt,1bc212ba-5e6c-11e4-84b4-f5653df4b171,GenericFile,FileConcatenatorP
>roductType,GenericFile,FileConcatenatorProductTypeId,GenericFile,FileConca
>tenatorProductTypeName,Flat,concatenatedOutputFile-2014-10-28T06:31:53.597
>Z.txt,concatenatedOutputFile-2014-10-28T06:31:53.597Z.txt,/Users/AngelaWan
>g/files/concatenatedOutputFile-2014-10-28T06:31:53.597Z.txt,text/plain,tex
>t,plain
>
>All the flowing 4 Lucene queries return 0 record.
>
>$./query_tool --url http://localhost:9000 --lucene -query
>'CAS.ProductTypeName:GenericFile'
>
>$./query_tool --url http://localhost:9000 --lucene -query
>'ProductTypeName:GenericFile'
>
>$./query_tool --url http://localhost:9000 --lucene -query
>'CAS.ProductTypeName:GenericFileName'
>
>$./query_tool --url http://localhost:9000 --lucene -query
>'ProductTypeName:GenericFileName'
>
>Oct 28, 2014 3:57:09 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>paginateQuery
>
>WARNING: Query: [q=CAS.ProductTypeName:GenericFile] for Product Type:
>[urn:oodt:GenericFile] returned no results
>
>Only this one gives me the correct result:
>
>$./query_tool --url http://localhost:9000 --lucene -query
>'CAS.ProductName:blah.txt'
>
>4ad79161-3d10-11e4-b2be-cf9a90c3077d
>
>
>Thanks.
>
>Best,
>
>Mengying (Angela) Wang
>
>
>
>
>
>
>On Tue, Oct 28, 2014 at 7:24 AM, Chris Mattmann 
>wrote:
>
>> Thanks for reporting this Angela. It would be great if you can
>> file an issue for this in JIRA? Does this happen with the SQL
>> Query only, or does it do so also with Lucene?
>>
>> Cheers,
>> Chris
>>
>> 
>> Chris Mattmann
>> chris.mattm...@gmail.com
>>
>>
>>
>>
>> -Original Message-
>> From: Mengying Wang 
>> Reply-To: 
>> Date: Tuesday, October 28, 2014 at 12:10 AM
>> To: 
>> Subject: Error occurs when using the query_tool script of the file
>>manager
>> for a new product type
>>
>> >Hi Everyone,
>> >
>> >According to the tutorial
>> >http://oodt.apache.org/components/maven/curator/user/basic.html, I
>>added
>> a
>> >new JSON product type to my file manager. However, if I run the
>>query_tool
>> >script before any json files ingested, it will raise some errors, as
>>shown
>> >here:
>> >
>> >$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >JSON"
>> >Oct 27, 2014 11:52:07 PM
>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>> >paginateQuery
>> >WARNING: Query: [q=] for Product Type: [urn:example:JSON] returned no
>> >results
>> >java.lang.NullPointerException
>> >at
>> 
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFile

Re: Review Request 22791: Streaming OODT Changes

[Michael Starch]

> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml, line 274
> > 
> >
> > Custom Maven repos are difficult since the Central repository is 
> > phasing them out:
> > 
> > 
> > http://blog.sonatype.com/2010/03/why-external-repos-are-being-phased-out-of-central/
> > 
> > If we absolutely need to ref this repo, can you make it a Maven 
> > profile, not enabled by default, so that users won't have the issue for 
> > this when downloading and building OODT?

These jars exist in the standard repo, so finding them there.


> On Sept. 4, 2014, 5:37 p.m., Chris Mattmann wrote:
> > http://svn.apache.org/repos/asf/oodt/trunk/resource/src/main/java/org/apache/oodt/cas/resource/mesos/ResourceMesosFrameworkFactory.java,
> >  line 1
> > 
> >
> > All of these files need ALv2 license headers.

Added ALv2 headers.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22791/#review52324
---


On Oct. 24, 2014, 10:35 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22791/
> ---
> 
> (Updated Oct. 24, 2014, 10:35 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This patch contains all the changes needed to add in "streaming oodt" into 
> the oodt svn repository.
> 
> There are four main portions:
>-Mesos Framework for Resource Manager (Prototype working)
>-Spark Runner for Workflow Manager (Prototype working)
>-Filemanager "streaming" type (In development)
>-Deployment and cluster management scripts (In development)
> 
> Where can this stuff be put so that it is available to use, even while it is 
> in development?
> 
> 
> Note: Filemanager work (and corrections here-in) have been moved to sub-patch 
> review: https://reviews.apache.org/r/27172
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/shutdown.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-master.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/mesos-slave.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/start-up/resource.bash
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/scripts/utilites.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/env-vars.sh.tmpl
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/hosts 
> PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/install.sh 
> PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/cluster-tools/setup/required-software.txt
>  PRE-CREATION 
>   http://svn.apache.org/repos/asf/oodt/trunk/core/pom.xml 1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/CoreMetExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/MimeTypeExtractor.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Product.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/structs/Reference.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManager.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/BasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/DateTimeVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/versioning/SingleFileBasicVersioner.java
>  1617800 
>   
> http://svn.apache.org/repos/asf/oodt/

Re: Why is pushpull not always working?

[Chris Mattmann]

Thanks Lewis - in my mind it boils down somewhat to
the protocols that are implemented and the way they
behave with FTP servers and so forth. Ultimately, we
should be using the ones that are in PushPull plugins,
but they are virally licensed and I haven¹t found a
better alternative.

Another @bfoster update here would be great, Brian?

Cheers,
Chris



Chris Mattmann
chris.mattm...@gmail.com




-Original Message-
From: Lewis John Mcgibbney 
Reply-To: 
Date: Saturday, November 1, 2014 at 12:56 AM
To: "dev@oodt.apache.org" 
Subject: Why is pushpull not always working?

>Hi Folks,
>I really couldn't think of a better title for this thread so I apologize
>in
>advance.
>I recently set up pushpull with FTP to crawl remote ftp site [0] which is
>publicly available.
>I am getting the following
>
>INFO: Launching Daemons . . .
>Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
>registerRMIServer
>INFO: Created Daemon ID = 90121 on RMI registry port 9012
>Nov 01, 2014 12:46:01 AM
>org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
>launchDaemons
>INFO: Creating Daemon with ID = 1
>Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:48:00 PDT 2014
>Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:51:00 PDT 2014
>Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:54:00 PDT 2014
>Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
>setAsRunning
>INFO: Daemon with ID = 90121 was given permission to run
>Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
>INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
>until Sat Nov 01 00:57:00 PDT 2014
>
>No errors... no pulling... no nothing!
>I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
>plugins page [2].
>My question is simple... why is pushpull known NOT to work on occasion? I
>am going to debug this tomorrow but I thought I would ask this
>none-the-less.
>Thanks
>Lewis
>
>[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
>[1]
>https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
>#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
>ningthe./pushpullscript.WhatshouldIdo
>?
>[2] 
>https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
>
>
>-- 
>*Lewis*

Re: Multiple RemoteSpecs files in pushpull components

[Chris Mattmann]

Hey Lewis,

I am not sure that the PushPull is single-threaded - check out
FileRetrievalSystem - which I believe is multi-threaded - maybe
that¹s why?

CC @bfoster, can you help here?

Cheers,
Chris


Chris Mattmann
chris.mattm...@gmail.com




-Original Message-
From: Lewis John Mcgibbney 
Reply-To: 
Date: Friday, October 31, 2014 at 10:16 PM
To: "dev@oodt.apache.org" 
Subject: Multiple RemoteSpecs files in pushpull components

>Hi Folks,
>I am looking for some details on why we would require multiple RemoteSpecs
>files within the pushpull component.
>An example remote specs file can be found here [0] where essentially we
>define individual pushpull client deamon's which do the heavily lifting
>for
>us in an automated manner.
>The problem I am having here is that we reserve and initiate a LinkedList
>data structure [1] for multiple RemoteSpecs.xml files and we then iterate
>through this data structure [2] in an attempt to locate and process
>configuration defined within each such file.
>All of this is held in memory when we run one specific daemon as AFAIK the
>pushpull component is a single threaded client.
>
>My questions are simple (and maybe trivial), do we require such a data
>structure? Does anyone ever pass in multiple RemoteSpecs.xml files?
>
>Thank you
>Lewis
>
>[0]
>https://svn.apache.org/repos/asf/oodt/trunk/pushpull/src/main/resources/ex
>amples/RemoteSpecsFiles/RemoteSpecs.xml
>[1]
>https://github.com/apache/oodt/blob/trunk/pushpull/src/main/java/org/apach
>e/oodt/cas/pushpull/daemon/DaemonLauncher.java#L63
>[2]
>https://github.com/apache/oodt/blob/trunk/pushpull/src/main/java/org/apach
>e/oodt/cas/pushpull/daemon/DaemonLauncher.java#L102
>
>-- 
>*Lewis*

Re: RADiX OPSUI Webapp Page Expired Error

[Chris Mattmann]

Guys also note I opened an issue for this a while
back:

https://issues.apache.org/jira/browse/OODT-751



Chris Mattmann
chris.mattm...@gmail.com




-Original Message-
From: "Verma, Rishi (398M)" 
Reply-To: 
Date: Friday, October 31, 2014 at 2:20 PM
To: MengYing Wang 
Cc: "dev@oodt.apache.org" , Chris Mattmann

Subject: Re: RADiX OPSUI Webapp Page Expired Error

>Hi Mengying,
>
>The ³fix code² is still there [1]. However, it seems this particular
>solution is not working anymore in 0.7.
>
>Could you try removing the fix and observing the results? This was a
>temporary fix, and a more fundamental one may be necessary. Check out the
>opsui code [2] to play around with potential solutions. Will look into it
>as well.
>
>Thanks much,
>Rishi
>
>[1] 
>http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apa
>che/oodt/pcs/opsui/OPSUIWebPage.java
>[2] 
>http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apa
>che/oodt/pcs/opsui/
>
>On Oct 30, 2014, at 3:23 PM, MengYing Wang
>mailto:mengyingwa...@gmail.com>> wrote:
>
>Hey Rishi,
>
>I see you have fixed this error in the OODT 0.6
>https://issues.apache.org/jira/browse/OODT-596. However, currently I am
>using the OODT 0.7 with RADiX, and keep receiving the error. Could you
>tell me what should I do to fix it, or is there any patch file that I
>could apply? Many thanks!
>
>--
>Best,
>Mengying (Angela) Wang
>
>---
>Rishi Verma
>NASA Jet Propulsion Laboratory
>California Institute of Technology
>4800 Oak Grove Drive, M/S 158-248
>Pasadena, CA 91109
>Tel: 1-818-393-5826
>

Re: OODT delete on server

[Chris Mattmann]

Thanks Etienne.

Apache OODT by itself doesn¹t call any compression
directly before sending the file, but we¹ve seen
people develop e.g., Chunk servers, and also kick
compression down into the file formats (e.g., since
HDF and NetCDF have compression, folks will use them
sometimes).

Hope that helps and hope others can chime in too!

Cheers,
Chris



Chris Mattmann
chris.mattm...@gmail.com




-Original Message-
From: Shakeh Khudikyan 
Date: Friday, October 31, 2014 at 10:49 AM
To: Etienne Koen 
Cc: Chris Mattmann 
Subject: Re: OODT delete on server

>Hi Etienne,
>
>
>I'm not sure about compressing data. I've never had to do that. OODT
>might have it, Chris?
>
>
>-Shakeh
>~~
>Shakeh E. Khudikyan
>Software Engineer
>Data Management Systems and Technologies
>Instrument Software and Science Data Systems Section (398)
>
>NASA Jet Propulsion Laboratory
>4800 Oak Grove Drive MS: 158-242
>Pasadena, CA  91109
>
>Phone: 818/354-5897
>Email:   shakeh.e.khudik...@jpl.nasa.gov
>
>~~
>
>
>
>
>From: Etienne Koen 
>Date: Wednesday, October 29, 2014 3:02 AM
>To: Shakeh Khudikyan 
>Cc: Chris Mattmann 
>Subject: Re: OODT delete on server
>
>
>
>Hi Shakeh,
>
>
>I can't seem to detect another query about  OODT pushpull in my inbox, so
>sorry if I am repeating myself...
>
>
>Does OODT have the capability of compressing data before it is sent or
>has other parties implemented this functionality successfully themselves?
>
>
>Regards
>Etienne
>
>
>On 29 October 2014 09:55, Etienne Koen  wrote:
>
>Hi Shakeh,
>
>
>Do you know whether OODT has the functionality of deleting a product in a
>similar way as a client would use pushpull to request a file being
>downloaded from a remote server?
>
>
>If so, what component of OODT would I use?
>
>
>Regards
>Etienne
>
>
>
>
>
>
>
>

Why is pushpull not always working?

[Lewis John Mcgibbney]

Hi Folks,
I really couldn't think of a better title for this thread so I apologize in
advance.
I recently set up pushpull with FTP to crawl remote ftp site [0] which is
publicly available.
I am getting the following

INFO: Launching Daemons . . .
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
registerRMIServer
INFO: Created Daemon ID = 90121 on RMI registry port 9012
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
launchDaemons
INFO: Creating Daemon with ID = 1
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:48:00 PDT 2014
Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:51:00 PDT 2014
Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:54:00 PDT 2014
Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 90121 was given permission to run
Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
until Sat Nov 01 00:57:00 PDT 2014

No errors... no pulling... no nothing!
I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
plugins page [2].
My question is simple... why is pushpull known NOT to work on occasion? I
am going to debug this tomorrow but I thought I would ask this
none-the-less.
Thanks
Lewis

[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
[1]
https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrunningthe./pushpullscript.WhatshouldIdo
?
[2] https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins


-- 
*Lewis*