Hi Chris,
I played with the SQL statement in my pge config file today and keep getting a
parsing error. I tooked at your PgeConfig_RatAggregator.xml (oh, you have a
cut-and-paste error on line 31, '[Ampersand]' should be '[Exclamation]' if I
understand correctly what you are doing there. And this typo propagated its way
to the other xml files as well), and I have tried to mimic the parts that apply
to what I am trying to do and I'm stumped by the SQL parsing error.
I am doing a very simple query just to try to prove that I *can* do a query. I
didn't really need the FORMAT agreement, but I wasn't sure the right syntax for
leaving it out. So I just specified the metadata key I'm searching for in the
FORMAT arguement. So here's what I have.
<metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT FileLocation
FROM EngineeringFile }"/>
One difference between this statement and yours is that you used $ signs in
your FORMAT specification. I have tried it both ways, with dollar signs and
without dollar signs and still get the parse error. (What is the significance
of the "$"?)
Here's the output from oodt.out, and my pge config file and tasks.xml follow
afterwards. The stack trace looks familiar from the standpoint that I've seen
parse errors before and fixed them. But, this one points out a suspicious
looking exception for a malformed URL for some reason. However, the output from
the log file indicates all of my urls are fine. No pge metadata is written for
this task and no script is created. Please let me know if you see anything
obvious that I'm missing.
Thanks,
Val
Oct 15, 2014 5:15:02 PM
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
checkTaskRequiredMetadata
INFO: Task: [DB-Builder-Task] has no required metadata fields
Oct 15, 2014 5:15:03 PM
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
executeTaskLocally
INFO: Executing task: [DB-Builder-Task] locally
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
INFO: Converting workflow configuration to static metadata...
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_CrawlerConfigFile] value
=
[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler/policy/crawler-config.xml]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PCS_WorkflowManagerUrl] value =
[http://localhost:9001]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_FileManagerUrl] value =
[http://localhost:9000]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [FEI_DROP_DIR] value =
[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGE_HOME] value =
[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key =
[PGETask_Query_ClientTransferServiceFactory] value =
[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_ActionIds] value =
[MoveFileToLevel0Dir]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_MimeExtractorRepo] value
=
[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/policy/mime-extractor-map.xml]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_DumpMetadata] value = [true]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Query_FileManagerUrl] value =
[http://localhost:9000]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Name] value = [DB-Builder-Task]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_ConfigFilePath] value =
[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/config/jedi-conduit-db-builder-config.xml]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
INFO: Loading workflow context metadata...
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
[44f791c9-54b0-11e4-a032-c9e22d0219d9]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding dynamic metadata: key = [JobId] value =
[44f791c9-54b0-11e4-a032-c9e22d0219d9]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
[http://slothrop.jhuapl.edu:9001]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding dynamic metadata: key = [TaskId] value = [urn:oodt:dbBuilderTask]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
createPgeMetadata
FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
[slothrop.jhuapl.edu]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig
INFO: Create PgeConfig...
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig
INFO: Using default PgeConfigBuilder:
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance run
SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
java.io.IOException: Failed to build PgeConfig : Failed to parse value:
SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:87)
at
org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:230)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
at
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
at
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Failed to parse value:
SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
at
org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245)
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(XmlFilePgeConfigBuilder.java:165)
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFilePgeConfigBuilder.java:120)
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:83)
... 6 more
Caused by: java.net.MalformedURLException
at java.net.URL.<init>(URL.java:619)
at java.net.URL.<init>(URL.java:482)
at java.net.URL.<init>(URL.java:431)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
... 11 more
Caused by: java.lang.NullPointerException
at java.net.URL.<init>(URL.java:524)
... 14 more
org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException:
PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
at
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
at
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to build PgeConfig : Failed to parse
value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:87)
at
org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:230)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
... 4 more
Caused by: java.lang.Exception: Failed to parse value:
SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
at
org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245)
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(XmlFilePgeConfigBuilder.java:165)
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFilePgeConfigBuilder.java:120)
at
org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:83)
... 6 more
Caused by: java.net.MalformedURLException
at java.net.URL.<init>(URL.java:619)
at java.net.URL.<init>(URL.java:482)
at java.net.URL.<init>(URL.java:431)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
... 11 more
Caused by: java.lang.NullPointerException
at java.net.URL.<init>(URL.java:524)
... 14 more
Oct 15, 2014 5:15:03 PM
org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
executeTaskLocally
WARNING: Exception executing task: [DB-Builder-Task] locally: Message: PGETask
FAILED!!! : Failed to build PgeConfig : Failed to parse value:
SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
<!--
** Filename: jedi-conduit-db-builder-config.xml
-->
<pgeConfig>
<import file="pge-common-config.xml" />
<!-- Note: do not use the "shell" attribute, "shellType is the correct
attribute here"-->
<exe
dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/jobs"
shellType="/usr/local/bin/bash">
<cmd>echo "Current Working Directory is `pwd`" >
[JobLogDir]/db-builder.log</cmd>
<cmd>echo "Current Environment is `/bin/env`" >>
[JobLogDir]/db-builder.log</cmd>
<cmd>[JavaHome] -cp [JediConduitJar] [LoadClass] [DirList]
[Arguements]</cmd>
</exe>
<!-- Custom metadata to add to output files -->
<customMetadata>
<metadata key="LoadClass" val="conduit.jedi.utils.JEDIFastDBBuilder" />
<metadata key="TelemetryFileDb" val="[JobDir]/valsTelemetryFile.db" />
<metadata key="Arguements" val="[TelemetryFileDb]" />
<metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT
FileLocation FROM EngineeringFile }" />
<!-- SQL example statement from the code
SQL
(FORMAT='FileLocation/Filename',SORT_BY='FileSize',FILTER=<StartDateTime,EndDateTime,DataVersion,TakeHighestPriority)
{ SELECT FileLocation,Filename,FileSize FROM IASI_L1C WHERE
ProductionDateTime >= '2007-12-01T00:00:00.000000Z' }
-->
</customMetadata>
</pgeConfig>
<!--
** Filename: tasks.xml
-->
<task id="urn:oodt:dbBuilderTask" name="DB-Builder-Task"
class="org.apache.oodt.cas.pge.StdPGETaskInstance">
<conditions/>
<configuration>
<property name="PGETask_Name" value="DB-Builder-Task"/>
<property name="PGETask_ConfigFilePath"
value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/config/jedi-conduit-db-builder-config.xml"
envReplace="true" />
<property name="PGETask_DumpMetadata" value="true"/>
<property name="PCS_WorkflowManagerUrl" value="http://localhost:9001";
/>
<property name="PGETask_Query_FileManagerUrl"
value="http://localhost:9000"; />
<property name="PGETask_Ingest_FileManagerUrl"
value="http://localhost:9000"/>
<property name="PGETask_Query_ClientTransferServiceFactory"
value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"/>
<property name="PGETask_Ingest_CrawlerConfigFile"
value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler/policy/crawler-config.xml"/>
<property name="PGETask_Ingest_MimeExtractorRepo"
value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/policy/mime-extractor-map.xml"/>
<property name="PGETask_Ingest_ActionIds" value="MoveFileToLevel0Dir"/>
<property name="PGE_HOME"
value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/>
<property name="FEI_DROP_DIR"
value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging"/>
</configuration>
<requiredMetFields/>
</task>
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory
> -----Original Message-----
> From: Chris Mattmann [mailto:[email protected]]
> Sent: Tuesday, October 14, 2014 4:11 PM
> To: [email protected]
> Subject: Re: Failed to build PgeConfig, exception in
> PathUtils.doDynamicReplacement
>
> Awesome Val, glad it helped and that you are moving further!
>
> Here?s a quick example of how to call your perl script as an extractor in
> CAS-PGE
> (barring yes, the duplication of specifying a regExp to select the type to
> run the
> extractor on, but I?ll get to that later):
>
> Under <output>:
>
> <files regExp=?\\*.hdf"
>
> metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorMetWri
> ter"
> args=?/path/to/your-extern-config.xml?>
>
>
> Your extern-config.xml file should work like this:
>
> http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources/exam
> ples/extern-config.xml
>
>
> Note that the ExternExtractor is documented here:
>
> http://oodt.apache.org/components/maven/metadata/user/basic.html
>
>
> That will get you your Perl script integrated as an extractor into the system.
>
> On to your question about selecting information from the File Manager:
>
> Look up the SQL(.. portions of the DRAT PGE configs to get an idea of how to
> do
> this. In short, yes you can do it, and the way to do it is to use the special
> CAS-
> PGE SQL-like language that you can prefix your met key with (make sure
> envReplace is turned on). You can do like SQL([options]){ SELECT FROM
> <product type name> WHERE <criteria>
>
> The results of your above query are stored into the metadata field name in
> your
> CAS-PGE config file.
>
> I?ll walk you through more of this later, but have to run to a meeting.
>
> Gluck!
>
> Cheers,
> Chris
>
>
> ------------------------
> Chris Mattmann
> [email protected]
>
>
>
>
> -----Original Message-----
> From: "Mallder, Valerie" <[email protected]>
> Reply-To: <[email protected]>
> Date: Tuesday, October 14, 2014 at 12:58 PM
> To: "[email protected]" <[email protected]>
> Subject: RE: Failed to build PgeConfig, exception in
> PathUtils.doDynamicReplacement
>
> >Hi Chris,
> >
> >Thanks for your emails from Friday and Saturday. They were extremely
> >helpful and I have now worked my way through all of my setup issues so
> >far. I was using the new style CASE-PGE keys because Rishi used those
> >in his working example and at one point I had been trying to mimic that
> >example. I didn't know anything about the legacy mode flag. But, after
> >changing the keys to the old style, everything started to come together
> >a little faster.
> >
> >To start my crawler at the beginning of my pipeline, I chose the
> >approach that Rishi had taken where the pgeConfig starts the
> >crawler_launcher script directly rather than rely on the definition of
> >the output files to trigger the crawler to run. This was because the
> >met extractor I am using is a perl script, and your DRAT system only
> >shows how to specify a class for the metFileWriterClass attribute. I
> >could not find anything to show me how to specify a perl script as the met
> >file
> writer instead.
> >Also, having to specify the file types to crawl for by using a regular
> >expression in the config file seemed like an unnecessary duplication of
> >the definition of the files types to crawl for (since the regular
> >expression just specifies a mime-type that is already defined in
> >mime-types.xml). I didn't want to specify the mime-type in two
> >different places.
> >
> >So, now I've been through the process of setting up my CAS-PGE to
> >ingest the raw science and spacecraft housekeeping telemetry files, and
> >run a postIngestSuccess action to copy some of those files to a new
> >directory for further processing.
> >
> >The next step in my pipeline may be a little tricky. I need to query
> >the filemgr and pass the results of the query (it could be a long list)
> >as an input parameter to the first algorithm in the pipeline which,
> >happens to be another perl script. Does anyone know how I might do
> >this in a PGE config file?
> >
> >Thanks!!
> >Val
> >
> >
> >
> >
> >
> >Valerie A. Mallder
> >New Horizons Deputy Mission System Engineer Johns Hopkins
> >University/Applied Physics Laboratory
> >
> >
> >> -----Original Message-----
> >> From: Mattmann, Chris A (3980) [mailto:[email protected]]
> >> Sent: Friday, October 10, 2014 7:59 PM
> >> To: [email protected]
> >> Subject: Re: Failed to build PgeConfig, exception in
> >> PathUtils.doDynamicReplacement
> >>
> >> Thanks Val you are close!
> >>
> >> Looking at what you show below, I think the issue is one of the
> >>following:
> >>
> >> 1. You are using the new style CAS-PGE Keys. Try using the old ones
> >>(e.g., the ones present here:
> >>
> >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/exam
> >>ple
> >>s/
> >> WorkflowTask/tasks.xml
> >>
> >> (note the ?_? and not the ?/? used in keys).
> >> These keys are used by CAS-PGE if you see something in your wmgr bin
> >>script (or your resource manager batch_stub script) stating
> >>?legacyMode=true?).
> >> If you
> >> are using RADIX, I believe that?s the case:
> >>
> >> http://s.apache.org/hO
> >>
> >> 2. Regarding better building environments. The m2e plugin for Eclipse
> >>is fantastic nowadays and can literally checkout a multi-module Maven
> >>project from SVN (when paired with Subversive as a plugin or
> >>Subclipse). Another thing to check out is this page on the wiki:
> >>
> >>
> >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develope
> >>r+H
> >>el
> >> p
> >>
> >>
> >> Let me know if that fixes it. Sorry for all the trouble :) Trust me
> >>it will be worth it.
> >> CAS-PGE is awesome once working.
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> +++++
> >> Chris Mattmann, Ph.D.
> >> Chief Architect
> >> Instrument Software and Science Data Systems Section (398) NASA Jet
> >> Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 168-519, Mailstop: 168-527
> >> Email: [email protected]
> >> WWW: http://sunset.usc.edu/~mattmann/
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> +++++
> >> Adjunct Associate Professor, Computer Science Department University
> >> of Southern California, Los Angeles, CA 90089 USA
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> +++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: <Mallder>, Valerie <[email protected]>
> >> Reply-To: "[email protected]" <[email protected]>
> >> Date: Friday, October 10, 2014 at 3:40 PM
> >> To: "[email protected]" <[email protected]>
> >> Subject: Failed to build PgeConfig, exception in
> >>PathUtils.doDynamicReplacement
> >>
> >> >Hi All,
> >> >
> >> >I'm still working on this! Still trying to get a CA-PGE Task to run.
> >> >I'm almost there. Now it at least trys to build the PGE config file.
> >>But the
> >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking up a
> >> >null string somewhere, but it also looks like it is crashing in a
> >> >strange place. It is unable to read the .xml file that contains my
> >> ><pgeConfig>...</pgeConfig> but, the correct path and file are shown
> >> >in the output log messages I've include below which also shows the
> >> >stack trace. I looks like it is failing during a recursive call to
> >> >PathUtils.doDynamicReplacement. I saw some chatter about some prior
> >> >errors in this code in the mailing list archives from 2010 and 2012.
> >> >So I am hoping someone might remember and be able to tell me if I am
> >> >doing something that is causing this error. I included my tasks.xml
> >> >and fei-crawler-pge-config.xml file after the log messages.
> >> >
> >> >At this point, I think need to start running these processes in an
> >> >environment where I can debug this better. The runtime output and
> >> >stack traces simply aren't enough for me to track this down. I am
> >> >newbie to using eclipse and mvn, so, does anyone have some notes on
> >> >the best way to import oodt-0.7 into an exclipe workspace and tell
> >> >it to build it using mvn?? I am hoping I can leverage other peoples
> >> >knowledge of how to do this so that I can do it quickly and not
> >> >waste another week. I imported the oodt-0.7 directory into eclipse
> >> >so I could view the files easier, but I just made eclipse link to
> >> >the where the files are located rather that copy them into a
> >> >workspace folder. If there's a better way, please tell me. And now,
> >> >how do I tell exclipse
> >>to build
> >> stuff?
> >> >
> >> >Thanks!!
> >> >Valerie
> >> >
> >> >
> >> >
> >> >Using CATALINA_BASE:
> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> >> >Using CATALINA_HOME:
> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> >> >Using CATALINA_TMPDIR:
> >>
> >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/t
> >>>emp
> >> >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55
> >> >Workflow Manager started PID file
> >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workfl
> >> >ow/
> >> >run
> >> >/cas.workflow.pid).
> >> >Oct 10, 2014 5:53:52 PM
> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >> >loadProperties
> >> >INFO: Loading Workflow Manager Configuration Properties from:
> >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workfl
> >> >ow/
> >> >etc
> >> >/workflow.properties]
> >> >Oct 10, 2014 5:53:52 PM
> >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory
> >> >getResmgrUrl
> >> >INFO: No Resource Manager URL provided or malformed URL: executing
> >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM
> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
> >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM
> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >> >handleEvent
> >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10,
> >> >2014
> >> >5:54:12 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >> >handleEvent
> >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event
> >> >startJediPipeline Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >checkTaskRequiredMetadata
> >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields Oct
> >> >10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >executeTaskLocally
> >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014
> >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >INFO: Converting workflow configuration to static metadata...
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl]
> >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Ingest/CrawlerConfigFile]
> >> >value =
> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/c
> >> >raw
> >> >ler
> >> >/policy/crawler-config.xml]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/Name] value =
> >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds]
> >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value =
> >> >[true] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Query/ClientTransferServiceFactory] value =
> >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGE_HOME] value =
> >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/Query/FileManagerUrl]
> >> >value = [http://localhost:9000] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Ingest/MimeExtractorRepo]
> >> >value =
> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e
> >> >xte
> >> >nsi
> >> >ons/policy/mime-extractor-map.xml]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] value
> >> >=
> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e
> >> >xte
> >> >nsi
> >> >ons/config/fei-crawler-pge-config.xml]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000] Oct
> >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >INFO: Loading workflow context metadata...
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
> >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [JobId] value =
> >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
> >> >[http://slothrop.jhuapl.edu:9001]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [TaskId] value =
> >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
> >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeConfig
> >> >INFO: Create PgeConfig...
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeConfig
> >> >INFO: Using default PgeConfigBuilder:
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run
> >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to
> >> >parse
> >> >value: null
> >> >java.io.IOException: Failed to build PgeConfig : Failed to parse value:
> >> >null
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:87)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInsta
> >> >nce
> >> >.ja
> >> >va:230)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ex
> >> >ecu
> >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ru
> >> >n(I
> >> >terativeWorkflowProcessorThread.java:321)
> >> > at
> >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> >>Source)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >Caused by: java.lang.Exception: Failed to parse value: null
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:77)
> >> > ... 6 more
> >> >Caused by: java.lang.NullPointerException
> >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> >> > at java.util.regex.Matcher.reset(Matcher.java:308)
> >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
> >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRep
> >> >lac
> >> >eme
> >> >nt(PathUtils.java:321)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pat
> >> >hUt
> >> >ils
> >> >.java:96)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> >> > ... 8 more
> >> >
> >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstance
> >> >Exc
> >> >ept
> >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
> >> >value: null
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ex
> >> >ecu
> >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ru
> >> >n(I
> >> >terativeWorkflowProcessorThread.java:321)
> >> > at
> >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> >>Source)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed
> >> >to parse value: null
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:87)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInsta
> >> >nce
> >> >.ja
> >> >va:230)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> >> > ... 4 more
> >> >Caused by: java.lang.Exception: Failed to parse value: null
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:77)
> >> > ... 6 more
> >> >Caused by: java.lang.NullPointerException
> >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> >> > at java.util.regex.Matcher.reset(Matcher.java:308)
> >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
> >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRep
> >> >lac
> >> >eme
> >> >nt(PathUtils.java:321)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pat
> >> >hUt
> >> >ils
> >> >.java:96)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> >> > ... 8 more
> >> >Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >executeTaskLocally
> >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally:
> >>Message:
> >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> >> >null
> >> >
> >> >
> >> >
> >> >
> >> >Content of tasks.xml
> >> >
> >> >
> >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";>
> >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName"
> >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance">
> >> > <conditions/>
> >> >
> >> > <configuration>
> >> > <property name="PGETask/Name" value="feiCrawlerTaskName"/>
> >> > <property name="PGETask/ConfigFilePath"
> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de
> >> >plo y/e xtensions/config/fei-crawler-pge-config.xml"/>
> >> > <property name="PGETask/DumpMetadata" value="true"/>
> >> > <property name="PGETask/WorkflowManagerUrl"
> >> >value="http://localhost:9001"; />
> >> > <property name="PGETask/Query/FileManagerUrl"
> >> >value="http://localhost:9000"; />
> >> > <property name="PGETask/Ingest/FileManagerUrl"
> >> >value="http://localhost:9000"/>
> >> >
> >> > <property name="PGETask/Query/ClientTransferServiceFactory"
> >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFac
> >> >tor
> >> >y"/
> >> >>
> >> > <property name="PGETask/Ingest/CrawlerConfigFile"
> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de
> >> >plo
> >> >y/c
> >> >rawler/policy/crawler-config.xml"/>
> >> > <property name="PGETask/Ingest/MimeExtractorRepo"
> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de
> >> >plo y/e xtensions/policy/mime-extractor-map.xml"/>
> >> > <property name="PGETask/Ingest/ActionIds"
> >> >value="MoveFileToLevel0Dir"/>
> >> > <property name="PGE_HOME"
> >>
> >>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"
> >>>/>
> >> > </configuration>
> >> >
> >> > <requiredMetFields/>
> >> > </task>
> >> ></cas:tasks>
> >> >
> >> >
> >> >Contents of fei-crawler-pge-config.xml
> >> >
> >> >
> >> ><pgeConfig>
> >> >
> >> > <!-- How to run the PGE -->
> >> > <!-just echoing the current directory to a file so I can see if
> >> >this thing ever starts -->
> >> > <exe
> >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/da
> >> >ta/
> >> >pge
> >> >/jobs" shell="/bin/sh">
> >> > <cmd>echo "Current Working Directory is `pwd`" >
> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pg
> >> >e.l
> >> >og<
> >> >/cmd>
> >> > </exe>
> >> >
> >> > <!-- Files to ingest -->
> >> > <output>
> >> > <!-- trying this approach. Telling PGE there is output files
> >> >should invoke the crawler. -->
> >> >
> >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/d
> >> >ata
> >> >/st
> >> >aging"/>
> >> > </output>
> >> >
> >> > <!-- Custom metadata to add to output files -->
> >> > <customMetadata>
> >> >
> >> > <!-- each of these directories exist -->
> >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs"
> >> >envReplace="true"/>
> >> > <metadata key="JobInputDir" val="[JobDir]/input"/>
> >> > <metadata key="JobOutputDir" val="[JobDir]/output"/>
> >> > <metadata key="JobLogDir" val="[JobDir]/logs"/>
> >> >
> >> > </customMetadata>
> >> ></pgeConfig>
> >> >
> >> >Valerie A. Mallder
> >> >
> >> >New Horizons Deputy Mission System Engineer The Johns Hopkins
> >> >University/Applied Physics Laboratory
> >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> >> >240-228-7846 (Office) 410-504-2233 (Blackberry)
> >> >
> >
>
