Hi Chris,
As a first step I got an XMLWorkflowRepository workflow running. I then
tried using the PackagedWorkflowRepository to create a workflow with the
same tasks. I got a sequential workflow up and running, similar to the
following:
<cas:workflows ...>
<sequential id="urn:test:MainWorkflow" name="MainWorkflow">
<task id-ref="urn:test:TaskA"/>
<task id-ref="urn:test:TaskB"/>
<task id-ref="urn:test:TaskC"/>
<task id-ref="urn:test:TaskD"/>
<task id-ref="urn:test:TaskE"/>
<task id-ref="urn:test:TaskF"/>
<task id-ref="urn:test:TaskG"/>
</sequential>
<!-- task definitions -->
<task>...</task>
</cas:workflows>
I found that I could run the workflow using ./wmgr-client --url
[WorkflowManagerUrl] --sendEvent --eventName [WorkflowId]
e.g. ./wmgr-client --url http://localhost:9001 --sendEvent --eventName
urn:test:MainWorkflow
After getting the sequential workflow running, I wanted to run tasks C and
D in parallel so I then tried adding a <parallel> section as follows:
<cas:workflows ...>
<sequential id="urn:test:MainWorkflow" name="MainWorkflow">
<task id-ref="urn:test:TaskA"/>
<task id-ref="urn:test:TaskB"/>
<parallel id="urn:test:ParallelWorkflow" name="ParallelWorkflow">
<task id-ref="urn:test:TaskC"/>
<task id-ref="urn:test:TaskD"/>
</parallel>
<task id-ref="urn:test:TaskE"/>
<task id-ref="urn:test:TaskF"/>
<task id-ref="urn:test:TaskG"/>
</sequential>
...
</cas:workflows>
The parallel section generates an additional task to be submitted called
'Redirector Task' that wants to be executed in the 'high' queue using the
BranchRedirector class:
INFO: Task: [Redirector Task] has no required metadata fields
May 13, 2015 1:27:29 AM
org.apache.oodt.cas.resource.system.XmlRpcResourceManagerClient submitJob
FINEST:
[
{
job.queueName=high,
job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob,
job.name=redirector-322db9d2-4663-4b89-9af0-f8f97ad74c80,
job.id=, job.status=, job.load=2,
job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput
},
{
task.instance.class=org.apache.oodt.cas.workflow.examples.BranchRedirector,
task.config={},
task.metadata={...}
}
]
I wasn't sure how to configure my setup to deal with this task. From
reading OODT-70 I gather that it might be trying to start up other
wmgr-clients on batch stubs to run these tasks in separate workflows. I
tried setting up a local batch stub that accepts jobs from the 'high'
queue. When running the full workflow I get the following exception from
the batch stub when it tries to execute the redirector task:
INFO: Assigning job: [redirector-8561de94-2681-477b-8bcb-331d83224d5b] to
node: [local-redirector]
May 13, 2015 1:36:51 AM
org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub genericExecuteJob
INFO: stub attempting to execute class:
[org.apache.oodt.cas.workflow.structs.TaskJob]
org.apache.xmlrpc.XmlRpcClientException: Failure writing request
at
org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequest(XmlRpcClientRequestProcessor.java:76)
at
org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequestBytes(XmlRpcClientRequestProcessor.java:102)
at
org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:70)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at
org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.sendEvent(XmlRpcWorkflowManagerClient.java:477)
at
org.apache.oodt.cas.workflow.examples.BranchRedirector.run(BranchRedirector.java:62)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
at
org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.xmlrpc.XmlRpcException: null values not supported by
XML-RPC
at org.apache.xmlrpc.XmlWriter.writeObject(XmlWriter.java:231)
at
org.apache.xmlrpc.XmlRpcClientRequestProcessor.encodeRequest(XmlRpcClientRequestProcessor.java:72)
... 10 more
org.apache.oodt.cas.resource.structs.exceptions.JobInputException: Failed
to run task
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
at
org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException:
Failure writing request
at
org.apache.oodt.cas.workflow.examples.BranchRedirector.run(BranchRedirector.java:64)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
... 2 more
May 13, 2015 1:36:51 AM
org.apache.oodt.cas.resource.batchmgr.XmlRpcBatchMgrProxy run
SEVERE: Job execution failed for jobId '2015-05-13T01:36:38.746-07:00' :
batchstub.executeJob returned false
java.lang.Exception: batchstub.executeJob returned false
at
org.apache.oodt.cas.resource.batchmgr.XmlRpcBatchMgrProxy.run(XmlRpcBatchMgrProxy.java:125)
It looks like my sequential and parallel workflows have the required names
and IDs. Do you know what could be going wrong? Maybe I'm missing a few
vital configuration steps.
Many thanks for your help,
Ross
On 8 May 2015 at 19:31, Ross Laidlaw <rlaidlaw.o...@gmail.com> wrote:
> Hi Chris,
>
> Many thanks for your advice. I'll check out the
> PackagedWorkflowRepository and will let you know how it goes.
>
> Ross
>
>
>
> > On May 8, 2015, at 12:58 PM, Mattmann, Chris A (3980) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
> >
> > Hey Ross:
> >
> > Great work! Try the PackagedWorkflowRepository. The best documentation
> > that exists of it is in OODT-70 [1].
> >
> > I would use that to support the type of workflows that you’re doing.
> > If this works we should make it the default since it’s supposed to
> > be back compat with the XMLWorkflowRepositoryManager.
> >
> > Cheers,
> > Chris
> >
> > P.S. see some of the ASO scripts I wrote to check on specific task/
> > PGE status, etc.
> >
> >
> > [1]https://issues.apache.org/jira/browse/OODT-70
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattm...@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Ross Laidlaw <rlaidlaw.o...@gmail.com>
> > Reply-To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> > Date: Friday, May 8, 2015 at 8:14 AM
> > To: "dev@oodt.apache.org" <dev@oodt.apache.org>
> > Subject: Request for advice on workflows
> >
> >> Hi All,
> >>
> >> I'm working on an OODT setup (v0.8.1) with several workflow tasks
> defined
> >> and working well individually. I can use the wmgr-client to fire off
> jobs
> >> separately and all appears to be working as planned. I'd like to
> combine
> >> these tasks into a workflow so that I don't have to run each set of
> jobs.
> >> For example, in the workflow I'd like to run several 'task A' jobs and
> >> when
> >> all these have finished I'd like the workflow to automatically start
> >> several 'task B' jobs, etc.
> >>
> >> I found some example workflows in the distribution, plus several guides
> up
> >> on the wiki. Is there a current recommended way of setting up a simple
> >> workflow? Should I use workflow v2 or is it best to stick with v1?
> Once
> >> I've set up a workflow, how do I start it off? It looks like it might
> be
> >> possible to nest workflows, is that functionality available yet?
> >>
> >> Additionally, what's the best way to check on the status of currently
> >> running jobs? I've used the pcs_stat tool a lot and it's great for
> >> checking on the overall status, but is there a recommended way to check
> >> e.g. if any 'task A' jobs are still running for a certain workflow and
> >> what
> >> state they're currently in?
> >>
> >> Many thanks in advance for your advice,
> >>
> >>
> >> Ross
> >
>
