I'm actually suspicious that this might be a missing environment variable setting. Doesn't the XmlHelper resolve the environment variable names to their values in an xml file? I can't remember.
Would be good to check the environment settings and make sure none of those are null as well. Valerie A. Mallder New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied Physics Laboratory > -----Original Message----- > From: Chris Mattmann [mailto:[email protected]] > Sent: Tuesday, June 09, 2015 11:44 AM > To: [email protected] > Subject: Re: OODT Documentation > > Good advice, Val. I think I may know the issue here, though. > I believe the issue is *definitely* associated with the legacy keys issue, as > you > stated. If you roll down to the NPE that Andrea is experience, it’s because > one of > the PGE config key values is null, one of the required ones that OODT expects. > Pro tip: filing a JIRA ticket for actually stating which one of these is null > would be > IMMENSELY helpful and a relatively simple fix. Andrea, or Val, if one of you > could > file that JIRA ticket I will commit to updating PGE to print out which key is > actually > null. This one has been a long time coming. > > Now: why is that key null? It’s null b/c I’m almost positive that the > legancyMode for > PGE is turned on by default in RADIX. > > Andrea: what happens if you grab this file: http://s.apache.org/kDs and > replace > your $OODT_HOME/resmgr/bin/batch_stub file with it. Stop the batch stub, turn > it > back on, submit the job again, and cross your fingers! > > Cheers, > Chris > > > ---- > Chris Mattmann > [email protected] > > > > > > > -----Original Message----- > From: "Mallder, Valerie" <[email protected]> > Reply-To: <[email protected]> > Date: Tuesday, June 9, 2015 at 8:05 AM > To: "[email protected]" <[email protected]> > Subject: RE: OODT Documentation > > >Hi Andrea, > > > >I have found it helpful when debugging, or just investigating how > >something works, to remove the resource manager from the equation and > >run without it at first. If you can get your tasks to run without the > >resource manager, then you should be able to get them to run WITH the > >resource manager. And if you can't, then at least you've narrowed down > >the area where the problem is located. > > > >Go into oodt/workflow/etc and edit the file workflow.properties and > >comment out line 36. (Line 39 if you are using the trunk version) This > >should be the line that specifies the resource manager url. Just > >comment that out. And then try running again. You will of course get > >a different set of log messages, but they might not look so cryptic > >this time. > > > >Let me know how it goes. > >Val > > > > > > > > > >Valerie A. Mallder > >New Horizons Deputy Mission System Engineer Johns Hopkins > >University/Applied Physics Laboratory > > > > > >> -----Original Message----- > >> From: Andrea DeMarco [mailto:[email protected]] > >> Sent: Tuesday, June 09, 2015 10:37 AM > >> To: Mattmann, Chris A (3980) > >> Cc: Wyngaard, Jane R (3980-Affiliate); Starch, Michael D (398M); > >>Kristian ZARB ADAMI; Ramirez, Paul M (398M); Alessio Magro; > >>[email protected] > >> Subject: Re: OODT Documentation > >> > >> Hi Chris, > >> > >> Some additional detail: > >> > >> When stopping/starting the workflow manager before kicking the PGE, > >>the terminal (post PGE submission) also prints out the following: > >> > >> andrea@issa:~/oodt/workflow/bin$ ./wmgr stop > >>andrea@issa:~/oodt/workflow/bin$ > >> ./wmgr start Workflow Manager started PID file > >>(/home/andrea/oodt/workflow/run/cas.workflow.pid). > >> andrea@issa:~/oodt/workflow/bin$ Jun 09, 2015 4:34:24 PM > >>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > >>loadProperties > >> INFO: Loading Workflow Manager Configuration Properties from: > >> [/home/andrea/oodt/workflow/etc/workflow.properties] > >> Jun 09, 2015 4:34:25 PM > >> org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init> > >> INFO: Workflow Manager started by andrea Jun 09, 2015 4:34:29 PM > >>org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent > >> INFO: WorkflowManager: Received event: fileconcatenator-pge Jun 09, > >>2015 > >> 4:34:29 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager > >> handleEvent > >> INFO: WorkflowManager: Workflow FileConcatenatorWorkflow retrieved > >>for event fileconcatenator-pge Jun 09, 2015 4:34:29 PM > >>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > >> setRClient > >> INFO: Resource Manager Job Submission enabled to: > >>[http://localhost:9002] Jun > >> 09, 2015 4:34:29 PM > >> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread > >> checkTaskRequiredMetadata > >> INFO: All required metadata fields present for task: > >>[FileConcatenator] Jun 09, > >> 2015 4:34:29 PM > >> org.apache.oodt.cas.resource.system.XmlRpcResourceManagerClient > >>submitJob > >> FINEST: [{job.queueName=high, > >> job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, > >> job.name=urn:oodt:FileConcatenator, > >> job.id=, job.status=, job.load=2, > >> > >>job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, > >> > >>{task.instance.class=org.apache.oodt.pge.examples.fileconcatenator.Fil > >>eCo > >>ncaten > >> atorPGETask, > >> > >>task.config={PGETask_ConfigFilePath=/home/andrea/oodt/pge/file_concate > >>nat > >>or/ > >> pge-configs/PGEConfig.xml, > >> > >>PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatrans > >>fer > >>.Local > >> DataTransferFactory, > >> > >>PCS_ActionRepoFile=file:/home/andrea/oodt/crawler/policy/crawler-confi > >>g.x > >>ml, > >> PCS_MetFileExtension=met, PGETask_DumpMetadata=true, > >>PCS_WorkflowManagerUrl= http://localhost:9001, > >>PCS_FileManagerUrl=http://localhost:9000, > >> PGETask_Name=FileConcatenator}, > >> task.metadata={TaskId=[urn:oodt:FileConcatenator], > >>WorkflowManagerUrl=[ http://issa:9001], > >>JobId=[a2eecfc3-0eb4-11e5-ac1b-fff04e088d71], > >> ProcessingNode=[issa], RunID=[testNumber1], > >>WorkflowInstId=[a2eecfc3-0eb4- > >> 11e5-ac1b-fff04e088d71]}}] > >> > >> Not sure what it all means, but it may help :) > >> > >> Cheers, > >> Andrea > >> > >> On 9 June 2015 at 16:03, Andrea DeMarco <[email protected]> > >> wrote: > >> > >> > Hi Chris, > >> > > >> > No problem - I can imagine that checklist to be quite extensive :) > >> > > >> > I'll try and help as best I can: > >> > > >> > 1) The output when starting the batch_stub is: > >> > andrea@issa:~/oodt/resmgr/bin$ ./batch_stub 2001 [1] 2535 > >> > andrea@issa:~/oodt/resmgr/bin$ Jun 09, 2015 3:53:04 PM > >> > org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub <init> > >> > INFO: XmlRpc Batch Stub started by andrea > >> > > >> > 2) The latest oodt.out log file (after running the below) is > >> > attached > >>. > >> > > >> > 3) Kicking of the PGE with the following command (and terminal > >>output): > >> > > >> > andrea@issa:~/oodt/workflow/bin$ ./wmgr-client --url > >> > http://localhost:9001 --operation --sendEvent --eventName > >> > fileconcatenator-pge --metaData --key RunID testNumber1 Sending > >> > event 'fileconcatenator-pge'... SUCCESS > >> > > >> > I have now shut down my VM and restarted everything. When kicking > >> > off the PGE, this is the terminal output from the same terminal > >> > that started the > >> > batch_stub: > >> > > >> > Jun 09, 2015 3:57:26 PM > >> > org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub > >> > genericExecuteJob > >> > INFO: stub attempting to execute class: > >> > [org.apache.oodt.cas.workflow.structs.TaskJob] > >> > Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance > >> > createPgeMetadata > >> > INFO: Converting workflow configuration to static metadata... > >> > Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance > >> > createPgeMetadata > >> > INFO: Loading workflow context metadata... > >> > Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance > >> > createPgeConfig > >> > INFO: Create PgeConfig... > >> > Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance > >> > createPgeConfig > >> > INFO: Using default PgeConfigBuilder: > >> > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder > >> > Jun 09, 2015 3:57:26 PM org.apache.oodt.cas.pge.PGETaskInstance run > >> > SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to > >> > parse > >> > value: null > >> > java.io.IOException: Failed to build PgeConfig : Failed to parse > >>value: > >> > null > >> > at > >> > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFil > >> > ePg > >> > eConfigBuilder.java:87) > >> > at > >> > org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInst > >> > anc > >> > e.java:231) at > >> > org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:12 > >> > 3) > >> > at > >> > org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:1 > >> > 02) > >> > at > >> > org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$Runnable > >> > Job > >> > .run(XmlRpcBatchStub.java:229) at > >> > java.lang.Thread.run(Thread.java:745) > >> > Caused by: java.lang.Exception: Failed to parse value: null at > >> > org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > >> > at > >> > org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > >> > at > >> > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFil > >> > ePg > >> > eConfigBuilder.java:77) > >> > ... 5 more > >> > Caused by: java.lang.NullPointerException at > >> > java.util.regex.Matcher.getTextLength(Matcher.java:1234) > >> > at java.util.regex.Matcher.reset(Matcher.java:308) > >> > at java.util.regex.Matcher.<init>(Matcher.java:228) > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) > >> > at > >> > org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRe > >> > pla > >> > cement(PathUtils.java:321) > >> > at > >> > org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pa > >> > thU > >> > tils.java:96) at > >> > org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:486) > >> > ... 7 more > >> > > >> > org.apache.oodt.cas.resource.structs.exceptions.JobInputException: > >> > Failed to run task at > >> > org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:1 > >> > 04) > >> > at > >> > org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$Runnable > >> > Job > >> > .run(XmlRpcBatchStub.java:229) at > >> > java.lang.Thread.run(Thread.java:745) > >> > Caused by: > >> > > >> > >>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceEx > >>cep > >>tion > >> : > >> > PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: > >> > null at > >> > org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:15 > >> > 0) > >> > at > >> > org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:1 > >> > 02) > >> > ... 2 more > >> > Caused by: java.io.IOException: Failed to build PgeConfig : Failed > >> > to parse value: null at > >> > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFil > >> > ePg > >> > eConfigBuilder.java:87) > >> > at > >> > org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInst > >> > anc > >> > e.java:231) at > >> > org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:12 > >> > 3) > >> > ... 3 more > >> > Caused by: java.lang.Exception: Failed to parse value: null at > >> > org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) > >> > at > >> > org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) > >> > at > >> > org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFil > >> > ePg > >> > eConfigBuilder.java:77) > >> > ... 5 more > >> > Caused by: java.lang.NullPointerException at > >> > java.util.regex.Matcher.getTextLength(Matcher.java:1234) > >> > at java.util.regex.Matcher.reset(Matcher.java:308) > >> > at java.util.regex.Matcher.<init>(Matcher.java:228) > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088) > >> > at > >> > org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRe > >> > pla > >> > cement(PathUtils.java:321) > >> > at > >> > org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pa > >> > thU > >> > tils.java:96) at > >> > org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:486) > >> > ... 7 more > >> > > >> > > >> > > >> > Cheers, > >> > Andrea > >> > > >> > > >> > On 9 June 2015 at 15:47, Mattmann, Chris A (3980) < > >> > [email protected]> wrote: > >> > > >> >> Thanks Andrea, OK bear with me as we go through our checklist :) > >> >> > >> >> Does anything show in $OODT_HOME/log/oodt.log? > >> >> > >> >> How are you kicking off the PGE? Does the terminal in which you > >> >> started the batch stub print anything? > >> >> > >> >> Cheers, > >> >> Chris > >> >> > >> >> > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> +++++ > >> >> Chris Mattmann, Ph.D. > >> >> Chief Architect > >> >> Instrument Software and Science Data Systems Section (398) NASA > >> >> Jet Propulsion Laboratory Pasadena, CA 91109 USA > >> >> Office: 168-519, Mailstop: 168-527 > >> >> Email: [email protected] > >> >> WWW: http://sunset.usc.edu/~mattmann/ > >> >> > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> +++++ > >> >> Adjunct Associate Professor, Computer Science Department > >> >> University of Southern California, Los Angeles, CA 90089 USA > >> >> > >> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >> +++++ > >> >> > >> >> > >> >> > >> >> > >> >> -----Original Message----- > >> >> From: Andrea DeMarco <[email protected]> > >> >> Date: Tuesday, June 9, 2015 at 6:44 AM > >> >> To: jpluser <[email protected]> > >> >> Cc: "Wyngaard, Jane R (3980-Affiliate)" > >> >> <[email protected]>, "Starch, Michael D (398M)" > >> >> <[email protected]>, Kristian ZARB ADAMI > >> <[email protected]>, "Ramirez, Paul M (398M)" > >> >> <[email protected]>, Alessio Magro <[email protected]>, > >> >> "[email protected]" <[email protected]> > >> >> Subject: Re: OODT Documentation > >> >> > >> >> >Hi Chris, > >> >> > > >> >> > > >> >> >Yup, the jar file is present in both resmgr/lib and workflow/lib. > >> >> >You can have a look at my JAR contents at: > >> >> > > >> >> > > >> >> https://www.dropbox.com/s/1h0siibxf8sv5ut/fileconcatenator-pge-1.0 > >> >> .ja > >> >> r?dl= > >> >> >0 > >> >> >< > >> >> https://www.dropbox.com/s/1h0siibxf8sv5ut/fileconcatenator-pge-1.0 > >> >> .ja > >> >> r?dl > >> >> >=0> - perhaps something's off there? > >> >> > > >> >> > > >> >> >Cheers, > >> >> >Andrea > >> >> > > >> >> > >> >> > >> > > >> > > >> > >> > >> -- > >> Best Regards, > >> Andrea DeMarco > >> Institute of Space Sciences and Astronomy University of Malta >
