RE: re： Question about OODT file manager
Thanks Professor Mattmann, not running batch_stub was the main culprit and there were some other issues such as missing jars; and sorry for not confirming this right away, my laptop was actually crashing, and i just had time to fix it; BTW, I was able to get the cas-pge example to work, (even though I saw the workflow failed to pass the pre-condition in the log, the combined file and some metadata files (i.e.3 files) were still successfully ingested and placed in the output directory) BTW, i think there are a lot of mistakes in the documents, do you want us to help correct the document(i.e. https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example )? If possible, I would like to please share my notes with some problem steps mentioned there. Anyway, thanks for your help and appreciated. Thanks Luke -Original Message- From: Mattmann, Chris A (3980) [mailto:chris.a.mattm...@jpl.nasa.gov] Sent: Saturday, November 1, 2014 10:48 AM To: Luke; dev@oodt.apache.org Cc: 'Christian Alan Mattmann'; zhouj...@usc.edu; xiaoy...@usc.edu; 'Zichuan Wang' Subject: Re: re： Question about OODT file manager Dear Luke, just confirming, we solved this in class right? It had to do with the batch stub not being turned on. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Luke shuai...@usc.edu Date: Tuesday, October 28, 2014 at 12:52 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov, dev@oodt.apache.org dev@oodt.apache.org Cc: Chris Mattmann mattm...@usc.edu, zhouj...@usc.edu zhouj...@usc.edu, xiaoy...@usc.edu xiaoy...@usc.edu, 'Zichuan Wang' zichu...@usc.edu Subject: RE: re： Question about OODT file manager Dear Professor Mattamnn, Thanks a lot Professor Mattmann for the kind help, it is appreciated, sorry for getting back to you with my appreciation, I have been conducting tests with OODT based on your advice, but unfortunately I am having another problem I am following the steps (https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example ) to get a sense of how to get workflow to work. The problem is that the File-Concatenator-PGE (by running the wmgr-client command-line) does not seems to be invoked or executed, but I am seeing the tasks are getting stacked up in the workflow manager with status either RSUBMIT or QUEUED, but they are not getting executed, PFA: workflow_monitor.jpg, please note, by default the workflow min pool size is 6; so here comes another problem, i have 6 submitted tasks with status RSUBMIT, but any new incoming tasks will be forwarded to the waiting QUEUE with status QUEUED...please refer to the workflow_monitor.jpg for details, where I have 3 QUEUED workflow task and 6 RSUMBITE tasks. Question 1): not sure why the workflow is not being executed, and hanging at the state of RSUBMIT, after enabling the log level, I am seeing the following entry in the log, not sure if this has anything to do with the hanging problem where workflow is not getting executed and hanging at state of RSUBMIT. Oct 28, 2014 3:35:07 AM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread safeCheckJobComplete WARNING: Exception checking completion status for job: [2014-10-28T01:59:32.813-07:00]: Messsage: java.lang.Exception: java.lang.NullPointerException Question 2): I think currently on my side any new incoming workflow task I am sending with the following command is being directed to the waiting QUEUE because of the min pool size (i.e. 6) (I can increase this to a larger number though), ./wmgr-client --url http://localhost:9200 --operation --sendEvent --eventName fileconcatenator-pge --metaData --key RunID testNumber1 If possible, I would like to please know if there is a way we can purge the queue and get rid of those workflow tasks either in RSUMBIT and QUEUED I have already sent, please kindly help. Very sorry for troubling you with this, to be honest I find OODT a bit challenging to grasp within a short time frame, probably because there is no book like OODT in action like Solr and what I am doing is just trial and error blended with guess, but I don’t want to make a blind guess, it will be appreciated if you can please also shed some lights on where I can get more information logging or other way where I can troubleshoot. I think it might be worth tracking what is happening when workflow reach the status RSUBMIT and how to get a
Re: re： Question about OODT file manager
Yes Luke, making the instructions better would be much appreciated! If you have an account on the wiki please share it, else sign up for an Apache OODT wiki account and please share it with me or anyone else on dev@oodt and we’ll add you. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Luke liu shuai...@usc.edu Date: Sunday, November 2, 2014 at 1:32 AM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov, dev@oodt.apache.org dev@oodt.apache.org Cc: Chris Mattmann mattm...@usc.edu, zhouj...@usc.edu zhouj...@usc.edu, xiaoy...@usc.edu xiaoy...@usc.edu, 'Zichuan Wang' zichu...@usc.edu Subject: RE: re： Question about OODT file manager Thanks Professor Mattmann, not running batch_stub was the main culprit and there were some other issues such as missing jars; and sorry for not confirming this right away, my laptop was actually crashing, and i just had time to fix it; BTW, I was able to get the cas-pge example to work, (even though I saw the workflow failed to pass the pre-condition in the log, the combined file and some metadata files (i.e.3 files) were still successfully ingested and placed in the output directory) BTW, i think there are a lot of mistakes in the documents, do you want us to help correct the document(i.e. https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example )? If possible, I would like to please share my notes with some problem steps mentioned there. Anyway, thanks for your help and appreciated. Thanks Luke -Original Message- From: Mattmann, Chris A (3980) [mailto:chris.a.mattm...@jpl.nasa.gov] Sent: Saturday, November 1, 2014 10:48 AM To: Luke; dev@oodt.apache.org Cc: 'Christian Alan Mattmann'; zhouj...@usc.edu; xiaoy...@usc.edu; 'Zichuan Wang' Subject: Re: re： Question about OODT file manager Dear Luke, just confirming, we solved this in class right? It had to do with the batch stub not being turned on. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Luke shuai...@usc.edu Date: Tuesday, October 28, 2014 at 12:52 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov, dev@oodt.apache.org dev@oodt.apache.org Cc: Chris Mattmann mattm...@usc.edu, zhouj...@usc.edu zhouj...@usc.edu, xiaoy...@usc.edu xiaoy...@usc.edu, 'Zichuan Wang' zichu...@usc.edu Subject: RE: re： Question about OODT file manager Dear Professor Mattamnn, Thanks a lot Professor Mattmann for the kind help, it is appreciated, sorry for getting back to you with my appreciation, I have been conducting tests with OODT based on your advice, but unfortunately I am having another problem I am following the steps (https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Exampl e ) to get a sense of how to get workflow to work. The problem is that the File-Concatenator-PGE (by running the wmgr-client command-line) does not seems to be invoked or executed, but I am seeing the tasks are getting stacked up in the workflow manager with status either RSUBMIT or QUEUED, but they are not getting executed, PFA: workflow_monitor.jpg, please note, by default the workflow min pool size is 6; so here comes another problem, i have 6 submitted tasks with status RSUBMIT, but any new incoming tasks will be forwarded to the waiting QUEUE with status QUEUED...please refer to the workflow_monitor.jpg for details, where I have 3 QUEUED workflow task and 6 RSUMBITE tasks. Question 1): not sure why the workflow is not being executed, and hanging at the state of RSUBMIT, after enabling the log level, I am seeing the following entry in the log, not sure if this has anything to do with the hanging problem where workflow is not getting executed and hanging at state of RSUBMIT. Oct 28, 2014 3:35:07 AM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread safeCheckJobComplete WARNING: Exception checking completion status for job:
NSF DataViz Hackathon for Polar CyberInfrastructure: New York, NY 11/3/2014 - 11/4/2014 Call for Remote Participation
Call for Remote Participation The NSF DataViz Hackathon for Polar CyberInfrastructure will bring together Polar researchers, Cyber Infrastructure experts, Data Visualiation experts, and members of the community interested in connecting technology, science and communication. The Hackathon website is located at: http://nsf-polar-cyberinfrastructure.github.io/datavis-hackathon/ The Hackathon will take place tomorrow, Monday, November 3, 2014 and Tuesday November 4, 2014, beginning at 9am ET and located at: The Orozco Room Parsons - The New School 66 W 12th St, 7th floor (Between Fifth Avenue and Sixth Avenue) New York, NY 10011 Though onsite attendance is closed, we are offering remote participation in the meeting via our Github repository: https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon You can participate in the meeting by reviewing, commenting and providing your feedback on our current sessions, located at: https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon/issues Please note that a Github account is required to participate. We look forward to your remote participation and to the hackathon and its results! Cheers, Chris Mattmann ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: Could not mvn install OODT from the trunk
Do you have any OODT related environment variables set? Tyler On Wed, Oct 29, 2014 at 7:31 PM, MengYing Wang mengyingwa...@gmail.com wrote: Hey, $ mvn -version Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700) Java version: 1.7.0_51 Java home: /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: mac os x version: 10.9.2 arch: x86_64 Family: mac Yes. The error is repeatable on my pc. I have tried several times, all failed with the same error. Best, Mengying (Angela) Wang On Wed, Oct 29, 2014 at 4:06 PM, Michael Starch starc...@umich.edu wrote: MengYing, I cannot duplicate it on Mac OS X 10.9.5, Java 7 nor CentOS with Java 7. Is the error repeatable on your end? Michael On Oct 29, 2014 4:01 PM, MengYing Wang mengyingwa...@gmail.com wrote: Hi Michael, $ java -version java version 1.7.0_51 Java(TM) SE Runtime Environment (build 1.7.0_51-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode). Thanks. Best, Mengying Wang On Wed, Oct 29, 2014 at 6:52 AM, Michael Starch starc...@umich.edu wrote: Mengying, I just tested this on Mac 1.9.5 using Java 7 and had no issues. Can you tell me your java version? In the meantime I will test with Java 6 as I suspect that this might be an issue. --Michael Hi Everyone, This is my command sequence and some log: $svn co http://svn.apache.org/repos/asf/oodt/trunk/ oodt_trunk $cd oodt_trunk/ $mvn clean install .. --- T E S T S --- Running org.apache.oodt.cas.resource.mux.TestQueueMuxMonitor Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.277 sec FAILURE! Running org.apache.oodt.cas.resource.queuerepo.TestXmlQueueRepository Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.182 sec Running org.apache.oodt.cas.resource.monitor.TestGangliaResourceMonitor Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.41 sec Running org.apache.oodt.cas.resource.cli.TestResourceCli Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.848 sec Running org.apache.oodt.cas.resource.jobqueue.TestJobStack Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 sec Running org.apache.oodt.cas.resource.system.TestXmlRpcResourceManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.346 sec Running org.apache.oodt.cas.resource.monitor.TestGangliaXMLParser Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.327 sec Running org.apache.oodt.cas.resource.mux.TestQueueMuxBatchmgr Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec Running org.apache.oodt.cas.resource.monitor.TestAssignmentMonitor Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.248 sec Running org.apache.oodt.cas.resource.util.TestUlimit Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec Results : Failed tests: Tests run: 48, Failures: 1, Errors: 0, Skipped: 0 [INFO] [ERROR] BUILD FAILURE [INFO] [INFO] There are test failures. Please refer to /Users/AngelaWang/Downloads/oodt_trunk/resource/target/surefire-reports for the individual test results. .. Attached is the detailed test result for the failed TestQueueMuxMonitor test. I am using Mac OS X 10.9.2. Thank you for your help! -- Best, Mengying (Angela) Wang -- Best, Mengying (Angela) Wang -- Best, Mengying (Angela) Wang
Re: Could not mvn install OODT from the trunk
I use maven 3.2 to build OODT. This might help. +1
Re: NSF DataViz Hackathon for Polar CyberInfrastructure: New York, NY 11/3/2014 - 11/4/2014 Call for Remote Participation
Looks like a great event! On 02/11/14 17:37, Mattmann, Chris A (3980) wrote: Call for Remote Participation The NSF DataViz Hackathon for Polar CyberInfrastructure will bring together Polar researchers, Cyber Infrastructure experts, Data Visualiation experts, and members of the community interested in connecting technology, science and communication. The Hackathon website is located at: http://nsf-polar-cyberinfrastructure.github.io/datavis-hackathon/ The Hackathon will take place tomorrow, Monday, November 3, 2014 and Tuesday November 4, 2014, beginning at 9am ET and located at: The Orozco Room Parsons - The New School 66 W 12th St, 7th floor (Between Fifth Avenue and Sixth Avenue) New York, NY 10011 Though onsite attendance is closed, we are offering remote participation in the meeting via our Github repository: https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon You can participate in the meeting by reviewing, commenting and providing your feedback on our current sessions, located at: https://github.com/NSF-Polar-Cyberinfrastructure/datavis-hackathon/issues Please note that a Github account is required to participate. We look forward to your remote participation and to the hackathon and its results! Cheers, Chris Mattmann ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -- *Tom Barber* | Technical Director meteorite bi *T:* +44 20 8133 3730 *W:* www.meteorite.bi | *Skype:* meteorite.consulting *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK
Re: RADiX filemgr-client script cannot delete product by id or name
Sounds like a bug in RADiX. MengYing would you mind adding an issue. The filemgr-client script should source the env.sh so that this variable would be set. --Paul Sent from my iPhone On Nov 1, 2014, at 7:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Yes, you have to reingest your data after restarting your file manager after deleting the catalog. Then try the query tool ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: MengYing Wang mengyingwa...@gmail.com Date: Saturday, November 1, 2014 at 1:47 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: dev@oodt.apache.org dev@oodt.apache.org, Chris Mattmann mattm...@apache.org Subject: Re: RADiX filemgr-client script cannot delete product by id or name Dear Prof. Mattmann, Now the ./query-tool sql query does work too. -_-! $ ./query-tool --url http://localhost:9000 --sql -query SELECT * FROM GenericFile org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClie ntResponseProcessor.java:104) at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClien tResponseProcessor.java:71) at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(Xm lRpcFileManagerClient.java:1003) at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java :252) at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242) Exception in thread main org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(Xm lRpcFileManagerClient.java:1009) at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java :252) at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:242) Best, Mengying (Angela) Wang On Sat, Nov 1, 2014 at 1:09 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Can you please remove your FILEMGR_HOME/catalog (or your RADIX_HOME/data/catalog) directory, restart file manager, and then try again? ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: MengYing Wang mengyingwa...@gmail.com Date: Saturday, November 1, 2014 at 1:03 PM To: dev@oodt.apache.org dev@oodt.apache.org Cc: Chris Mattmann mattm...@apache.org Subject: Re: RADiX filemgr-client script cannot delete product by id or name Dear Prof. Mattmann, Thank you for your help! Actually, I have set the FILEMGR_URL. Below are the variables that I have set: export OODT_HOME=/Users/AngelaWang/Documents/programs/oodt-0.7 export CRAWLER_HOME=/Users/AngelaWang/Documents/programs/oodt-0.7/crawler export FILEMGR_HOME=/Users/AngelaWang/Documents/programs/oodt-0.7/filemgr export WORKFLOW_HOME=/Users/AngelaWang/Documents/programs/oodt-0.7/workflow export PGE_ROOT=/Users/AngelaWang/Documents/programs/oodt-0.7/pge export WORKFLOW_URL=http://localhost:9200; export FILEMGR_URL=http://localhost:9000; Best, Mengying (Angela) Wang On Sat, Nov 1, 2014 at 10:46 AM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: This is an environment variable issue - your FILEMGR_URL is not set. Can you check it Angela? Cheers, Chris
Re: Assignment2: CAS-PGE file concontenator example FAILURE
Hmm, so it sounds like your workflow manager is running, on port 9001, and that you are sending an event to it via the workflow manager client, hopefully via http://localhost:9001, correct? If so, you should see the notice you received in the workflow logs, e.g., Event received for.. My question is - did you copy the appropriate policy into events.xml from the file concatenate pge example? You need that event to be mapped to your specific workflow and it sounds like it isn¹t. Please also include dev@oodt.apache.org on your emails. Thanks. Cheers, Chris Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Aakarsh Medleri Hire Math medle...@usc.edu Date: Sunday, November 2, 2014 at 10:44 AM To: Chris Mattmann mattm...@usc.edu Cc: Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu Subject: Re: Assignment2: CAS-PGE file concontenator example FAILURE It doesnt give me any other error log. The only log I get is in workflow as below: Nov 01, 2014 5:35:46 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent INFO: WorkflowManager: Received event: fileconcatenator-pge The failure doesnt say anything but sending failure like the one in the snapshot i sent earlier. I dont say any events/workflow tasks in the console as well. It is empty. Thanks, Aakarsh On Sun, Nov 2, 2014 at 9:40 AM, Christian Alan Mattmann mattm...@usc.edu wrote: Hm ok workflow manager is running. That verifies that. Can you suggest what the failure you are getting is? Cheers, Chris Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Aakarsh Medleri Hire Math medle...@usc.edu Date: Sunday, November 2, 2014 at 10:37 AM To: Chris Mattmann mattm...@usc.edu Subject: Re: Assignment2: CAS-PGE file concontenator example FAILURE telnet localhost 9001 gives this professor: telnet localhost 9001 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. On Sun, Nov 2, 2014 at 8:38 AM, Christian Alan Mattmann mattm...@usc.edu wrote: Sorry I meant telnet localhost 9001 Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Aakarsh Medleri Hire Math medle...@usc.edu Date: Saturday, November 1, 2014 at 11:18 PM To: Chris Mattmann mattm...@usc.edu Subject: Re: Assignment2: CAS-PGE file concontenator example FAILURE Worflow, Resource File managers are all running professor. telnet 9001 gives me this: aakarsh@aakarsh-VirtualBox:~/Desktop/csci572/oodt-07/oodt-deploy/workflo w $ telnet 9001 Trying 0.0.35.41... telnet: Unable to connect to remote host: Invalid argument Thanks, Aakarsh On Sat, Nov 1, 2014 at 10:08 PM, Christian Alan Mattmann mattm...@usc.edu wrote: Can you please make sure your Workflow Manager is running? What happens when you telnet 9001? Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Aakarsh Medleri Hire Math medle...@usc.edu Date: Saturday, November 1, 2014 at 6:55 PM To: Chris Mattmann mattm...@usc.edu Subject: Assignment2: CAS-PGE file concontenator example FAILURE Hi Professor, I am trying to invoke FileConcatenatorPGE by running the wmgr-client and I get a failure as below: However, there is no log on what exactly is causing the failure. I check oodt.out, worflow, file and resource manager logs. There is no indication of any errors. Resource manager batch_stub is also running on port 2001. I am stuck here and have exhausted all possible options I could find to resolve this. Could you please help me figure this out? Note: I am running OODT 0.7 and following environment variables are being set. export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386/ export JCC_JDK=/usr/lib/jvm/java-7-openjdk-i386/ export JRE_HOME=/usr/lib/jvm/java-7-openjdk-i386/jre/ export OODT_HOME=/home/aakarsh/Desktop/csci572/oodt-07/oodt-deploy/ export
Re: FileConcatenator not generating output
Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edumailto:jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: TikaCmdLineMetExtractor does not generate .met file
Hi Zichen, Thanks for your mail. If you use MetExtractorProductCrawler, met is generated, but it¹s never serialized to disk. I think that explains it. Let me know if that makes sense. Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Zichen Nie zichen...@gmail.com Date: Saturday, November 1, 2014 at 4:47 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Subject: TikaCmdLineMetExtractor does not generate .met file Dear Professor: I followed the instruction on how to use OODT cas-crawler, and tried to generate .met file using TikaCmdLineExtractor. I can see from the log that Tika is extracting my metadata but it does not generate .met file for my json file. Here is my command line: ./crawler_launcher --operation --launchMetCrawler -filemgrUrl http://localhost:9000 http://localhost:9000 --clientTransferer org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory --productPath /Users/threeears/Documents/572/Assignment2/oodt-deploy/cas-crawler-0.7/dat a/test/0.json --metExtractor org.apache.oodt.cas.metadata.extractors.TikaCmdLineMetExtractor --metExtractorConfig /Users/threeears/Documents/572/Assignment2/oodt-deploy/cas-crawler-0.7/ext ractors/tikaextractor/tikaextractor.config --metFileExtension met I thought MetCrawler should generate meta file before ingestion, it's weird that my ingestion is successful and met file is not shown. Am I using the right extractor and crawler? Are there any necessary configurations that I missed? Best, Zichen
Re: FileConcatenator not generating output
Hi Professor, I have defined the PGE_ROOT. Infact I have set them in my .bashrc file.so it ll be set always My PGE_ROOT is defined as below. root@homework1:~# echo $PGE_ROOT /root/oodt-deploy/cas-pge-0.7 cas-pge-0.7 folder has filecontenator folder where jobs output should reside. Not sure what I am missing Thanks a lot helping us with this debugging. Thanks Ravi On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann mattm...@usc.edu wrote: I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: Chris Mattmann mattm...@usc.edu, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: Regarding workflow
Hi Meghamsh, Please try again with 0.7 oodt, and you should be good. Cheers, Chris Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Meghamsh Utkur ut...@usc.edu Date: Sunday, November 2, 2014 at 7:03 PM To: Chris Mattmann mattm...@usc.edu Subject: Regarding workflow Hello Professor, I was working with the pge concatenator example specified in the workflow manager wiki page.I have installed the 0.6 version of oodt .I came across a post which said that inorder to make workflow manager work I have to change the version of oodt to 0.7/0.3 and reinstall workflow pge concatenator.I would like to know if even oodt needs to be installed again after making the chanes to pom.xml? For some reason I still get resource not found while installing pge concatenator. can you please throw some light on it. Thanks and Regards Meghamsh Utkur -- Meghamsh UtkurGraduate Student,University of Southern California ut...@usc.edu (503)881-4791
Re: FileConcatenator not generating output
I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: Chris Mattmann mattm...@usc.edu, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: Error while executing CAS-PGE
I think you are missing a variable that you are referencing in your PgeConfig.xml. Can you please put up a paste.bin and send the link? Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Saurabh Damodar Patwardhan patwa...@usc.edu Date: Sunday, November 2, 2014 at 3:19 PM To: Chris Mattmann mattm...@usc.edu Subject: Error while executing CAS-PGE Hi Professor, I am trying to execute the CASPGE example and have added appropriate jars in workflow and resource manager directories. But I am facing an issue in the batch stub when I try to run the file concatenation example: SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: null java.io.IOException: Failed to build PgeConfig : Failed to parse value: null at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon figBuilder.java:87) at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja va:230) at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123) at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102) at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run (XmlRpcBatchStub.java:229) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.Exception: Failed to parse value: null at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501) at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480) at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCon figBuilder.java:77) Caused by: java.lang.NullPointerException at java.util.regex.Matcher.getTextLength(Matcher.java:1234) at java.util.regex.Matcher.reset(Matcher.java:308) at java.util.regex.Matcher.init(Matcher.java:228) at java.util.regex.Pattern.matcher(Pattern.java:1088) at org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisReplaceme nt(PathUtils.java:321) at org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(PathUtils .java:96) at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488) Please let me know, if I am doing anything incorrect. Warm Regards, Saurabh Patwardhan Masters in Computer Science University of Southern California
Re: Assignment2: CAS-PGE file concontenator example FAILURE
It worked, see the message: INFO: Successful ingest of product: [/home/aakarsh/Desktop/csci572/oodt-07/oodt-deploy/pge/file_concatenator/ou tput/jobs/job-2014-11-02T21:21:03.991Z/concatenatedOutputFile-2014-11-02T21 :21:03.991Z.txt] Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Aakarsh Medleri Hire Math medle...@usc.edu Date: Sunday, November 2, 2014 at 4:36 PM To: Chris Mattmann mattm...@usc.edu Cc: Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev@oodt.apache.org dev@oodt.apache.org Subject: Re: Assignment2: CAS-PGE file concontenator example FAILURE Sorry Professor. I had missed a tag in events.xml because of which it was not identifying the task. I got couple of more errors with respect to jars not being in right place. After fixing them the job went through. Now I can see the concatenated file in the output folder, however i dont get the message saying file was ingested. I get as below. Will this be a problem or am I good to go? - aakarsh@aakarsh-VirtualBox:~/Desktop/csci572/oodt-07/oodt-deploy/workflow/ bin$ ./wmgr-client --url http://localhost:9001 --operation --sendEvent --eventName fileconcatenator-pge --metaData --key RunID testNumber1 Sending event 'fileconcatenator-pge'... SUCCESS aakarsh@aakarsh-VirtualBox:~/Desktop/csci572/oodt-07/oodt-deploy/workflow/ bin$ -- - I checked the logs for batch_stub and I see this: -- - INFO: Successfully ingested product: [/home/aakarsh/Desktop/csci572/oodt-07/oodt-deploy/pge/file_concatenator/o utput/jobs/job-2014-11-02T21:21:03.991Z/concatenatedOutputFile-2014-11-02T 21:21:03.991Z.txt]: product id: 29919f44-62d6-11e4-8736-abf940e2f70f Nov 02, 2014 1:21:09 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile INFO: Successful ingest of product: [/home/aakarsh/Desktop/csci572/oodt-07/oodt-deploy/pge/file_concatenator/o utput/jobs/job-2014-11-02T21:21:03.991Z/concatenatedOutputFile-2014-11-02T 21:21:03.991Z.txt] Nov 02, 2014 1:21:09 PM org.apache.oodt.cas.pge.PGETaskInstance verifyIngests WARNING: Product was not ingested [file='/home/aakarsh/Desktop/csci572/oodt-07/oodt-deploy/pge/file_concaten ator/output/jobs/job-2014-11-02T21:21:03.991Z/sciPgeExeScript_FileConcaten ator',result='PRECONDS_FAILED',msg='Failed to pass preconditions'] Nov 02, 2014 1:21:09 PM org.apache.oodt.cas.pge.PGETaskInstance verifyIngests WARNING: Product was not ingested [file='/home/aakarsh/Desktop/csci572/oodt-07/oodt-deploy/pge/file_concaten ator/output/jobs/job-2014-11-02T21:21:03.991Z/concatenatedOutputFile-2014- 11-02T21:21:03.991Z.txt.met',result='PRECONDS_FAILED',msg='Failed to pass preconditions'] -- - Thanks, Aakarsh On Sun, Nov 2, 2014 at 11:56 AM, Christian Alan Mattmann mattm...@usc.edu wrote: Hmm, so it sounds like your workflow manager is running, on port 9001, and that you are sending an event to it via the workflow manager client, hopefully via http://localhost:9001, correct? If so, you should see the notice you received in the workflow logs, e.g., Event received for.. My question is - did you copy the appropriate policy into events.xml from the file concatenate pge example? You need that event to be mapped to your specific workflow and it sounds like it isn¹t. Please also include dev@oodt.apache.org on your emails. Thanks. Cheers, Chris Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Aakarsh Medleri Hire Math medle...@usc.edu Date: Sunday, November 2, 2014 at 10:44 AM To: Chris Mattmann mattm...@usc.edu Cc: Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu Subject: Re: Assignment2: CAS-PGE file concontenator example FAILURE It doesnt give me any other error log. The only log I get is in workflow as below: Nov 01, 2014 5:35:46 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager handleEvent INFO: WorkflowManager: Received event: fileconcatenator-pge The failure doesnt say anything but sending failure like the one in the snapshot i sent earlier. I dont say any events/workflow tasks in the console as well. It is empty. Thanks, Aakarsh On Sun, Nov 2, 2014 at 9:40 AM, Christian Alan Mattmann mattm...@usc.edu wrote: Hm ok workflow manager
Re: FileConcatenator not generating output
Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- *dir path=[JobDir] createBeforeExe=false* Thanks Ravi ?xml version=1.0 encoding=UTF-8? pgeConfig !-- How to run the PGE -- exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=true !-- one or more of these ** regExp or name can be used-- files regExp=.*\.txt metFileWriterClass=org.apache.oodt.pge.examples.fileconcatenator.writers.ConcactenatingFilenameExtractorWriter args=[PGE_ROOT]/file_concatenator/extractors/concatenatingfilename.extractor.config.xml/ files regExp=.*\.txt metFileWriterClass=org.apache.oodt.cas.pge.writers.metlist.MetadataListPcsMetFileWriter args=[PGE_ROOT]/file_concatenator/extractors/metlistwriter/metout.xml/ /dir /output !-- Custom metadata to add to output files -- customMetadata !-- helpful keys -- metadata key=LessThan val=#x3C;/ metadata key=LessThanOrEqualTo val=[LessThan]=/ metadata key=GreaterThan val=#x3E;/ metadata key=GreaterThanOrEqualTo val=[GreaterThan]=/ metadata key=Exclamation val=#33;/ metadata key=Ampersand val=#38;/ metadata key=NotEqualTo val=[Ampersand]=/ metadata key=LogicalAnd val=[Ampersand][Ampersand]/ metadata key=CshPipeToStdOutAndError val=[GreaterThan][Ampersand][Exclamation]/ metadata key=ProductionDateTime val=[DATE.UTC]/ metadata key=JobDir val=[PGE_ROOT]/file_concatenator/output/jobs/job-[ProductionDateTime]/ metadata key=InputFile1 val=[PGE_ROOT]/file_concatenator/files/concatenatingInputFile1.txt/ metadata key=InputFile2 val=[PGE_ROOT]/file_concatenator/files/concatenatingInputFile2.txt/ metadata key=OutputFile val=[PGE_ROOT]/file_concatenator/output/jobs/concatenatedOutputFile-[ProductionDateTime].txt/ /customMetadata /pgeConfig
Re: cwiki-CAS-PGE-learn by example
Sure can you update the page? You just need a wiki account which you can register for. ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Dongni Zhao dongn...@usc.edu Date: Sunday, November 2, 2014 at 3:07 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Subject: cwiki-CAS-PGE-learn by example Hi, Professor: It seems like an error happened on page :https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Learn+by+Example where: Is that throws instead of thows? Thanks. Dongni Zhao M.S. in Computer Science University of Southern California
Intership / Mobile Application Project
Dear students, my name is Guillaume, I am a French student studying abroad at Dornsife, and trying to build a mobile application called Trace during this academic year that predicts its users about future meeting opportunities between friends who do not live in the same area. I am in search of a developer as co-founder or co-worker that is interested in solving the problem of missing out meeting opportunities. For more information, visit trace.strikingly.com http://s.bl-1.com/h/kCGrg7Q or simply send me an e-mail to perd...@usc.edu. I am looking forward to meeting all of you who are interested ! All the best, Guillaume
