Re: FileConcatenator not generating output
Hi Ravi, are you still having trouble here? Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 10:00 PM To: Chris Mattmann mattm...@usc.edu Cc: Chris Mattmann chris.a.mattm...@jpl.nasa.gov, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I have defined the PGE_ROOT. Infact I have set them in my .bashrc file.so it ll be set always My PGE_ROOT is defined as below. root@homework1:~# echo $PGE_ROOT /root/oodt-deploy/cas-pge-0.7 cas-pge-0.7 folder has filecontenator folder where jobs output should reside. Not sure what I am missing Thanks a lot helping us with this debugging. Thanks Ravi On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann mattm...@usc.edu wrote: I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: Chris Mattmann mattm...@usc.edu, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: FileConcatenator not generating output
Great to hear that this is working now Ravi. Note that yes the File Manager (and Workflow Manager) will have the metrics on ingestion since CAS-PGE will help to capture this metadata for you while running ETLLib. Cheers! Chris Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Wednesday, November 5, 2014 at 6:14 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Chris Mattmann mattm...@usc.edu, Gouthami Kondakindi konda...@usc.edu, dev dev@oodt.apache.org, Preethi Ramesh pram...@usc.edu, rmura...@usc.edu rmura...@usc.edu Subject: Re: FileConcatenator not generating output Hi Professor, It works now.Looks like the wiki page was recently updated and when I followed that steps (which includes downloading .7 version of oodt ) it works fine . Thanks for checking with us. We were wondering how the overall picture looks like ? As per our current understanding * Json files will be injected into the filemgr * Workflow will be used to execute ETLlib which indexe JSON files from the filemgr archive into solr. * Solr's searh algo will be modified as per our requirement. But if this is the case how come filemgr will have metric on solr ingestion? Its little confusing here. Thanks Ravikumar Jeyaraman On Nov 5, 2014 4:59 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Hi Ravi, are you still having trouble here? Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 10:00 PM To: Chris Mattmann mattm...@usc.edu Cc: Chris Mattmann chris.a.mattm...@jpl.nasa.gov, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I have defined the PGE_ROOT. Infact I have set them in my .bashrc file.so it ll be set always My PGE_ROOT is defined as below. root@homework1:~# echo $PGE_ROOT /root/oodt-deploy/cas-pge-0.7 cas-pge-0.7 folder has filecontenator folder where jobs output should reside. Not sure what I am missing Thanks a lot helping us with this debugging. Thanks Ravi On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann mattm...@usc.edu wrote: I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: Chris Mattmann mattm...@usc.edu, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end
Re: FileConcatenator not generating output
Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edumailto:jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: FileConcatenator not generating output
Hi Professor, I have defined the PGE_ROOT. Infact I have set them in my .bashrc file.so it ll be set always My PGE_ROOT is defined as below. root@homework1:~# echo $PGE_ROOT /root/oodt-deploy/cas-pge-0.7 cas-pge-0.7 folder has filecontenator folder where jobs output should reside. Not sure what I am missing Thanks a lot helping us with this debugging. Thanks Ravi On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann mattm...@usc.edu wrote: I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: Chris Mattmann mattm...@usc.edu, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: FileConcatenator not generating output
I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman jeyar...@usc.edu Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann chris.a.mattm...@jpl.nasa.gov Cc: Chris Mattmann mattm...@usc.edu, rmura...@usc.edu rmura...@usc.edu, saish...@usc.edu saish...@usc.edu, Vineet Ghatge Hemantkumar heman...@usc.edu, Gouthami Kondakindi konda...@usc.edu, Preethi Ramesh pram...@usc.edu, dev dev@oodt.apache.org Subject: Re: FileConcatenator not generating output Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=false Thanks Ravi
Re: FileConcatenator not generating output
Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, Ravikumar Jeyaraman jeyar...@usc.edu wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- *dir path=[JobDir] createBeforeExe=false* Thanks Ravi ?xml version=1.0 encoding=UTF-8? pgeConfig !-- How to run the PGE -- exe dir=[JobDir] shell=/bin/bash !-- cd to PGE root -- cmdcd [PGE_ROOT]/file_concatenator/cmd cmdcp [InputFile1] [OutputFile]/cmd cmdcat [InputFile2] [OutputFile]/cmd /exe !-- Files to ingest -- output !-- one or more of these -- dir path=[JobDir] createBeforeExe=true !-- one or more of these ** regExp or name can be used-- files regExp=.*\.txt metFileWriterClass=org.apache.oodt.pge.examples.fileconcatenator.writers.ConcactenatingFilenameExtractorWriter args=[PGE_ROOT]/file_concatenator/extractors/concatenatingfilename.extractor.config.xml/ files regExp=.*\.txt metFileWriterClass=org.apache.oodt.cas.pge.writers.metlist.MetadataListPcsMetFileWriter args=[PGE_ROOT]/file_concatenator/extractors/metlistwriter/metout.xml/ /dir /output !-- Custom metadata to add to output files -- customMetadata !-- helpful keys -- metadata key=LessThan val=#x3C;/ metadata key=LessThanOrEqualTo val=[LessThan]=/ metadata key=GreaterThan val=#x3E;/ metadata key=GreaterThanOrEqualTo val=[GreaterThan]=/ metadata key=Exclamation val=#33;/ metadata key=Ampersand val=#38;/ metadata key=NotEqualTo val=[Ampersand]=/ metadata key=LogicalAnd val=[Ampersand][Ampersand]/ metadata key=CshPipeToStdOutAndError val=[GreaterThan][Ampersand][Exclamation]/ metadata key=ProductionDateTime val=[DATE.UTC]/ metadata key=JobDir val=[PGE_ROOT]/file_concatenator/output/jobs/job-[ProductionDateTime]/ metadata key=InputFile1 val=[PGE_ROOT]/file_concatenator/files/concatenatingInputFile1.txt/ metadata key=InputFile2 val=[PGE_ROOT]/file_concatenator/files/concatenatingInputFile2.txt/ metadata key=OutputFile val=[PGE_ROOT]/file_concatenator/output/jobs/concatenatedOutputFile-[ProductionDateTime].txt/ /customMetadata /pgeConfig
