Re: FileConcatenator not generating output
Hi Professor, It works now.Looks like the wiki page was recently updated and when I followed that steps (which includes downloading .7 version of oodt ) it works fine . Thanks for checking with us. We were wondering how the overall picture looks like ? As per our current understanding - Json files will be injected into the filemgr - Workflow will be used to execute ETLlib which indexe JSON files from the filemgr archive into solr. - Solr's searh algo will be modified as per our requirement. But if this is the case how come filemgr will have metric on solr ingestion? Its little confusing here. Thanks Ravikumar Jeyaraman On Nov 5, 2014 4:59 PM, "Mattmann, Chris A (3980)" < chris.a.mattm...@jpl.nasa.gov> wrote: > Hi Ravi, > are you still having trouble here? > > Cheers, > Chris > > > > ++ > Chris Mattmann, Ph.D. > Chief Architect > Instrument Software and Science Data Systems Section (398) > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 168-519, Mailstop: 168-527 > Email: chris.a.mattm...@nasa.gov > WWW: http://sunset.usc.edu/~mattmann/ > ++ > Adjunct Associate Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++ > > > > > > > -Original Message- > From: Ravikumar Jeyaraman > Date: Sunday, November 2, 2014 at 10:00 PM > To: Chris Mattmann > Cc: Chris Mattmann , "rmura...@usc.edu" > , "saish...@usc.edu" , Vineet Ghatge > Hemantkumar , Gouthami Kondakindi , > Preethi Ramesh , dev > Subject: Re: FileConcatenator not generating output > > >Hi Professor, > > > > I have defined the PGE_ROOT. Infact I have set them in > >my .bashrc file.so it ll be set always > > > >My PGE_ROOT is defined as below. > > > >root@homework1:~# echo $PGE_ROOT > >/root/oodt-deploy/cas-pge-0.7 > > > > > >cas-pge-0.7 folder has filecontenator folder where jobs output should > >reside. > > > > > >Not sure what I am missing > > > > > >Thanks a lot helping us with this debugging. > > > > > >Thanks > > > >Ravi > > > > > >On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann > > wrote: > > > >I think your PGE_ROOT environment variable is not defined. > >Can you try echo $PGE_ROOT and tell me what it shows? If it¹s > >not defined, please define it, then restart OODT and try again. > > > > > >Chris Mattmann, Ph.D. > >Adjunct Associate Professor, Computer Science Department > >University of Southern California > >Los Angeles, CA 90089 USA > >Email: mattm...@usc.edu > >WWW: http://sunset.usc.edu/~mattmann/ > >++++ > > > > > > > > > >-Original Message- > >From: Ravikumar Jeyaraman > >Date: Sunday, November 2, 2014 at 3:13 PM > >To: Chris Mattmann > >Cc: Chris Mattmann , "rmura...@usc.edu" > >, "saish...@usc.edu" , Vineet Ghatge > >Hemantkumar , Gouthami Kondakindi , > >Preethi Ramesh , dev > >Subject: Re: FileConcatenator not generating output > > > >>Hi Professor, > >> > >> I tried setting JobDir to > >>$PGE_ROOT/file_concatenator/output/jobs manually. > >> > >> > >>After executing JobDir is still a empty dir has nothing in it. > >> > >> > >>I have attached my PGEConfig.xml with this mail in case you need it. > >> > >> > >> > >>On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) > >> wrote: > >> > >> > >>Thanks Ravi can u show what the job dir has in it? > >> > >>Sent from my iPhone > >> > >>On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" > >>wrote: > >> > >> > >> > >>Hi Professor, > >> > >> We are still trying out file concatenate work-flow > >>example . We were able to execute it with out error. But concatenated > >>output file is not generated at the end of execution. > >> > >> > >> > >>We are doubting some path setting in PGEConfig.xml . > >> > >> > >>I remember you suggested one of the student to change createBeforeExe to > >>true. I tried that also. But still files are not generated. I also tried > >>setting manually JobDir env variable. Still no luck . Any suggestions ? > >> > >> > >> > >>cd [PGE_ROOT]/file_concatenator > >>cp [InputFile1] [OutputFile] > >>cat [InputFile2] >> [OutputFile] > >> > >> > >> > >> > >> > >> > >> > >> > >>Thanks > >> > >>Ravi > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > > > > > > > > > > > > > > > > > >
Re: FileConcatenator not generating output
Great to hear that this is working now Ravi. Note that yes the File Manager (and Workflow Manager) will have the metrics on ingestion since CAS-PGE will help to capture this metadata for you while running ETLLib. Cheers! Chris Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman Date: Wednesday, November 5, 2014 at 6:14 PM To: Chris Mattmann Cc: "saish...@usc.edu" , Vineet Ghatge Hemantkumar , Chris Mattmann , Gouthami Kondakindi , dev , Preethi Ramesh , "rmura...@usc.edu" Subject: Re: FileConcatenator not generating output >Hi Professor, > It works now.Looks like the wiki page was recently >updated and when I followed that steps (which includes downloading .7 >version of oodt ) it works fine . >Thanks for checking with us. > >We were wondering how the overall picture looks like ? >As per our current understanding > > >* Json files will be injected into the filemgr >* Workflow will be used to execute ETLlib which indexe JSON files from >the filemgr archive into solr. > >* Solr's searh algo will be modified as per our requirement. > > > > >But if this is the case how come filemgr will have metric on solr >ingestion? Its little confusing here. > > >Thanks > >Ravikumar Jeyaraman >On Nov 5, 2014 4:59 PM, "Mattmann, Chris A (3980)" > wrote: > >Hi Ravi, >are you still having trouble here? > >Cheers, >Chris > > > >++ >Chris Mattmann, Ph.D. >Chief Architect >Instrument Software and Science Data Systems Section (398) >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >Office: 168-519, Mailstop: 168-527 >Email: chris.a.mattm...@nasa.gov >WWW: http://sunset.usc.edu/~mattmann/ >++ >Adjunct Associate Professor, Computer Science Department >University of Southern California, Los Angeles, CA 90089 USA >++ > > > > > > >-Original Message- >From: Ravikumar Jeyaraman >Date: Sunday, November 2, 2014 at 10:00 PM >To: Chris Mattmann >Cc: Chris Mattmann , "rmura...@usc.edu" >, "saish...@usc.edu" , Vineet Ghatge >Hemantkumar , Gouthami Kondakindi , >Preethi Ramesh , dev >Subject: Re: FileConcatenator not generating output > >>Hi Professor, >> >> I have defined the PGE_ROOT. Infact I have set them in >>my .bashrc file.so it ll be set always >> >>My PGE_ROOT is defined as below. >> >>root@homework1:~# echo $PGE_ROOT >>/root/oodt-deploy/cas-pge-0.7 >> >> >>cas-pge-0.7 folder has filecontenator folder where jobs output should >>reside. >> >> >>Not sure what I am missing >> >> >>Thanks a lot helping us with this debugging. >> >> >>Thanks >> >>Ravi >> >> >>On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann >> wrote: >> >>I think your PGE_ROOT environment variable is not defined. >>Can you try echo $PGE_ROOT and tell me what it shows? If it¹s >>not defined, please define it, then restart OODT and try again. >> >> >>Chris Mattmann, Ph.D. >>Adjunct Associate Professor, Computer Science Department >>University of Southern California >>Los Angeles, CA 90089 USA >>Email: mattm...@usc.edu >>WWW: http://sunset.usc.edu/~mattmann/ >> >> >> >> >> >>-Original Message- >>From: Ravikumar Jeyaraman >>Date: Sunday, November 2, 2014 at 3:13 PM >>To: Chris Mattmann >>Cc: Chris Mattmann , "rmura...@usc.edu" >>, "saish...@usc.edu" , Vineet Ghatge >>Hemantkumar , Gouthami Kondakindi , >>Preethi Ramesh , dev >>Subject: Re: FileConcatenator not generating output >> >>>Hi Professor, >>> >>> I tried setting JobDir to >>>$PGE_ROOT/file_concatenator/output/jobs manually. >>> >>> >>>After executing JobDir is still a empty dir has nothing in it. >>> >>> >>>I have attached my PGEConfig.xml with this mail in case you need it. >>> >>> >>> >>>On
Re: FileConcatenator not generating output
Hi Ravi, are you still having trouble here? Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Ravikumar Jeyaraman Date: Sunday, November 2, 2014 at 10:00 PM To: Chris Mattmann Cc: Chris Mattmann , "rmura...@usc.edu" , "saish...@usc.edu" , Vineet Ghatge Hemantkumar , Gouthami Kondakindi , Preethi Ramesh , dev Subject: Re: FileConcatenator not generating output >Hi Professor, > > I have defined the PGE_ROOT. Infact I have set them in >my .bashrc file.so it ll be set always > >My PGE_ROOT is defined as below. > >root@homework1:~# echo $PGE_ROOT >/root/oodt-deploy/cas-pge-0.7 > > >cas-pge-0.7 folder has filecontenator folder where jobs output should >reside. > > >Not sure what I am missing > > >Thanks a lot helping us with this debugging. > > >Thanks > >Ravi > > >On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann > wrote: > >I think your PGE_ROOT environment variable is not defined. >Can you try echo $PGE_ROOT and tell me what it shows? If it¹s >not defined, please define it, then restart OODT and try again. > > >Chris Mattmann, Ph.D. >Adjunct Associate Professor, Computer Science Department >University of Southern California >Los Angeles, CA 90089 USA >Email: mattm...@usc.edu >WWW: http://sunset.usc.edu/~mattmann/ > > > > > >-Original Message- >From: Ravikumar Jeyaraman >Date: Sunday, November 2, 2014 at 3:13 PM >To: Chris Mattmann >Cc: Chris Mattmann , "rmura...@usc.edu" >, "saish...@usc.edu" , Vineet Ghatge >Hemantkumar , Gouthami Kondakindi , >Preethi Ramesh , dev >Subject: Re: FileConcatenator not generating output > >>Hi Professor, >> >> I tried setting JobDir to >>$PGE_ROOT/file_concatenator/output/jobs manually. >> >> >>After executing JobDir is still a empty dir has nothing in it. >> >> >>I have attached my PGEConfig.xml with this mail in case you need it. >> >> >> >>On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) >> wrote: >> >> >>Thanks Ravi can u show what the job dir has in it? >> >>Sent from my iPhone >> >>On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" >>wrote: >> >> >> >>Hi Professor, >> >> We are still trying out file concatenate work-flow >>example . We were able to execute it with out error. But concatenated >>output file is not generated at the end of execution. >> >> >> >>We are doubting some path setting in PGEConfig.xml . >> >> >>I remember you suggested one of the student to change createBeforeExe to >>true. I tried that also. But still files are not generated. I also tried >>setting manually JobDir env variable. Still no luck . Any suggestions ? >> >> >> >>cd [PGE_ROOT]/file_concatenator >>cp [InputFile1] [OutputFile] >>cat [InputFile2] >> [OutputFile] >> >> >> >> >> >> >> >> >>Thanks >> >>Ravi >> >> >> >> >> >> >> >> >> >> >> >> >> > > > > > > > >
Re: FileConcatenator not generating output
Hi Professor, I tried setting JobDir to $PGE_ROOT/file_concatenator/output/jobs manually. After executing JobDir is still a empty dir has nothing in it. I have attached my PGEConfig.xml with this mail in case you need it. On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) < chris.a.mattm...@jpl.nasa.gov> wrote: > > Thanks Ravi can u show what the job dir has in it? > > Sent from my iPhone > > On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" > wrote: > >Hi Professor, > We are still trying out file concatenate work-flow > example . We were able to execute it with out error. But concatenated > output file is not generated at the end of execution. > > We are doubting some path setting in PGEConfig.xml . > > I remember you suggested one of the student to change createBeforeExe to > true. I tried that also. But still files are not generated. I also tried > setting manually JobDir env variable. Still no luck . Any suggestions ? > > > > cd [PGE_ROOT]/file_concatenator > cp [InputFile1] [OutputFile] > cat [InputFile2] >> [OutputFile] > > > > > > ** > > Thanks > Ravi > > cd [PGE_ROOT]/file_concatenator cp [InputFile1] [OutputFile] cat [InputFile2] >> [OutputFile]
Re: FileConcatenator not generating output
I think your PGE_ROOT environment variable is not defined. Can you try echo $PGE_ROOT and tell me what it shows? If it¹s not defined, please define it, then restart OODT and try again. Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ -Original Message- From: Ravikumar Jeyaraman Date: Sunday, November 2, 2014 at 3:13 PM To: Chris Mattmann Cc: Chris Mattmann , "rmura...@usc.edu" , "saish...@usc.edu" , Vineet Ghatge Hemantkumar , Gouthami Kondakindi , Preethi Ramesh , dev Subject: Re: FileConcatenator not generating output >Hi Professor, > > I tried setting JobDir to >$PGE_ROOT/file_concatenator/output/jobs manually. > > >After executing JobDir is still a empty dir has nothing in it. > > >I have attached my PGEConfig.xml with this mail in case you need it. > > > >On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) > wrote: > > >Thanks Ravi can u show what the job dir has in it? > >Sent from my iPhone > >On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" >wrote: > > > >Hi Professor, > > We are still trying out file concatenate work-flow >example . We were able to execute it with out error. But concatenated >output file is not generated at the end of execution. > > > >We are doubting some path setting in PGEConfig.xml . > > >I remember you suggested one of the student to change createBeforeExe to >true. I tried that also. But still files are not generated. I also tried >setting manually JobDir env variable. Still no luck . Any suggestions ? > > > >cd [PGE_ROOT]/file_concatenator >cp [InputFile1] [OutputFile] >cat [InputFile2] >> [OutputFile] > > > > > > > > >Thanks > >Ravi > > > > > > > > > > > > >
Re: FileConcatenator not generating output
Hi Professor, I have defined the PGE_ROOT. Infact I have set them in my .bashrc file.so it ll be set always My PGE_ROOT is defined as below. root@homework1:~# echo $PGE_ROOT /root/oodt-deploy/cas-pge-0.7 cas-pge-0.7 folder has filecontenator folder where jobs output should reside. Not sure what I am missing Thanks a lot helping us with this debugging. Thanks Ravi On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann wrote: > I think your PGE_ROOT environment variable is not defined. > Can you try echo $PGE_ROOT and tell me what it shows? If it¹s > not defined, please define it, then restart OODT and try again. > > > Chris Mattmann, Ph.D. > Adjunct Associate Professor, Computer Science Department > University of Southern California > Los Angeles, CA 90089 USA > Email: mattm...@usc.edu > WWW: http://sunset.usc.edu/~mattmann/ > > > > > > -Original Message- > From: Ravikumar Jeyaraman > Date: Sunday, November 2, 2014 at 3:13 PM > To: Chris Mattmann > Cc: Chris Mattmann , "rmura...@usc.edu" > , "saish...@usc.edu" , Vineet Ghatge > Hemantkumar , Gouthami Kondakindi , > Preethi Ramesh , dev > Subject: Re: FileConcatenator not generating output > > >Hi Professor, > > > > I tried setting JobDir to > >$PGE_ROOT/file_concatenator/output/jobs manually. > > > > > >After executing JobDir is still a empty dir has nothing in it. > > > > > >I have attached my PGEConfig.xml with this mail in case you need it. > > > > > > > >On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) > > wrote: > > > > > >Thanks Ravi can u show what the job dir has in it? > > > >Sent from my iPhone > > > >On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" > >wrote: > > > > > > > >Hi Professor, > > > > We are still trying out file concatenate work-flow > >example . We were able to execute it with out error. But concatenated > >output file is not generated at the end of execution. > > > > > > > >We are doubting some path setting in PGEConfig.xml . > > > > > >I remember you suggested one of the student to change createBeforeExe to > >true. I tried that also. But still files are not generated. I also tried > >setting manually JobDir env variable. Still no luck . Any suggestions ? > > > > > > > >cd [PGE_ROOT]/file_concatenator > >cp [InputFile1] [OutputFile] > >cat [InputFile2] >> [OutputFile] > > > > > > > > > > > > > > > > > >Thanks > > > >Ravi > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: FileConcatenator not generating output
Thanks Ravi can u show what the job dir has in it? Sent from my iPhone On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" mailto:jeyar...@usc.edu>> wrote: Hi Professor, We are still trying out file concatenate work-flow example . We were able to execute it with out error. But concatenated output file is not generated at the end of execution. We are doubting some path setting in PGEConfig.xml . I remember you suggested one of the student to change createBeforeExe to true. I tried that also. But still files are not generated. I also tried setting manually JobDir env variable. Still no luck . Any suggestions ? cd [PGE_ROOT]/file_concatenator cp [InputFile1] [OutputFile] cat [InputFile2] >> [OutputFile] Thanks Ravi
