Great to hear that this is working now Ravi. Note that yes the File Manager (and Workflow Manager) will have the metrics on ingestion since CAS-PGE will help to capture this metadata for you while running ETLLib.
Cheers! Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: mattm...@usc.edu WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Ravikumar Jeyaraman <jeyar...@usc.edu> Date: Wednesday, November 5, 2014 at 6:14 PM To: Chris Mattmann <chris.a.mattm...@jpl.nasa.gov> Cc: "saish...@usc.edu" <saish...@usc.edu>, Vineet Ghatge Hemantkumar <heman...@usc.edu>, Chris Mattmann <mattm...@usc.edu>, Gouthami Kondakindi <konda...@usc.edu>, dev <dev@oodt.apache.org>, Preethi Ramesh <pram...@usc.edu>, "rmura...@usc.edu" <rmura...@usc.edu> Subject: Re: FileConcatenator not generating output >Hi Professor, > It works now.Looks like the wiki page was recently >updated and when I followed that steps (which includes downloading .7 >version of oodt ) it works fine . >Thanks for checking with us. > >We were wondering how the overall picture looks like ? >As per our current understanding > > >* Json files will be injected into the filemgr >* Workflow will be used to execute ETLlib which indexe JSON files from >the filemgr archive into solr. > >* Solr's searh algo will be modified as per our requirement. > > > > >But if this is the case how come filemgr will have metric on solr >ingestion? Its little confusing here. > > >Thanks > >Ravikumar Jeyaraman >On Nov 5, 2014 4:59 PM, "Mattmann, Chris A (3980)" ><chris.a.mattm...@jpl.nasa.gov> wrote: > >Hi Ravi, >are you still having trouble here? > >Cheers, >Chris > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >Chris Mattmann, Ph.D. >Chief Architect >Instrument Software and Science Data Systems Section (398) >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >Office: 168-519, Mailstop: 168-527 >Email: chris.a.mattm...@nasa.gov >WWW: http://sunset.usc.edu/~mattmann/ >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >Adjunct Associate Professor, Computer Science Department >University of Southern California, Los Angeles, CA 90089 USA >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > >-----Original Message----- >From: Ravikumar Jeyaraman <jeyar...@usc.edu> >Date: Sunday, November 2, 2014 at 10:00 PM >To: Chris Mattmann <mattm...@usc.edu> >Cc: Chris Mattmann <chris.a.mattm...@jpl.nasa.gov>, "rmura...@usc.edu" ><rmura...@usc.edu>, "saish...@usc.edu" <saish...@usc.edu>, Vineet Ghatge >Hemantkumar <heman...@usc.edu>, Gouthami Kondakindi <konda...@usc.edu>, >Preethi Ramesh <pram...@usc.edu>, dev <dev@oodt.apache.org> >Subject: Re: FileConcatenator not generating output > >>Hi Professor, >> >> I have defined the PGE_ROOT. Infact I have set them in >>my .bashrc file.so it ll be set always >> >>My PGE_ROOT is defined as below. >> >>root@homework1:~# echo $PGE_ROOT >>/root/oodt-deploy/cas-pge-0.7 >> >> >>cas-pge-0.7 folder has filecontenator folder where jobs output should >>reside. >> >> >>Not sure what I am missing >> >> >>Thanks a lot helping us with this debugging. >> >> >>Thanks >> >>Ravi >> >> >>On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann >><mattm...@usc.edu> wrote: >> >>I think your PGE_ROOT environment variable is not defined. >>Can you try echo $PGE_ROOT and tell me what it shows? If it¹s >>not defined, please define it, then restart OODT and try again. >> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>Chris Mattmann, Ph.D. >>Adjunct Associate Professor, Computer Science Department >>University of Southern California >>Los Angeles, CA 90089 USA >>Email: mattm...@usc.edu >>WWW: http://sunset.usc.edu/~mattmann/ >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >>-----Original Message----- >>From: Ravikumar Jeyaraman <jeyar...@usc.edu> >>Date: Sunday, November 2, 2014 at 3:13 PM >>To: Chris Mattmann <chris.a.mattm...@jpl.nasa.gov> >>Cc: Chris Mattmann <mattm...@usc.edu>, "rmura...@usc.edu" >><rmura...@usc.edu>, "saish...@usc.edu" <saish...@usc.edu>, Vineet Ghatge >>Hemantkumar <heman...@usc.edu>, Gouthami Kondakindi <konda...@usc.edu>, >>Preethi Ramesh <pram...@usc.edu>, dev <dev@oodt.apache.org> >>Subject: Re: FileConcatenator not generating output >> >>>Hi Professor, >>> >>> I tried setting JobDir to >>>$PGE_ROOT/file_concatenator/output/jobs manually. >>> >>> >>>After executing JobDir is still a empty dir has nothing in it. >>> >>> >>>I have attached my PGEConfig.xml with this mail in case you need it. >>> >>> >>> >>>On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) >>><chris.a.mattm...@jpl.nasa.gov> wrote: >>> >>> >>>Thanks Ravi can u show what the job dir has in it? >>> >>>Sent from my iPhone >>> >>>On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" <jeyar...@usc.edu> >>>wrote: >>> >>> >>> >>>Hi Professor, >>> >>> We are still trying out file concatenate work-flow >>>example . We were able to execute it with out error. But concatenated >>>output file is not generated at the end of execution. >>> >>> >>> >>>We are doubting some path setting in PGEConfig.xml . >>> >>> >>>I remember you suggested one of the student to change createBeforeExe to >>>true. I tried that also. But still files are not generated. I also tried >>>setting manually JobDir env variable. Still no luck . Any suggestions ? >>> >>> <exe dir="[JobDir]" shell="/bin/bash"> >>> <!-- cd to PGE root --> >>> <cmd>cd [PGE_ROOT]/file_concatenator</cmd> >>> <cmd>cp [InputFile1] [OutputFile]</cmd> >>> <cmd>cat [InputFile2] >> [OutputFile]</cmd> >>> </exe> >>> >>> <!-- Files to ingest --> >>> <output> >>> <!-- one or more of these --> >>> <dir path="[JobDir]" createBeforeExe="false"> >>> >>> >>>Thanks >>> >>>Ravi >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> >> >> > > > > >
