Great to hear that this is working now Ravi. Note that yes the File Manager (and Workflow Manager) will have the metrics on ingestion since CAS-PGE will help to capture this metadata for you while running ETLLib.
Cheers! Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Adjunct Associate Professor, Computer Science Department University of Southern California Los Angeles, CA 90089 USA Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Ravikumar Jeyaraman <[email protected]> Date: Wednesday, November 5, 2014 at 6:14 PM To: Chris Mattmann <[email protected]> Cc: "[email protected]" <[email protected]>, Vineet Ghatge Hemantkumar <[email protected]>, Chris Mattmann <[email protected]>, Gouthami Kondakindi <[email protected]>, dev <[email protected]>, Preethi Ramesh <[email protected]>, "[email protected]" <[email protected]> Subject: Re: FileConcatenator not generating output >Hi Professor, > It works now.Looks like the wiki page was recently >updated and when I followed that steps (which includes downloading .7 >version of oodt ) it works fine . >Thanks for checking with us. > >We were wondering how the overall picture looks like ? >As per our current understanding > > >* Json files will be injected into the filemgr >* Workflow will be used to execute ETLlib which indexe JSON files from >the filemgr archive into solr. > >* Solr's searh algo will be modified as per our requirement. > > > > >But if this is the case how come filemgr will have metric on solr >ingestion? Its little confusing here. > > >Thanks > >Ravikumar Jeyaraman >On Nov 5, 2014 4:59 PM, "Mattmann, Chris A (3980)" ><[email protected]> wrote: > >Hi Ravi, >are you still having trouble here? > >Cheers, >Chris > > > >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >Chris Mattmann, Ph.D. >Chief Architect >Instrument Software and Science Data Systems Section (398) >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >Office: 168-519, Mailstop: 168-527 >Email: [email protected] >WWW: http://sunset.usc.edu/~mattmann/ >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >Adjunct Associate Professor, Computer Science Department >University of Southern California, Los Angeles, CA 90089 USA >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > >-----Original Message----- >From: Ravikumar Jeyaraman <[email protected]> >Date: Sunday, November 2, 2014 at 10:00 PM >To: Chris Mattmann <[email protected]> >Cc: Chris Mattmann <[email protected]>, "[email protected]" ><[email protected]>, "[email protected]" <[email protected]>, Vineet Ghatge >Hemantkumar <[email protected]>, Gouthami Kondakindi <[email protected]>, >Preethi Ramesh <[email protected]>, dev <[email protected]> >Subject: Re: FileConcatenator not generating output > >>Hi Professor, >> >> I have defined the PGE_ROOT. Infact I have set them in >>my .bashrc file.so it ll be set always >> >>My PGE_ROOT is defined as below. >> >>root@homework1:~# echo $PGE_ROOT >>/root/oodt-deploy/cas-pge-0.7 >> >> >>cas-pge-0.7 folder has filecontenator folder where jobs output should >>reside. >> >> >>Not sure what I am missing >> >> >>Thanks a lot helping us with this debugging. >> >> >>Thanks >> >>Ravi >> >> >>On Sun, Nov 2, 2014 at 8:39 PM, Christian Alan Mattmann >><[email protected]> wrote: >> >>I think your PGE_ROOT environment variable is not defined. >>Can you try echo $PGE_ROOT and tell me what it shows? If it¹s >>not defined, please define it, then restart OODT and try again. >> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>Chris Mattmann, Ph.D. >>Adjunct Associate Professor, Computer Science Department >>University of Southern California >>Los Angeles, CA 90089 USA >>Email: [email protected] >>WWW: http://sunset.usc.edu/~mattmann/ >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >>-----Original Message----- >>From: Ravikumar Jeyaraman <[email protected]> >>Date: Sunday, November 2, 2014 at 3:13 PM >>To: Chris Mattmann <[email protected]> >>Cc: Chris Mattmann <[email protected]>, "[email protected]" >><[email protected]>, "[email protected]" <[email protected]>, Vineet Ghatge >>Hemantkumar <[email protected]>, Gouthami Kondakindi <[email protected]>, >>Preethi Ramesh <[email protected]>, dev <[email protected]> >>Subject: Re: FileConcatenator not generating output >> >>>Hi Professor, >>> >>> I tried setting JobDir to >>>$PGE_ROOT/file_concatenator/output/jobs manually. >>> >>> >>>After executing JobDir is still a empty dir has nothing in it. >>> >>> >>>I have attached my PGEConfig.xml with this mail in case you need it. >>> >>> >>> >>>On Sun, Nov 2, 2014 at 1:52 PM, Mattmann, Chris A (3980) >>><[email protected]> wrote: >>> >>> >>>Thanks Ravi can u show what the job dir has in it? >>> >>>Sent from my iPhone >>> >>>On Nov 2, 2014, at 1:48 PM, "Ravikumar Jeyaraman" <[email protected]> >>>wrote: >>> >>> >>> >>>Hi Professor, >>> >>> We are still trying out file concatenate work-flow >>>example . We were able to execute it with out error. But concatenated >>>output file is not generated at the end of execution. >>> >>> >>> >>>We are doubting some path setting in PGEConfig.xml . >>> >>> >>>I remember you suggested one of the student to change createBeforeExe to >>>true. I tried that also. But still files are not generated. I also tried >>>setting manually JobDir env variable. Still no luck . Any suggestions ? >>> >>> <exe dir="[JobDir]" shell="/bin/bash"> >>> <!-- cd to PGE root --> >>> <cmd>cd [PGE_ROOT]/file_concatenator</cmd> >>> <cmd>cp [InputFile1] [OutputFile]</cmd> >>> <cmd>cat [InputFile2] >> [OutputFile]</cmd> >>> </exe> >>> >>> <!-- Files to ingest --> >>> <output> >>> <!-- one or more of these --> >>> <dir path="[JobDir]" createBeforeExe="false"> >>> >>> >>>Thanks >>> >>>Ravi >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> >> >> > > > > >
