Val, I would inspect your file manager catalog. Can you use the OPSUI or some other browser (Lucene) to see the metadata that is recorded for your files? I think this is part of the problem - from the query the only reason that $FileLocation/$Filename wouldn’t be resolved in your queries is that they aren’t recorded as metadata fields for your products.
Can you check? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Lewis John Mcgibbney <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, August 27, 2015 at 5:07 PM To: "[email protected]" <[email protected]> Subject: Re: catalog queries in pge config files and filemgr-client problem >Hi Val, >Wise idea to start debugging this from the filemgr-client. >Can you please try changing logging levels in the following file to DEBUG > >https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/loggi >ng.properties#L40-L74 > >If you try your queries again then hopefully you should see some >patterns/indications within FileMgr logs to indicate what is going on. > > >On Wednesday, August 26, 2015, Mallder, Valerie ><[email protected]> >wrote: > >> Hi All, >> >> So, I'm phasing off of the New Horizons project for now and I am back to >> my science data pipeline work. I am in the process of trying to pick up >> where I left off a few months ago - but with a twist J I graduated from >> RADiX and now I am trying to get my pipeline system set up to run with >>the >> 'regular' OODT. I have made extremely good progress so far. I have >>pretty >> much replicated all of the functionality of filemgr and workflow manager >> that I had been using with RADiX, with the exception of one thing that >> doesn't seem to be working the way it used too. And I'm hoping someone >>out >> there can give me some ideas on how to fix it. >> >> I have a catalog query in one of my PGE config files. I copied the >>syntax >> for the query statement directly from one of Chris' PGE config files >> (PgeConfig_RatAggregator.xml) in the 'drat-master' application. >> >> Here is Chris' statement: >> >> <metadata key="InputFiles" >> >>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim >>e'){SELECT >> FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}"/> >> >> Here is my statement: >> >> <metadata key="InitialDirList" >> >>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim >>e'){SELECT >> FileLocation,Filename,CAS.ProductReceivedTime FROM ScienceFile}"/> >> >> And here are the results from the query: >> >> >> >>$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$ >>FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F >>ileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fi >>leLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil >>eLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$File >>Location/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL >>ocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLo >>cation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc >>ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoca >>tion/$Filename, >> >> And those results are not exactly what I wanted J >> >> So I ran some experiments with ./filemgr-client on the command line to >> make sure the files I expected to be in the catalog were in fact in the >> catalog. >> >> The following operation worked just as expected. The expected number of >> products were listed. >> ./filemgr-client --url http://localhost:9016 --operation --getFirstPage >> --productTypeName ScienceFile >> >> However, this query gave some unexpected results. And this may be >>related >> the problem with the query in my PGE config file, but I am not sure. >> ./filemgr-client --url http://localhost:9016 --operation -sqlQuery >>-query >> "SELECT FileLocation,Filename FROM ScienceFile" >> >> That query only outputs several blank lines to the screen and nothing >>else. >> >> I am not very familiar with all of the catalog related code yet so I am >> not sure where to start looking for the problem. If you know this area >>of >> the code well and have some debugging ideas, please let me know. >> >> Thanks so much! >> >> Val >> >>
