Crawler actions...
Hey guys... I've been slowly chuggin away at integrating cas-cli into cas-crawler... busy few weeks at work... but I'm also kinda stuck on something... currently the crawler supports command line options which configure crawler actions (not command line actions), however there is no way to display the help for these (i.e. whether they are required or not)... this is because they become required only if another option (i.e. -ais) is specified with their action id... new cas-cli doesn't support this either and I initially thought about adding this feature to cas-cli but didn't for simplicity sake (cas-cli is complex enough)... so I'm think of just make crawler action configurable via java properties... this will not change the currently implemented classes at all or the way actions are written in the future... basically spring will just set its properties for these actions by using java properties instead... what u guys thing? -brian
Review Request: XMLPS should be able to stream large results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3543/ --- Review request for oodt. Summary --- See OODT-341: https://issues.apache.org/jira/browse/OODT-341 * CDEResult extends org.apache.oodt.xmlquery.Result * CDEResult mimetype is always text/plain * CDEResult size is always 0 * CDEResult inputstream is CDEResultInputStream * CDEResultInputStream has IO methods and wraps a CDEResult * CDEResult wraps a ResultSet and returns rows as Strings, applying MappingFuncs if a Mapping is provided, and appending constant fields if a List of CDEValues is provided. * ProductQueryServlet relies on servlet container to handle Content-Length (and possibly Transfer-Encoding: chunked) This addresses bug OODT-341. https://issues.apache.org/jira/browse/OODT-341 Diffs - trunk/grid/src/main/java/org/apache/oodt/grid/ProductQueryServlet.java 1183564 trunk/xmlps/pom.xml 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/DBMSExecutor.java 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/XMLPSProductHandler.java 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/structs/CDEResult.java 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/structs/CDEResultInputStream.java PRE-CREATION trunk/xmlps/src/test/java/org/apache/oodt/xmlps/structs/TestCDEResult.java PRE-CREATION trunk/xmlps/src/test/java/org/apache/oodt/xmlps/structs/TestCDEResultInputStream.java PRE-CREATION Diff: https://reviews.apache.org/r/3543/diff Testing --- Runs in Tomcat 7. I've used it at CHLA. NOT tested in other app servers. Added 2 unit tests: TestCDEResult and TestCDEResultInputStream, which both pass. Thanks, rickdn
Re: Review Request: XMLPS should be able to stream large results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3543/#review4462 --- trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/DBMSExecutor.java https://reviews.apache.org/r/3543/#comment10013 The MappingFuncs are applied in CDEResult#applyMappingFuncs() trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/XMLPSProductHandler.java https://reviews.apache.org/r/3543/#comment10014 Line 272 (latest) corresponds to line 250 (patched). trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/XMLPSProductHandler.java https://reviews.apache.org/r/3543/#comment10015 Appending constant values to the end of the row is now done in CDEResult#addConstValues() - Ricky On 2012-01-19 02:25:16, Ricky Nguyen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3543/ --- (Updated 2012-01-19 02:25:16) Review request for oodt. Summary --- See OODT-341: https://issues.apache.org/jira/browse/OODT-341 * CDEResult extends org.apache.oodt.xmlquery.Result * CDEResult mimetype is always text/plain * CDEResult size is always 0 * CDEResult inputstream is CDEResultInputStream * CDEResultInputStream has IO methods and wraps a CDEResult * CDEResult wraps a ResultSet and returns rows as Strings, applying MappingFuncs if a Mapping is provided, and appending constant fields if a List of CDEValues is provided. * ProductQueryServlet relies on servlet container to handle Content-Length (and possibly Transfer-Encoding: chunked) This addresses bug OODT-341. https://issues.apache.org/jira/browse/OODT-341 Diffs - trunk/grid/src/main/java/org/apache/oodt/grid/ProductQueryServlet.java 1183564 trunk/xmlps/pom.xml 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/DBMSExecutor.java 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/product/XMLPSProductHandler.java 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/structs/CDEResult.java 1233127 trunk/xmlps/src/main/java/org/apache/oodt/xmlps/structs/CDEResultInputStream.java PRE-CREATION trunk/xmlps/src/test/java/org/apache/oodt/xmlps/structs/TestCDEResult.java PRE-CREATION trunk/xmlps/src/test/java/org/apache/oodt/xmlps/structs/TestCDEResultInputStream.java PRE-CREATION Diff: https://reviews.apache.org/r/3543/diff Testing --- Runs in Tomcat 7. I've used it at CHLA. NOT tested in other app servers. Added 2 unit tests: TestCDEResult and TestCDEResultInputStream, which both pass. Thanks, Ricky
Re: Crawler actions...
Hey Brian, On Jan 18, 2012, at 3:02 PM, Brian Foster wrote: Hey guys... I've been slowly chuggin away at integrating cas-cli into cas-crawler... busy few weeks at work... but I'm also kinda stuck on something... currently the crawler supports command line options which configure crawler actions (not command line actions), however there is no way to display the help for these (i.e. whether they are required or not)... this is because they become required only if another option (i.e. -ais) is specified with their action id... new cas-cli doesn't support this either and I initially thought about adding this feature to cas-cli but didn't for simplicity sake (cas-cli is complex enough)... so I'm think of just make crawler action configurable via java properties... this will not change the currently implemented classes at all or the way actions are written in the future... basically spring will just set its properties for these actions by using java properties instead... what u guys thing? I'd be +1 for this if the command line for the crawler worked the same way. Are you proposing that CrawlerLauncher would now simply read command line actions and then have the Crawler's CommandLine actions set Spring java properties which would then be ready by the CrawlerActions? Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
