Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
On 2012-03-15 00:35:26, brian Foster wrote: Why can't DataSourceCatalog be extended and just modified per hypersonic needs?... what are the differences exactly between DataSource and Hyersonic versions of the Catalogs? Hey Brian, great question. The reality is that this catalog was hidden away in the File Manager test suite inside of TestDataSourceCatalog. There was something wiggy about the DataSourceCatalog code that Hypersonic SQL didn't like so I had to extend DataSourceCatalog for internal testing since it made sense to use Hypersonic SQL for that (could load up the DBMS in memory, etc.) I know that there was like 1 method that used incompatible SQL syntax that Hypersonic SQL didn't like. Then when I overrode that one method, I had to override like 4-5 others to inject my subclass method into the call chain for that which is what you see in the patch. It seems that DataSourceCatalog has since evolved and it's entirely possible that it would support Hypersonic SQL now, so here's what I see as 2 options: 1. we try and simply remove Hypersonic SQL catalog period, and then see if the TestDataSourceCatalog passes. If it does, this issue is moot and I will close this ReviewBoard and the issue. 2. If #1 doesn't succeed, and suggests changes, we could: a) flow those changes into DataSourceCatalog -- someone would have to have time to do that :) It also could happen later; or b) suggest that since this catalog exists and we know it works, temporarily in 0.4 and ongoing until someone refactors or improves DataSourceCatalog ship the HypersonicSQLFriendlyCatalog for folks who want to use it. Sound right? Thoughts? - Chris --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/#review5978 --- On 2012-03-14 23:40:55, Chris Mattmann wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/ --- (Updated 2012-03-14 23:40:55) Review request for oodt, brian Foster, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- DataSourceCatalog compatible with HypersonicSQL This addresses bug OODT-410. https://issues.apache.org/jira/browse/OODT-410 Diffs - ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/HsqlDbFriendlyDataSourceCatalog.java PRE-CREATION ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/HsqlDbFriendlyDataSourceCatalogFactory.java PRE-CREATION ./trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java 1294690 Diff: https://reviews.apache.org/r/4358/diff Testing --- Thanks, Chris
Re: Review Request: Port CAS-PGE's PcsMetadataKeys to PgeTaskMetKeys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4359/#review5989 --- Ship it! LGTM! trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java https://reviews.apache.org/r/4359/#comment12984 love the javadoc! - Chris On 2012-03-15 08:25:38, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4359/ --- (Updated 2012-03-15 08:25:38) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- Will allow for specification of new keys while still supported legacy keys This addresses bug OODT-411. https://issues.apache.org/jira/browse/OODT-411 Diffs - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1300799 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1300799 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PcsMetadataKeys.java 1300798 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1300799 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1300799 trunk/pge/src/test/org/apache/oodt/cas/pge/metadata/TestPgeTaskMetKeys.java 1300799 Diff: https://reviews.apache.org/r/4359/diff Testing --- existing unit-tests for PgeTaskMetKeys Thanks, brian
Re: Review Request: Port CAS-PGE's PcsMetadataKeys to PgeTaskMetKeys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4359/ --- (Updated 2012-03-15 08:08:37.402954) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- Since required only a small change, added support for isVector() to PgeTaskMetKeys which allows workflow static metadata to be split on comma if it is a multi-value type (but still allowing scalar values to contain commas without getting split into multi-values) Summary --- Will allow for specification of new keys while still supported legacy keys This addresses bug OODT-411. https://issues.apache.org/jira/browse/OODT-411 Diffs (updated) - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1300799 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1300799 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PcsMetadataKeys.java 1300798 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1300799 trunk/pge/src/test/org/apache/oodt/cas/pge/metadata/TestPgeTaskMetKeys.java 1300799 Diff: https://reviews.apache.org/r/4359/diff Testing --- existing unit-tests for PgeTaskMetKeys Thanks, brian
Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, and Thomas Bennett. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1300633 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/#review6014 --- Ship it! LGTM. Nice catch Ricky -- what about a small unit test for regression? - Chris On 2012-03-16 01:51:02, Ricky Nguyen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 01:51:02) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, and Thomas Bennett. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1300633 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: svn commit: r1301268 - /oodt/trunk/CHANGES.txt
no prob!... lots more to come too... don't think i'm even half done with cas-pge yet... lol-brianOn Mar 15, 2012, at 06:24 PM, "Mattmann, Chris A (388J)" chris.a.mattm...@jpl.nasa.gov wrote:You are my hero! Thanks for the wengine help bud! Cheers, Chris On Mar 15, 2012, at 4:34 PM, bfos...@apache.org bfos...@apache.org wrote: Author: bfoster Date: Thu Mar 15 23:34:51 2012 New Revision: 1301268URL: http://svn.apache.org/viewvc?rev=1301268view=rev Log: - Updated CHANGES.txt-- OODT-411Modified: oodt/trunk/CHANGES.txtModified: oodt/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/oodt/trunk/CHANGES.txt?rev=1301268r1=1301267r2=1301268view=diff == --- oodt/trunk/CHANGES.txt (original) +++ oodt/trunk/CHANGES.txt Thu Mar 15 23:34:51 2012 @@ -4,6 +4,8 @@ Apache OODT Change Log Release 0.4: Current Development +* OODT-411 Port CAS-PGE's PcsMetadataKeys to PgeTaskMetKeys (bfoster) + * OODT-409 Convert CAS-PGE metadata keys and workflow statuses from static Strings to enums (bfoster)* OODT-406 Add CAS-PGE support for multiple Property Adders (bfoster) ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/#review6017 --- Ship it! You da man Ricky. LGTM. - Chris On 2012-03-16 03:18:23, Ricky Nguyen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 03:18:23) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, Sheryl John, and Thomas Bennett. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1301315 trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1301315 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Review Request: Add support to ExecUtils callProgram to take OutputStreams for forwarding stdout and stderr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4377/#review6018 --- Ship it! LGTM. - Chris On 2012-03-16 03:00:01, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4377/ --- (Updated 2012-03-16 03:00:01) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- Allow for any OutputStream to be supplied while still supporting existing Logger methods This addresses bug OODT-415. https://issues.apache.org/jira/browse/OODT-415 Diffs - trunk/commons/src/main/java/org/apache/oodt/commons/exec/ExecUtils.java 1301316 Diff: https://reviews.apache.org/r/4377/diff Testing --- Thanks, brian
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 03:04:13.893499) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, and Thomas Bennett. Changes --- added unit test Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs (updated) - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1301315 trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1301315 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/#review6019 --- Ship it! Unit test looks good too! - Sheryl On 2012-03-16 03:18:23, Ricky Nguyen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 03:18:23) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, Sheryl John, and Thomas Bennett. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1301315 trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1301315 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
On 2012-03-15 00:35:26, brian Foster wrote: Why can't DataSourceCatalog be extended and just modified per hypersonic needs?... what are the differences exactly between DataSource and Hyersonic versions of the Catalogs? Chris Mattmann wrote: Hey Brian, great question. The reality is that this catalog was hidden away in the File Manager test suite inside of TestDataSourceCatalog. There was something wiggy about the DataSourceCatalog code that Hypersonic SQL didn't like so I had to extend DataSourceCatalog for internal testing since it made sense to use Hypersonic SQL for that (could load up the DBMS in memory, etc.) I know that there was like 1 method that used incompatible SQL syntax that Hypersonic SQL didn't like. Then when I overrode that one method, I had to override like 4-5 others to inject my subclass method into the call chain for that which is what you see in the patch. It seems that DataSourceCatalog has since evolved and it's entirely possible that it would support Hypersonic SQL now, so here's what I see as 2 options: 1. we try and simply remove Hypersonic SQL catalog period, and then see if the TestDataSourceCatalog passes. If it does, this issue is moot and I will close this ReviewBoard and the issue. 2. If #1 doesn't succeed, and suggests changes, we could: a) flow those changes into DataSourceCatalog -- someone would have to have time to do that :) It also could happen later; or b) suggest that since this catalog exists and we know it works, temporarily in 0.4 and ongoing until someone refactors or improves DataSourceCatalog ship the HypersonicSQLFriendlyCatalog for folks who want to use it. Sound right? Thoughts? brian Foster wrote: i'd be cool with you checking it in for now with a TODO at the top of the class and maybe a JIRA issue to merge the 2 at some point Chris Mattmann wrote: Cool, I will try and use the DataSourceCatalog, see if it passes, and if so, just forget this one. If it doesn't pass, I'll consider the above your +1 and then move forward with this. Chris Mattmann wrote: Hey Brian, here was the issue, check it: java.sql.SQLException: Not in aggregate function or group by clause: org.hsqldb.Expression@16ba5c7a in statement [SELECT COUNT(DISTINCT p.product_id) AS numResults FROM GenericFile_metadata p ORDER BY p.product_id DESC ] at org.hsqldb.jdbc.Util.sqlException(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source) Hypersonic SQL doesn't like COUNT functions and ORDER BY to be used together. See: http://www.dcm4che.org/jira/browse/ARR-37 So, what I did with this catalog is override the getResultListSize, which then caused me to have to override paginateQuery, and then caused me to have to intercept the pagedQuery and then the full paging API, which is what this catalog does, and I think correctly. If I don't hear any further objections, I am going to commit this tomorrow. Ricky Nguyen wrote: Why can't we remove the ORDER clause? SELECT COUNT(DISTINCT p.product_id) AS numResults FROM GenericFile_metadata p since the order doesn't matter when you're just returning a count. What do you guys think? I'd be OK removing the ORDER clause but it will be a change from the SQL that the DataSourceCatalog used to generate. But, you're right, it doesn't really matter. Wonder why Oracle, and MySQL and Postgres allow it? - Chris --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/#review5978 --- On 2012-03-14 23:40:55, Chris Mattmann wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/ --- (Updated 2012-03-14 23:40:55) Review request for oodt, brian Foster, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- DataSourceCatalog compatible with HypersonicSQL This addresses bug OODT-410. https://issues.apache.org/jira/browse/OODT-410 Diffs - ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/HsqlDbFriendlyDataSourceCatalog.java PRE-CREATION ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/HsqlDbFriendlyDataSourceCatalogFactory.java PRE-CREATION ./trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java 1294690 Diff: https://reviews.apache.org/r/4358/diff Testing --- Thanks, Chris
