Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
On 2012-03-15 00:35:26, brian Foster wrote: Why can't DataSourceCatalog be extended and just modified per hypersonic needs?... what are the differences exactly between DataSource and Hyersonic versions of the Catalogs? Chris Mattmann wrote: Hey Brian, great question. The reality is that this catalog was hidden away in the File Manager test suite inside of TestDataSourceCatalog. There was something wiggy about the DataSourceCatalog code that Hypersonic SQL didn't like so I had to extend DataSourceCatalog for internal testing since it made sense to use Hypersonic SQL for that (could load up the DBMS in memory, etc.) I know that there was like 1 method that used incompatible SQL syntax that Hypersonic SQL didn't like. Then when I overrode that one method, I had to override like 4-5 others to inject my subclass method into the call chain for that which is what you see in the patch. It seems that DataSourceCatalog has since evolved and it's entirely possible that it would support Hypersonic SQL now, so here's what I see as 2 options: 1. we try and simply remove Hypersonic SQL catalog period, and then see if the TestDataSourceCatalog passes. If it does, this issue is moot and I will close this ReviewBoard and the issue. 2. If #1 doesn't succeed, and suggests changes, we could: a) flow those changes into DataSourceCatalog -- someone would have to have time to do that :) It also could happen later; or b) suggest that since this catalog exists and we know it works, temporarily in 0.4 and ongoing until someone refactors or improves DataSourceCatalog ship the HypersonicSQLFriendlyCatalog for folks who want to use it. Sound right? Thoughts? brian Foster wrote: i'd be cool with you checking it in for now with a TODO at the top of the class and maybe a JIRA issue to merge the 2 at some point Chris Mattmann wrote: Cool, I will try and use the DataSourceCatalog, see if it passes, and if so, just forget this one. If it doesn't pass, I'll consider the above your +1 and then move forward with this. Chris Mattmann wrote: Hey Brian, here was the issue, check it: java.sql.SQLException: Not in aggregate function or group by clause: org.hsqldb.Expression@16ba5c7a in statement [SELECT COUNT(DISTINCT p.product_id) AS numResults FROM GenericFile_metadata p ORDER BY p.product_id DESC ] at org.hsqldb.jdbc.Util.sqlException(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source) Hypersonic SQL doesn't like COUNT functions and ORDER BY to be used together. See: http://www.dcm4che.org/jira/browse/ARR-37 So, what I did with this catalog is override the getResultListSize, which then caused me to have to override paginateQuery, and then caused me to have to intercept the pagedQuery and then the full paging API, which is what this catalog does, and I think correctly. If I don't hear any further objections, I am going to commit this tomorrow. Ricky Nguyen wrote: Why can't we remove the ORDER clause? SELECT COUNT(DISTINCT p.product_id) AS numResults FROM GenericFile_metadata p since the order doesn't matter when you're just returning a count. Chris Mattmann wrote: What do you guys think? I'd be OK removing the ORDER clause but it will be a change from the SQL that the DataSourceCatalog used to generate. But, you're right, it doesn't really matter. Wonder why Oracle, and MySQL and Postgres allow it? brian Foster wrote: I vote ditch ORDER! Chris Mattmann wrote: Hmm VERY interesting. So, I tried to ditch ORDER, and just use DataSourceCatalog. Here's what I got: [chipotle:oodt/trunk/filemgr] mattmann% more target/surefire-reports/org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog.txt --- Test set: org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog --- Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.828 sec FAILURE! testFirstProductOnlyOnFirstPage(org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog) Time elapsed: 0.124 sec FAILURE! junit.framework.ComparisonFailure: expected:[ShouldBeFirstForPage.tx]t but was:[tes]t at junit.framework.Assert.assertEquals(Assert.java:81) at junit.framework.Assert.assertEquals(Assert.java:87) at org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog.testFirstProductOnlyOnFirstPage(TestDataSourceCatalog.java:278) [chipotle:oodt/trunk/filemgr] mattmann% I need to look more, but something is wiggy here. When i ditch DataSourceCatalog line number
Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/ --- (Updated 2012-03-16 14:14:17.061621) Review request for oodt, brian Foster, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- - Updated diff to not include specific HypersonicSQL catalog and to update by removing ORDER BY from the getResultListSize in DataSourceCatalog. Tests don't pass for me. Summary --- DataSourceCatalog compatible with HypersonicSQL This addresses bug OODT-410. https://issues.apache.org/jira/browse/OODT-410 Diffs (updated) - ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java 1301321 ./trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java 1301321 Diff: https://reviews.apache.org/r/4358/diff Testing --- Thanks, Chris
Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
On 2012-03-15 00:35:26, brian Foster wrote: Why can't DataSourceCatalog be extended and just modified per hypersonic needs?... what are the differences exactly between DataSource and Hyersonic versions of the Catalogs? Chris Mattmann wrote: Hey Brian, great question. The reality is that this catalog was hidden away in the File Manager test suite inside of TestDataSourceCatalog. There was something wiggy about the DataSourceCatalog code that Hypersonic SQL didn't like so I had to extend DataSourceCatalog for internal testing since it made sense to use Hypersonic SQL for that (could load up the DBMS in memory, etc.) I know that there was like 1 method that used incompatible SQL syntax that Hypersonic SQL didn't like. Then when I overrode that one method, I had to override like 4-5 others to inject my subclass method into the call chain for that which is what you see in the patch. It seems that DataSourceCatalog has since evolved and it's entirely possible that it would support Hypersonic SQL now, so here's what I see as 2 options: 1. we try and simply remove Hypersonic SQL catalog period, and then see if the TestDataSourceCatalog passes. If it does, this issue is moot and I will close this ReviewBoard and the issue. 2. If #1 doesn't succeed, and suggests changes, we could: a) flow those changes into DataSourceCatalog -- someone would have to have time to do that :) It also could happen later; or b) suggest that since this catalog exists and we know it works, temporarily in 0.4 and ongoing until someone refactors or improves DataSourceCatalog ship the HypersonicSQLFriendlyCatalog for folks who want to use it. Sound right? Thoughts? brian Foster wrote: i'd be cool with you checking it in for now with a TODO at the top of the class and maybe a JIRA issue to merge the 2 at some point Chris Mattmann wrote: Cool, I will try and use the DataSourceCatalog, see if it passes, and if so, just forget this one. If it doesn't pass, I'll consider the above your +1 and then move forward with this. Chris Mattmann wrote: Hey Brian, here was the issue, check it: java.sql.SQLException: Not in aggregate function or group by clause: org.hsqldb.Expression@16ba5c7a in statement [SELECT COUNT(DISTINCT p.product_id) AS numResults FROM GenericFile_metadata p ORDER BY p.product_id DESC ] at org.hsqldb.jdbc.Util.sqlException(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source) Hypersonic SQL doesn't like COUNT functions and ORDER BY to be used together. See: http://www.dcm4che.org/jira/browse/ARR-37 So, what I did with this catalog is override the getResultListSize, which then caused me to have to override paginateQuery, and then caused me to have to intercept the pagedQuery and then the full paging API, which is what this catalog does, and I think correctly. If I don't hear any further objections, I am going to commit this tomorrow. Ricky Nguyen wrote: Why can't we remove the ORDER clause? SELECT COUNT(DISTINCT p.product_id) AS numResults FROM GenericFile_metadata p since the order doesn't matter when you're just returning a count. Chris Mattmann wrote: What do you guys think? I'd be OK removing the ORDER clause but it will be a change from the SQL that the DataSourceCatalog used to generate. But, you're right, it doesn't really matter. Wonder why Oracle, and MySQL and Postgres allow it? brian Foster wrote: I vote ditch ORDER! Chris Mattmann wrote: Hmm VERY interesting. So, I tried to ditch ORDER, and just use DataSourceCatalog. Here's what I got: [chipotle:oodt/trunk/filemgr] mattmann% more target/surefire-reports/org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog.txt --- Test set: org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog --- Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.828 sec FAILURE! testFirstProductOnlyOnFirstPage(org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog) Time elapsed: 0.124 sec FAILURE! junit.framework.ComparisonFailure: expected:[ShouldBeFirstForPage.tx]t but was:[tes]t at junit.framework.Assert.assertEquals(Assert.java:81) at junit.framework.Assert.assertEquals(Assert.java:87) at org.apache.oodt.cas.filemgr.catalog.TestDataSourceCatalog.testFirstProductOnlyOnFirstPage(TestDataSourceCatalog.java:278) [chipotle:oodt/trunk/filemgr] mattmann% I need to look more, but something is wiggy here. brian Foster wrote: When i ditch
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4374/#review6026
---
trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java
https://reviews.apache.org/r/4374/#comment13010
What about:
String element = elements.get(i);
if (fullMetadata.containsKey(element) {
reducedMetadata.addMetadata(element,
fullMetadata.getAllMetadata(element);
}
also add addMetadata() really should be replaceMetadata()... although in
this cause unless the same element is in the elements List more than once
addMetadata() will work the same
- brian
On 2012-03-16 03:18:23, Ricky Nguyen wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4374/
---
(Updated 2012-03-16 03:18:23)
Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, Sheryl
John, and Thomas Bennett.
Summary
---
filemgr query throws NPE when some products have undefined metadata values
This addresses bug OODT-413.
https://issues.apache.org/jira/browse/OODT-413
Diffs
-
trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java
1301315
trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java
1301315
Diff: https://reviews.apache.org/r/4374/diff
Testing
---
tested at CHLA with command:
./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q
SELECT VpsEpisodeStartTime FROM CernerEvents
Thanks,
Ricky
Re: Review Request: Create Java Logger Handler for CAS-PGE
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4375/ --- (Updated 2012-03-16 08:25:17.505045) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- Updated to use new methods in oodt-commons ExecUtils Summary --- This handler will allow for multiple CAS-PGE instances to exist in the same JVM while routing the logging from each to their own log files in their execution directory - This patch also includes: - adding support for PgeMetadata to take PgeTaskMetKeys directly - some cleanup and reorganizing of PgeTaskInstance This addresses bug OODT-414. https://issues.apache.org/jira/browse/OODT-414 Diffs (updated) - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1301267 trunk/pge/src/main/java/org/apache/oodt/cas/pge/logging/PgeLogHandler.java PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/logging/PgeLogRecord.java PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1300798 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1301267 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1301267 trunk/pge/src/test/org/apache/oodt/cas/pge/logging/TestPgeLogHandler.java PRE-CREATION Diff: https://reviews.apache.org/r/4375/diff Testing --- Added and updated unit-tests Thanks, brian
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 03:18:23.793698) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, Sheryl John, and Thomas Bennett. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1301315 trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1301315 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Google Summer of Code 2012
Hi Ross, Awesome, that sounds great! I saw your message on OODT-402 and responded there as well. I will be happy to mentor this issue and am looking forward to you contributing to GSoC and more broadly, to Apache OODT as a whole! Cheers, Chris On Mar 16, 2012, at 5:23 AM, Ross Laidlaw wrote: Hi Chris, I'm a student at Oxford University in England, studying for a masters degree in Software Engineering. I'm very interested in the OODT project and I'd like to get involved. I'm hoping to participate in Google Summer of Code 2012. This would be an ideal opportunity for me to deliver a package of work within a set period of time and hopefully demonstrate my abilities to the OODT team. After that, if the OODT team feel that my work is satisfactory and would like me to contribute more, I would very much like to continue contributing to the project beyond the end of GSoC. Therefore, please could I put myself forward as a candidate to deliver OODT-402 for GSoC 2012? I've started reading through the documentation for OODT, in particular the CAS FileManager user guide and developer guide, with the aim of producing a draft proposal for GSoC. Apologies if you received this message twice - I posted a similar message to the OODT-402 page in JIRA (https://issues.apache.org/jira/browse/OODT-402). Many thanks for your time, Ross On Sun, 04 Mar 2012 17:23:01 GMT, Chris Mattmann wrote: Hey Guys, FYI here: https://issues.apache.org/jira/browse/OODT-402 I just wrote up what I think would be a cool Google Summer of Code project. I'll forward along the GSoC email information. If you have great OODT ideas that you'd like to see turned into a GSoC project, and you have the time to mentor a student, we have a shot at it being selected. So, bring em' on! Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: Review Request: Port CAS-PGE's PcsMetadataKeys to PgeTaskMetKeys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4359/#review6031 --- Ship it! +1 - Paul On 2012-03-15 08:25:38, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4359/ --- (Updated 2012-03-15 08:25:38) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- Will allow for specification of new keys while still supported legacy keys This addresses bug OODT-411. https://issues.apache.org/jira/browse/OODT-411 Diffs - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1300799 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1300799 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PcsMetadataKeys.java 1300798 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1300799 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1300799 trunk/pge/src/test/org/apache/oodt/cas/pge/metadata/TestPgeTaskMetKeys.java 1300799 Diff: https://reviews.apache.org/r/4359/diff Testing --- existing unit-tests for PgeTaskMetKeys Thanks, brian
Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/#review6043 --- ./trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java https://reviews.apache.org/r/4358/#comment13016 I think HsqlDbFriendlyDataSourceCatalog.paginateQuery was missing an ORDER BY ... DESC clause here, whereas DataSourceCatalog (line 1960) does a descending sort. Since the primary key product_id is incremented (src/testdata/testcat.sql), and ShouldBeFirstForPage.txt is inserted LAST, DataSourceCatalog returns ShouldBeFirstForPage.txt as the first item on the FIRST page when ordered DESC. To me, it seems like DataSourceCatalog works properly and the test case needs to be modified. - Ricky On 2012-03-16 14:14:17, Chris Mattmann wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/ --- (Updated 2012-03-16 14:14:17) Review request for oodt, brian Foster, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- DataSourceCatalog compatible with HypersonicSQL This addresses bug OODT-410. https://issues.apache.org/jira/browse/OODT-410 Diffs - ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java 1301321 ./trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java 1301321 Diff: https://reviews.apache.org/r/4358/diff Testing --- Thanks, Chris
Re: Review Request: Create Java Logger Handler for CAS-PGE
On 2012-03-16 14:28:24, Chris Mattmann wrote: trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java, line 108 https://reviews.apache.org/r/4375/diff/2/?file=93053#file93053line108 what makes this throw an exception now? I'm gradually adding throws Exception to all the methods in CAS-PGE's PGETaskInstance so if they are overridden by the extending class, the extending class can easily kill CAS-PGE (and give the reason why it should stop) by just throwing any Exception - brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4375/#review6035 --- On 2012-03-16 08:25:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4375/ --- (Updated 2012-03-16 08:25:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- This handler will allow for multiple CAS-PGE instances to exist in the same JVM while routing the logging from each to their own log files in their execution directory - This patch also includes: - adding support for PgeMetadata to take PgeTaskMetKeys directly - some cleanup and reorganizing of PgeTaskInstance This addresses bug OODT-414. https://issues.apache.org/jira/browse/OODT-414 Diffs - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1301267 trunk/pge/src/main/java/org/apache/oodt/cas/pge/logging/PgeLogHandler.java PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/logging/PgeLogRecord.java PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1300798 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1301267 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1301267 trunk/pge/src/test/org/apache/oodt/cas/pge/logging/TestPgeLogHandler.java PRE-CREATION Diff: https://reviews.apache.org/r/4375/diff Testing --- Added and updated unit-tests Thanks, brian
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 19:11:34.840950) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, Sheryl John, and Thomas Bennett. Changes --- I agree. reducedMet should equal fullMet for each element in the list, and replaceMetadata() guarantees that. Also, used list iterator syntactic sugar. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs (updated) - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1300633 trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1300633 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Review Request: OODT-413: filemgr query throws NPE when some products have undefined metadata values
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/#review6056 --- Ship it! Gotta love that syntactic sugar! LGTM! - Chris On 2012-03-16 19:11:34, Ricky Nguyen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4374/ --- (Updated 2012-03-16 19:11:34) Review request for oodt, Chris Mattmann, brian Foster, Paul Ramirez, Sheryl John, and Thomas Bennett. Summary --- filemgr query throws NPE when some products have undefined metadata values This addresses bug OODT-413. https://issues.apache.org/jira/browse/OODT-413 Diffs - trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/LuceneCatalog.java 1300633 trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestLuceneCatalog.java 1300633 Diff: https://reviews.apache.org/r/4374/diff Testing --- tested at CHLA with command: ./filemgr-client -u $FILEMGR_URL -op -sql -of '$VpsEpisodeStartTime' -q SELECT VpsEpisodeStartTime FROM CernerEvents Thanks, Ricky
Re: Review Request: Create Java Logger Handler for CAS-PGE
On 2012-03-16 14:28:24, Chris Mattmann wrote: trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java, line 568 https://reviews.apache.org/r/4375/diff/2/?file=93048#file93048line568 I like this having an explicit step now. after this patch goes in, the trunk CAS-PGE will have a feature which was attempted in wengine branch CAS-PGE but i never got working right... when multiple CAS-PGEs run in the same JVM, which is the normal case since several will end up in a CAS-Resource's BatchStub, they will log on top of each other since the logger was based on the class name and CAS-PGE usually had the same class name so was impossible to filter loggers by that... so it was really hard to sift through the logs finding relevant logger for a given CAS-PGE instance... this patch fixed that by making logging base on PGETask/Name (or legacy PGETask_Name) - brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4375/#review6035 --- On 2012-03-16 08:25:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4375/ --- (Updated 2012-03-16 08:25:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- This handler will allow for multiple CAS-PGE instances to exist in the same JVM while routing the logging from each to their own log files in their execution directory - This patch also includes: - adding support for PgeMetadata to take PgeTaskMetKeys directly - some cleanup and reorganizing of PgeTaskInstance This addresses bug OODT-414. https://issues.apache.org/jira/browse/OODT-414 Diffs - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1301267 trunk/pge/src/main/java/org/apache/oodt/cas/pge/logging/PgeLogHandler.java PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/logging/PgeLogRecord.java PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1300798 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1301267 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1301267 trunk/pge/src/test/org/apache/oodt/cas/pge/logging/TestPgeLogHandler.java PRE-CREATION Diff: https://reviews.apache.org/r/4375/diff Testing --- Added and updated unit-tests Thanks, brian
Re: Review Request: OODT-410: DataSourceCatalog compatible with HypersonicSQL
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4358/ --- (Updated 2012-03-17 01:40:37.106062) Review request for oodt, brian Foster, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- OK guys, I got it working. Thanks for the comments to date, Ricky and Brian et al! All tests pass for me now. Summary --- DataSourceCatalog compatible with HypersonicSQL This addresses bug OODT-410. https://issues.apache.org/jira/browse/OODT-410 Diffs (updated) - ./trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java 1301321 ./trunk/filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSourceCatalog.java 1301321 Diff: https://reviews.apache.org/r/4358/diff Testing --- Thanks, Chris
Re: Review Request: Improved CAS-PGE logging, product ingest fail CAS-PGE will fail, and PgeConfigBuilder configurable via PgeTaskMetKeys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4393/#review6060 --- trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13030 Thanks for keeping this! trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13032 Typo, should be //Run the PGE and process its data. trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13031 Typo, should be //Ingest products. trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13033 Should we LOG.fine this? Maybe: log(Level.FINE, File: [+file+] matched.); trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13034 extraneous removal of this.build... for consistency? trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13035 Love it! trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13036 Should be: Verifying trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4393/#comment13037 Hahahahahha! Nice removal! - Chris On 2012-03-16 23:32:12, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4393/ --- (Updated 2012-03-16 23:32:12) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- The 2 main issues here (and they are small changes) are: - CAS-PGE now fails when products fail to ingest * this was the changes found in createPgeConfig() and the addition of the key to PgeTaskMetKeys - PgeConfigBuilder is configurable via PgeTaskMetKeys * this was the removal of the Exception catching in: runIngestCrawler The other issue is just overall improvement to PgeTaskInstance logging ALSO: run(Metadata, WorkflowTaskConfiguration) method was moved to the top for better visibility This addresses bugs OODT-418, OODT-419 and OODT-420. https://issues.apache.org/jira/browse/OODT-418 https://issues.apache.org/jira/browse/OODT-419 https://issues.apache.org/jira/browse/OODT-420 Diffs - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1301762 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1301761 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1301762 Diff: https://reviews.apache.org/r/4393/diff Testing --- Still need to write unit-test for this patch... Thanks, brian
Review Request: Improved CAS-PGE logging, product ingest fail CAS-PGE will fail, and PgeConfigBuilder configurable via PgeTaskMetKeys
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4393/ --- Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- The 2 main issues here (and they are small changes) are: - CAS-PGE now fails when products fail to ingest * this was the changes found in createPgeConfig() and the addition of the key to PgeTaskMetKeys - PgeConfigBuilder is configurable via PgeTaskMetKeys * this was the removal of the Exception catching in: runIngestCrawler The other issue is just overall improvement to PgeTaskInstance logging ALSO: run(Metadata, WorkflowTaskConfiguration) method was moved to the top for better visibility This addresses bugs OODT-418, OODT-419 and OODT-420. https://issues.apache.org/jira/browse/OODT-418 https://issues.apache.org/jira/browse/OODT-419 https://issues.apache.org/jira/browse/OODT-420 Diffs - trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1301762 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1301761 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1301762 Diff: https://reviews.apache.org/r/4393/diff Testing --- Thanks, brian
