[GSoC] OODT-436 and OODT-437
(apologies for the cross post) Hi All, FYI, I'd like to call your attention to OODT-436 [1] and OODT-437 [2], two new spatial projects proposed for GSOC 2012. One is to enable the File Manager to support a new GeoJSON service which will make OODT products available in GIS systems like qGIS, etc., as layers. This will also have a tie in to the work going on in SIS right now and the current thoughts about loading data from OODT into SIS related to GeoRSS, etc., and the work that Ross has been doing in OODT-402 [3]. In addition, the other issue is to support an image overlay service in OODT so that we can begin to support a raster like capability with OODT data. I'm super excited about both of these issues, so feel free to ask questions and if there are any GSoC potential students out there interested in these projects, please contact myself, or Paul Ramirez (who is in the process of signing up to become a mentor for GSoC). Thanks! Cheers, Chris [1] https://issues.apache.org/jira/browse/OODT-436 [2] https://issues.apache.org/jira/browse/OODT-437 [3] https://issues.apache.org/jira/browse/OODT-402 ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6686 --- trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4628/#comment14489 this seems like an ancillary change to this patch. However, it's a useful functionality so I don't feel strongly about separating it out. Just be wary of stuff like this (b/c as it grows) it can take away from the purpose of the patch ;) trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java https://reviews.apache.org/r/4628/#comment14496 +like - Chris On 2012-04-03 21:56:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-03 21:56:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Organization of Java-SQL-DB Interface
All, I am currently working on designing a catalog implementation that interacts with an oracle database for use with the filemanager component. The current version uses sql that is inline with the java code. However, a recent need for maintenance to this SQL has highlighted the need to pull the sql into separate files for easier maintenance. What is the preferred way in oodt to store such information and load it? -Michael Starch
Re: Organization of Java-SQL-DB Interface
hey mike,Are you looking do something like this? Storing queries in a file and adding in values utilizing PathUtils replacement:SELECT DISTINCT product_idfrom MOA_IASI_L1C_Metadata where element_id = 'urn:peate:NominalDate' and element_value = '[NominalDate]';While this is cool and convenient, i see a few problems with dynamically loading the queries:1) The inline queries in Java are able to be well unit-tested2) Making the queries too easy to change can be a production code nightmare... This is the SQL which determines if the Product's metadata gets ingested correctly or not... i wouldn't want to just be able to easily changes these without going through a rigorous testing process to be sure they work correctlyWhile i noted these problems, i'm not adverse to this being checked in as a new implementation... i'd be interested to see what you've got... could you create a JIRA issue and attach your code?-brianNot sure you want these SQL commands to be easy to change... you pull them out then they become hard to unit-test... alsoOn Apr 04, 2012, at 01:00 PM, "Starch, Michael D (388L)" michael.d.sta...@jpl.nasa.gov wrote:All, I am currently working on designing a catalog implementation that interacts with an oracle database for use with the filemanager component. The current version uses sql that is inline with the java code. However, a recent need for maintenance to this SQL has highlighted the need to pull the sql into separate files for easier maintenance. What is the preferred way in oodt to store such information and load it? -Michael Starch
Re: Google Summer of Code 2012
Hi Chris, I've submitted a first draft of my GSoC 2012 proposal to the GSoC website. Here's a link to the proposal: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/rlaidlaw/1 If you have time, please could you review it and let me know if it's up to scratch? If you'd like me to make any amendments or additions, please let me know and I'll update it. Many thanks, Ross On 17 March 2012 13:10, Ross Laidlaw rlaidlaw.o...@gmail.com wrote: Hi Chris, Many thanks for your reply. That's great news! I'm really keen to work on this project and very happy that you will be mentoring. I'll try to update you regularly on my progress and I'll definitely post any questions to the mailing lists, as requested. Ross On 16 March 2012 14:46, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Ross, Awesome, that sounds great! I saw your message on OODT-402 and responded there as well. I will be happy to mentor this issue and am looking forward to you contributing to GSoC and more broadly, to Apache OODT as a whole! Cheers, Chris On Mar 16, 2012, at 5:23 AM, Ross Laidlaw wrote: Hi Chris, I'm a student at Oxford University in England, studying for a masters degree in Software Engineering. I'm very interested in the OODT project and I'd like to get involved. I'm hoping to participate in Google Summer of Code 2012. This would be an ideal opportunity for me to deliver a package of work within a set period of time and hopefully demonstrate my abilities to the OODT team. After that, if the OODT team feel that my work is satisfactory and would like me to contribute more, I would very much like to continue contributing to the project beyond the end of GSoC. Therefore, please could I put myself forward as a candidate to deliver OODT-402 for GSoC 2012? I've started reading through the documentation for OODT, in particular the CAS FileManager user guide and developer guide, with the aim of producing a draft proposal for GSoC. Apologies if you received this message twice - I posted a similar message to the OODT-402 page in JIRA (https://issues.apache.org/jira/browse/OODT-402). Many thanks for your time, Ross On Sun, 04 Mar 2012 17:23:01 GMT, Chris Mattmann wrote: Hey Guys, FYI here: https://issues.apache.org/jira/browse/OODT-402 I just wrote up what I think would be a cool Google Summer of Code project. I'll forward along the GSoC email information. If you have great OODT ideas that you'd like to see turned into a GSoC project, and you have the time to mentor a student, we have a shot at it being selected. So, bring em' on! Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: Google Summer of Code 2012
http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/rlaidlaw/1 Argh. When I try to view it, I get: This proposal is not made public, and you are not the student who submitted the proposal, nor are you a mentor for the organization it was submitted to. Maybe another OODT'er can see it? --k
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6670 --- trunk/pge/src/main/resources/examples/Crawler/action-beans.xml https://reviews.apache.org/r/4628/#comment14432 I'd define these properties in another file and then include them here. This is only a suggestion and not a just but I see the properties as something that could likely be changed or set to a fixed value and if we factor it out of here we can keep people from touching this file too much. I think this file just makes peoples heads spin at first but the properties don't (i.e. it hides the Spring goodness in a good way). trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml https://reviews.apache.org/r/4628/#comment14431 Put these examples inside comment tags as they wouldn't work as they existed anyhow. Also putting a longer description in the comment would help (i.e. one or more of these is not as helpful as what it does functionally. Why did we remove the files tag? Is this no longer supported? If it is then I recommend putting it back in but commented out. For instance, I'd expect that instead of metadata keys you want to set more of what will be done with that custom metadata would be of use. Also an example of multivalued metadata. - Paul On 2012-04-03 21:56:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-03 21:56:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Re: Organization of Java-SQL-DB Interface
Hi Brian, I shared your concerns, *and* your enthusiasm. Mike: I think you can use JIRA issue https://issues.apache.org/jira/browse/OODT-393 and we'll be happy to help you shepherd this. Cheers, Chris On Apr 4, 2012, at 1:37 PM, holeno...@me.com holeno...@me.com wrote: hey mike, Are you looking do something like this? Storing queries in a file and adding in values utilizing PathUtils replacement: SELECT DISTINCT product_id from MOA_IASI_L1C_Metadata where element_id = 'urn:peate:NominalDate' and element_value = '[NominalDate]'; While this is cool and convenient, i see a few problems with dynamically loading the queries: 1) The inline queries in Java are able to be well unit-tested 2) Making the queries too easy to change can be a production code nightmare... This is the SQL which determines if the Product's metadata gets ingested correctly or not... i wouldn't want to just be able to easily changes these without going through a rigorous testing process to be sure they work correctly While i noted these problems, i'm not adverse to this being checked in as a new implementation... i'd be interested to see what you've got... could you create a JIRA issue and attach your code? -brian Not sure you want these SQL commands to be easy to change... you pull them out then they become hard to unit-test... also On Apr 04, 2012, at 01:00 PM, Starch, Michael D (388L) michael.d.sta...@jpl.nasa.gov wrote: All, I am currently working on designing a catalog implementation that interacts with an oracle database for use with the filemanager component. The current version uses sql that is inline with the java code. However, a recent need for maintenance to this SQL has highlighted the need to pull the sql into separate files for easier maintenance. What is the preferred way in oodt to store such information and load it? -Michael Starch ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: Google Summer of Code 2012
Hi Ross, Thanks. LGTM :) I commented as much on the proposal. I also like that you saw the recent issues submitted by Paul and Cam, and are interested in those too. We have a synergistic opportunity here! Glad that you are a part. Cheers, Chris On Apr 4, 2012, at 2:23 PM, Ross Laidlaw wrote: Hi Chris, I've submitted a first draft of my GSoC 2012 proposal to the GSoC website. Here's a link to the proposal: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/rlaidlaw/1 If you have time, please could you review it and let me know if it's up to scratch? If you'd like me to make any amendments or additions, please let me know and I'll update it. Many thanks, Ross ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
