Re: Needed OODT Improvments
Sounds like some excellent maintenance activities. +1 -Cam On Wed, Apr 15, 2015 at 3:27 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: ACK On Wed, Apr 15, 2015 at 5:39 PM, Michael Starch starc...@umich.edu wrote: Dev, I've been talking with Tom and we've noted a list of things that should be improved on OODT. - pom/maven: - fix dependency circles in radix - make extra components with large number of dependencies optional (pending pull request) - move dependency versions to top-level pom making versions consistent - update dependency versions - replace xml-rpc development-only server - separate REST interfaces and client code for ops ui and cas-curator - cleanup - deprecate unused or superseded sub-components - spin-off non-essential functionality into sibling projects - documentation Any thoughts? -Michael -- *Lewis*
Re: New Committer: Dana Freeborn
That's excellent news Tom. Thanks for keeping the process moving Cameron On Friday, March 27, 2015, Tom Barber tom.bar...@meteorite.bi wrote: The Project Management Committee (PMC) for Apache OODT has asked Dana Freeborn to become a committer and we are pleased to announce that they have accepted. -- Tom -- Sent from a Tin Can attached to a String
Re: i am lost
Welcome Eva, I second Lewis and please know there are all different levels of expert on this mailing list which is great since OODT can do a lot. One idea that I've heard before (I cannot take credit ) is that OODT is more like a data management framework and less like an application. Starting out typically feels like someone just dumped a bunch of Legos on the floor and you don't really know where to start. That is normal, I still feel like that at times. As Lewis mentioned if you give us a use case or a problem you're working on we'll be more than happy to help. Cheers, Cameron On Thursday, March 19, 2015, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Eva, Your at the right place :) I would guide you towards the wiki documentation if I knew what you were trying to do. Anything in particular you are struggling with? Thanks Lewis On Thursday, March 19, 2015, Eva Schlauch eschla...@mpifr-bonn.mpg.de javascript:; wrote: Hello there, so I am new to OODT and the mailing list. The nice people at our computing center have installed an OODT for me but I don't seem to get a grip on it. How can I find threads that may already answer all my questions? Where to start? Where can I send my newby this-is-all-too-much-please-help-me requests? Best regards, Eva -- *Lewis*
Re: OODT Tests
Hey Folks,
I realize I am late to the discussion but in the past when one person has a
working build and another person is having issues the problem/solution
(depending on how you look at it) is in the local maven repo.
On my local machine Mac OS X 10.9.5 I blew out my maven repo and did an
install from a fresh checkout of trunk and I am getting the same error that
Lewis has. I believe that Jenkins does it's build/test against an empty
local maven repo.
This is kind of like asking if there is petrol in the tank when a car
breaks down, but I figure it couldn't hurt to mention it.
Good Luck,
Cameron
On Fri, Jan 23, 2015 at 9:37 AM, Tom Barber tom.bar...@meteorite.bi wrote:
Its a null pointer disguised as a broken url. I'll take a look later.
Tom
On 23 Jan 2015 17:20, Ramirez, Paul M (398M)
paul.m.rami...@jpl.nasa.gov
wrote:
Weird one. Is there something wrong with your network connection? Is that
URL being blocked? The stack trace says URL does not exist but I'm able
to
access it here on my phone.
--Paul
Sent from my iPhone
On Jan 23, 2015, at 9:10 AM, Lewis John Mcgibbney
lewis.mcgibb...@gmail.com wrote:
Hi Tom,
Thanks for heads up.
I am kinda confused as to local failure, I'm reproducing failures on
Jenkins
Regression on
org.apache.oodt.cas.protocol.http.TestHttpProtocol.testLSandCD
Error Message
Failed to cd to {parent = 'null', path =
'repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http',
isDir = 'true'} : URL does not exist
'
http://svn.apache.org/repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http
'
Stacktrace
org.apache.oodt.cas.protocol.exceptions.ProtocolException: Failed to
cd to {parent = 'null', path =
'repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http',
isDir = 'true'} : URL does not exist
'
http://svn.apache.org/repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http
'
at org.apache.oodt.cas.protocol.http.HttpProtocol.cd
(HttpProtocol.java:78)
at
org.apache.oodt.cas.protocol.http.TestHttpProtocol.testLSandCD(TestHttpProtocol.java:52)
Caused by: java.io.IOException: URL does not exist
'
http://svn.apache.org/repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http
'
at
org.apache.oodt.cas.protocol.http.util.HttpUtils.isDirectory(HttpUtils.java:126)
at org.apache.oodt.cas.protocol.http.HttpProtocol.cd
(HttpProtocol.java:72)
... 23 more
Caused by: java.lang.NullPointerException
at
org.apache.oodt.cas.metadata.util.MimeTypeUtils.autoResolveContentType(MimeTypeUtils.java:223)
at
org.apache.oodt.cas.metadata.util.MimeTypeUtils.autoResolveContentType(MimeTypeUtils.java:132)
at
org.apache.oodt.cas.protocol.http.util.HttpUtils.isDirectory(HttpUtils.java:122)
... 24 more
On Fri, Jan 23, 2015 at 8:45 AM, Tom Barber tom.bar...@meteorite.bi
wrote:
The only real problem with the build is that the Jersey Client 1.x POM
has
been knackered for god knows how long, but as long as I can remember
working with REST stuff, and the Hadoop Client jar in the Resource
Manager
has it hardcoded as a dependency. So the other night I forced it to
use
a
newer version, so it failed a few times and I stepped through the
required
upgraded jars.
Apart from that and a few minor tweaks I think its alright.
Tom
On 23/01/15 16:41, Lewis John Mcgibbney wrote:
Hi Folks,
Builds have been dodgy for a while now.
Anyone have a clue what happened?
I just looked at out Jenkins build record for trunk and quite frankly
it
kinda appalling.
I'm building RADiX for a customer right now then I'm going to take
some
time looking at tests again.
Lewis
--
*Tom Barber* | Technical Director
meteorite bi
*T:* +44 20 8133 3730
*W:* www.meteorite.bi | *Skype:* meteorite.consulting
*A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2
7YG, UK
--
*Lewis*
Re: Shutdown of the OODT Apache User Mailing List
+1. Thanks for bringing this up Sean. -Cam On Mon, Dec 29, 2014 at 10:47 AM, BW bw...@mysoftcloud.com wrote: +1 On Monday, December 29, 2014, Tyler Palsulich tpalsul...@gmail.com wrote: +1, shut it down. Tyler On Mon, Dec 29, 2014 at 10:43 AM, ke...@apache.org wrote: Folks: We have two public mailing lists: - dev@oodt.apache.org, ostensibly for discussion of the development of Apache OODT - u...@oodt.apache.org, allegedly for end users to ask questions and find answers about Apache OODT Greg Stein noted during the last Apache board meeting that since the user@ traffic is so low that we're needlessly segmenting our community. OODT itself is software that's more for developers than end users since it requires such high levels of integration into existing architectures. Following the K.I.S.S. Principle [2], I'd like to call for a community vote to shut down the u...@oodt.apache.org mailing list: [ ] +1, shut down u...@oodt.apache.org [ ] ±0, don't care about u...@oodt.apache.org's fate [ ] -1, do NOT shut down u...@oodt.apache.org, because … I'll leave the vote open at least 72 hours. And here's my +1. Best wishes --k [1] https://svn.apache.org/repos/private/foundation/board/board_agenda_2014_12_17.txt [2] http://en.wikipedia.org/wiki/KISS_principle
Re: Why is pushpull not always working?
Hey Lewis, Last time I dug into this I was running toward the Active vs. Passive FTP settings, but I am not sure that is the root cause. One option or fix would be to add in a timeout parameter, where if the pushpull daemon doesn't get a reply in a set amount of time, it should stop and restart. Every time I have seen pushpull stall or hang, a simple stop and restart gets it working again. I know Ross Laidlaw is using pushpull on a his Near Real Time snow processing pipeline, so I am sure he will be very interested in your work Lewis. Thanks, Cameron On Sat, Nov 1, 2014 at 10:58 AM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Thanks Chris. OK doke. i am going to debug this today and find out what is happening. Thanks for heads up. Lewis On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann chris.mattm...@gmail.com wrote: Thanks Lewis - in my mind it boils down somewhat to the protocols that are implemented and the way they behave with FTP servers and so forth. Ultimately, we should be using the ones that are in PushPull plugins, but they are virally licensed and I haven¹t found a better alternative. Another @bfoster update here would be great, Brian? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Lewis John Mcgibbney lewis.mcgibb...@gmail.com Reply-To: dev@oodt.apache.org Date: Saturday, November 1, 2014 at 12:56 AM To: dev@oodt.apache.org dev@oodt.apache.org Subject: Why is pushpull not always working? Hi Folks, I really couldn't think of a better title for this thread so I apologize in advance. I recently set up pushpull with FTP to crawl remote ftp site [0] which is publicly available. I am getting the following INFO: Launching Daemons . . . Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon registerRMIServer INFO: Created Daemon ID = 90121 on RMI registry port 9012 Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonLauncher launchDaemons INFO: Creating Daemon with ID = 1 Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning INFO: Daemon with ID = 90121 was given permission to run Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep until Sat Nov 01 00:48:00 PDT 2014 Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning INFO: Daemon with ID = 90121 was given permission to run Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep until Sat Nov 01 00:51:00 PDT 2014 Nov 01, 2014 12:50:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning INFO: Daemon with ID = 90121 was given permission to run Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep until Sat Nov 01 00:54:00 PDT 2014 Nov 01, 2014 12:53:59 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning INFO: Daemon with ID = 90121 was given permission to run Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep until Sat Nov 01 00:57:00 PDT 2014 No errors... no pulling... no nothing! I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris' plugins page [2]. My question is simple... why is pushpull known NOT to work on occasion? I am going to debug this tomorrow but I thought I would ask this none-the-less. Thanks Lewis [0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/ [1] https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide #OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun ningthe./pushpullscript.WhatshouldIdo ? [2] https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins -- *Lewis* -- *Lewis* -- Sent from a Tin Can attached to a String
Re: what is batch stub? Is it necessary?
Valerie, This could be nothing, or it could be the root cause...your output XML tags are malformed. !-- Files to ingest -- output/ /output Should be: !-- Files to ingest -- output /output No trailing slash in the opening tag. It might be failing since it cannot parse the XML cleanly. Doesn't explain the batchstub stuff, but might be related to the latest challenge you sighted: *The problem is that the ExternScriptTaskInstance is unable to recognize the command line arguments that I want to pass to the crawler_launcher script.* In my experience the XML is super finicky. Good Luck, and keep the questions coming. We are here to help. -Cameron On Wed, Oct 8, 2014 at 7:37 PM, Ramirez, Paul M (398J) paul.m.rami...@jpl.nasa.gov wrote: +1 billion --Paul Sent from my iPhone On Oct 8, 2014, at 5:55 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Folks, Is it possible to create a parent issue for defining XSD's for all of the XML file we need ti OODT? I do not know them all, but from this thread alone, it is clear that we could do with setting some kind of restrictions on what can be included within task and configuration XML within OODT. Thoughts? Lewis On Wed, Oct 8, 2014 at 5:44 PM, Verma, Rishi (398J) rishi.ve...@jpl.nasa.gov wrote: Hi Val, Yep - here’s a link to the tasks.xml file: https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt-netscan/workflow/src/main/resources/policy/tasks.xml The problem is that the ExternScriptTaskInstance is unable to recognize the command line arguments that I want to pass to the crawler_launcher script. Hmm.. could you share your workflow manager log, or better yet, the batch_stub output? Curious to see what error is thrown. Is a script file being generated for your PGE? For example, inside your [PGE_HOME] directory, and within the particular job directory created for your execution of a workflow, you will see some files starting with “sciPgeExeScript_…”. You’ll find one for your pgeConfig, and you can check to see what the PGE commands actually translate into, with respect to a shell script format. If that file is there, take a look at it, and validate whether the command works within the script (i.e. copy/paste and run the crawler command manually). Another suggestion is to take a step back, and build up slowly, i.e.: 1. Do an “echo” command within your PGE first. (e.g. cmd echo “Hello APL.” /tmp/test.txt/cmd) 2. If above works, do a crawler_launcher empty command(e.g. cmd/path/to/oodt/crawler/bin/crawler_launcher/cmd) and verify the batch_stub or Workflow Manager prints some kind of output when you run the workflow. 3. Build up your crawler_launcher command piece by piece to see where it is failing Thanks, Rishi On Oct 8, 2014, at 4:24 PM, Mallder, Valerie valerie.mall...@jhuapl.edu wrote: Hi Rishi, Thank you very much for pointing me to your working example. This is very helpful. My pgeConfig looks very similar to yours. So, I commented out the resource manager like you suggested and tried running again without the resource manager. And my problem still exists. The problem is that the ExternScriptTaskInstance is unable to recognize the command line arguments that I want to pass to the crawler_launcher script. Could you send me a link to your tasks.xml file? I'm curious as to how you defined your task. My pgeConfig and tasks.xml are below. Thanks! Val ?xml version=1.0 encoding=UTF-8? pgeConfig !-- How to run the PGE -- exe dir=[JobDir] shell=/bin/sh envReplace=true cmd[CRAWLER_HOME]/bin/crawler_launcher --operation --launchAutoCrawler \ --filemgrUrl [FILEMGR_URL] \ --clientTransferer org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory \ --productPath [JobInputDir] \ --mimeExtractorRepo [OODT_HOME]/extensions/policy/mime-extractor-map.xml \ --actionIds MoveFileToLevel0Dir/cmd /exe !-- Files to ingest -- output/ /output !-- Custom metadata to add to output files -- customMetadata metadata key=JobDir val=[OODT_HOME]/ metadata key=JobInputDir val=[FEI_DROP_DIR]/ metadata key=JobOutputDir val=[JobDir]/data/pge/jobs/ metadata key=JobLogDir val=[JobDir]/data/pge/logs/ /customMetadata /pgeConfig !-- tasks.xml **-- cas:tasks xmlns:cas=http://oodt.jpl.nasa.gov/1.0/cas; task id=urn:oodt:crawlerLauncherId name=crawlerLauncherName class=org.apache.oodt.cas.workflow.examples.ExternScriptTaskInstance conditions/ !-- There are no pre execution conditions right now -- configuration property name=ShellType value=/bin/sh / property name=PathToScript value=[CRAWLER_HOME]/bin/crawler_launcher envReplace=true / property name=PGETask_Name
Re: OODT Officially Stable - Hats off to Ross Laidlaw
Great work Ross. I know it hasn't been easy getting those tests fixed up, so thank you for being persistent and seeing it through. -Cam On Fri, Aug 8, 2014 at 2:05 PM, Ross Laidlaw rlaidlaw.o...@gmail.com wrote: Thanks all! Glad to contribute! There are some remaining issues to discuss/resolve with file manager tests (e.g. see OODT-736 and OODT-741), but we're getting there! There are also some issues with the protocol tests that Chris highlighted in OODT-644. So Jenkins builds will probably still fail sometimes, but hopefully we can fix these soon. Ross On 8 August 2014 13:47, Verma, Rishi (398J) rishi.ve...@jpl.nasa.gov wrote: Way to go Ross, way to go everyone! Rishi Sent from my iPhone On Aug 8, 2014, at 1:29 PM, Chris Mattmann chris.mattm...@gmail.com wrote: Fscking awesome!! Thank you Ross and everyone! Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tom Barber tom.bar...@meteorite.bi Reply-To: u...@oodt.apache.org Date: Friday, August 8, 2014 1:07 PM To: u...@oodt.apache.org Subject: Re: OODT Officially Stable - Hats off to Ross Laidlaw Wp On 8 Aug 2014 21:05, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Folks, The OODT trunk (0.7-SNAPSHOT) codebase is stable [0] now for the first time in a while which is extremely good news. I personally (and I'm sure I speak on behalf of many of us) am extremely appreciative of the work Ross has recently put in to ensuring the codebase is stabalized. Hats off Ross. The drinks are on me... and make sure to remind me about it. Have a great weekend folks. Lewis [0] https://builds.apache.org/job/oodt-trunk/1438/ -- Lewis
Re: [oodt-dev] pushpull with passive FTP
Tim and Chris, In the past I have run into a similar issue and I ended up using something else (wget or python) to fetch the data. That being said I also lost the ability to check with FileManager to see if I had already downloaded and archived a file. All this to say I think it would be a good idea to setup an explicit option/setting which would be active by default, but could be set to passive for cases like Tim's. This option becomes increasingly important as more and more of us are deploying code within VMs. I think this at least warrants a ticket in JIRA. -Cameron On Wed, Jul 23, 2014 at 5:15 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Moving this conversation to Apache OODT :) ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Stough, Timothy M (398F) timothy.m.sto...@jpl.nasa.gov Reply-To: OODT Developers oodt-...@list.jpl.nasa.gov Date: Wednesday, July 23, 2014 3:52 PM To: oodt-dev oodt-...@jpl.nasa.gov Subject: [oodt-dev] pushpull with passive FTP Hello All, I'm trying to use pushpull with Radix in a virtual machine. The FTP site that I'm talking to gives me a 500 error because, I think, I'm trying to use active FTP from within a NAT connected VM. So, how can I convince pushpull to use passive FTP? Thanks, Tim - Tim Stough NASA/Caltech Jet Propulsion Lab Senior System ArchitectAssociate Program Manager, NASA Applied Sciences Program, Disaster Management for Solid Earth Hazards Data Understanding Group (Section 398) 626-644-6574 (cell) 818-393-5347 (office) -
Re: [jira] [Commented] (OODT-694) Upgrade xmlrpc library in OODT 2.0.1 -- 3.0a1
Lewis, I poked around the project a bit and all the download links and the one to subversion all 404'd. We should probably email their list and see if the code is even around anymore. I was hoping that the lack of development just meant it is stable and the XMLRPC spec hasn't changed much. -Cam On Wed, May 14, 2014 at 2:05 PM, Lewis John McGibbney (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/OODT-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998018#comment-13998018] Lewis John McGibbney commented on OODT-694: --- http://ws.apache.org/xmlrpc/apidocs/index.html Upgrade xmlrpc library in OODT 2.0.1 -- 3.0a1 -- Key: OODT-694 URL: https://issues.apache.org/jira/browse/OODT-694 Project: OODT Issue Type: Improvement Components: build proces, file manager Affects Versions: 0.6 Reporter: Lewis John McGibbney Fix For: 0.8 A _newer_ version of this library is available [0]... however there may be a more actively maintained xmlrpc lib we could use as this one looks to be pretty stagnant. I am digging in to XmlRpcClientExceptions being thrown within filemgr tests and it was lovely to realize that an upgrade of the xmlkrpc lib is also required. [0] http://search.maven.org/#artifactdetails|xmlrpc|xmlrpc|3.0a1|N%2FA -- This message was sent by Atlassian JIRA (v6.2#6252) -- Sent from a Tin Can attached to a String
Re: Query Tool Deprecation
Tom, TLDR: Run ./filemgr-client and review the long list of options. -sql and -lucene are in there to replace the query tool. Below is a detailed list of how I checked this. The longer version: == I *think* that the OODT 'cli' module was going to unify all of the Command Line Interfaces to the OODT components. The README for cli is here: http://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt The latest filemgr-client script (0.7-SNAPSHOT) depends on the cli component, and requires a cmd-line-options.xml and cmd-line-actions.xml. I just updated my local checkout to the latest version and ran: $ cd oodt/trunk $ mvn clean install $ cd filemgr $ mvn clean install $ cd target $ tar -zxvf cas-filemgr-0.7-SNAPSHOT-dist.tar.gz # This will just untar the dist in place for testing purposes $ cd cas-filemgr-0.7-SNAPSHOT/bin/ $ vi filemgr-client # Need to set $JAVA_HOME, etc... $ ./filemgr-client - | Short | Long | Description - -op,--operationDeclare that you wish to present an operation SubOptions: Required: Optional: -addPT, --addProductType Triggers addProductType Action ... ... ... -sql, --sqlQueryTriggers sqlQuery Action Requirement Rules: [sqlQuery : REQUIRED] -dmet, --dumpMetadataTriggers dumpMetadata Action Requirement Rules: [dumpMetadata : REQUIRED] -lucene,--luceneQuery Triggers luceneQuery Action Requirement Rules: [luceneQuery : REQUIRED] So Tom i think those cli options (-sql, or -lucene) are the replacements for query_tool. -Cameron On Mon, Mar 17, 2014 at 1:38 PM, Tom Barber tom.bar...@meteorite.bi wrote: Hi guys I want to look at querying the filemgr, but I look at the querytool source and its been deprecated. Whats its replacement? Thanks Tom -- *Tom Barber* | Technical Director meteorite bi *T:* +44 20 8133 3730 *W:* www.meteorite.bi | *Skype:* meteorite.consulting *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK -- Sent from a Tin Can attached to a String
Re: borked links
Juha, The URL will get updated soon. In the meantime the proper URL is: http://www.apache.org/dyn/closer.cgi/oodt/apache-oodt-0.6-src.zip Thanks for reporting this. -Cameron On Wed, Jan 29, 2014 at 3:19 PM, Juha Vierinen x...@mit.edu wrote: Downloads - 0.6 Release http://www.apache.org/dyn/closer.cgi/oodt/apache-oodt-0.6.zip juha -- Sent from a Tin Can attached to a String
Re: ApacheCon 2014
Tom, OODT as a business application is a novel use case for sure. Perhaps to help, if you took some of the architecture diagrams and tried to relabel the pieces with Business jargon instead of NASA/Science jargon it might help. Just this week several of the JPL folks were at Caltech meeting with researchers from MIT Haystack and Cambridge University, and they had a lot of questions about acronyms like PGE, and Chris told them to replace it with algorithm. Sorry it isn't a workable non-science use case, but it might help fill in the gap between science and business. Good luck, Cameron On Wed, Jan 29, 2014 at 8:54 PM, Chris Mattmann chris.mattm...@gmail.comwrote: Chris, smells like another talk. Please submit :) Seriously, you have it written below :) Submit! :) Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Christopher Warner christopher.war...@gmail.com Reply-To: dev@oodt.apache.org Date: Wednesday, January 29, 2014 8:45 AM To: dev@oodt.apache.org Subject: Re: ApacheCon 2014 One of the more difficult things for me trying to sell OODT was espousing why it's useful for structured processes whether they be government initiated protocols for data handling or workflow handling of that data. In some cases I'm dealing with large data sets that exist in things like excel, word, or some other unstructured crazy format. So part of the problem is social and dealing with researchers outside of the science paradigm whom aren't readily used to processing or extracting information from their data in that way. I think it's a great idea that would give an immediate visually accessible way of how it's useful for data sets not-related to science alone. From New York.. On Mon, Jan 27, 2014 at 2:08 PM, Tom Barber tom.bar...@meteorite.bi wrote: So I saw the call go out that they were short of applications and it got me thinking once again. My issue with Apache OODT is the fact currently a load of science bods use it (no offence ;) ) and the stuff you see written about and presented whilst a) is cool as hell b) lacks day to day real world applications for non scientists because if people do use it for non science applications, its not publicised very well which is where I want to try and advocate OODT But with my current clients don't have an obvious use case, but there must be a bunch if I sit down and think about them, so I was thinking about pitching a talk along the lines of Utilising the power of Apache OODT with mainstream Business Intelligence tools Where I can discuss a few real world scenarios and I was thinking I could demonstrate OODT integration with Pentaho Data Integration(Apache Licensed) using ETL to control ingestion and use Pentaho Data Integration to query and extract data from the Catalog. I could also use the Pentaho BI server to create a Dashboard framework to display relevant OODT information and as an alternative ingestion platform to the default File manager. Clearly non of this is yet written and I guess I'd write it if the talk got accepted and I got my fellow colleagues to help sponsor the trip, but i thought the idea was worth putting out there to garner feedback from the development community. Thoughts on a postcard please. Thanks Tom On 17/01/14 00:12, Chris Mattmann wrote: Hi Tom, I was thinking of submitting one on the use of Apache OODT in the Airborne Snow Observatory (ASO) project at JPL. or on the Wicket-based web interfaces. Do you have any ideas of what to submit? Cheers, Chris Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Tom Barber tom.bar...@meteorite.bi Reply-To: dev@oodt.apache.org Date: Tuesday, January 14, 2014 2:34 PM To: dev@oodt.apache.org dev@oodt.apache.org Subject: ApacheCon 2014 Hi folks, Just a quick query. I was wondering if any OODT developers were planning to submit any talk proposals into ApacheCon? Thanks Tom -- *Tom Barber* | Technical Director meteorite bi *T:* +44 20 8133 3730 *W:* www.meteorite.bi | *Skype:* meteorite.consulting *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK -- *Tom Barber* | Technical Director meteorite bi *T:* +44 20 8133 3730 *W:* www.meteorite.bi | *Skype:* meteorite.consulting *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK -- Christopher Warner http://cwarner.kernelcode.com -- Sent from a Tin Can attached to a String
Resource Manager 0.7-SNAPSHOT and Ganglia Monitor
Hey Rajith and Devs, I am upgrading my resource manager 0.3 to the latest version (currently 0.7-SNAPSHOT) and I have run into the Ganglia Monitor Plugin with the following errors: SEVERE: The required nodeId is not available: sneffels-11 :null java.lang.NullPointerException at org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73) at org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108) at java.lang.Thread.run(Thread.java:662) Dec 9, 2013 1:17:05 PM org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor getLoad SEVERE: The required nodeId is not available: kush-11 :null java.lang.NullPointerException at org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73) at org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108) at java.lang.Thread.run(Thread.java:662) I get this error message for all 44 of the nodes I have defined in my nodes.xml config. A couple questions: 1. Can I simply run resource manager without the Plugin via a change in config? I tried to comment out the config lines, but it didn't work. 2. If I cannot simply disconnect the Ganglia Dependency, then do I have to setup and run a Ganglia metadata daemon? I look forward to your reply, Cameron -- Sent from a Tin Can attached to a String
Re: Resource Manager 0.7-SNAPSHOT and Ganglia Monitor
Thanks Rajith for the prompt reply. I will check into getting that installed on our machine. Cameron On Dec 9, 2013 6:57 PM, Rajith Siriwardana rajithsiriward...@gmail.com wrote: hi Cameron, AFAIK there is no way to disable the Ganglia plugin, so you'll have to setup and run a Ganglia metadata daemon (Gmetad). I'll check that again. Cheers, Rajith On Tue, Dec 10, 2013 at 2:53 AM, Cameron Goodale sigep...@gmail.com wrote: Hey Rajith and Devs, I am upgrading my resource manager 0.3 to the latest version (currently 0.7-SNAPSHOT) and I have run into the Ganglia Monitor Plugin with the following errors: SEVERE: The required nodeId is not available: sneffels-11 :null java.lang.NullPointerException at org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73) at org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108) at java.lang.Thread.run(Thread.java:662) Dec 9, 2013 1:17:05 PM org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor getLoad SEVERE: The required nodeId is not available: kush-11 :null java.lang.NullPointerException at org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73) at org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134) at org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108) at java.lang.Thread.run(Thread.java:662) I get this error message for all 44 of the nodes I have defined in my nodes.xml config. A couple questions: 1. Can I simply run resource manager without the Plugin via a change in config? I tried to comment out the config lines, but it didn't work. 2. If I cannot simply disconnect the Ganglia Dependency, then do I have to setup and run a Ganglia metadata daemon? I look forward to your reply, Cameron -- Sent from a Tin Can attached to a String
Re: PushPull DaemonLauncher
Hey Brian,
Thanks for looking into this for me. Here is an excerpt from the logs
after I turned them up to FINE (trying to see anything I might be missing)
and I also added in some of my own log statements to track the Daemon
unregister method. I added in a 3 star (***) prefix to note the log
messages I added.
One thing to note is after my log messages the process should have shutdown
but it would just hang there, so I think something outside the startDaemon
method is keeping the process alive. Perhaps the missing notify() method
you mentioned can resolve it.
FINE: Should close connection in response to directive: close
Dec 3, 2013 9:46:13 AM org.apache.commons.httpclient.HttpConnection
releaseConnection
FINE: Releasing connection back to connection manager.
Dec 3, 2013 9:46:13 AM
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
addToDownloadQueue
WARNING: Skipping file {parent = 'null', path =
'/allData/1/MOD09GA/Recent/MOD09GA.A2013336.h25v05.005.NRT.hdf', isDir =
'false'} because it is already in staging area
Dec 3, 2013 9:46:16 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90111 on RMI registry port 9011 is shutting down
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon
unregister
***INFO: Unregistered the MBean
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon
unregister
***INFO: UnicastRemoteObject undone
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon
unregister
***INFO: After daemonListener
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
***INFO: Unregistration Completed
To be complete, here is a Patch of where I added in the Log messages (just
in case I added them incorrectly):
Index: Daemon.java
===
--- Daemon.java (revision 1546755)
+++ Daemon.java (working copy)
@@ -239,6 +239,8 @@
+ Daemon.this.getDaemonID() + on RMI registry
port
+ Daemon.this.rmiRegPort + is shutting down);
Daemon.this.unregister();
+LOG.log(Level.INFO, Unregistration Completed);
+
}
}).start();
}
@@ -251,8 +253,11 @@
this.mbs.unregisterMBean(new ObjectName(
org.apache.oodt.cas.pushpull.daemon:type=Daemon
+ this.getDaemonID()));
+LOG.log(Level.INFO, Unregistered the MBean);
UnicastRemoteObject.unexportObject(this, true);
+LOG.log(Level.INFO, UnicastRemoteObject undone);
this.daemonListener.wasUnregisteredWith(this);
+LOG.log(Level.INFO, After daemonListener);
} catch (Exception e) {
e.printStackTrace();
}
-Cameron
On Tue, Dec 3, 2013 at 11:41 AM, Brian Foster holeno...@me.com wrote:
Hey Cameron,
can you attach the logs when you come across another case where the
pushpull gets stuck?
i was doing a quick pass through the code... it looks like the
DaemonController stays up all the time... wait() is called on it... i
thought there was a notify() call when daemon queue became empty, which
would cause the DaemonController to wake up and terminate, but i don't see
it... i will see if i can get a fix in for that sometime this week.
-Brian
On Nov 30, 2013, at 03:08 PM, Cameron Goodale sigep...@gmail.com wrote:
Hey Brian and Jordan,
I have a similar issue with trying to run pushpull within a cron because
for some reason (i need to sort out) my pushpull daemon will just hang
after running and sleeping successfully using
the org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol. I haven't
taken the time to find out how many sleep/wake cycles it takes to
eventually hang, but I have seen it happen several times now.
I looked into the code and it seems like updating the runInfo tag so
period= should short circuit the code from re-running, but when I have
done this in testing I get a message like this:
Nov 30, 2013 10:07:32 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Nov 30, 2013 10:07:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90111 on RMI registry port 9011 is shutting down
but the process doesn't actually stop. Looking at Daemon.java I see that:
Daemon.this.unregister(); is called right after that log message, but it
doesn't seem to quit the process. That command is run within a Java Thread
object, but nothing seems to stop the Thread, so the daemon will never
exit.
If you guys have made any progress on this I would love to hear about it.
I am also not an expert on Java Threads, so I could have missed something
Re: PushPull DaemonLauncher
Hey Brian and Jordan,
I have a similar issue with trying to run pushpull within a cron because
for some reason (i need to sort out) my pushpull daemon will just hang
after running and sleeping successfully using
the org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol. I haven't
taken the time to find out how many sleep/wake cycles it takes to
eventually hang, but I have seen it happen several times now.
I looked into the code and it seems like updating the runInfo tag so
period= should short circuit the code from re-running, but when I have
done this in testing I get a message like this:
Nov 30, 2013 10:07:32 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Nov 30, 2013 10:07:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90111 on RMI registry port 9011 is shutting down
but the process doesn't actually stop. Looking at Daemon.java I see that:
Daemon.this.unregister(); is called right after that log message, but it
doesn't seem to quit the process. That command is run within a Java Thread
object, but nothing seems to stop the Thread, so the daemon will never exit.
If you guys have made any progress on this I would love to hear about it.
I am also not an expert on Java Threads, so I could have missed something
in the code, but it seems like the Thread is started and never terminated.
Thanks for reading,
Cameron
On Thu, Nov 21, 2013 at 1:43 PM, Brian Foster holeno...@me.com wrote:
Hey Jordan,
Try not specifying the runInfo element for your daemons in
RemoteSpecs.xml file. It should just run once through for those sites. I
believe the pushpull shuts down automatically when all deamons have
terminated, so after you run one pass it should just terminate. It's been
a while since i've worked with push-pull code, if that doesn't work i'll
have to dig further into the code.
if you looking to do just a on off download a given file, then you might
consider just using cas-protocol directly. Take a look at:
https://svn.apache.org/repos/asf/oodt/trunk/protocol/api/src/main/resources/policy/cmd-line-actions.xml.
You can run cas-protocol via this bin script:
https://svn.apache.org/repos/asf/oodt/trunk/protocol/api/src/main/bin/protocol.
Just build and deploy like all other cas products and that script will end
up in the bin directory. If you would like to add custom action to
cas-protocol there is documentation for this at:
https://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt
-Brian
On Nov 07, 2013, at 05:31 PM, Jordan Padams jordan.pad...@gmail.com
wrote:
Hello all,
I am trying to develop a pretty simple wrapper for the PushPull where I
call the DaemonLauncher to run through the daemons in my config once and
then shut down altogether. We currently don't have a need to have the
software running at intervals.
On that note, here are a few of the methods I've tried but haven't worked:
1. DaemonLauncher.main() - The software hangs because of the wait() waiting
for a notify().
2. DaemonLauncher.viewDaemonWaitingList() - I've tried monitoring the
daemon waiting list, and quit() the DaemonLauncher once it gets to 0, but
then it misses the last daemon in my config.
Is there a way for me to wait on the DaemonLauncher until all daemons have
completed, then quit?
I've implemented a quick workaround by creating the following method so I
can see if daemons are still running:
public boolean hasRunningDaemons() {
return this.dm.hasRunningDaemons();
}
I don't have a lot of experience with daemon processes so this may just be
something I'm blatantly missing. Appreciate the help.
Thanks,
Jordan
--
Jordan Padams
Software Engineer
NASA Jet Propulsion Laboratory
--
Sent from a Tin Can attached to a String
Re: How to process files in a sorted order
Konstantinos, My name is Cameron and I am a committer on the Apache OODT project. I am not familiar with the internals of crawler, but I don't believe there is a way to accomplish your goal of enforcing a sorting algorithm within the crawler config. I think you will have to write your own crawler that will implement your sorting logic. Sincerely, Cameron Goodale On Thu, Nov 7, 2013 at 7:44 PM, Konstantinos Mavrommatis kmavromma...@celgene.com wrote: Hi, In my environment I am using cas-crawler to process directories of 1000s of files. The metadata for these files are extracted automatically using the mimetypes definitions and small wrapper scripts. In these directories some of the files are derived from other files and metadata from the older files need to be transferred to the newer file. In order to achieve this I need to have the files processed by the cas-crawler starting from the older file to the newer file or in other cases in alphabetical order.. Any ideas how this can be achieved? The crawler command I currently use is: ./crawler_launcher --operation --launchAutoCrawler --productPath $FILEPATH --filemgrUrl $FMURL --clientTransferer org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory --mimeExtr actorRepo ../policy/mime-extractor-map.xml Thanks in advance for your help Konstantinos * THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE. If the reader is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please reply to the sender to notify us of the error and delete the original message. Thank You. * -- Sent from a Tin Can attached to a String
Re: [jira] [Commented] (OODT-622) svn:ignore - request to add Eclipse configuration files to ignore list for several modules
Thanks for taking the time to clean up the little things. +1 -Cameron On Sun, Jun 9, 2013 at 7:10 AM, Chris A. Mattmann (JIRA) j...@apache.orgwrote: [ https://issues.apache.org/jira/browse/OODT-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13679068#comment-13679068] Chris A. Mattmann commented on OODT-622: +1 commit away! :) svn:ignore - request to add Eclipse configuration files to ignore list for several modules -- Key: OODT-622 URL: https://issues.apache.org/jira/browse/OODT-622 Project: OODT Issue Type: Wish Affects Versions: 0.6 Reporter: Ross Laidlaw Assignee: Ross Laidlaw Priority: Trivial Labels: properties, repository, subversion Fix For: 0.6 Attachments: OODT-622.rlaidlaw.2013-06-09.patch.txt For many modules in OODT, the svn:ignore lists already contain the Eclipse configuration files and directory (.classpath, .project and .settings). But there are a few modules that don't yet have this setting. I'd like to add the Eclipse config files to the ignore lists for the remaining modules. I've produced a patch to do this and I'm happy to commit the changes if there are no objections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: JIRA label for new users
Mike, Your list of tags + meaning works for me. +1 -Cameron On Tue, May 21, 2013 at 8:14 AM, Michael Joyce jo...@apache.org wrote: +1 on this Gavin. Re. actual label names, how about a name followed by an (explanation) of the difficulty level: Don't Know Newcomer (Easy) - Everyone can do this level Committer (Medium) - This is for regular contributors/committers. Where the vast majority of issues would fall. Expert (Hard) or Guru or ChrisMattmann or whatever everyone likes for hard - Used sparingly when an issue requires expert component knowledge We could default the field to Don't Know so every issue always has some level? I like the CouchDB approach but I feel they have too many options. Easy Easy - Medium Medium - Hard Really Really Hard (Their Guru level) So, when something is Medium, which category does it fall into? I think the most important part of this system is identifying the difficulty for newcomers. The other levels are just nice to have in my opinion. That being said, I think simpler is better here. What do you guys think? -- Joyce On Sun, May 19, 2013 at 8:30 AM, Rajith Siriwardana rajithsiriward...@gmail.com wrote: +1 for having a label for issues that new users can help to resolve. On Sun, May 19, 2013 at 8:51 PM, Mattmann, Chris A (398J) chris.a.mattm...@jpl.nasa.gov wrote: Thanks Gav. +1 to using a controlled value field to make the labels meaningful. Let's see what others think but I am for sure +1. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Gavin McDonald ga...@16degrees.com.au Organization: 16 degrees complete web solutions Reply-To: dev@oodt.apache.org dev@oodt.apache.org, ga...@16degrees.com.au ga...@16degrees.com.au Date: Sunday, May 19, 2013 3:37 AM To: dev@oodt.apache.org dev@oodt.apache.org Subject: RE: JIRA label for new users -Original Message- From: Mattmann, Chris A (398J) [mailto: chris.a.mattm...@jpl.nasa.gov] Sent: Friday, 17 May 2013 1:37 AM To: d...@climate.incubator.apache.org Cc: dev@oodt.apache.org Subject: Re: JIRA label for new users Great idea, please implement on OODT too! (cc to dev@oodt for peeps) :) We can start to use labels now for jira tickets, we just need an agreed and documented set of labels. Why? Too many labels and we run the risk of there being too many and they sort of Lose the impact and meaning. So a few well-chosen labels for attention grabbing is needed. Every Jira ticket has a Labels section so this is available now. However, my main point of this mail, is that we can add custom fields to our issues. Some projects have done such a thing, some, like the couchdb project, have implemented a drop down field in their jiras so that anyone at a glance can determine if the ticket is newcomer, contributor, committer, hardcore project guru or Greg Stein level. (OK so I made that last level up, but really, the choice of drop down field name is ours to decide) Pop on over to the couchdb jira and take a look. I am happy to implement this in the OODT Jira if you agree it a good idea. I'm a big +1 ftr Gav... Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Michael Joyce jo...@apache.org Reply-To: d...@climate.incubator.apache.org d...@climate.incubator.apache.org Date: Wednesday, May 15, 2013 8:34 PM To: dev d...@climate.incubator.apache.org Subject: JIRA label for new users I sent this link around to a few people via twitter but I figured it warranted being sent here as well! http://nikcodes.com/2013/05/10/new-contributor-jump-in/ TLDR: Every OSS
Re: New list ?
+0 Gavin, it doesn't bother me. So if it goes either way, it's cool. Cameron On Apr 30, 2013 3:04 AM, Gavin McDonald ga...@16degrees.com.au wrote: A personal peeve of mine, all the jira notifications going to the commits list, makes it hard to see whats going on. (Yes I could filter but..) How about an issues@ list where we can send the jira stuff ? Gav...
Re: [ANNOUNCE] Welcome Gavin McDonald as an OODT PMC member and committer!
Gavin, First off welcome to OODT. Secondly, thank you for your help with the svn dump migration and finding the wrm-rcmes-project debacle and fixing it up. Monday I will be sure to scope the test svn repo you setup. Cheers! -Cameron On Wed, Apr 17, 2013 at 8:24 AM, Mattmann, Chris A (398J) chris.a.mattm...@jpl.nasa.gov wrote: -Original Message- From: Gavin McDonald ga...@16degrees.com.au Organization: 16 degrees complete web solutions Reply-To: dev@oodt.apache.org dev@oodt.apache.org, ga...@16degrees.com.au ga...@16degrees.com.au Date: Monday, April 15, 2013 5:55 PM To: dev@oodt.apache.org dev@oodt.apache.org Subject: RE: [ANNOUNCE] Welcome Gavin McDonald as an OODT PMC member and committer! -Original Message- From: Sean Kelly [mailto:ke...@apache.org] Sent: Wednesday, 10 April 2013 7:56 AM To: dev@oodt.apache.org Cc: gmcdon...@apache.org Subject: [ANNOUNCE] Welcome Gavin McDonald as an OODT PMC member and committer! Folks: I'm happy to report that Gavin McDonald has accepted our invitation to become a committer on Apache OODT and to join the OODT Project Management Committee. WOOT! Welcome, Gav. Please feel free to introduce yourself. Thanks Sean, Hi All, I hope to help out more in the near future, starting with making the PEAR stuff more reliable/compatible/working. Then onto other stuff. Dude you rock! Welcome Gav! I also do some ASF sysadmin type jobs around the place. :) I look forward to working with you all . Me too dude! Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -- Sent from a Tin Can attached to a String
Re: Review Request: OODT-597: Improve theme/styling of OPSUI
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10535/#review19454 --- Ship it! Rishi the style is exactly what you set out to achieve. Simple, clean and high contrast. Well done. - Cameron Goodale On April 19, 2013, 12:30 a.m., Rishi Verma wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10535/ --- (Updated April 19, 2013, 12:30 a.m.) Review request for oodt, Chris Mattmann, Andrew Hart, Paul Ramirez, and Cameron Goodale. Description --- Hey all, Per JIRA Issue: https://issues.apache.org/jira/browse/OODT-597, I proposed we update some of OPSUI's look and feel to make it more readable. The attached diff resolves all points addressed in OODT-597, so please take a look at that for more details. I've attached some screenshots of what the new OPSUI looks like - for your convenience. I know A LOT of work must have gone into the original OPSUI look and feel. So I want to make sure my improvements to it (and in some cases changes) are transparent and reviewed before we push. Like I said in OODT-597, the old OPSUI was very slick. I feel it was a little bit to read though on low-contrast screens, and that's why I proposed an improvement to the theme. Also, I think its good if we keep some consistency between how we do OODT logos, coloring, and themeing to make sure we are generating a consistent product. Anyways, let me know what you think! I'll push the commits when I get a 'ship it'! :D rishi This addresses bug OODT-597. https://issues.apache.org/jira/browse/OODT-597 Diffs - http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.css 1468173 http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.html 1468173 http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.java 1468173 http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage_cleanwhite.css PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage_cleanwhite.html PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage_navyblue.css PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/DefaultSkin.css 1468173 http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage.css 1468173 http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage.java 1468173 http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage_cleanwhite.css PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage_navyblue.css PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/OPSUIWebPage.java PRE-CREATION http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/project_logo.png UNKNOWN http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/splash.png UNKNOWN http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/META-INF/context.xml 1468173 Diff: https://reviews.apache.org/r/10535/diff/ Testing --- Local testing. See screenshots. Screenshots --- Splash page https://reviews.apache.org/r/10535/s/20/ Workflow monitor page https://reviews.apache.org/r/10535/s/21/ Workflow tasks diagram https://reviews.apache.org/r/10535/s/22/ Workflow task info https://reviews.apache.org/r/10535/s/23/ File Manager browser page https://reviews.apache.org/r/10535/s/24/ Thanks, Rishi Verma
Re: Interested in participating GSoC 2013 with OODT
Welcome to OODT Rajith, It does sound like your skills will be a great asset to the project over the summer. I look forward to working with you soon. Cheers, Cameron On Wed, Mar 27, 2013 at 8:44 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Rajith! Thank you for your interest in OODT-219! Would love to have you help out on the project. The mailing list doesn't accept attachments, so can you head over the OODT-219 and attach the diagram there? Thank you and looking forward to working with you on this! Cheers, Chris From: Rajith Siriwardana rajithsiriward...@ieee.orgmailto: rajithsiriward...@ieee.org Reply-To: dev@oodt.apache.orgmailto:dev@oodt.apache.org dev@oodt.apache.orgmailto:dev@oodt.apache.org Date: Wednesday, March 27, 2013 9:20 AM To: dev dev@oodt.apache.orgmailto:dev@oodt.apache.org Subject: Re: Interested in participating GSoC 2013 with OODT hi small mistake, forgot to attach the high level deployment architecture diagram which i was talking about. note that this architecture diagram changed to use mk_livestatus replacing Nagios XML engine to query stats. I can provide more details upon request about the the project I carried out. Thank you, T.W.R.O. Siriwardana StMIEEE Undergraduate Department of Computer Science Engineering Faculty of Engineering University of Moratuwa Sri Lanka. On Wed, Mar 27, 2013 at 9:38 PM, Rajith Siriwardana rajithsiriward...@ieee.orgmailto:rajithsiriward...@ieee.org wrote: hi, I'm a 4th year (final year) undergraduate of Faculty of Engineering University of Moratuwa majoring in Computer Science and Engineering and I would like contribute to OODT. I did my internship last year which spanned for 9 months in WSO2[1]. which is a middle-ware company and I worked with their WSO2 Stratos product[2]. Basically I was assigned for developing WSO2 Stratos system level health monitoring component and WSO2 Stratos heartbeat monitoring package. For WSO2 Stratos health monitoring component[3] before finalizing Nagios[4] for monitoring with mk_livestatus[5] replacing NagiosXML engine for query stats, which is shown in the architecture image, I did some work with Ganglia (how to use it for WSO2 Stratos system level health monitoring). Furthermore I took part in test automation hackathon carried out for WSO2 ESB[6] around 2 months. So I believe I have enough knowledge to complete the GSoC 2013 project Monitor that plugs into ganglia OODT-219 https://issues.apache.org/jira/browse/OODT-219 [7] with some guidance. Please give me any advice, guidance you'd like to extend. You can access my Linkedin profile via [8] [1]http://wso2.com/ [2]http://wso2.com/cloud/stratos/ [3]http://mail.wso2.org/mailarchive/architecture/2012-June/008827.html [4]http://www.nagios.org/ [5]http://mathias-kettner.de/checkmk_livestatus.html [6]http://wso2.com/products/enterprise-service-bus/ [7]https://issues.apache.org/jira/browse/OODT-219 [8]http://lk.linkedin.com/in/rajithsiriwardana/ Thank you, T.W.R.O. Siriwardana StMIEEE Undergraduate Department of Computer Science Engineering Faculty of Engineering University of Moratuwa Sri Lanka. -- Sent from a Tin Can attached to a String
Re: Min Pool Size in WorkflowManager Thread Pool (0.3)
Gabe, On Snow I raised the minPoolSize to the number of jobs I am able to run concurrently on our servers via Resource Manager. I am running minPoolSize=40 AND maxPoolSize=40 and everything works fine. I have noticed that if I flood the Workflow manager too quickly the jobs tend to wait until all 40 jobs are ready before all of them will move from STAGING to PGE_EXEC. Hope that helps. -Cam On Mon, Mar 18, 2013 at 10:37 AM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Gabe, The best docs I can point you at are here: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/package-s ummary.html#package_description I would recommend also reading: http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/Pooled Executor.html That really explains what the purpose of those diff properties are. Cheers, Chris On 3/14/13 1:57 PM, Resneck, Gabriel M (388J) gabriel.m.resn...@jpl.nasa.gov wrote: Hi, guys! I've run into a problem with the Workflow Manager in release 0.3. The number of active threads allowed by the pool seems to be dictated by the minimum thread count when the queue supplied to the pool object is unlimited. The default number (6) is a bit low for our purposes, so I was wondering how you guys have dealt with this issue in the past. Did you simply increase the minimum pool size or implement another solution? If you increased the minimum pool size, what was the highest that you have used and did you see any issues as a result? Thanks! Gabe =) -- Sent from a Tin Can attached to a String
Re: FW: [GSoC Mentors] Google Summer of Code 2013
Chris, I think it would be a good idea to check in with Ross who was our GSoC student last summer, and ask him for feedback on his experience with OODT. I hope he is listening to the dev list still, but if not we should check-in and see what he is doing now. My hope is that if Ross gives us feedback (good or bad or both) we can work to improve this summer for our next GSoC student. My apologies in advance if you already did this when the GSoC last year was closed out. Just a few questions to get the feedback pump primed would be: Was the mini-project challenging enough? Did the OODT Devs provide enough information and support? Did you encounter anything that was frustrating or unnecessarily difficult? Any thoughts? -Cam chee On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: [Sorry for cross posting] Guys, FYI please note that you can participate as a mentor from a PMC via Apache as they are a GSoC org. ComDev will coordinate our participation but start thinking about what projects we may want to do. Cheers, Chris From: Carol Smith car...@google.commailto:car...@google.com Date: Monday, February 11, 2013 11:02 AM To: Google Summer of Code Mentors List google-summer-of-code-mentors-l...@googlegroups.commailto: google-summer-of-code-mentors-l...@googlegroups.com Subject: [GSoC Mentors] Google Summer of Code 2013 Hi GSoC mentors and org admins, We've announced that we're doing Google Summer of Code 2013 [1]. Yay! If you would like to help spread the word about GSoC, we have presentations [2], logos [3], and flyers [4] for you to use. Please host meetups, tell your friends and colleagues about the program, go to conferences, talk to people about the program, and just generally do all the awesome word-of-mouth stuff you do every year to promote the program. The GSoC calendar, FAQ, and events timeline have all been updated with this year's important dates, so please refer to those for the milestones for this year's program. NB: the normal timeline for the program has been modified for this year. You'll probably want to examine the dates closely to make sure you know when important things are happening. Please consider translating the presentations and/or flyers into your native language and submitting them directly to me to post on the wiki. Localization for our material is integral to reaching the widest possible audience around the world. If you decide to translate a flyer, please fill out our form to request a thank you gift for your effort. [5] If you decide to host a meetup, please email me to let me know the date, time, and location so I can put it on the GSoC calendar. Also, remember to take pictures at your meetup and write up a blog post for our blog using our provided template for formatting [6]. If you need promotional items for your attendees, please fill out our form [7] to request some; we're happy to send some along. We can provide up to about 25 pens, notebooks, or stickers and/or a few t-shirts. Please keep in mind, though, that shipping restrictions and timeline vary country-to-country; request items early to make sure they get there on time! If you have questions about hosting meetups, please see the section in our FAQ [8]. Please consider applying to participate as an organization again this year or maybe joining as a mentor for your favorite organization if they are selected this year. We rely on you for your help for the success of this program, so thank you in advance for all the work you do! [1] - http://google-opensource.blogspot.com/2013/02/flip-bits-not-burgers-google-summer-of.html [2] - http://code.google.com/p/google-summer-of-code/wiki/ProgramPresentations [3] - http://code.google.com/p/google-summer-of-code/wiki/GsocLogos [4] - http://code.google.com/p/google-summer-of-code/wiki/GsocFlyers [5] - http://goo.gl/gEHDO [6] - http://goo.gl/wbZrt [7] - http://goo.gl/0BsR8 [8] - http://goo.gl/2NGfp Cheers, Carol -- You received this message because you are subscribed to the Google Groups Google Summer of Code Mentors List group. To unsubscribe from this group and stop receiving emails from it, send an email to google-summer-of-code-mentors-list+unsubscr...@googlegroups.com mailto:google-summer-of-code-mentors-list+unsubscr...@googlegroups.com. To post to this group, send email to google-summer-of-code-mentors-l...@googlegroups.commailto: google-summer-of-code-mentors-l...@googlegroups.com. Visit this group at http://groups.google.com/group/google-summer-of-code-mentors-list?hl=en-US . For more options, visit https://groups.google.com/groups/opt_out.
Re: [HEADS UP] Apache OODT now builds with Maven3 too
Chris, This is a huge win! Now I can let Apple install Maven 3 with my next Java update and I don't have to revert. -Cam On Sun, Feb 3, 2013 at 3:41 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hey Guys, With r1441987, OODT 0.6 trunk now builds with Maven3. Yay! Trunk still builds with Maven2 as well. [INFO] CAS Product Server SUCCESS [2.791s] [INFO] CAS Workflow Manager Monitor Web App .. SUCCESS [2.341s] [INFO] Catalog and Archive File Management Browser ... SUCCESS [1.561s] [INFO] Catalog and Archive Workflow Management GUI Editor SUCCESS [9.351s] [INFO] Process Control System Operator Interface Webapp .. SUCCESS [4.225s] [INFO] OODT Process Control System JAX-RS service layer .. SUCCESS [4.131s] [INFO] Apache OODT ... SUCCESS [37.794s] [INFO] [INFO] [INFO] BUILD SUCCESSFUL [INFO] [INFO] Total time: 7 minutes 6 seconds [INFO] Finished at: Sun Feb 03 15:40:18 PST 2013 [INFO] Final Memory: 100M/191M [INFO] [chipotle:~/tmp/oodt0.6] mattmann% Cheers, Chris
Re: OODT 0.3 branch
Chintu, I see that your test data volume is 262GB, but I am curious about the make up of the data. On average what is your file size and how many files? The reason I ask is because the process of extraction and ingestion can vary wildly. On the LMMP project I was ingesting 12GB DEMs over NFS and it was a slow process. It was basically serial with 1CR+1FM, but we didn't have a requirement to push large volumes of data. On our recent Snow Data System I am processing 160 workflow jobs in parallel and OODT could handle the load, it turned out the filesystem was our major bottleneck. We used a SAN initially when doing development, but when we increased the number of jobs in parallel the I/O became so bad we moved to GlusterFS. GlusterFS had speed improvements over the SAN, but we had to be careful about heavy writing, moving, deleting since the clustering would try to replicate the data. Turns out Gluster is great for heavy writting OR heavy reading, but not both at the same time. Finally we are using NAS and it works great. My point is the file system plays a major role in performance when ingesting data. The ultimate speed test would be if you could actually write the data into the final archive directory and basically do an ingestion in place (skip data transfer entirely), but I know that is rarely possible. This is an interesting challenge to see what configuration will yield the best through put/performance. I look forward to hearing more about your progress on this. Best Regards, Cameron On Wed, Dec 12, 2012 at 7:28 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Chintu, From: Mistry, Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0) chintu.mis...@nasa.govmailto:chintu.mis...@nasa.gov Date: Wednesday, December 12, 2012 12:02 PM To: jpluser chris.a.mattm...@jpl.nasa.govmailto: chris.a.mattm...@jpl.nasa.gov, dev@oodt.apache.orgmailto: dev@oodt.apache.org dev@oodt.apache.orgmailto:dev@oodt.apache.org Subject: Re: OODT 0.3 branch If you are saying that FM can handle multiple connections at one time, Yep I'm saying that it can. then multiple crawlers pointing to same FM should increase performance significantly. Well that really depends to be honest. It sounds like you guys are hitting an IO bottleneck potentially in data transfer? What file sizes are you transferring? If you are IO bound on the data transfer part, the product isn't fully ingested until: 1. it's entry is added to the catalog 2. The data transfer finishes Are you checking the FM for status along the way? Also realize that the FM will never be faster than the file system, so if it takes the file system X minutes to transfer a file F1, Y to transfer F2, and Z to transfer F3, then you still have to wait at least the max(X,Y,Z) time, regardless for the 3 ingestions to complete. But that’s not what we saw in our tests. For example, I saw barely 2 minutes performance difference between 2FM-6CR and 3FM-6CR. 1) 2 hour 6 minutes to process 262G (1FM 3CR - 3CR to 1FM) 2) 1 hour 58 minutes to process 262G (1FM 6CR - 6CR to 1FM) 3) 1 hour 39 minutes to process 262G (2FM 6CR - 3CR to 1FM) 4) 1 hour 39 minutes to process 262G (2FM 9CR - 4+CR to 1FM) 5) 1 hour 37 minutes to process 262G (3FM 9CR - 3CR to 1FM) 6) 2 hourto process 262G (3FM 20CR - 6+CR to 1FM) 7) 28 minutesto process 262G (6FM 9CR - 1+CR to 1FM) = This is my latest test and this is good number. What would be interesting is simply looking at the speed for how long it takes to cp the files (which I bet is what's happening) versus mv'ing the files by hand. If mv is faster, I'd: 1. Implement a Data Transfer implementation that simply replaces the calls to FileUtils.copyFile or .moveFile with systemCalls (see ExecHelper from oodt-commons) to UNIX equivalents. 2. Plug that data transfer in to your crawler invocations via the cmd line. HTH! Cheers, Chris From: Mattmann, Chris A chris.a.mattm...@jpl.nasa.govmailto: chris.a.mattm...@jpl.nasa.gov Date: Wednesday, December 12, 2012 2:51 PM To: Mistry, Chintu (GSFC-586.0)[COLUMBUS TECHNOLOGIES AND SERVICES INC] chintu.mis...@nasa.govmailto:chintu.mis...@nasa.gov, dev@oodt.apache.orgmailto:dev@oodt.apache.org dev@oodt.apache.org mailto:dev@oodt.apache.org Subject: Re: OODT 0.3 branch Hey Chintu, From: Mistry, Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0) chintu.mis...@nasa.govmailto:chintu.mis...@nasa.gov Date: Tuesday, December 11, 2012 2:41 PM To: jpluser chris.a.mattm...@jpl.nasa.govmailto: chris.a.mattm...@jpl.nasa.gov, dev@oodt.apache.orgmailto: dev@oodt.apache.org dev@oodt.apache.orgmailto:dev@oodt.apache.org Subject: Re: OODT 0.3 branch Answers inline below. ---snip Gotcha, so you are using different product types. So, each crawler is crawling various product types in each one of the staging area dirs, that looks like e.g., /STAGING_AREA_BASE
Re: 1100+ Javadoc Warnings...
Hey Brian,
I think the real issue is with the setup of the maven javadoc plugin.
There is a way to ask the javadoc to search within the current project to
create the links between docs, I just haven't been able to get it working
for the maven plugin. I agree that adding the full path will be incredibly
cumbersome so i hope with some maven magic 99% of the javadoc warnings will
go away.
Thanks for the reply to this.
-Cam
On Fri, Aug 24, 2012 at 10:52 PM, Brian Foster holeno...@me.com wrote:
hey cameron... sorry totally forgot about this email... i think you can
just add an import statement for the {@link classname}... putting the
full path in the {@link} will just make the javadoc messy looking... i will
try to fix my eclipse setup so it complains about that and asks me to
import so i remember to do it from now on
-brian
On Aug 13, 2012, at 7:02 AM, Cameron Goodale wrote:
Hey Brian, (I am addressing you because my focus for javadoc clean up is
the cli component)
and Paul (since you are the maven guru)
90% of the javadoc problems we (Mike Joyce and I) are finding are related
to the javadoc tool not searching within the java package to find and
fill
in a complete reference.
Here is an example of the DIFF I created from the cli module after
cleaning
up over 100 javadoc errors. 2 or 3 were merely typos, but most of them
where like these:
===
--- src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
(revision
1371572)
+++ src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
(working
copy)
@@ -17,7 +17,7 @@
package org.apache.oodt.cas.cli.option;
/**
- * {@link CmdLineOption} for specifying {@link CmdLineAction}.
*+ * {@link CmdLineOption} for specifying {@link
org.apache.oodt.cas.cli.action.CmdLineAction}.*
*
* @author bfoster (Brian Foster)
*/
Index:
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
===
---
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
(revision
1371572)
+++
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
(working
copy)
@@ -20,7 +20,7 @@
import org.apache.oodt.cas.cli.option.CmdLineOptionInstance;
/**
- * Validator for specified values of {@link CmdLineOption}s.
*+ * Validator for specified values of {@link
org.apache.oodt.cas.cli.option.CmdLineOption}s.*
*
* @author bfoster (Brian Foster)
*/
So it seems like we have to use fully qualified package names
when referencing methods and objects within javadocs. I was really
hoping
that some kinda maven plugin magic would be able to search and resolve
these paths for us during the build process.
I have tried a couple different things like using mvn javadoc:fix but
that doesn't seem to do much.
So if anyone has any ideas to try on this I am all ears.
Thanks,
Cameron
--
Sent from a Tin Can attached to a String
1100+ Javadoc Warnings...
Hey Brian, (I am addressing you because my focus for javadoc clean up is
the cli component)
and Paul (since you are the maven guru)
90% of the javadoc problems we (Mike Joyce and I) are finding are related
to the javadoc tool not searching within the java package to find and fill
in a complete reference.
Here is an example of the DIFF I created from the cli module after cleaning
up over 100 javadoc errors. 2 or 3 were merely typos, but most of them
where like these:
===
--- src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
(revision
1371572)
+++ src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
(working
copy)
@@ -17,7 +17,7 @@
package org.apache.oodt.cas.cli.option;
/**
- * {@link CmdLineOption} for specifying {@link CmdLineAction}.
*+ * {@link CmdLineOption} for specifying {@link
org.apache.oodt.cas.cli.action.CmdLineAction}.*
*
* @author bfoster (Brian Foster)
*/
Index:
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
===
---
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
(revision
1371572)
+++
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
(working
copy)
@@ -20,7 +20,7 @@
import org.apache.oodt.cas.cli.option.CmdLineOptionInstance;
/**
- * Validator for specified values of {@link CmdLineOption}s.
*+ * Validator for specified values of {@link
org.apache.oodt.cas.cli.option.CmdLineOption}s.*
*
* @author bfoster (Brian Foster)
*/
So it seems like we have to use fully qualified package names
when referencing methods and objects within javadocs. I was really hoping
that some kinda maven plugin magic would be able to search and resolve
these paths for us during the build process.
I have tried a couple different things like using mvn javadoc:fix but
that doesn't seem to do much.
So if anyone has any ideas to try on this I am all ears.
Thanks,
Cameron
--
Sent from a Tin Can attached to a String
CSS issue with JIRA?
Hey OODTers, I have been trying to access JIRA and I can, but the css files seem to be missing. Does anyone else see this same problem with: https://issues.apache.org/jira/browse/OODT-328 ? Screenshot of what I see is attached. Thanks. -Cameron
Re: CSS issue with JIRA?
Ross and Sheryl, Thanks for double checking on this for me. I wanted to make sure I didn't have some odd Browser/Network settings on my machine. I just jumped over to issues.apache.org/jira and saw Maintenance in Progress so it looks like Ross guessed correctly. In the future we can check out this page for details on the ASF infrastructure status. http://monitoring.apache.org/status/ I guess i will just keep checking up on this through out the day. Cheers, Cameron On Fri, Aug 10, 2012 at 2:05 AM, Ross Laidlaw rlaidlaw.o...@gmail.comwrote: Hi Cam Sheryl, I think JIRA might be down at the moment. I can't access it here in the UK - I get a page saying 'Service Temporarily Unavailable'. Perhaps some maintenance/upgrades going on I guess... Ross On 10 Aug 2012, at 07:11, Sheryl John shery...@gmail.com wrote: Yep it does look like the css files are missing. Didn't get the screenshot. On Thu, Aug 9, 2012 at 11:06 PM, Cameron Goodale good...@apache.org wrote: Hey OODTers, I have been trying to access JIRA and I can, but the css files seem to be missing. Does anyone else see this same problem with: https://issues.apache.org/jira/browse/OODT-328 ? Screenshot of what I see is attached. Thanks. -Cameron -- -Sheryl
Re: [jira] [Commented] (OODT-310) Port WEngine to trunk
Sheryl, I get the same error on another project using review board. The issue is caused by filling out too much or too little of the base repository url. If/when you get the right combination to work, make note of it or grab a screen shot. It'll help keep your sanity. Cameron On Aug 6, 2012 11:34 AM, Brian Foster (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/OODT-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429325#comment-13429325] Brian Foster commented on OODT-310: --- EngineRunnerFactory additions fixed in r1369914 Port WEngine to trunk - Key: OODT-310 URL: https://issues.apache.org/jira/browse/OODT-310 Project: OODT Issue Type: Sub-task Components: workflow manager Reporter: Chris A. Mattmann Assignee: Chris A. Mattmann Fix For: 0.5 Attachments: OODT-310.2012-08-03.bfoster.patch.txt, OODT-310.Mattmann.082311.patch.txt, OODT-310.Mattmann.100911.patch.txt, OODT-310.sherylj.101711.patch.txt This issue will track the porting of the wengine-branch WorkflowEngine interface and associated classes to trunk. This involves changing to the Processor model (Sequential, Condition, etc.) which has already begun (as of OODT-70), and also the actual Engine implementation itself, that doesn't block and that includes a queue-based model developed by [~bfoster]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (OODT-310) Port WEngine to trunk
That is great news. At JPL my issue was using a shortcut url for our svn repo. http://oodt/repo vs. http://oodt.jpl.nasa.gov/repo but the same basic deal. -Cameron On Mon, Aug 6, 2012 at 8:51 PM, Sheryl John shery...@gmail.com wrote: Thanks Cameron! It worked after changing the url to http. On Mon, Aug 6, 2012 at 7:43 PM, Cameron Goodale sigep...@gmail.com wrote: Sheryl, I get the same error on another project using review board. The issue is caused by filling out too much or too little of the base repository url. If/when you get the right combination to work, make note of it or grab a screen shot. It'll help keep your sanity. Cameron On Aug 6, 2012 11:34 AM, Brian Foster (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/OODT-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429325#comment-13429325 ] Brian Foster commented on OODT-310: --- EngineRunnerFactory additions fixed in r1369914 Port WEngine to trunk - Key: OODT-310 URL: https://issues.apache.org/jira/browse/OODT-310 Project: OODT Issue Type: Sub-task Components: workflow manager Reporter: Chris A. Mattmann Assignee: Chris A. Mattmann Fix For: 0.5 Attachments: OODT-310.2012-08-03.bfoster.patch.txt, OODT-310.Mattmann.082311.patch.txt, OODT-310.Mattmann.100911.patch.txt, OODT-310.sherylj.101711.patch.txt This issue will track the porting of the wengine-branch WorkflowEngine interface and associated classes to trunk. This involves changing to the Processor model (Sequential, Condition, etc.) which has already begun (as of OODT-70), and also the actual Engine implementation itself, that doesn't block and that includes a queue-based model developed by [~bfoster]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira -- -Sheryl -- Sent from a Tin Can attached to a String
Re: Problem when running pushpull
YunHee, Thank you for testing out the PushPull User Guide. I have been waiting for someone to run through it and have success (which it appears you have). I just added in a small block about how to Launch the PushPull Daemon, but it seems that you got it running already. If there is anything that you think should be added to the User Guide, or if any part was confusing I would appreciate hearing from you. Cheers! Cameron On Sat, Jul 21, 2012 at 10:15 PM, YunHee Kang yunh.k...@gmail.com wrote: HI chris, Ahh gotcha. You've found a config file bug in oodt0.4. Can you please: 1. Create yourself an account at our JIRA issue tracker: https://issues.apache.org/jira/browse/OODT 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects version to 0.4) I just created the isse named OODT-474 related with config file bug in oodt0.4. It is the first time to write an issue report. Can you check the issue that I wrote ? Ahh, what FTP site are you trying to access? That way I can try and test this myself in my own local checkout. I tried to test downloading a file based on WIKI site https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html;. There is no problem to download the file as follows: $ ftp l4ftl01.larc.nasa.gov Connected to l4ftl01.larc.nasa.gov. 220- 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 220- 220- This US Government computer is for authorized users only. 220- By accessing this system you are consenting to complete 220- monitoring with no expectation of privacy. Unauthorized 220- access or use may subject you to disciplinary action and 220- criminal prosecution. 220- 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 220- 220- 220- 220 FTP server ready. Name (l4ftl01.larc.nasa.gov:yhkang): anonymous 331 Guest login ok, send your complete e-mail address as password. Password: 230 Guest login ok, access restrictions apply. Remote system type is UNIX. Using binary mode to transfer files. ftp cd /TES/TL2CO2N.005 250 CWD command successful. ftp cd 2004.09.20 250 CWD command successful. ftp get TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5 local: TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5 remote: TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5 200 PORT command successful. 150 Opening BINARY mode data connection for TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5 (92150280 bytes). 226 Transfer complete. Checksum matched. 92150280 bytes received in 171.84 secs (523.7 kB/s) Thanks, Yunhee 2012/7/21 Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov: Hi YunHee, On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote: I just found a problem that is in a file $CAS_PP_HOME/policy/ProtocolFactoryInfo.xml in oodt 0.4. The problem is caused by missing classes related with an element protocolFactory in the file described above. To fix this problem, It needs to replace a wrong class name with a correct class name. For example there is no class named org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar file in the directory, $CAS_PP_HOME/lib. To run the script $CAS_PP_HOME/bin/pushpull, CogJGlobusFtpClientFactory needs to be replaced with CogJGlobusFtpProtocolFactory. Ahh gotcha. You've found a config file bug in oodt0.4. Can you please: 1. Create yourself an account at our JIRA issue tracker: https://issues.apache.org/jira/browse/OODT 2. File an issue (create a new issue/bug) against oodt 0.4 (set the affects version to 0.4) If you are willing to, we'd love to have a patch file from you that fixes this config file. The patch process would be something like: 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5 2. edit odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and fix XML policy file. 3. svn diff OODT-xxx.ykang.yyMMdd.patch.txt where xxx is the JIRA issue ID from #2 above. 4. attach your patch to the JIRA issue from #2. But I don't know how to replace org.apache.oodt.cas.protocol.local.LocalClientFactory. I'm not sure you have to -- this is probably OK. To be honest, I am not familiar with OODT. protocol type=ftp !--protocolFactory class=org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory/-- !-- cas-protocol-ftp-0.4.jar -- protocolFactory class=org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory/ !--protocolFactory class=org.apache.oodt.cas.protocol.ftp.CommonsNetFtpClientFactory/-- protocolFactory class=org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory/ /protocol protocol type=http !--protocolFactory class=org.apache.oodt.cas.protocol.http.HttpClientFactory/-- !-- cas-protocol-http-0.4.jar-- protocolFactory class=org.apache.oodt.cas.protocol.http.HttpProtocolFactory/ /protocol protocol type=sftp
Re: OODT SoCal Meetup/Hackathon
I think the pool/house idea sounds great. If we start early enough, then we can hack for like 4 - 6 hours, then use the pool as a reward for a job well done. My hope is that we can get some long stretches of time to work on OODT, without email, twitter, IM, or facebook distractions...well the exception being the Google Hangout (of course). -Cam On Thu, May 17, 2012 at 7:06 PM, Ramirez, Paul M (388J) paul.m.rami...@jpl.nasa.gov wrote: Your place sounds good to me. Now we just need to lock down a date. 1st or 2nd Saturday of June work? +1 to Google hangout. Probably best if people do a SVN checkout before we meetup. --Paul On May 17, 2012, at 10:45 AM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Now *that* is a great idea bfost. We can set up a Google Hangout for this. OK, then I am going to throw out my pool/house for the meetup location, depending on what people think! Cheers, Chris On May 16, 2012, at 11:32 PM, Brian Foster wrote: google+ hangout 4 those out of state! On May 16, 2012, at 11:54 AM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hey Paul, This sounds like a great idea and I'm in (of course!) One thing we'll need to do is make sure that we do a really good job of trying to include everyone, even folks that are not in the SoCal area. That means potentially using the Wiki, and other forms of communication (JIRA, ReviewBoard, etc.) to make sure everyone (including those not able to attend) feels included. Great idea! Cheers, Chris On May 15, 2012, at 2:21 PM, Ramirez, Paul M (388J) wrote: Hi All, Anyone interested in doing a meetup/hackathon at the beginning of June. The goal would be to get some updates to our code base but really focused on getting 0.4 out the door. If there is enough interest we would need a place to host it. Anyone and everyone would be welcome as it would be a good way to understand the ways to contribute which go well beyond just code. If enough people seem interested I will work out the details. My initial thought is this would be on a weekend. --Paul ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: OODT SoCal Meetup/Hackathon
If you schedule it, I will try to be there. June I might be in Paris, or on Holiday (no these are no the same thing). Just need some good wiFi, so hopefully we can grab some from the neighbors. _C On Tue, May 15, 2012 at 9:08 PM, David Woollard wooll...@gmail.com wrote: I think I could make a weekend in June work. We should do a Balmer Peak-a-thon (http://xkcd.com/323/). -Dave On May 15, 2012, at 5:21 PM, Ramirez, Paul M (388J) wrote: Hi All, Anyone interested in doing a meetup/hackathon at the beginning of June. The goal would be to get some updates to our code base but really focused on getting 0.4 out the door. If there is enough interest we would need a place to host it. Anyone and everyone would be welcome as it would be a good way to understand the ways to contribute which go well beyond just code. If enough people seem interested I will work out the details. My initial thought is this would be on a weekend. --Paul
Re: Resource Manager client question
Hey Mike and Sheryl, Mike was asking me for some similar advice and I plain ran outta talent on this topic. From what I can tell Mike would like to run his python scripts on Resource Manager without the need for setting up Workflow or PGE. At the time I hadn't really thought through all the configuration files needed, but having stewed on it I thought I should reply. Now my current SnowDS implementation is to have the Workflow Task reference a CAS-PGE (which contains the execution block for my python program i want to run). Then the Workflow is merely configured to farm the jobs out to the Resource Manager. Here is a list of questions that I have started to wonder about with Mike's help, any answers would be appreciated: 1. Can Resource Manager + Batchstubs be used without any additional OODT components? 2. Is PGE required to run/wrap non-Java programs so they can run within Resource Manager? Closing comments to Mike: If you are planning to use OODT for data management, it is initially very tempting to only setup and configure the minimal set of components because you will feel productive and it feels like progress is being made. Trust me I know since I was in your shoes about 6 months ago when trying to get some image processing IDL code to run and I bably needed to see progress (notice I didn't use the works make progress). Because I wanted to use (what I thought was) the easier solution I ended up hardcoding paths to resources my python code needed in the code instead of passing the parameters into the code in the first place. This worked reasonably well as long as everything stayed the samebut then it didn't so I had to re-visit my easier setup and fix it. Recently I have been working to undo my mistakes and python has been very forgiving, but the best part was that all the strange and mystic Workflow setups and PGEConfig.xml files actually started to make a whole lot more sense. I am now able to configure and stand up a complete workflow config, then jump into PGEConfig and get the input parameters to my python code. This means if the input files i need to process changes I don't need to change my python code, instead I can merely pass in a different set of parameters into the workflow and they will persist to my wrapped python. In short I know that combing through all the xml config is tough, especially when things are not working as quickly as you would like. I understand how defeated and frustrating it can be to have the component fail and just feel lost, not knowing what is causing the problem. I know the documentation isn't perfect and sometimes it is missing altogether, but the people that are on this list will bend over backwards to help you understand (some will even share their config files with line-by-line comments included at no extra charge ;) Thank you Sheryl for being awesome and helpful (you always are). Mike keep the questions coming and I will be sure to add in my $0.02 when I am able to. Best Regards, Cameron On Mon, May 7, 2012 at 5:09 PM, Sheryl John shery...@gmail.com wrote: Hi Mike, Yup, you can run your python scripts, java programs etc. from CAS-PGE which is used with the Workflow Manager. Check out this cas-pge guide [1] and the other wiki pages related to workflow. You can use Resource Manager to run tasks sent from the Workflow Manager. I've recently started testing this but there are others on the list who can guide you more on the Resource Manager. HTH! Sheryl [1] https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html On Mon, May 7, 2012 at 3:43 PM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] michael.iwu...@nasa.gov wrote: I have two questions, I am able to run the Resource Manager with no issues. I have some python scripts and possibly some other programs I would like to run using the Resource Manager. From what I know so far I believe the cas-pge component needs to be used in conjunction with the Resource Manager and is used as a wrapper program for running my scripts. Can someone give me more information on how this can be accomplished or are there any examples to view? I would also like to be able to utilize the Job Scheduler, Monitor and Job queue classes that are part of the Resource Manager. I can't find any examples of how they are used anywhere. And if examples do exist can someone point me in the right direction or give me more information on this? Thanks Mike -- -Sheryl
Added a Resource Manager FAQ Page to the Wiki
Hey OODT Community, Last night I was pulling my hair out trying to figure out why Resource Manager was falling over on my project. This morning I finally found the error in my policy setups. With solution in place I was encouraged to start an FAQ (hat tip: Chris Mattmann) page on the Wiki to help others avoid repeating my mistake. The page is here: https://cwiki.apache.org/confluence/display/OODT/FAQ+-+Resource+Manager It has been linked from the Main Wiki Page as well. If you have other gotcha's or anything else to add, please feel free to do so. I tried to include the full error message in hopes that Google would index it and direct others to the page. Prost! Cameron
Re: OODT-402 progress update
Ross,
I agree with Chris that you are doing a great job on this. Thank you for
all the hard work in setting this up. Having a standard geospatial
interface from the FileManager to SIS will prove extremely helpful to
several JPL/NASA Earth Science Missions in the future.
Just wanted to say thanks!
Cameron
On Sat, Mar 31, 2012 at 2:47 PM, Mattmann, Chris A (388J)
chris.a.mattm...@jpl.nasa.gov wrote:
(sorry for the cross post, but of interest to both communities)
Hey Ross,
That is awesome to hear! Comments below:
On Mar 31, 2012, at 4:04 AM, Ross Laidlaw wrote:
Hi Chris,
Many thanks for your comments and advice. I have now installed SIS on
my system and I have the demo up and running. I've also been
experimenting with the connection between File Manager and SIS and I
have a basic connection working. Here are the steps I have carried so
far:
Firstly, I added elements for 'Latitude' and 'Longitude' to the
GenericFile policy. Then I created a set of test files (simple text
files) with corresponding '.met' metadata files. I added 'Latitude'
and 'Longitude' metadata to the '.met' files. I then ingested these
into my File Manager repository.
That's perfect.
After that, I enabled the 'georss' tags in the 'rssconf.xml'
configuration file (I added tags for geo:lat and geo:long to start
with) and viewed the RSS output from the CAS REST API in my browser.
I then added the URL
('http://localhost:8080/fmprod/viewRecent?channel=ALL') as an RSS feed
to the 'sis-location-config.xml' config in my SIS webapp.
Next, I ran a bounding box query from the SIS demo.jsp and my file
manager files appeared on the map! Awesome!
That is SWEET!
I have uploaded a set of screenshots showing my progress, hopefully
they're accessible and make sense (I'm new to flickr, so apologies if
things look a bit disorganised!):
http://www.flickr.com/photos/rlaidlaw/sets/72157629342438940/with/7031581313/
They look great, I checked them out. Would you be willing to do the
following?
1. Log onto the SIS wiki and add a little tutorial (small) based on the
above email
and screenshots?
wiki here: https://cwiki.apache.org/confluence/display/SIS/Home
2. Link the SIS tutorial via the OODT wiki, here:
https://cwiki.apache.org/confluence/display/OODT/Home
I tried to order the screenshots to follow the steps I outlined above,
with the final screenshot being the output from the SIS demo showing
the File Manager files on the map.
That's awesome.
For next steps, I could start to formalize some of my experimental
code and begin to address the key points you have outlined on the
OODT-402 jira page (i.e. writing a default GeoRSS config and a default
file manager policy for a 'LocationAwareProduct' type).
That would be perfect!
I will follow
up with specific questions about both of these, but for now I just
wanted to give you an update on my progress and to check that I'm
heading in the right direction.
You are definitely heading in the right direction that's precisely what I
was
hoping you would figure out how to do. Thank you Ross for the update
and your results look VERY promising, great job so far!
Cheers,
Chris
++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++
Re: question on push pull
Luca, Good catch. I was typing faster than my brain was working. I meant to say Push Pull instead of Crawler. Sorry for the confusion. Chris laid it all out really well. Guess I was just a little to excited to answer you and goofed in my haste. To answer your question, I am not sure where the code is in PUSHPULL (got it right this time ;) since I haven't had reason to use it yet. I did check the etc/examples directory in PushPull and I only found file, ftp, and sftp examples. If you do sort out the scp and scp -r versions it would be great to add them to the etc/examples area. It could be part of the Apache Extras that Chris mentioned: http://code.google.com/a/apache-extras.org/p/oodt-pushpull-plugins/ Good Luck. -Cameron On Wed, Feb 29, 2012 at 10:23 AM, Cinquini, Luca (3880) luca.cinqu...@jpl.nasa.gov wrote: Hi Cameron, Maybe I am confused, but I was actually asking about the push-pull capabilities - does the crawler plug into the push-pull framework ? (sorry about my ignorance here). If push-pull supports scp, would you know the name of the protocol transfer factory to use - I haven't found one. thanks a lot, Luca On Feb 28, 2012, at 8:40 AM, Cameron Goodale wrote: Luca, I haven't tried this exact use case within Crawler, but Crawler does support scp and I have used 'scp -r' to recursively download a folder and all content housed within. I can only imagine ftp has a similar recursive option as well. Maybe another more Crawler Savy dev can shine some light on the recursion use case when using Crawler. -Cameron P.S. When we get a final answer let's add this to the Crawler User Guide Wiki too as an example use case. Glad you found the Crawler Wiki page useful. On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) luca.cinqu...@jpl.nasa.gov wrote: Hi all, I have a quick question concerning the pushpull framework : is there any way to transfer full directory trees, as opposed to single files ? And which of the currently implemented transfer protocols would allow that ? I haven't see any examples on that, though I might have missed it. thanks a lot, Luca P.S.: Cameron, thanks for writing the push-pull user guide - it's great. -- Sent from a Tin Can attached to a String -- Sent from a Tin Can attached to a String
Re: [jira] [Commented] (OODT-393) Column Based Catalog for Filemanager
I think this is a great idea to help bring more JPL code to Apache so everyone can benefit. Cameron On Feb 27, 2012 8:20 PM, Chris A. Mattmann (Commented) (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/OODT-393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13217879#comment-13217879] Chris A. Mattmann commented on OODT-393: Hey Michael, I'll be happy to work with you to shepherd this in. Column Based Catalog for Filemanager Key: OODT-393 URL: https://issues.apache.org/jira/browse/OODT-393 Project: OODT Issue Type: New Feature Components: file manager Affects Versions: 0.4 Reporter: Michael Starch Assignee: Chris A. Mattmann Priority: Minor This issue is to port the ColumnedBasedCatalog back-end that works with the JPL filemanager to a place at Apache. It uses a column-based table structure in the database back-end, which is more efficient for most databases as it is a true relational structure. This will also add in the ComplexQuery feature that exists in the JPL branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Question about xmlrpc
Irina et al, I have run into this same issue with exceeding the 100 connections to xml-rpc filemanager, and workflow manager and here were the steps I took to try and avoid hitting the limit ever. 1. Optimize the Lucene Index (if you are using the Lucene Catalog and have 100,000's of entries this can help improve how quickly your requests are handled and this will free up used connections faster). 2. I used lsof to detect how many active connections where made to the FileManager, if the number exceeded 85 (to be safe) I would have my submission code sleep for 10 seconds and try again. Not the most performant fix, but I never lose a job submission. Option 2 was written in Python, and if you want a copy of it just let me know and will be happy to fwd it along. Good luck. -Cameron On Tue, Feb 21, 2012 at 2:21 PM, holenoter holeno...@me.com wrote: hey irina, how many retries do you have set for each task and how long do is your interval between retries? -brian On Feb 21, 2012, at 09:56 AM, Tkatcheva, Irina N (388D) irina.n.tkatch...@jpl.nasa.gov wrote: Hi Brian and all, I have noticed that the system does recover after the System overload: Maximum number of concurren trequests (100) exceeded message, but usually some jobs stay in 'Waiting on resource (executing)' condition and never proceed further. I have seen it every time after the overload messages. I usually run a test that runs a bunch of jobs overnight. If there is no overload messages, all jobs are completed; if there are overload messages, usually in the morning some jobs are stuck in 'Waiting on resource (executing)' state. So it looks to me that the system does not recover completely. Irina On Feb 17, 2012, at 9:17 AM, Brian Foster wrote: Hey Chris, ya I'm in favor of adding the property but let's make it use 100 by default if the property is not set and I would even say let's add it to the properties file but comment it out or something.. that's a really advanced flag which only needs to be changed to get rid of that logging message... CAS works fine even when that message is being thrown... I think it prints to sndout, otherwise I would have just turned the logging for that off back when I added the client retry handlers that fixed the issue... oh and this is another thing your probably gonna want to port to trunk workflow :) -Brian Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.govmailto: chris.a.mattm...@jpl.nasa.gov wrote: Thanks Brian, I was thinking this too, +1, which is why I cautioned against any number greater than 256 in terms of thread count in my reply email too, since the risk is either that (a) you have to increase the ulimit (which extends the boundaries from devops oriented updates to sysops on the sysadmin side); and (b) the JVM will likely start trashing unless there is an inordinate amount of RAM, or swap space, etc. I think the best solution here is to simply make it a configurable property and then encourage projects to use a sensible default that's not too large... Cheers, Chris On Feb 16, 2012, at 12:52 AM, Brian Foster wrote: You have to be careful with the number you set that too because you are basically telling XML-RPC that it is now allowed to create 2000 threads in the same JVM... not a good practice... I don't remember the exact number but the JVM will crash if it creates a certain number of threads because there is a limit to the number of threads one process can create and I believe this is restricted at the operating system level... and i believe this number is less than 2000... The trunk filemgr and wengine already have built-in client retry handling support and are configurable via java properties (i.e. org.apache.oodt.cas.filemgr.system.xmlrpc.connection.retries and o.a.o.c.filemger.system.connection.retry.interval.seconds and there are similar ones for wengine)... The message you are seeing is XML-RPC server logging that it already using a 100 worker threads... you will see this message if you create a 100+ jobs in the RM (e.g. Workflow Conditions and Tasks) and they all start talking to the workflow manager or file manger at the same time... the client retry handlers will catch this error and just wait and retry again... you shouldn't be loosing any data... the only inconvenience I guess is that message is cluttering the logs -Brian On Feb 15, 2012, at 10:42 PM, Cheng, Cecilia S (388K) cecilia.s.ch...@jpl.nasa.govmailto:cecilia.s.ch...@jpl.nasa.gov wrote: Hi Chris, Sure we can discuss this in dev@oodt.apache.orgmailto:dev@oodt.apache.org . If you feel comfortable w/ the 2000 number, of course I can push the patch upstream into Apache OODT. But what kind of tests, if any, should we do before we deliver the patch? Our projects are concerned that if we arbitrarily set a number, we don't know what other problems it might cause. Thanks, Cecilia On 2/15/12 10:07 PM, Mattmann, Chris A (388J)
Re: Question about xmlrpc
Irina, I just sent the python script to the dev listbut I am now thinking it was stripped off. I will email a copy directly to you and Michael as a back up. -Cameron On Wed, Feb 22, 2012 at 4:35 PM, Tkatcheva, Irina N (388D) irina.n.tkatch...@jpl.nasa.gov wrote: Hi Cameron, Thank you for your input. I think your script could be useful in the operations (but it is for Michael (Starch) to decide). In any case, please forward it to us. Thanks, Irina On Feb 22, 2012, at 7:02 AM, Cameron Goodale wrote: Irina et al, I have run into this same issue with exceeding the 100 connections to xml-rpc filemanager, and workflow manager and here were the steps I took to try and avoid hitting the limit ever. 1. Optimize the Lucene Index (if you are using the Lucene Catalog and have 100,000's of entries this can help improve how quickly your requests are handled and this will free up used connections faster). 2. I used lsof to detect how many active connections where made to the FileManager, if the number exceeded 85 (to be safe) I would have my submission code sleep for 10 seconds and try again. Not the most performant fix, but I never lose a job submission. Option 2 was written in Python, and if you want a copy of it just let me know and will be happy to fwd it along. Good luck. -Cameron On Tue, Feb 21, 2012 at 2:21 PM, holenoter holeno...@me.com wrote: hey irina, how many retries do you have set for each task and how long do is your interval between retries? -brian On Feb 21, 2012, at 09:56 AM, Tkatcheva, Irina N (388D) irina.n.tkatch...@jpl.nasa.gov wrote: Hi Brian and all, I have noticed that the system does recover after the System overload: Maximum number of concurren trequests (100) exceeded message, but usually some jobs stay in 'Waiting on resource (executing)' condition and never proceed further. I have seen it every time after the overload messages. I usually run a test that runs a bunch of jobs overnight. If there is no overload messages, all jobs are completed; if there are overload messages, usually in the morning some jobs are stuck in 'Waiting on resource (executing)' state. So it looks to me that the system does not recover completely. Irina On Feb 17, 2012, at 9:17 AM, Brian Foster wrote: Hey Chris, ya I'm in favor of adding the property but let's make it use 100 by default if the property is not set and I would even say let's add it to the properties file but comment it out or something.. that's a really advanced flag which only needs to be changed to get rid of that logging message... CAS works fine even when that message is being thrown... I think it prints to sndout, otherwise I would have just turned the logging for that off back when I added the client retry handlers that fixed the issue... oh and this is another thing your probably gonna want to port to trunk workflow :) -Brian Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.govmailto: chris.a.mattm...@jpl.nasa.gov wrote: Thanks Brian, I was thinking this too, +1, which is why I cautioned against any number greater than 256 in terms of thread count in my reply email too, since the risk is either that (a) you have to increase the ulimit (which extends the boundaries from devops oriented updates to sysops on the sysadmin side); and (b) the JVM will likely start trashing unless there is an inordinate amount of RAM, or swap space, etc. I think the best solution here is to simply make it a configurable property and then encourage projects to use a sensible default that's not too large... Cheers, Chris On Feb 16, 2012, at 12:52 AM, Brian Foster wrote: You have to be careful with the number you set that too because you are basically telling XML-RPC that it is now allowed to create 2000 threads in the same JVM... not a good practice... I don't remember the exact number but the JVM will crash if it creates a certain number of threads because there is a limit to the number of threads one process can create and I believe this is restricted at the operating system level... and i believe this number is less than 2000... The trunk filemgr and wengine already have built-in client retry handling support and are configurable via java properties (i.e. org.apache.oodt.cas.filemgr.system.xmlrpc.connection.retries and o.a.o.c.filemger.system.connection.retry.interval.seconds and there are similar ones for wengine)... The message you are seeing is XML-RPC server logging that it already using a 100 worker threads... you will see this message if you create a 100+ jobs in the RM (e.g. Workflow Conditions and Tasks) and they all start talking to the workflow manager or file manger at the same time... the client retry handlers will catch this error and just wait and retry again... you shouldn't be loosing any data
Re: [jira] [Created] (OODT-380) Pypes UX
Adam, I am digging this tool, especially since right now I am in working in SNOW workflow heaven and i think a visual tool would really help everyone see what is happening in the pipeline. The License is ALv2 so that is a good start, and it is written in Python ;) I would be up for throwing a couple hours at trying to set it up and install in on my machine and test it out. -Cam On Wed, Feb 15, 2012 at 5:01 PM, Adam Estrada (Created) (JIRA) j...@apache.org wrote: Pypes UX Key: OODT-380 URL: https://issues.apache.org/jira/browse/OODT-380 Project: OODT Issue Type: New Feature Components: website Affects Versions: 0.3 Environment: All Reporter: Adam Estrada Priority: Minor Fix For: 0.4 What would you think about integrating http://www.pypes.org/ as the UI for the workflow manager? Seems like a decent idea as most of OODT is about building workflows, right? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
OODT Meet Up at PyCon 2012
Hey OODTers, Paul R and I will be attending PyCon in Santa Clarita from March 7, 2012 to March 11, 2012. We are planning to organize an OODT/ASF meet up with anyone else on this mailing list who will also be attending PyCon this year. The tentative meet-up will happen on Friday March 10th after the first day of sessions. The idea is to select a venue that is within walking distance of the conference so everyone can attend without a lot of logistic head-aches. If you have any questions, comments, or want to confirm you are attending PyCon please reply-all to this message. Details on PyCon: https://us.pycon.org/2012/about/https://us.pycon.org/2012/about/ Thanks for your Time, -Cameron
Re: SOPA PIPA for OODT and other 4-letter acronyms
Sean, You have my +1. We only have 4 days left if we are gonna do this. Do you have any idea's how we can make the site updates needed? I would imagine this would require majority support from the OODT PMC as well. -Cameron On Sat, Jan 14, 2012 at 1:15 AM, Mark Struberg strub...@yahoo.de wrote: basically +1 we could do this with CSS tricks, to overlay a png applied to body with a 'black hole' graphic which gets a higher Z axis or kind of trick. We should not need to change each and every page if possible. LieGrue, strub - Original Message - From: Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov To: dev@oodt.apache.org dev@oodt.apache.org Cc: memb...@apache.org memb...@apache.org Sent: Saturday, January 14, 2012 4:57 AM Subject: Re: SOPA PIPA for OODT and other 4-letter acronyms +1 from me. Cheers, Chris On Jan 13, 2012, at 6:43 PM, Sean Kelly wrote: Colleagues: The Internet is a global resource and I'm thankful that the majority of our Apache and OODT users needn't put up with any of the shenanigans that the governments of Syria, Iran, and China employ routinely to throttle the free distribution of software, open exchange of ideas, and unmoderated dialog of critique deployed against their respective peoples. Sadly, the United States of America is considering legislation [1], [2] in its dual legislative houses of government that is on par with these oppressive regimes[3]—all in a misguided attempt to stop piracy of both durable and logical goods. While we are an international organization that freely distributes software beyond national borders, we've got to admit that the practices employed within the United States has global ramifications, especially since our software, being free and open source, enjoys a worldwide audience and runs the majority of the web's infrastructure. Other efforts within the Apache Software Foundation have already stepped forward with visible, tangible displays of their commitment against the SOPA and PIPA acts currently under consideration by the USA House of Representatives and Senate, respectively. Some of these efforts have elected to participate in an internet strike by displaying banners and/or going dark on the 18th of January, 2012. I would strongly urge the Object Oriented Data Technology (OODT) project of the Apache Software Foundation (ASF) to join in this solidarity and demonstrate a commitment towards the unregulated and open exchange of ideas that we expect on the internet. Towards that end, I propose we shutter the OODT website at http://oodt.apache.org/ and replace every page accessible under that domain with a singular white-text-on-black-background that conveys, without any doubt, our commitment to an open, un-regulated, free-of-government control internet that eschews the ideas of both SOPA and PIPA. I hope OODT will join in this protest. And, if it is at all possible, ALL of apache.org will follow suit. The Internet is at a vital nexus point, and we face an opportunity for change we dare not squander. Yours faithfully, Sean. [1] http://en.wikipedia.org/wiki/Stop_Online_Piracy_Act [2] http://edrn.me/S [3] http://edrn.me/t ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
PushPull Question....
Hey Devs, I have been working on building an OODT PushPull User Guide from an Example I am trying to get working for a project, and I have encountered a little issue (at least I hope it is little). Here is the guide (work in progress): https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide The setups listed in the User Guide are the same ones I am working against. Now here is the log messages I am seeing. Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager setAsRunning INFO: Daemon with ID = 91131 was given permission to run org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol INFO: Testing protocol org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient . . . this may take a few minutes . . . Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.Protocol cdToHOME INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/ isDir=true' Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol SEVERE: Protocol org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient failed compatibility test : [CogJGlobusFtpClient] Failed to get list of files : Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 500 'ORT 97,93,139,127,210,161': command not understood.] Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler getAppropriateProtocolBySite WARNING: ProtocolFactory org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClientFactory is not compatible with server at ftp://l4ftl01.larc.nasa.gov Dec 27, 2011 10:59:01 AM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol INFO: Testing protocol org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient . . . this may take a few minutes . . . Dec 27, 2011 10:59:01 AM org.apache.oodt.cas.pushpull.protocol.Protocol cdToHOME INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/ isDir=true' Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol cdToROOT INFO: Changing to ROOT directory '/' Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol cdToHOME INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/ isDir=true' Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with protocol 'org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient' and username 'anonymous' Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol cd INFO: Changing to directory 'ftp://l4ftl01.larc.nasa.gov/TES/TL2CO2N.005isDir=true' PageSize: 8 PageLoc: 0 FileList size: 491 Dec 27, 2011 10:59:03 AM org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect INFO: Disconnecting protocol from ftp://l4ftl01.larc.nasa.gov Dec 27, 2011 10:59:03 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to sleep until Tue Dec 27 11:00:00 PST 2011 First PushPull tries to use the CogJGlobusFtpClient and it encounters an error so it switches to CommonsNetFtpClient does a couple cd commands, connects successfully then disconnects and goes back to sleep. It will then wake up after the 3 minutes and repeat the same process. Has anyone seen this issue or have any suggestions for a fix? Thank you in advance. -Cameron P.S. If you have any feedback on the PushPull User Guide be sure to leave comments or make changes to the wiki.
Re: PushPull Question....
Brian,
Thanks for the info. I don't have my eclipse setup properly to run that
java file as an application so I will have to try that with some help from
Paul or Chris next week perhaps. I looked up the regex for Java and the \w
is supposed to be a word character matching alphanumeric and underscore.
I ran my regex against this online tool I found and it seems to be a valid
match.
http://www.regexplanet.com/advanced/java/index.html
Regex: TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5
Input: TES-Aura_L2-CO2-Nadir_r002931_F06_08.he5
Thanks for the feedback and help on this. I have tried so many different
tweaks with backslashes, etc. I am going to just walk away now and look at
it later.
-Cam
On Tue, Dec 27, 2011 at 2:58 PM, Brian Foster holeno...@me.com wrote:
Hey cameron... I think the problem is with ur
DirStructXmlParserFiles/TESL2CO2.xml file... for the file name regExp u are
specifying whitespace (i.e. \w instead of underscore)... pushpull looks
like it is behaving normally... it is acting like it is not finding any
files that match ur regExps... I wrote an app which may help u with your
regular expressions... in the pushpull util package in eclipse right click
on ExpressionValidator.java and Run As: Java Application... you can test
your regExp in the GUI that pops up.
-brian
Cameron Goodale good...@apache.org wrote:
Hey Devs,
I have been working on building an OODT PushPull User Guide from an
Example
I am trying to get working for a project, and I have encountered a little
issue (at least I hope it is little).
Here is the guide (work in progress):
https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
The setups listed in the User Guide are the same ones I am working
against.
Now here is the log messages I am seeing.
Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 91131 was given permission to run
org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
Dec 27, 2011 10:59:00 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
INFO: Testing protocol
org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient . . . this
may take a few minutes . . .
Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToHOME
INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/isDir=true'
Dec 27, 2011 10:59:00 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
SEVERE: Protocol
org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient failed
compatibility test : [CogJGlobusFtpClient] Failed to get list of files :
Server refused performing the request. Custom message: (error code 1)
[Nested exception message: Custom message: Unexpected reply: 500 'ORT
97,93,139,127,210,161': command not understood.]
Dec 27, 2011 10:59:00 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler
getAppropriateProtocolBySite
WARNING: ProtocolFactory
org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClientFactory is
not
compatible with server at ftp://l4ftl01.larc.nasa.gov
Dec 27, 2011 10:59:01 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
INFO: Testing protocol
org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient . . . this
may take a few minutes . . .
Dec 27, 2011 10:59:01 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToHOME
INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/isDir=true'
Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToROOT
INFO: Changing to ROOT directory '/'
Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToHOME
INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/isDir=true'
Dec 27, 2011 10:59:02 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with protocol
'org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient' and
username 'anonymous'
Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol cd
INFO: Changing to directory
'ftp://l4ftl01.larc.nasa.gov/TES/TL2CO2N.005isDir=true'
PageSize: 8 PageLoc: 0
FileList size: 491
Dec 27, 2011 10:59:03 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol from ftp://l4ftl01.larc.nasa.gov
Dec 27, 2011 10:59:03 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to sleep
until Tue Dec 27 11:00:00 PST 2011
First PushPull tries to use the CogJGlobusFtpClient and it encounters an
error so it switches to CommonsNetFtpClient does a couple cd commands,
connects successfully then disconnects and goes back to sleep. It will
then wake up after the 3 minutes and repeat the same process. Has anyone
seen this issue or have any suggestions for a fix?
Thank you in advance
Re: [ANNOUNCE] Welcome Ricky Nguyen to the Apache OODT PMC and as a committer
Welcome Ricky! ApacheCon 2011 will go down in the History Books as a great time...from pizza to Irish pubs+car-bombs. Cheers! -Cameron On Thu, Dec 15, 2011 at 12:10 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: #YouAreFullOfWin RICKY! Glad to have you on board! On Dec 15, 2011, at 11:38 AM, Nguyen, Ricky wrote: Hi all! Thanks for welcoming me into the OODT community so warmly and so quickly. It's been great working with Chris, Andrew, Cameron and Dan these past few months as they've helped me with the CHLA VPICU project (@vpicu) (along with Sheryl, Paul Vee, and Dave). I also met Paul Ramirez at ApacheCon in Vancouver, where my eyes were opened to the amazing Apache community. Yesterday I just met Steve Hughes who offered great insight into ontologies. Brian, Sean, Rishi, and Thomas, I've seen your great work, on the mailing lists, in the code, and on the wikis. I look forward to working with you all and the others on this list. A bit about myself… I left the beaches of Hawaii for the snow of Boston to get my degree from MIT. Without professional sports in Hawaii, I became an adopted Red Sox fan and fond of all things Boston. My sister and parents moved to the Bay Area, so I joined them and worked for Oracle for 4 years. My girlfriend earned a pediatric residency at CHLA this fall, so I moved down and coincidentally the VPICU at CHLA had an amazing position open for me! I've been working with OODT ever since. I am sad that Bay Area football decided to suddenly become awesome as soon as I left, happy that the NBA lockout is over and CP3+Griffin=WIN, and excited to contribute to and learn from the OODT community! :D Ricky On Dec 14, 2011, at 1:20 PM, Mattmann, Chris A (388J) wrote: Hi Folks, Ricky Nguyen has recently been elected to join our ranks as an Apache OODT PMC and committer. Welcome Ricky! Please feel free to share a bit about yourself. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ mg_info.txt ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: CAS-CLI option standardiztion...
I say as long as --help can tell me what I need to know to get it working then we all win. No matter which you choose, please keep up the great docs. Great docs are always in fashion. :-D Cameron On Nov 18, 2011 6:00 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: You rox! Cheers, Chris On Nov 18, 2011, at 5:23 PM, holenoter wrote: oh that's right... filemgr... how could i forget about the filemgr???.. i'll do that one next... i'll integrate all the random tools into the client's supported actions as well... such as the delete and query tools.. then in the mean time we can talk about how we want the crawler command-line to look... i'll create an issue for both the filemgr and crawler with cas-cli so we have a place to keep this conversation going... onto the filemgr conversion then... -brian On Nov 18, 2011, at 03:01 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hey duder, On Nov 18, 2011, at 2:28 PM, holenoter wrote: hey guys, okay so the workflow and resource manager have been upgraded to using cas-cli... they maintained the same look as they had before... They look amazing. A true testament to the great generic cli framework you've made. now the question i have is: the command-line look they have now, is that the look we want going forward or do we want to give the command-line a new look... if we decide to change the command-line, backward compatibility can be easily maintained by keeping the current cas-cli spring xml files around and all any user has to do is point at those spring files and workflow and resource manager command-line will work and look like it always has... i don't really have an opinion one way of the other, Yeah I'm kind of the same way. I'm used to the FM WM and RM working that way (with the --operation action step), so I'm not necessarily looking for them to change much. We also haven't had a lot of complaints. I've heard the occasional complaints about --operation, but I haven't seen a lot of folks willing to put in the effort that you have to get them all integrated and unified together, and/or changed. Because of that, I'm tentative to actually suggest they should be changed. Furthermore, I guess in a way we could consider the FM, WM and RM to be different beasts than e.g., crawler and/or pushpull and cas-pge since they are the core *services* and the crawler/pushpull and cas-pge are really client-side frameworks. See more below. but my next component which is going to get cas-cli is the crawler, and it's command-line doesn't look like the workflow or resource manager (probably my fault) Nah, you were just ahead of your time per usual :-) ... so i would like to make it either look like the workflow and resource manager (i.e. --operation --action) or make the workflow and resource manager look more like the crawler... or given them all a new look... either way i would like to make all the components' command-line work in a way such that if you learn one, you learn them all... let me know what you guys think. I'm +1 for your statement about knowing one of them, and then knowing them all. However, I think you've done that from the perspective of having your statement now be true with FM, WM and RM. And I think you're going to do that for the other client frameworks and they will look the same. Having said that, the differences are pretty small (whether or not an --operation flag has to be passed to fire off the action with name action, compared to baking in the action name into the --flag. That's not too different. I think being on cas-cli will help bring them all together in the long run anyways, but I don't see a huge value added from changing the CLI contracts as they have existed for a while. On the other hand, I see a HUGE advantage for standardizing the software framework with a flexible CLI framework like you've created to help that. My 2 cents. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer
Re: [ANNOUNCE] Welcome Sheryl John as new Apache OODT committer and PMC member
Welcome to OODT Sheryl! I am glad to have you on the team. -Cameron On Sat, Nov 12, 2011 at 1:01 PM, Sheryl John shery...@gmail.com wrote: Hello everyone! First of all, thanks to all from the OODT lists and the JPL team who've helped me in the past couple of months. And, a special shout out to Prof Mattmann for mentoring me all along. :) About myself: A native of south India, grew up in Dubai, and then completed my Bachelors in Computer Science and Engineering from National Institute of Technology,Trichy (India) on May 2010. I joined University of Southern California last Fall for MS in CS. I was introduced to Apache OODT world by Prof. Mattmann during his Software Architecture class last Spring and, since this summer, I've been working as an intern at Children's Hospital Los Angeles for the Virtual Pediatric Intensive Care Unit (VPICU). At VPICU, I'm learning how to leverage OODT's components for clinical data workflows and medical data archiving and management. It's been awesome working with OODT and I hope to contribute more to the project and the community. Go OODT! :-D Thanks again for having me aboard! Sheryl On Sat, Nov 12, 2011 at 12:21 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Folks, A while back I nominated Sheryl John for OODT PMC and committership. The VOTE results are back and I'm happy to report that Sheryl is now a member of the OODT PMC and a committer! Welcome, Sheryl. Please feel free to say a bit about yourself. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -- -Sheryl -- Sent from a Tin Can attached to a String
Re: [jira] [Updated] (OODT-345) Integration CAS-Workflow with CAS-CLI
Brian,
This isn't really technically relevant, but I must comment that you are a
MACHINE! I am actually hoping that I have a project that will use your new
CLI interface. Just wanted to let you know that your work is appreciated.
-Cam
On Mon, Nov 7, 2011 at 1:32 PM, Brian Foster (Updated) (JIRA)
j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/OODT-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
Brian Foster updated OODT-345:
--
Attachment: SwitchWorkflowToUseCasCli.patch.txt
- attached patch
- code can be reviewed here: https://reviews.apache.org/r/2755/
Integration CAS-Workflow with CAS-CLI
-
Key: OODT-345
URL: https://issues.apache.org/jira/browse/OODT-345
Project: OODT
Issue Type: Improvement
Components: workflow manager
Affects Versions: 0.4
Environment: none
Reporter: Brian Foster
Assignee: Brian Foster
Priority: Minor
Fix For: 0.4
Attachments: SwitchWorkflowToUseCasCli.patch.txt
Here are some example outputs of what the new command line will look
like:
{code}
$ ./wmgr-client -h
-
| Short | Long | Description
-
-u, --url urlWorkflow
Manager URL
Handler:
Sets the
following Java Properties:
[org.apache.oodt.cas.workflow.url]
-op,--operationDeclare that
you wish to present an operation
-p, --pageNum page Page Number
Requirement Rules:
[GetNextPage : REQUIRED, GetPrevPage : REQUIRED]
Handler:
Will
invoke 'setPageNum' on action selected,
except
for the following actions: []
-m, --metaData Declare that
you wish to present metadata
Requirement Rules:
[SendEvent : OPTIONAL, DynMetadata : OPTIONAL]
-en,--eventName event-name Name of the
Event to trigger
Requirement Rules:
[SendEvent : REQUIRED, GetWorkflowsByEvent :
REQUIRED]
Handler:
Will
invoke 'setEventName' on action selected,
except
for the following actions: []
-s, --status status Workflow
status
Requirement Rules:
[GetFirstPage : OPTIONAL, GetNextPage : OPTIONAL,
GetPrevPage : OPTIONAL, GetLastPage : OPTIONAL]
Handler:
Will
invoke 'setStatus' on action selected, except
for the
following actions: []
-tids, --taskIds task-ids List of
workflow task ids
Requirement Rules:
[DynWorkflow : REQUIRED]
Handler:
Will
invoke 'setTaskIds' on action selected,
except
for the following actions: []
-id,--id id ID
Requirement Rules:
[GetTaskById : REQUIRED, GetConditionById :
REQUIRED,
GetWorkflowById : REQUIRED,
GetWorkflowInst : REQUIRED, GetWallClockTime :
REQUIRED,
GetTaskWallClockTime : REQUIRED,
StopWorkflowInst : REQUIRED, PauseWorkflowInst :
REQUIRED,
ResumeWorkflowInst : REQUIRED,
GetWorkflowInstMet : REQUIRED]
Handler:
Will
invoke 'setId' on action selected, except for
the
following actions: [GetTaskById : setTaskId,
GetConditionById :
Re: Getting started
Thien, Thanks for the feedback on the site. Below is my take on OODT, and I am sure others can chime in with maybe some publications you can read. I hope you find this helpful. OODT is not a turnkey solution. (Just wanted to get that out of the way, since many people see OODT as a product and not a framework) OODT is a framework for data management (this means ANY data, from Space to Climate to Medical to Images can be archive and processed with OODT) The power of OODT is the flexibility of the framework, but this also makes it challenging to grasp and understand everything it can do. The power of OODT is in the community of Devs and Users. Chances are someone else in the community has encountered the same challenge or a similar one. Keep the Questions and Comments coming. -Cameron On Fri, Oct 28, 2011 at 9:27 AM, Nguyen, Thien thien.nguy...@va.gov wrote: Hello, I'm a medical informatics software developer at the Boston Department of Veteran Affairs. We are reviewing tools to prototype for use in our systems that would help manage, share, and track research information. We have genomic data, health records, phenotypic data, and results of nlp/machine learning techniques. OODT sounds like a nice package that we would like to try out, but, honestly, the website says very little. I'm not even sure where to begin to even try interacting with the downloaded source. Any direction on how to get started or lower-level resources to read up on would be greatly appreciated. Thanks, Thien
Re: [jira] [Commented] (OODT-52) Update the CAS File Manager User Guide
Great catch and solution on the Maven 3 vs. 2 issue on Mac. The filemanager xdocs used to tell you how to first install oodt-core, then ONLY install filemanager. I tried this last night and it would not build properly due to missing jars, etc... My plan is to update the xdoc for File Manager to instruct the user to checkout and built OODT (the complete suite) it will just be easier. Once they get the build success message I will send them to the wiki. I will also be sure to include the warning message for Mac OS X users. Even in 10.5.x the Mac Java updates also update to Maven 3. Getting closer to closing this issue out. -Cam On Tue, Sep 27, 2011 at 4:44 PM, Thomas Bennett tbenn...@ska.ac.za wrote: Hi Cameron, I previously built another wiki page that is dedicated to File Manager Policy. If you ever decide to tackle that part of File Manager I have a pretty good start you can fold back into the User Guide. https://cwiki.apache.org/confluence/display/OODT/Everything+you+want+to+know+about+File+Manger+Policy Excellent! Thank you. I'll work it into my current wiki. I have a some free cycles this weekend so I will start updating the xdoc page for the Basic User Guide on the site. I will use the page to explain the maven install and build process. Then point the user off to your great wiki page. Great! Thanks. This is one aspect that would be extremely useful but that I've been avoiding - so thank you. I did however make some brief notes on getting the environment working on a Mac. Please feel free to use. *Java and Maven on OS X 10.7* *Maven* Mac's ship with maven 3 by default. OODT needs maven 2. Should you wish to get maven 2 set up as your default, you can use the following steps: 1. Download apache-maven-2.2.1-bin tar file. 2. Install it to /usr/share/ by running sudo tar xzvf apache-maven-2.2.1-bin.tar.gz -C /usr/share 3. Now hook it in as the default maven. $ sudo rm /usr/bin/mvn $ sudo rm /usr/share/maven $ sudo ln -s /usr/share/apache-maven-2.2.1 /usr/share/maven $ sudo ln -s /usr/share/maven/bin/mvn /usr/bin/mvn *Java VM* To get the java environment working, you need to get the java_home command into your *$PATH.* $ ln -s /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home /usr/bin/java_home Now set up the environment variable. Add this line to your ~/.profile or file. JAVA_HOME=$( java_home ) Or source it directly: JAVA_HOME=$( System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home ) Cheers, Tom -- Sent from a Tin Can attached to a String
Re: Deploying CAS-FileManager within Windows
Rishi, I tried to get filemgr started on a windows machine about 2 years ago and I ran into a lot of issues. It is definitely made for a *nix type OS. If you do figure out the main sticking points I hope we can get some docs or issues setup to guide other Win users. OODT-187: https://issues.apache.org/jira/browse/OODT-187 Deals with the file pathing within the properties file. This could be an issue that might solve the problem. I would hope that file:/// would be supported by Win and *nix systems, but I haven't done the research to confirm. Good luck, Cameron On Fri, Sep 2, 2011 at 4:45 PM, Verma, Rishi (317I) rishi.ve...@jpl.nasa.gov wrote: Hello fellow developers, I'm trying to deploy CAS-Filemanager (0.4-SNAPSHOT) to a Windows XP machine, but am witnessing some warnings due to paths being set for a windows environment. Two questions I want to ask: [1] Does filemgr.properties ASSUME a UNIX-like path for the following two props (org.apache.oodt.cas.filemgr.repositorymgr.dirs, org.apache.oodt.cas.filemgr.validation.dirs)? [2] Is the prefix file:// for the above two properties applicable only to a unix environment? I say this because I have tried (on the Windows machine) to place paths using Windows-style forward-slash \ paths, as well as using Cygwin style paths to the same corresponding directories; however, neither method eliminates the below warnings when attempting to start filemgr within Windows (running within Cygwin I might add). Thanks! Rishi --- Below is an attempt to start filemgr: $ ./filemgr start Starting cas file manager: OK Sep 2, 2011 2:47:30 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager load Configuration INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.proper ties] Sep 2, 2011 2:47:30 PM org.apache.oodt.cas.filemgr.validation.XMLValidationLayer loadElements WARNING: Element directory: file:///cygdrive/c/proteome/proteome-filemgr-win/pol icy is not a directory: skipping element loading from it. Sep 2, 2011 2:47:30 PM org.apache.oodt.cas.filemgr.validation.XMLValidationLayer loadProductTypeMap WARNING: Element directory: file:///cygdrive/c/proteome/proteome-filemgr-win/pol icy is not a directory: skipping product type element map loading from it. Sep 2, 2011 2:47:31 PM org.apache.oodt.cas.filemgr.repository.XMLRepositoryManag er loadProductTypes WARNING: Product type directory: file:///cygdrive/c/proteome/proteome-filemgr-wi n/policy is not a directory: skipping product type loading from it. Sep 2, 2011 2:47:31 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClien t init INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.proper ties] Sep 2, 2011 2:47:31 PM org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransfe rer setFileManagerUrl INFO: Local Data Transfer to: [http://localhost:9000] enabled Sep 2, 2011 2:47:31 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager ini t INFO: File Manager started -- Sent from a Tin Can attached to a String
Re: [jira] [Updated] (OODT-306) File Manager Tools for BASH and TCSH
So I have been mulling this over since I wrote this JIRA issue and I think I
will break fmquery into:
lucene_query and sql_query since the query_tool that is being called under
the code is able to query both, and prefixing fm seems a bit redundant.
-Cameron
On Fri, Jul 22, 2011 at 6:03 PM, Cameron Goodale (JIRA) j...@apache.orgwrote:
[
https://issues.apache.org/jira/browse/OODT-306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
Cameron Goodale updated OODT-306:
-
Description:
*Chris Mattmann* has been writing some really useful shell alias commands
to help with some of the really repetitive tasks that tend to appear when
using the File Manager component. This issue will be used to share and
commit 2 files that contain environment settings that can be easily sourced,
or just copied into a users .bashrc or .tcshrc startup file.
The files will contain the scripting commands and some basic documentation.
I plan to drop the files into oodt/filemgr/src/main/resources since they
really only apply to the filemanager.
Here is my initial working Docu-Code (for BASH) I plan to clean up and
commit.
{code}
# Copy and Paste this Block into the .bashrc of your deployment user
account ##
#
# The following aliases must be used within a filemgr installation's
# bin directory since relative pathing is being used. This block also
# assumes that the filemgr is running on port 9000 (the default port of
filemgr)
#
alias fmquery=java
-Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
-Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url
http://localhost:9000 --lucene -query
#
alias fmdel=java
-Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
-Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.DeleteProduct
--fileManagerUrl http://localhost:9000 --read
#
alias metdump=java -Djava.ext.dirs=../lib
org.apache.oodt.cas.filemgr.tools.MetadataDumper --url $FILEMGR_URL --out .
--productId
#
END OF BLOCK ###
{code}
h2. Previous Commands Explained
h3. fmquery
This will execute a Lucene Query against the filemanager catalog and return
a list of ProductIDs. These Product IDs can then be piped into the fmdel
alias to remove all products that match the query criteria.
*Usage:*
- Return the ProductIds to the terminal window -
fmquery ProductType:ISMRawData
- Pipe the ProductIds to a file -
fmquery ProductType:ISMRawData ProductID_List.txt
h3. fmdel
Use this command to delete a single product from the catalog. This will
remove the lucene index entry from the catalog and will remove the archived
product from disk. (If you just want to see what products will be removed
from the archive, then you will need to add the --nocommit parameter to the
fmdel command alias after the --read param)
*Usage:*
- Remove one product from the catalog
fmdel 0691ee17-96ec-11e0-8556-3b3ca12e4f64
- Remove a set of products from the catalog using the fmquery command
[fmquery Metadata_Key:Matching_Key_Value | fmdel]
For example: To remove all Products with a GenericFile ProductType use
fmquery ProductType:GenericFile | fmdel
h3. metdump
Just added to the list. This will take in a product_id and dump out the
product's metadata into the current directory. *This one needs a little
more testing and docs.*
Comments are welcome.
was:
Chris Mattmann has been writing some really useful shell alias commands to
help with some of the really repetitive tasks that tend to appear when using
the File Manager component. This issue will be used to share and commit 2
files that contain environment settings that can be easily sourced, or just
copied into a users .bashrc or .tcshrc startup file.
The files will contain the scripting commands and some basic documentation.
I plan to drop the files into oodt/filemgr/src/main/resources since they
really only apply to the filemanager.
Here is my initial working Docu-Code (for BASH) I plan to clean up and
commit.
# Copy and Paste this Block into the .bashrc of your deployment user
account ##
#
# The following aliases must be used within a filemgr installation's
# bin directory since relative pathing is being used. This block also
# assumes that the filemgr is running on port 9000 (the default port of
filemgr)
#
alias fmquery=java
-Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
-Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url
http://localhost:9000 --lucene -query
#
alias fmdel=java
-Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
-Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.DeleteProduct
--fileManagerUrl http://localhost:9000 --read
#
alias metdump=java -Djava.ext.dirs=../lib
org.apache.oodt.cas.filemgr.tools.MetadataDumper --url $FILEMGR_URL -
-out
Re: [jira] [Updated] (OODT-296) Workflow Editor GUI
Chris, I saw the screen shot you attached and it looks really good. Hats off to you and bfoster on this one. Cameron On Jul 6, 2011 9:41 AM, Chris A. Mattmann (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/OODT-296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] Chris A. Mattmann updated OODT-296: --- Attachment: OODT-296.070611.Mattmann.patch.txt - latest patch. I'm trying to figure out how to make Jgraph and JUNG play nicely together. I *sort* of have it down but am still learning. Consequently, the editor doesn't load the workspace anymore, NPE, but you can see where I'm going. Workflow Editor GUI --- Key: OODT-296 URL: https://issues.apache.org/jira/browse/OODT-296 Project: OODT Issue Type: Sub-task Components: workflow manager Reporter: Chris A. Mattmann Assignee: Chris A. Mattmann Fix For: 0.4 Attachments: OODT-296.070611.Mattmann.patch.txt, OODT-296.Mattmann.070611.patch.txt, Screen shot 2011-07-05 at 2.28.51 PM.png Internally at JPL, we built a Workflow 2 GUI (thanks [~bfoster]!) :-) I'm working on preparing a patch to port it from the old internal JPL workflow manager (wengine) code to Apache OODT. I'll use this patch to track the progress. I need to migrate: * namespaces from g.j.n.oodt to o.a.oodt * use of jgraph library to http://jung.sourceforge.net/doc/api/index.html * some old deps on wengine-branch internal to JPL I'll attach a screenshot to show what it looks like (and will look like) shortly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Closed] (INFRA-3707) Enable svnpubsub on https://svn.apache.org/repos/asf/oodt/site
Chris, This is great news. I need to push out updates to the website, so I will try it without an svn up at the end and see what happens. If all goes well I will update the Wiki on how to build the website to remove that step. Thanks for setting this up. -Cameron On Wed, Jun 29, 2011 at 4:51 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: All, FYI, svnpubsub now enabled on the oodt/site directory. Cam, when you check in site updates, now it will automatically push it out to the mirrors, no need to SVN up anymore... Cheers, Chris Begin forwarded message: From: Gavin (JIRA) j...@apache.org Date: June 29, 2011 4:43:28 PM PDT To: Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov Subject: [jira] [Closed] (INFRA-3707) Enable svnpubsub on https://svn.apache.org/repos/asf/oodt/site [ https://issues.apache.org/jira/browse/INFRA-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] Gavin closed INFRA-3707. Resolution: Fixed Assignee: (was: Gavin) All done, apologies for delay Enable svnpubsub on https://svn.apache.org/repos/asf/oodt/site -- Key: INFRA-3707 URL: https://issues.apache.org/jira/browse/INFRA-3707 Project: Infrastructure Issue Type: Task Security Level: public(Regular issues) Components: SvnPubSub Environment: OODT site Reporter: Chris A. Mattmann Priority: Minor Hi Guys, Can you please enable svnpubsub on https://svn.apache.org/repos/asf/oodt/site/? Thanks! Cheers, Chris -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -- Sent from a Tin Can attached to a String
Re: A Question About OODT Documentation
Phil, Thanks for the input. Documentation is always important and something I have been trying to address. Currently our documentation is grouped by component, and some of the more mature components have more documentation. For Example FileManager: http://oodt.apache.org/components/maven/filemgr/ Has 3 Software Documentation Pages (one is still under construction) If you need more information on a specific component, or have suggestions for including JPL Docs @Apache I welcome the input. I appreciate that you took the time to dig around looking for our old docs. Best Regards, Cameron On Fri, Jun 24, 2011 at 6:53 AM, Phillip Rhodes motley.crue@gmail.comwrote: Hey guys, quick question... I'm interested in getting more familiar with OODT, so I started poking around the website looking for docs. Then I started Googling for stuff, and at one time had found the old JPL page which had links to a lot of stuff on it. This is even mentioned in the OODT Incubator Proposal There is a wealth of documentation available on OODT. The best starting point is the existing OODT JPL website (which will be ported to be sync'ed or just a pointer to the Apache website)http://oodt.jpl.nasa.gov; But as far as I can tell, since the oodt.jpl.nasa.gov page was re-pointed to oodt.apache.org, no corresponding documentation page was ever created that links to that stuff. I was able to go through the Wayback Machine and find the docs from the old site, but for the benefit of new-comers, it might be good to recreate that here? Or if it has already been recreated, maybe the link should be more prominent? (I didn't find one, but maybe I just missed it). Thanks, Phil
Re: A Question About OODT Documentation
Phil, Consider me inspired by your drive to learn about OODT. It wasn't that long ago that I didn't understand what a FileManager was, or when to use the Crawler, but in working with OODT I have lost sight of the steep learning curve as I have started to climb it. So to help you (and anyone else that has questions and wants answers) I created a wiki page that we can use to collaborate. It is specifically focused on New User Questions. Head on over and check it out. You should be able to create a new account (if you don't have one already) and start making updates, adding in questions you want answered (even if they are related to the 'old JPL docs') and we will do our best to answer and explain and get a New User Overview out of the wiki and published to the oodt website. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27361963 Bookmark it, and jot down questions and comments as you explore our project. I encourage the rest of the community to also follow/Watch the page and see how you can help grow it since we all have certain components we know better than others. I am looking forward to working with you Phil, Cameron On Fri, Jun 24, 2011 at 7:51 AM, Phillip Rhodes motley.crue@gmail.comwrote: Regarding the old docs: I kinda think it would be good to link to pretty much all of those same papers (assuming they're still available and that it's legal, blah, blah) that were up on the JPL site. Even if some of them are dated now, they provide historical context and something of an overview. Otherwise, the one thing that I think would be OODT n00bs (like myself) would be a nice, up to date, overview that lays out What are all these pieces, how do they fit together, and how does the whole system work? Just jumping into what's at http://oodt.apache.org, it's not necessarily real apparent which piece (or pieces) are needed, which one to focus on first, etc. Maybe something of a New User Overview? I'd be happy to help with such a thing, but I currently lack the knowledge to contribute one myself. I'm still just going through the existing docs and muddling through, trying to get my head around it all. :-) Phil On Fri, Jun 24, 2011 at 10:44 AM, Cameron Goodale good...@apache.org wrote: Phil, Thanks for the input. Documentation is always important and something I have been trying to address. Currently our documentation is grouped by component, and some of the more mature components have more documentation. For Example FileManager: http://oodt.apache.org/components/maven/filemgr/ Has 3 Software Documentation Pages (one is still under construction) If you need more information on a specific component, or have suggestions for including JPL Docs @Apache I welcome the input. I appreciate that you took the time to dig around looking for our old docs. Best Regards, Cameron On Fri, Jun 24, 2011 at 6:53 AM, Phillip Rhodes motley.crue@gmail.comwrote: Hey guys, quick question... I'm interested in getting more familiar with OODT, so I started poking around the website looking for docs. Then I started Googling for stuff, and at one time had found the old JPL page which had links to a lot of stuff on it. This is even mentioned in the OODT Incubator Proposal There is a wealth of documentation available on OODT. The best starting point is the existing OODT JPL website (which will be ported to be sync'ed or just a pointer to the Apache website)http://oodt.jpl.nasa.gov; But as far as I can tell, since the oodt.jpl.nasa.gov page was re-pointed to oodt.apache.org, no corresponding documentation page was ever created that links to that stuff. I was able to go through the Wayback Machine and find the docs from the old site, but for the benefit of new-comers, it might be good to recreate that here? Or if it has already been recreated, maybe the link should be more prominent? (I didn't find one, but maybe I just missed it). Thanks, Phil
Re: [VOTE] Apache OODT 0.3 Release
Team, Here is my check list: mvn package - SUCCESS mvn site - FAIL [INFO] [ERROR] BUILD ERROR [INFO] [INFO] Failed to resolve dependencies for one or more projects in the reactor. Reason: Missing: -- 1) org.apache.oodt:oodt-commons:jar:0.3 ... Path to dependency: 1) org.apache.oodt:pcs-input:jar:0.3 2) org.apache.oodt:oodt-commons:jar:0.3 -- 1 required artifact is missing. for artifact: org.apache.oodt:pcs-input:jar:0.3 from the specified remote repositories: javax-transaction (http://download.java.net/maven/2), apache.snapshots (http://repository.apache.org/snapshots/), globus-repo ( http://projects.iplantcollaborative.org/archiva/repository/internal), central (http://repo1.maven.org/maven2), oodt-m2-repo (http://oodt.jpl.nasa.gov/download/private), apache.org (http://people.apache.org/repo/m2-snapshot-repository), maven.topazproject.org (http://maven.topazproject.org/maven2), hadoop.apache.org (http://people.apache.org/~kalle/mahout/maven2/) startup FileManager - SUCCESS Ingest the blah.txt - SUCCESS start up Workflow - SUCCESS run the test workflow - SUCCESS Running mvn site is not a blocker in my opinion, I will just sort it out post-release when I update the website. +1 From me. -Cameron On Fri, Jun 24, 2011 at 7:06 AM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: On Jun 24, 2011, at 6:06 AM, Sean Kelly wrote: Signatures and checksums are good. I confirmed the Maven build as well. How did the Python build go? Good question. I didn't run the python tests but looks like woollard ran them and they look good. I just tried them out via: python setup.py test in the agility/ folder and... Test the `Documentable` and `DocumentableField` classes. ... ok Doctest: oodt.xmlutils.add ... ok Doctest: oodt.xmlutils.text ... ok -- Ran 34 tests in 0.070s OK All is well. Did they run on your machine? Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: svn commit: r1138739 - in /oodt/trunk: core/pom.xml pom.xml
Paul, Great job tracking this down. You are one of a small few who would track this down and fix it, so thank you for that. I am sure this will benefit all future Releases and Release Managers. -Cam On Wed, Jun 22, 2011 at 10:22 PM, prami...@apache.org wrote: Author: pramirez Date: Thu Jun 23 05:22:53 2011 New Revision: 1138739 URL: http://svn.apache.org/viewvc?rev=1138739view=rev Log: The maven release plugin is real picky. Having the scm parameters configured in the common parent pom confused the generation of the scm.url during the release:prepare. The scm parameters were simply moved from core/pom.xml to pom.xml. Unfortunately this can't be fully tested until it is committed as the release:prepare goal bails unless everything is commited in the current check out. It does however get far enough to see that the scm.url in the release.properties is corrected. Modified: oodt/trunk/core/pom.xml oodt/trunk/pom.xml Modified: oodt/trunk/core/pom.xml URL: http://svn.apache.org/viewvc/oodt/trunk/core/pom.xml?rev=1138739r1=1138738r2=1138739view=diff == --- oodt/trunk/core/pom.xml (original) +++ oodt/trunk/core/pom.xml Thu Jun 23 05:22:53 2011 @@ -255,12 +255,6 @@ the License. distributionmanual/distribution /license /licenses -scm -connectionscm:svn:https://svn.apache.org/repos/asf/oodt/trunk /connection -developerConnectionscm:svn: https://svn.apache.org/repos/asf/oodt/trunk/developerConnection -urlhttp://svn.apache.org/viewvc/oodt/trunk/url -tagHEAD/tag -/scm organization nameApache OODT/name urlhttp://oodt.apache.org//url Modified: oodt/trunk/pom.xml URL: http://svn.apache.org/viewvc/oodt/trunk/pom.xml?rev=1138739r1=1138738r2=1138739view=diff == --- oodt/trunk/pom.xml (original) +++ oodt/trunk/pom.xml Thu Jun 23 05:22:53 2011 @@ -34,7 +34,14 @@ the License. description Apache OODT is middleware for metadata. /description - + +scm +connectionscm:svn:https://svn.apache.org/repos/asf/oodt/trunk /connection +developerConnectionscm:svn: https://svn.apache.org/repos/asf/oodt/trunk/developerConnection +urlhttp://svn.apache.org/viewvc/oodt/trunk/url +tagHEAD/tag +/scm + modules modulecore/module modulecommons/module -- Sent from a Tin Can attached to a String
Re: Updated RADIX web page data dir
Guys, Sorry for being largely absent from the discussion, but I think you guys have it well under control. My vision for RADiX was an 80% solution so Chris if you want to be in the other 20% that is cool with me ;) I think we should make it clear that RADiX is an OPTION, and not the ONLY way to build and deploy OODT. Our desire is to help OODT Devs and Users by giving them a sort of 'default' development and deployment structure. I think the Ruby on Rails guys said it best, Convention over configuration since RADiX will already be configured to use it's convention. Great input and feedback Chris. I wonder if anyone else has given the Wiki a look and wants to give their input, we are open to it. https://cwiki.apache.org/OODT/radix-powered-by-oodt.html Prost! Cameron On Thu, May 19, 2011 at 10:13 PM, Mattmann, Chris A (388J) chris.a.mattm...@jpl.nasa.gov wrote: +1 to /failure/ingest and /failure/jobs, etc. That makes sense to me...though unfortunately will cause grumpy people like me to have to change to take advantage of the power of RADIX. I'll learn though (slowly) over time :) Cheers, Chris On May 19, 2011, at 7:40 AM, Ramirez, Paul M (388J) wrote: Hey Chris, The failure was just meant to be more generic as there could be other items, from say workflow, that would also go in there. How about having /failure/ingest? That still does remove the files word though so /failure/ingest_files is a possibility. The reasoning behind not having files in the directory name is that at the highest level we talk about ingestion of products and not files. Thanks, Paul On May 18, 2011, at 6:21 PM, Mattmann, Chris A (388J) wrote: Hi Paul, met could be for either the crawler or for the curator. As for failed_ingest_files, why the rename? Just to be shorter? I think failed_ingest_files, though more verbose, conveys more meaning than failure... Cheers, Chris On May 18, 2011, at 7:50 AM, Ramirez, Paul M (388J) wrote: Hey Chris, The met directory was for generated metadata by the crawler or was it for the curator? I updated the page with a description of the paths and renamed failed_ingest_files to failure. On the next pass I'll start posting a description of the source structure that will be available as output from the RADiX archetype. Thanks, Paul On May 17, 2011, at 7:45 PM, Mattmann, Chris A (388J) wrote: Hey Guys, FYI: https://cwiki.apache.org/confluence/display/OODT/RADiX+Powered+By+OODT I updated the page to propose a: /data /archive /staging /work /met /failed_ingest_files Dir structure for RADIX to more closely match our existing OODT deployments. Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -- Sent from a Tin Can attached to a String
Re: [jira] [Resolved] (OODT-156) Rewrite Workflow Monitor webapp using Apache Wicket
Chris, I have seen this webapp in action and it really is impressive. Good job learning wicket. I am looking forward to a 3rd wicket app...*hint *hint. For anyone on the list who is wondering what the workflow monitor looks like in action, take a look at this: http://min.us/mvfGVMn HTH Cameron On Fri, May 13, 2011 at 8:43 AM, Chris A. Mattmann (JIRA) j...@apache.orgwrote: [ https://issues.apache.org/jira/browse/OODT-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] Chris A. Mattmann resolved OODT-156. Resolution: Fixed - fixed in r1102794. Wow, this was a massive undertaking! I removed the old workflow JSP webapp as part of r1102794. Also, this is my 2nd Wicket webapp so I'm getting better, but tons of room for improvement! Comments/updates welcome! Rewrite Workflow Monitor webapp using Apache Wicket --- Key: OODT-156 URL: https://issues.apache.org/jira/browse/OODT-156 Project: OODT Issue Type: Improvement Components: workflow manager Reporter: Chris A. Mattmann Assignee: Chris A. Mattmann Labels: java, monitor, webapp, wicket, workflow Fix For: 0.3 Attachments: OODT-156.Mattmann.050411.patch.txt, OODT-156.Mattmann.050911.patch.txt Similar to OODT-155, except for the Workflow Monitor web application. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira -- Sent from a Tin Can attached to a String
Re: CAS_PGE's ExternExtractorMetWriter config file
Faranak,
I haven't tried the code yet, but I think the problem is you call python
code like this:
python CARVE_KML_Extractor.py
I think using ./ tells the she'll to run it. The shell doesn't understand
how to import modules.
Try using python instead of the ./ and let me know how it works.
Cameron
On Apr 15, 2011 9:24 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.gov wrote:
Due to having a problem with my eyes, I was hesitant to go through this
documentation. I finally endured the pain(literally) and went through it.
Apparently those lines are for running that mp3 java code. Please answer
my question below:
1.
The python extractor I have generates the file bellow:
?xml version=1.0 encoding=UTF-8?
cas:metadata xmlns:cas=http://oodt.jpl.nasa.gov/1.0/cas;
keyval
keyMasterQualityFlag/key
valGood/val
/keyval
keyval
keyProcessingLevel/key
valL1/val
/keyval
/cas:metadata
I do: chmod +x script.py however, it when I try to run it as ./script.py.
It wouldn't and complains about :
./CARVE_KML_Extractor.py: line 20: import: command not found
./CARVE_KML_Extractor.py: line 21: import: command not found
./CARVE_KML_Extractor.py: line 22: import: command not found
from: can't read /var/mail/xml.dom.minidom
However if I simply run like: python script.py [inputProduct][outputdir],
it works fine. Do I need to be able to run that python script like
./script.py to be able to use it as an metadata extractor.
Cameron, since this is your code. Would you help me and see what you can
do to make it run like: ./script.py.
2. I noticed that in the link
http://oodt.apache.org/components/maven/curator/user/basic.html, the
python code you just specify the input file:
./mp3PythonExtractor.py /usr/local/staging/products/mp3/Bach-SuiteNo2.mp3.
Since I need to integrate this to my cas-pge crawler, and it needs to
perform this metadata extraction on the product just generated, what
exactly I need to pass as argument in terms of [inputFile][outmetpath]?
Thanks,
Faranak
On 4/15/11 7:51 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.gov wrote:
Based on the document you have sent me, for the simple python script that
runs like this: kml.python [input][outputmet]. I need to have the config
file like:
?xml version=1.0 encoding=UTF-8?
cas:externextractor xmlns:cas=http://oodt.jpl.nasa.gov/1.0/cas;
exec workingDir=
extractorBinPath/usr/local/extractors/mp3extractor/mp3PythonExtractor.py
/extractorBinPath
args
arg isDataFile=true/
/args
/exec
/cas:externextractor
Do I have to specify the working directory like the first line: exec
workingDir=. Because I don't see that line in the peate's sample file.
Also I see the python extractor has these lines that mines doesn't. Is
this the reason my python script doesn't get run? The python file I have
simply parses the file and generates the output met.
cmd = java -jar /Users/woollard/Desktop/extractors/mp3extractor/
cmd += tika-app-0.5-SNAPSHOT.jar -m +fullPath+ | awk -F:
cmd += 'BEGIN {print \cas:metadata xmlns:cas=
cmd += \\\http://oodt.jpl.nasa.gov/1.0/cas\\\;\}
cmd += {print \keyvalkey\$1\/keyval\substr($2,2)\
cmd += /val/keyval\}' +fileName+.met
#!/usr/bin/python
import os
import sys
fullPath = sys.argv[1]
pathElements = fullPath.split(/);
fileName = pathElements[len(pathElements)-1]
fileLocation = fullPath[:(len(fullPath)-len(fileName))]
productType = MP3
cmd = java -jar /Users/woollard/Desktop/extractors/mp3extractor/
cmd += tika-app-0.5-SNAPSHOT.jar -m +fullPath+ | awk -F:
cmd += 'BEGIN {print \cas:metadata xmlns:cas=
cmd += \\\http://oodt.jpl.nasa.gov/1.0/cas\\\;\}
cmd += {print \keyvalkey\$1\/keyval\substr($2,2)\
cmd += /val/keyval\}' +fileName+.met
os.system(cmd)
f = open(fileName+.met, 'a')
f.write('keyvalkeyProductType/keyval'+productType)
f.write('/val/keyval\nkeyvalkeyFilename/keyval')
f.write(fileName+'/val/keyval\n'keyvalkeyFileLocation')
f.write('/keyval'+fileLocation+'/val/keyval\n')
f.write('/cas:metadata')
f.close()
On 4/15/11 6:22 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.gov wrote:
Thanks Brian. The document was actually very helpful.
--Faranak
From: holenoter holeno...@me.commailto:holeno...@me.com
Reply-To: dev@oodt.apache.orgmailto:dev@oodt.apache.org
dev@oodt.apache.orgmailto:dev@oodt.apache.org
Date: Fri, 15 Apr 2011 14:19:04 -0700
To: dev@oodt.apache.orgmailto:dev@oodt.apache.org
dev@oodt.apache.orgmailto:dev@oodt.apache.org
Cc: dev@oodt.apache.orgmailto:dev@oodt.apache.org
dev@oodt.apache.orgmailto:dev@oodt.apache.org
Subject: Re: CAS_PGE's ExternExtractorMetWriter config file
http://oodt.apache.org/components/maven/metadata/user/basic.html
On Apr 15, 2011, at 02:09 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.govmailto:faranak.davo...@jpl.nasa.gov
wrote:
I have a couple of out put products that I am trying to extract extra
metadata and add them to the final .met files. Here is how I run my
files:
Python [someBinDir]/ncdump
Re: CAS_PGE's ExternExtractorMetWriter config file
Faranak,
I just dug into this a little deeper and you just need to add one line of
code to the Python Extractor. Make the FIRST line:
#!/usr/bin/python
Then retry running the Extractor as ./CARVE_KML_Extractor etc
With that line first, the shell will realize the script is a python
program and run it properly.
It will send a lot of messages to the Terminal, but it will run. I
designed the script to be run using python.
Hope that helps, and sorry to hear about your eyes.
-Cameron
On Sat, Apr 16, 2011 at 1:25 PM, Cameron Goodale sigep...@gmail.com wrote:
Faranak,
I haven't tried the code yet, but I think the problem is you call python
code like this:
python CARVE_KML_Extractor.py
I think using ./ tells the she'll to run it. The shell doesn't understand
how to import modules.
Try using python instead of the ./ and let me know how it works.
Cameron
On Apr 15, 2011 9:24 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.gov wrote:
Due to having a problem with my eyes, I was hesitant to go through this
documentation. I finally endured the pain(literally) and went through it.
Apparently those lines are for running that mp3 java code. Please answer
my question below:
1.
The python extractor I have generates the file bellow:
?xml version=1.0 encoding=UTF-8?
cas:metadata xmlns:cas=http://oodt.jpl.nasa.gov/1.0/cas;
keyval
keyMasterQualityFlag/key
valGood/val
/keyval
keyval
keyProcessingLevel/key
valL1/val
/keyval
/cas:metadata
I do: chmod +x script.py however, it when I try to run it as ./script.py.
It wouldn't and complains about :
./CARVE_KML_Extractor.py: line 20: import: command not found
./CARVE_KML_Extractor.py: line 21: import: command not found
./CARVE_KML_Extractor.py: line 22: import: command not found
from: can't read /var/mail/xml.dom.minidom
However if I simply run like: python script.py [inputProduct][outputdir],
it works fine. Do I need to be able to run that python script like
./script.py to be able to use it as an metadata extractor.
Cameron, since this is your code. Would you help me and see what you can
do to make it run like: ./script.py.
2. I noticed that in the link
http://oodt.apache.org/components/maven/curator/user/basic.html, the
python code you just specify the input file:
./mp3PythonExtractor.py
/usr/local/staging/products/mp3/Bach-SuiteNo2.mp3.
Since I need to integrate this to my cas-pge crawler, and it needs to
perform this metadata extraction on the product just generated, what
exactly I need to pass as argument in terms of [inputFile][outmetpath]?
Thanks,
Faranak
On 4/15/11 7:51 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.gov wrote:
Based on the document you have sent me, for the simple python script that
runs like this: kml.python [input][outputmet]. I need to have the config
file like:
?xml version=1.0 encoding=UTF-8?
cas:externextractor xmlns:cas=http://oodt.jpl.nasa.gov/1.0/cas;
exec workingDir=
extractorBinPath/usr/local/extractors/mp3extractor/mp3PythonExtractor.py
/extractorBinPath
args
arg isDataFile=true/
/args
/exec
/cas:externextractor
Do I have to specify the working directory like the first line: exec
workingDir=. Because I don't see that line in the peate's sample file.
Also I see the python extractor has these lines that mines doesn't. Is
this the reason my python script doesn't get run? The python file I have
simply parses the file and generates the output met.
cmd = java -jar /Users/woollard/Desktop/extractors/mp3extractor/
cmd += tika-app-0.5-SNAPSHOT.jar -m +fullPath+ | awk -F:
cmd += 'BEGIN {print \cas:metadata xmlns:cas=
cmd += \\\http://oodt.jpl.nasa.gov/1.0/cas\\\;\}
cmd += {print \keyvalkey\$1\/keyval\substr($2,2)\
cmd += /val/keyval\}' +fileName+.met
#!/usr/bin/python
import os
import sys
fullPath = sys.argv[1]
pathElements = fullPath.split(/);
fileName = pathElements[len(pathElements)-1]
fileLocation = fullPath[:(len(fullPath)-len(fileName))]
productType = MP3
cmd = java -jar /Users/woollard/Desktop/extractors/mp3extractor/
cmd += tika-app-0.5-SNAPSHOT.jar -m +fullPath+ | awk -F:
cmd += 'BEGIN {print \cas:metadata xmlns:cas=
cmd += \\\http://oodt.jpl.nasa.gov/1.0/cas\\\;\}
cmd += {print \keyvalkey\$1\/keyval\substr($2,2)\
cmd += /val/keyval\}' +fileName+.met
os.system(cmd)
f = open(fileName+.met, 'a')
f.write('keyvalkeyProductType/keyval'+productType)
f.write('/val/keyval\nkeyvalkeyFilename/keyval')
f.write(fileName+'/val/keyval\n'keyvalkeyFileLocation')
f.write('/keyval'+fileLocation+'/val/keyval\n')
f.write('/cas:metadata')
f.close()
On 4/15/11 6:22 PM, Davoodi, Faranak (388J)
faranak.davo...@jpl.nasa.gov wrote:
Thanks Brian. The document was actually very helpful.
--Faranak
From: holenoter holeno...@me.commailto:holeno...@me.com
Reply-To: dev@oodt.apache.orgmailto:dev@oodt.apache.org
dev
[DISCUSS] - PGE Documentation
Hey Team, I opened OODT-171 https://issues.apache.org/jira/browse/OODT-171 to track the progress of making documentation for the PGE component. The goal is to get a Basic User Guide out the door in the next 2 weeks. I know that several of you have used PGE with great success, so if you can contribute to the documentation it would really be appreciated. Right now I am just collecting thoughts, ideas and brainstorming on this wiki page: https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Documentation Once we have enough content to warrant a Basic User Guide I will capture the content and feed it back into the trunk and site. If you want to add in Advanced User Guide material, that is great too, since we will need one of those as well, I just wanted to cover the basics during this first iteration. Thanks to everyone and I look forward to your contributions. Best Regards, Cameron -- Sent from a Tin Can attached to a String
Maven3 Update on OS X...and how to get back to Maven2
For everyone on this list who has been building OODT on a Mac, be sure to star,flag,save this email for later. If you haven't encountered the Maven 2 to Maven 3 gremlin yet, it will show up eventually. Here is a great blog post on the errors you might see and a really simple and straight forward way to revert back to Maven2. http://cyntech.wordpress.com/2011/03/09/maven-2-error/ Hope this helps. -Cameron -- Sent from a Tin Can attached to a String
File Manager Policy Wiki is UP!
Everyone, After several discussions surrounding 'How should I define my FileManager Policy?' I decided (with the help of John Tran and Chris Mattmann) to start a wiki page on the topic of policy for File Manager. You can find it here: https://cwiki.apache.org/confluence/display/OODT/Everything+you+want+to+know+about+File+Manger+Policy Please give it a look and leave comments or questions, or even add in your own experiences. File Manager Policy is really whatever you make of it, so the wiki page is meant to collect some 'Best Practice' solutions to help folks setup their policy correctly the first time. Cheers! Cameron
Re: Branched components refer to nonexistent incubator version
Paul, I have only seen SK give it a +1 so far. I wanted to run through the FileManager and Workflow components before I give my +1...it built ok (ignoring the maven stalls getting external jars) I just wanted to check a little deeper on the two components I am familiar with. Look for mine tomorrow morning. -Cameron On Wed, Jan 19, 2011 at 2:24 PM, Ramirez, Paul M (388J) paul.m.rami...@jpl.nasa.gov wrote: Hey All, It was brought to my attention that I forgot to update the pom in some of the branched components when I went around changing 0.2-incubating to 0.2. If I update this should I roll another RC? Anyhow, let me know. Also JPL email is being weird and I haven't seen anyone else vote yet except SK. Thanks, Paul -- Sent from a Tin Can attached to a String
