Re: Needed OODT Improvments

[Cameron Goodale]

Sounds like some excellent maintenance activities.

+1

-Cam

On Wed, Apr 15, 2015 at 3:27 PM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> ACK
>
>
> On Wed, Apr 15, 2015 at 5:39 PM, Michael Starch 
> wrote:
>
> > Dev,
> >
> > I've been talking with Tom and we've noted a list of things that should
> be
> > improved on OODT.
> >
> >   - pom/maven:
> >   - fix dependency circles in radix
> >   - make extra components with large number of dependencies optional
> > (pending pull request)
> >   - move dependency versions to top-level pom making versions
> > consistent
> >   - update dependency versions
> >
> >   - replace xml-rpc development-only server
> >   - separate REST interfaces and client code for ops ui and cas-curator
> >
> >   - cleanup
> >   - deprecate unused or superseded sub-components
> >   - spin-off non-essential functionality into sibling projects
> >   - documentation
> >
> > Any thoughts?
> >
> > -Michael
> >
>
>
>
> --
> *Lewis*
>

Re: New Committer: Dana Freeborn

[Cameron Goodale]

That's excellent news Tom. Thanks for keeping the process moving

Cameron

On Friday, March 27, 2015, Tom Barber  wrote:

> The Project Management Committee (PMC) for Apache OODT has asked Dana
> Freeborn to become a committer and we are pleased to announce that they
> have accepted.
>
>
> -- Tom
>
>

-- 

Sent from a Tin Can attached to a String

Re: i am lost

[Cameron Goodale]

Welcome Eva,

I second Lewis and please know there are all different levels of expert on
this mailing list which is great since OODT can do a lot.

One idea that I've heard before (I cannot take credit ) is that OODT is
more like a data management framework and less like an application.

Starting out typically feels like someone just dumped a bunch of Legos on
the floor and you don't really know where to start.  That is normal, I
still feel like that at times.

As Lewis mentioned if you give us a use case or a problem you're working on
we'll be more than happy to help.

Cheers,


Cameron

On Thursday, March 19, 2015, Lewis John Mcgibbney 
wrote:

> Hi Eva,
> Your at the right place :)
> I would guide you towards the wiki documentation if I knew what you were
> trying to do.
> Anything in particular you are struggling with?
> Thanks
> Lewis
>
> On Thursday, March 19, 2015, Eva Schlauch  >
> wrote:
>
> > Hello there,
> >
> > so I am new to OODT and the mailing list. The nice people at our
> computing
> > center have installed an OODT for me but I don't seem to get a grip on
> it.
> > How can I find threads that may already answer all my questions? Where to
> > start?
> >
> > Where can I send my newby this-is-all-too-much-please-help-me requests?
> >
> > Best regards, Eva
> >
>
>
> --
> *Lewis*
>

Re: OODT Tests

[Cameron Goodale]

Hey Folks,

I realize I am late to the discussion but in the past when one person has a
working build and another person is having issues the problem/solution
(depending on how you look at it) is in the local maven repo.

On my local machine Mac OS X 10.9.5 I blew out my maven repo and did an
install from a fresh checkout of trunk and I am getting the same error that
Lewis has.  I believe that Jenkins does it's build/test against an empty
local maven repo.

This is kind of like asking if there is petrol in the tank when a car
breaks down, but I figure it couldn't hurt to mention it.

Good Luck,


Cameron

On Fri, Jan 23, 2015 at 9:37 AM, Tom Barber  wrote:

> Its a null pointer disguised as a broken url. I'll take a look later.
>
> Tom
> On 23 Jan 2015 17:20, "Ramirez, Paul M (398M)" <
> paul.m.rami...@jpl.nasa.gov>
> wrote:
>
> > Weird one. Is there something wrong with your network connection? Is that
> > URL being blocked? The stack trace says URL does not exist but I'm able
> to
> > access it here on my phone.
> >
> > --Paul
> >
> > Sent from my iPhone
> >
> > > On Jan 23, 2015, at 9:10 AM, Lewis John Mcgibbney <
> > lewis.mcgibb...@gmail.com> wrote:
> > >
> > > Hi Tom,
> > > Thanks for heads up.
> > > I am kinda confused as to local failure, I'm reproducing failures on
> > Jenkins
> > >
> > > Regression on
> > org.apache.oodt.cas.protocol.http.TestHttpProtocol.testLSandCD
> > > Error Message
> > >
> > > Failed to cd to {parent = 'null', path =
> > >
> >
> 'repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http',
> > > isDir = 'true'} : URL does not exist
> > > '
> >
> http://svn.apache.org/repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http
> > '
> > >
> > > Stacktrace
> > >
> > > org.apache.oodt.cas.protocol.exceptions.ProtocolException: Failed to
> > > cd to {parent = 'null', path =
> > >
> >
> 'repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http',
> > > isDir = 'true'} : URL does not exist
> > > '
> >
> http://svn.apache.org/repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http
> > '
> > >at org.apache.oodt.cas.protocol.http.HttpProtocol.cd
> > (HttpProtocol.java:78)
> > >at
> >
> org.apache.oodt.cas.protocol.http.TestHttpProtocol.testLSandCD(TestHttpProtocol.java:52)
> > > Caused by: java.io.IOException: URL does not exist
> > > '
> >
> http://svn.apache.org/repos/asf/oodt/trunk/protocol/http/src/main/java/org/apache/oodt/cas/protocol/http
> > '
> > >at
> >
> org.apache.oodt.cas.protocol.http.util.HttpUtils.isDirectory(HttpUtils.java:126)
> > >at org.apache.oodt.cas.protocol.http.HttpProtocol.cd
> > (HttpProtocol.java:72)
> > >... 23 more
> > > Caused by: java.lang.NullPointerException
> > >at
> >
> org.apache.oodt.cas.metadata.util.MimeTypeUtils.autoResolveContentType(MimeTypeUtils.java:223)
> > >at
> >
> org.apache.oodt.cas.metadata.util.MimeTypeUtils.autoResolveContentType(MimeTypeUtils.java:132)
> > >at
> >
> org.apache.oodt.cas.protocol.http.util.HttpUtils.isDirectory(HttpUtils.java:122)
> > >... 24 more
> > >
> > >
> > >
> > >> On Fri, Jan 23, 2015 at 8:45 AM, Tom Barber 
> > wrote:
> > >>
> > >> The only real problem with the build is that the Jersey Client 1.x POM
> > has
> > >> been knackered for god knows how long, but as long as I can remember
> > >> working with REST stuff, and the Hadoop Client jar in the Resource
> > Manager
> > >> has it hardcoded as a dependency. So the other night I forced it to
> use
> > a
> > >> newer version, so it failed a few times and I stepped through the
> > required
> > >> upgraded jars.
> > >>
> > >> Apart from that and a few minor tweaks I think its alright.
> > >>
> > >> Tom
> > >>
> > >>
> >  On 23/01/15 16:41, Lewis John Mcgibbney wrote:
> > >>>
> > >>> Hi Folks,
> > >>> Builds have been dodgy for a while now.
> > >>> Anyone have a clue what happened?
> > >>> I just looked at out Jenkins build record for trunk and quite frankly
> > it
> > >>> kinda appalling.
> > >>> I'm building RADiX for a customer right now then I'm going to take
> some
> > >>> time looking at tests again.
> > >>> Lewis
> > >>
> > >> --
> > >> *Tom Barber* | Technical Director
> > >>
> > >> meteorite bi
> > >> *T:* +44 20 8133 3730
> > >> *W:* www.meteorite.bi | *Skype:* meteorite.consulting
> > >> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2
> > 7YG, UK
> > >
> > >
> > >
> > > --
> > > *Lewis*
> >
>

Re: Shutdown of the OODT Apache User Mailing List

[Cameron Goodale]

+1.  Thanks for bringing this up Sean.

-Cam

On Mon, Dec 29, 2014 at 10:47 AM, BW  wrote:

> +1
>
>
> On Monday, December 29, 2014, Tyler Palsulich 
> wrote:
>
>> +1, shut it down.
>>
>> Tyler
>>
>> On Mon, Dec 29, 2014 at 10:43 AM,  wrote:
>>
>>> Folks:
>>>
>>> We have two public mailing lists:
>>>
>>> - dev@oodt.apache.org, ostensibly for discussion of the development of
>>> Apache OODT
>>> - u...@oodt.apache.org, allegedly for end users to ask questions and
>>> find answers about Apache OODT
>>>
>>> Greg Stein noted during the last Apache board meeting that since the
>>> user@ traffic is so low that we're needlessly segmenting our
>>> community.  OODT itself is software that's more for developers than end
>>> users since it requires such high levels of integration into existing
>>> architectures.
>>>
>>> Following the K.I.S.S. Principle [2], I'd like to call for a community
>>> vote to shut down the u...@oodt.apache.org mailing list:
>>>
>>> [ ] +1, shut down u...@oodt.apache.org
>>> [ ] ±0, don't care about u...@oodt.apache.org's fate
>>> [ ] -1, do NOT shut down u...@oodt.apache.org, because …
>>>
>>> I'll leave the vote open at least 72 hours. And here's my +1.
>>>
>>> Best wishes
>>> --k
>>>
>>> [1]
>>> https://svn.apache.org/repos/private/foundation/board/board_agenda_2014_12_17.txt
>>> [2] http://en.wikipedia.org/wiki/KISS_principle
>>>
>>
>>

Re: Why is pushpull not always working?

[Cameron Goodale]

Hey Lewis,

Last time I dug into this I was running toward the "Active vs. Passive" FTP
settings, but I am not sure that is the root cause.  One option or fix
would be to add in a timeout parameter, where if the pushpull daemon
doesn't get a reply in a set amount of time, it should stop and restart.
Every time I have seen pushpull stall or hang, a simple stop and restart
gets it working again.

I know Ross Laidlaw is using pushpull on a his Near Real Time snow
processing pipeline, so I am sure he will be very interested in your work
Lewis.

Thanks,


Cameron

On Sat, Nov 1, 2014 at 10:58 AM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> Thanks Chris. OK doke. i am going to debug this today and find out what is
> happening.
> Thanks for heads up.
> Lewis
>
> On Sat, Nov 1, 2014 at 10:31 AM, Chris Mattmann 
> wrote:
>
> > Thanks Lewis - in my mind it boils down somewhat to
> > the protocols that are implemented and the way they
> > behave with FTP servers and so forth. Ultimately, we
> > should be using the ones that are in PushPull plugins,
> > but they are virally licensed and I haven¹t found a
> > better alternative.
> >
> > Another @bfoster update here would be great, Brian?
> >
> > Cheers,
> > Chris
> >
> >
> > 
> > Chris Mattmann
> > chris.mattm...@gmail.com
> >
> >
> >
> >
> > -Original Message-
> > From: Lewis John Mcgibbney 
> > Reply-To: 
> > Date: Saturday, November 1, 2014 at 12:56 AM
> > To: "dev@oodt.apache.org" 
> > Subject: Why is pushpull not always working?
> >
> > >Hi Folks,
> > >I really couldn't think of a better title for this thread so I apologize
> > >in
> > >advance.
> > >I recently set up pushpull with FTP to crawl remote ftp site [0] which
> is
> > >publicly available.
> > >I am getting the following
> > >
> > >INFO: Launching Daemons . . .
> > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> > >registerRMIServer
> > >INFO: Created Daemon ID = 90121 on RMI registry port 9012
> > >Nov 01, 2014 12:46:01 AM
> > >org.apache.oodt.cas.pushpull.daemon.DaemonLauncher
> > >launchDaemons
> > >INFO: Creating Daemon with ID = 1
> > >Nov 01, 2014 12:46:01 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:46:01 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:48:00 PDT 2014
> > >Nov 01, 2014 12:48:00 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:48:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:51:00 PDT 2014
> > >Nov 01, 2014 12:50:59 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:51:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:54:00 PDT 2014
> > >Nov 01, 2014 12:53:59 AM
> org.apache.oodt.cas.pushpull.daemon.DaemonManager
> > >setAsRunning
> > >INFO: Daemon with ID = 90121 was given permission to run
> > >Nov 01, 2014 12:54:00 AM org.apache.oodt.cas.pushpull.daemon.Daemon
> sleep
> > >INFO: Daemon with ID = 90121 on RMI registry port 9012 is going to sleep
> > >until Sat Nov 01 00:57:00 PDT 2014
> > >
> > >No errors... no pulling... no nothing!
> > >I also came across this FAQ (thanks Cam/Chris) [1] leading to Chris'
> > >plugins page [2].
> > >My question is simple... why is pushpull known NOT to work on occasion?
> I
> > >am going to debug this tomorrow but I thought I would ask this
> > >none-the-less.
> > >Thanks
> > >Lewis
> > >
> > >[0] ftp://n5eil01u.ecs.nsidc.org/SAN/AQUARIUS/AQ3_SSS.004/
> > >[1]
> > >
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
> >
> >#OODTPush-PullUserGuide-Nodatafileisdownloadedtomystagingdirectoryafterrun
> > >ningthe./pushpullscript.WhatshouldIdo
> > >?
> > >[2]
> > >https://cwiki.apache.org/confluence/display/OODT/OODT+Push+Pull+Plugins
> > >
> > >
> > >--
> > >*Lewis*
> >
> >
> >
>
>
> --
> *Lewis*
>



-- 

Sent from a Tin Can attached to a String

Re: what is batch stub? Is it necessary?

[Cameron Goodale]

Valerie,

This could be nothing, or it could be the root cause...your output XML tags
are malformed.

 
   
   

Should be:
 
   
   

No trailing slash in the opening tag.  It might be failing since it cannot
parse the XML cleanly.  Doesn't explain the batchstub stuff, but might be
related to the latest challenge you sighted:

*"The problem is that the ExternScriptTaskInstance is unable to recognize
the command line arguments that I want to pass to the crawler_launcher
script."*

In my experience the XML is super finicky.

Good Luck, and keep the questions coming.  We are here to help.

-Cameron


On Wed, Oct 8, 2014 at 7:37 PM, Ramirez, Paul M (398J) <
paul.m.rami...@jpl.nasa.gov> wrote:

> +1 billion
>
> --Paul
>
> Sent from my iPhone
>
> > On Oct 8, 2014, at 5:55 PM, Lewis John Mcgibbney <
> lewis.mcgibb...@gmail.com> wrote:
> >
> > Folks,
> > Is it possible to create a parent issue for defining XSD's for all of the
> > XML file we need ti OODT?
> > I do not know them all, but from this thread alone, it is clear that we
> > could do with setting some kind of restrictions on what can be included
> > within task and configuration XML within OODT.
> > Thoughts?
> > Lewis
> >
> > On Wed, Oct 8, 2014 at 5:44 PM, Verma, Rishi (398J) <
> > rishi.ve...@jpl.nasa.gov> wrote:
> >
> >> Hi Val,
> >>
> >> Yep - here’s a link to the tasks.xml file:
> >>
> >>
> https://github.com/riverma/xdata-jpl-netscan/blob/master/oodt-netscan/workflow/src/main/resources/policy/tasks.xml
> >>
> >>> The problem is that the ExternScriptTaskInstance is unable to recognize
> >> the command line arguments that I want to pass to the crawler_launcher
> >> script.
> >>
> >>
> >> Hmm.. could you share your workflow manager log, or better yet, the
> >> batch_stub output? Curious to see what error is thrown.
> >>
> >> Is a script file being generated for your PGE? For example, inside your
> >> [PGE_HOME] directory, and within the particular job directory created
> for
> >> your execution of a workflow, you will see some files starting with
> >> “sciPgeExeScript_…”. You’ll find one for your pgeConfig, and you can
> check
> >> to see what the PGE commands actually translate into, with respect to a
> >> shell script format. If that file is there, take a look at it, and
> validate
> >> whether the command works within the script (i.e. copy/paste and run the
> >> crawler command manually).
> >>
> >> Another suggestion is to take a step back, and build up slowly, i.e.:
> >> 1. Do an “echo” command within your PGE first. (e.g.  echo “Hello
> >> APL.” > /tmp/test.txt)
> >> 2. If above works, do a crawler_launcher empty command(e.g.
> >> /path/to/oodt/crawler/bin/crawler_launcher) and verify the
> >> batch_stub or Workflow Manager prints some kind of output when you run
> the
> >> workflow.
> >> 3. Build up your crawler_launcher command piece by piece to see where it
> >> is failing
> >>
> >> Thanks,
> >> Rishi
> >>
> >> On Oct 8, 2014, at 4:24 PM, Mallder, Valerie <
> valerie.mall...@jhuapl.edu>
> >> wrote:
> >>
> >>> Hi Rishi,
> >>>
> >>> Thank you very much for pointing me to your working example. This is
> >> very helpful.  My pgeConfig looks very similar to yours.  So, I
> commented
> >> out the resource manager like you suggested and tried running again
> without
> >> the resource manager. And my problem still exists. The problem is that
> the
> >> ExternScriptTaskInstance is unable to recognize the command line
> arguments
> >> that I want to pass to the crawler_launcher script. Could you send me a
> >> link to your tasks.xml file? I'm curious as to how you defined your
> task.
> >> My pgeConfig and tasks.xml are below.
> >>>
> >>> Thanks!
> >>> Val
> >>>
> >>>
> >>> 
> >>> 
> >>>
> >>>  
> >>>  
> >>>   [CRAWLER_HOME]/bin/crawler_launcher --operation
> >> --launchAutoCrawler \
> >>>   --filemgrUrl [FILEMGR_URL] \
> >>>   --clientTransferer
> >> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory \
> >>>   --productPath [JobInputDir] \
> >>>   --mimeExtractorRepo
> >> [OODT_HOME]/extensions/policy/mime-extractor-map.xml \
> >>>   --actionIds MoveFileToLevel0Dir
> >>>  
> >>>
> >>>  
> >>>  
> >>>  
> >>>
> >>> 
> >>>  
> >>> 
> >>> 
> >>> 
> >>> 
> >>>  
> >>>
> >>> 
> >>>
> >>>
> >>>
> >>> 
> >>>
> >>> http://oodt.jpl.nasa.gov/1.0/cas";>
> >>>
> >>>   >> class="org.apache.oodt.cas.workflow.examples.ExternScriptTaskInstance">
> >>>   
> >>> 
> >>>
> >>> 
> >>>  >> value="[CRAWLER_HOME]/bin/crawler_launcher" envReplace="true" />
> >>>
> >>> 
> >>>  >> value="[OODT_HOME]/extensions/config/crawler-pge-config.xml"
> >> envReplace="true" />
> >>> 
> >>>  
> >>>
> >>> 
> >>>
> >>> Valerie A. Mallder
> >>> New Horizons Deputy Mission System Engineer
> >>> Johns Hopkins University/Applied Physics Laboratory
> >>>
> >>>
>  -Original Message-
>  From: Verma, Rishi (398J) [mailto:rishi.ve...@jpl.nasa.gov]
>  Sent: Wednesday, October 08, 20

Re: OODT Officially Stable - Hats off to Ross Laidlaw

[Cameron Goodale]

Great work Ross.  I know it hasn't been easy getting those tests fixed up,
so thank you for being persistent and seeing it through.

-Cam


On Fri, Aug 8, 2014 at 2:05 PM, Ross Laidlaw 
wrote:

> Thanks all!  Glad to contribute!  There are some remaining issues to
> discuss/resolve with file manager tests (e.g. see OODT-736 and OODT-741),
> but we're getting there!
>
> There are also some issues with the protocol tests that Chris highlighted
> in OODT-644.  So Jenkins builds will probably still fail sometimes, but
> hopefully we can fix these soon.
>
>
> Ross
>
>
>
> On 8 August 2014 13:47, Verma, Rishi (398J) 
> wrote:
>
>> Way to go Ross, way to go everyone!
>>
>> Rishi
>> Sent from my iPhone
>>
>> On Aug 8, 2014, at 1:29 PM, "Chris Mattmann" 
>> wrote:
>>
>> > Fscking awesome!!
>> >
>> > Thank you Ross and everyone!
>> >
>> > 
>> > Chris Mattmann
>> > chris.mattm...@gmail.com
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Tom Barber 
>> > Reply-To: 
>> > Date: Friday, August 8, 2014 1:07 PM
>> > To: 
>> > Subject: Re: OODT Officially Stable - Hats off to Ross Laidlaw
>> >
>> >> Wp
>> >> On 8 Aug 2014 21:05, "Lewis John Mcgibbney" > >
>> >> wrote:
>> >>
>> >> Folks,
>> >>
>> >> The OODT trunk (0.7-SNAPSHOT) codebase is stable [0] now for the first
>> >> time in a while which is extremely good news.
>> >>
>> >> I personally (and I'm sure I speak on behalf of many of us) am
>> extremely
>> >> appreciative of the work Ross has recently put in to ensuring the
>> >> codebase is stabalized.
>> >>
>> >> Hats off Ross. The drinks are on me... and make sure to remind me about
>> >> it.
>> >>
>> >> Have a great weekend folks.
>> >>
>> >> Lewis
>> >>
>> >> [0] https://builds.apache.org/job/oodt-trunk/1438/
>> >>
>> >>
>> >> --
>> >> Lewis
>> >
>> >
>>
>
>

Re: [oodt-dev] pushpull with passive FTP

[Cameron Goodale]

Tim and Chris,

In the past I have run into a similar issue and I ended up using something
else (wget or python) to fetch the data.  That being said I also lost the
ability to check with FileManager to see if I had already downloaded and
archived a file.

All this to say I think it would be a good idea to setup an explicit
option/setting which would be active by default, but could be set to
passive for cases like Tim's.  This option becomes increasingly important
as more and more of us are deploying code within VMs.

I think this at least warrants a ticket in JIRA.


-Cameron


On Wed, Jul 23, 2014 at 5:15 PM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Moving this conversation to Apache OODT :)
>
>
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
>
> -Original Message-
> From: , "Timothy M   (398F)" 
> Reply-To: OODT Developers 
> Date: Wednesday, July 23, 2014 3:52 PM
> To: oodt-dev 
> Subject: [oodt-dev] pushpull with passive FTP
>
> >
> >
> >
> >Hello All,
> >
> >
> >I'm trying to use pushpull with Radix in a virtual machine.  The FTP site
> >that I'm talking to gives me a 500 error because, I think, I'm trying to
> >use active FTP from within a NAT connected VM.  So, how can I convince
> >pushpull to use passive FTP?
> >
> >
> >Thanks,
> >Tim
> >
> >-
> >Tim Stough
> >NASA/Caltech Jet Propulsion Lab
> >Senior System ArchitectAssociate
> > Program Manager,
> >
> > NASA Applied Sciences Program,
> >
> > Disaster Management for Solid Earth Hazards
> >Data Understanding Group (Section 398)
> >626-644-6574 (cell)
> >818-393-5347 (office)
> >-
> >
> >
> >
> >
>
>

Re: [jira] [Commented] (OODT-694) Upgrade xmlrpc library in OODT 2.0.1 --> 3.0a1

[Cameron Goodale]

Lewis,

I poked around the project a bit and all the download links and the one to
subversion all 404'd.  We should probably email their list and see if the
code is even around anymore.

I was hoping that the lack of development just meant it is stable and the
XMLRPC spec hasn't changed much.

-Cam


On Wed, May 14, 2014 at 2:05 PM, Lewis John McGibbney (JIRA) <
j...@apache.org> wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998018#comment-13998018]
>
> Lewis John McGibbney commented on OODT-694:
> ---
>
> http://ws.apache.org/xmlrpc/apidocs/index.html
>
> > Upgrade xmlrpc library in OODT 2.0.1 --> 3.0a1
> > --
> >
> > Key: OODT-694
> > URL: https://issues.apache.org/jira/browse/OODT-694
> > Project: OODT
> >  Issue Type: Improvement
> >  Components: build proces, file manager
> >Affects Versions: 0.6
> >Reporter: Lewis John McGibbney
> > Fix For: 0.8
> >
> >
> > A _newer_ version of this library is available [0]... however there may
> be a more actively maintained xmlrpc lib we could use as this one looks to
> be pretty stagnant.
> > I am digging in to XmlRpcClientExceptions being thrown within filemgr
> tests and it was lovely to realize that an upgrade of the xmlkrpc lib is
> also required.
> > [0] http://search.maven.org/#artifactdetails|xmlrpc|xmlrpc|3.0a1|N%2FA
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.2#6252)
>



-- 

Sent from a Tin Can attached to a String

Re: java.lang.NoClassDefFoundError: com/google/common/collect/Sets - Crawler Action ERROR

[Cameron Goodale]

Hey Lewis,

Thanks for the advice.  You are correct that this class is in the guava
lib.  I rebuilt OODT core and then re-built the crawler and I now have a
fresh guava-10.0.1.jar in my crawler lib and it is working.

Big thanks to you and Chris for helping me out.


-Cam


On Fri, Apr 25, 2014 at 6:06 PM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> Hi Cameron I would suggest manually looking into the crawler pom.xml to see
> if the google dependency (possibly guava) has been explicitly excluded.
> Additionally you can try mvn dependency:tree to see if it may be being
> excluded at another level.
> What I suggest, if the above does not work, is to include the missing dep
> in your crawler pom.xml, also log a jira issue and we can include it as
> part of trunk.
> Hth
> Lewis
> On Apr 25, 2014 4:47 PM, "Cameron Goodale"  wrote:
>
> > Hey Devs and Brian,
> >
> > I am trying to run a crawler action after a PGE has been run on a batch
> > stub and I am getting a java.lang.NoClassDefFoundError:
> > com/google/common/collect/Sets error, so I thought I'd ask the resident
> > Googler for some help.
> >
> > I am not sure if this class needs to be included in the cas-crawler.jar
> > file or if maven is supposed to fetch it from somewhere else.
> >
> > Any thoughts or ideas would be appreciated.  Longer error message is
> below.
> >
> > I am running:
> >
> > Workflow-0.7-SNAPSHOT
> > Crawler-0.7-SNAPSHOT
> > Resource-0.6  <--- this is by choice
> > cli-0.7-SNAPSHOT
> > pge-0.3-SNAPSHOT   <-- This is by choice
> >
> >
> > Thanks,
> >
> > Cameron
> >
> > INFO: Updating status to workflow as CRAWLING
> > Apr 25, 2014 4:11:25 PM sun.net.www.protocol.http.HttpURLConnection
> > writeRequests
> > FINE: sun.net.www.MessageHeader@154e4e319 pairs: {POST / HTTP/1.1:
> > null}{Content-Length: 232}{Content-Type: text/xml}{Cache-Control:
> > no-cache}{Pragma: no-cache}{User-Agent: Java/1.6.0_21}{Host:
> > localhost:9201}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*;
> > q=.2}{Connection: keep-alive}
> > Apr 25, 2014 4:11:25 PM sun.net.www.protocol.http.HttpURLConnection
> > getInputStream
> > FINE: sun.net.www.MessageHeader@47ef7de45 pairs: {null: HTTP/1.1 200
> > OK}{Server: Apache XML-RPC 1.0}{Connection: close}{Content-Type:
> > text/xml}{Content-Length: 121}
> > Apr 25, 2014 4:11:25 PM org.apache.oodt.cas.pge.PGETaskInstance
> > runIngestCrawler
> > INFO: Executing StdProductCrawler in productPath:
> > [/snow-stor/output/MODSCAGv2NRT/1398467390502/h07v03]
> > Apr 25, 2014 4:11:25 PM
> > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry
> > getSingleton
> > FINE: Creating shared instance of singleton bean 'DeleteDataFile'
> > Apr 25, 2014 4:11:25 PM
> >
> >
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
> > createBean
> > FINE: Creating instance of bean 'DeleteDataFile' with merged definition
> > [Root bean: class [org.apache.oodt.cas.crawl.action.DeleteFile];
> > scope=singleton; abstract=false; lazyInit=true; autowireCandidate=false;
> > autowireMode=0; dependencyCheck=0; factoryBeanName=null;
> > factoryMethodName=null; initMethodName=null; destroyMethodName=null;
> > defined in URL [file:/usr/local/snow/crawler/policy/action-beans.xml]]
> > Apr 25, 2014 4:11:25 PM
> >
> >
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
> > createBean
> > FINE: Eagerly caching bean 'DeleteDataFile' to allow for resolving
> > potential circular references
> > Exception in thread "Thread-14" java.lang.NoClassDefFoundError:
> > com/google/common/collect/Sets
> > at
> >
> >
> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.getActions(CrawlerActionRepo.java:49)
> > at
> >
> >
> org.apache.oodt.cas.crawl.ProductCrawler.validateActions(ProductCrawler.java:236)
> > at
> >
> >
> org.apache.oodt.cas.crawl.ProductCrawler.loadAndValidateActions(ProductCrawler.java:230)
> > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:83)
> > at
> >
> >
> org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
> > at
> >
> >
> org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
> > at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
> > at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
> > at
> >
> >
> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
> > at java.lang.Thread.run(Thread.java:619)
> >
> > --
> >
> > Sent from a Tin Can attached to a String
> >
>



-- 

Sent from a Tin Can attached to a String

java.lang.NoClassDefFoundError: com/google/common/collect/Sets - Crawler Action ERROR

[Cameron Goodale]

Hey Devs and Brian,

I am trying to run a crawler action after a PGE has been run on a batch
stub and I am getting a java.lang.NoClassDefFoundError:
com/google/common/collect/Sets error, so I thought I'd ask the resident
Googler for some help.

I am not sure if this class needs to be included in the cas-crawler.jar
file or if maven is supposed to fetch it from somewhere else.

Any thoughts or ideas would be appreciated.  Longer error message is below.

I am running:

Workflow-0.7-SNAPSHOT
Crawler-0.7-SNAPSHOT
Resource-0.6  <--- this is by choice
cli-0.7-SNAPSHOT
pge-0.3-SNAPSHOT   <-- This is by choice


Thanks,

Cameron

INFO: Updating status to workflow as CRAWLING
Apr 25, 2014 4:11:25 PM sun.net.www.protocol.http.HttpURLConnection
writeRequests
FINE: sun.net.www.MessageHeader@154e4e319 pairs: {POST / HTTP/1.1:
null}{Content-Length: 232}{Content-Type: text/xml}{Cache-Control:
no-cache}{Pragma: no-cache}{User-Agent: Java/1.6.0_21}{Host:
localhost:9201}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*;
q=.2}{Connection: keep-alive}
Apr 25, 2014 4:11:25 PM sun.net.www.protocol.http.HttpURLConnection
getInputStream
FINE: sun.net.www.MessageHeader@47ef7de45 pairs: {null: HTTP/1.1 200
OK}{Server: Apache XML-RPC 1.0}{Connection: close}{Content-Type:
text/xml}{Content-Length: 121}
Apr 25, 2014 4:11:25 PM org.apache.oodt.cas.pge.PGETaskInstance
runIngestCrawler
INFO: Executing StdProductCrawler in productPath:
[/snow-stor/output/MODSCAGv2NRT/1398467390502/h07v03]
Apr 25, 2014 4:11:25 PM
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry
getSingleton
FINE: Creating shared instance of singleton bean 'DeleteDataFile'
Apr 25, 2014 4:11:25 PM
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
createBean
FINE: Creating instance of bean 'DeleteDataFile' with merged definition
[Root bean: class [org.apache.oodt.cas.crawl.action.DeleteFile];
scope=singleton; abstract=false; lazyInit=true; autowireCandidate=false;
autowireMode=0; dependencyCheck=0; factoryBeanName=null;
factoryMethodName=null; initMethodName=null; destroyMethodName=null;
defined in URL [file:/usr/local/snow/crawler/policy/action-beans.xml]]
Apr 25, 2014 4:11:25 PM
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory
createBean
FINE: Eagerly caching bean 'DeleteDataFile' to allow for resolving
potential circular references
Exception in thread "Thread-14" java.lang.NoClassDefFoundError:
com/google/common/collect/Sets
at
org.apache.oodt.cas.crawl.action.CrawlerActionRepo.getActions(CrawlerActionRepo.java:49)
at
org.apache.oodt.cas.crawl.ProductCrawler.validateActions(ProductCrawler.java:236)
at
org.apache.oodt.cas.crawl.ProductCrawler.loadAndValidateActions(ProductCrawler.java:230)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:83)
at
org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
at
org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
at
org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:619)

-- 

Sent from a Tin Can attached to a String

Re: Query Tool Deprecation

[Cameron Goodale]

Tom,

TLDR:  Run ./filemgr-client and review the long list of options.  -sql and
-lucene are in there to replace the query tool.  Below is a detailed list
of how I checked this.


The longer version:
==

I *think* that the OODT 'cli' module was going to unify all of the Command
Line Interfaces to the OODT components.

The README for cli is here:

http://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt

The latest filemgr-client script (0.7-SNAPSHOT) depends on the cli
component, and requires a cmd-line-options.xml and cmd-line-actions.xml.

I just updated my local checkout to the latest version and ran:

$ cd oodt/trunk
$ mvn clean install
$ cd filemgr
$ mvn clean install
$ cd target
$ tar -zxvf cas-filemgr-0.7-SNAPSHOT-dist.tar.gz  # This will just untar
the dist in place for testing purposes
$ cd cas-filemgr-0.7-SNAPSHOT/bin/
$ vi filemgr-client  # Need to set $JAVA_HOME, etc...
$ ./filemgr-client

-
| Short | Long | Description
-

 -op,--operationDeclare that
you wish to present an operation
   SubOptions:
   > Required:
   > Optional:
  -addPT, --addProductType  Triggers
addProductType Action
...
...
...
-sql,   --sqlQueryTriggers sqlQuery
Action
  Requirement
Rules:
   [sqlQuery :
REQUIRED]
  -dmet,  --dumpMetadataTriggers
dumpMetadata Action
  Requirement
Rules:

 [dumpMetadata : REQUIRED]
  -lucene,--luceneQuery Triggers
luceneQuery Action
  Requirement
Rules:
   [luceneQuery
: REQUIRED]


So Tom i think those cli options (-sql, or -lucene) are the replacements
for query_tool.


-Cameron





On Mon, Mar 17, 2014 at 1:38 PM, Tom Barber  wrote:

> Hi guys
>
> I want to look at querying the filemgr, but I look at the querytool source
> and its been deprecated. Whats its replacement?
>
> Thanks
>
> Tom
> --
> *Tom Barber* | Technical Director
>
> meteorite bi
> *T:* +44 20 8133 3730
> *W:* www.meteorite.bi | *Skype:* meteorite.consulting
> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2 7YG, UK
>



-- 

Sent from a Tin Can attached to a String

Re: ApacheCon 2014

[Cameron Goodale]

Tom,

OODT as a business application is a novel use case for sure.  Perhaps to
help, if you took some of the architecture diagrams and tried to relabel
the pieces with Business jargon instead of NASA/Science jargon it might
help.

Just this week several of the JPL folks were at Caltech meeting with
researchers from MIT Haystack and Cambridge University, and they had a lot
of questions about acronyms like PGE, and Chris told them to replace it
with algorithm.

Sorry it isn't a workable non-science use case, but it might help fill in
the gap between science and business.

Good luck,


Cameron


On Wed, Jan 29, 2014 at 8:54 PM, Chris Mattmann wrote:

> Chris, smells like another talk. Please submit :) Seriously, you have it
> written below :) Submit! :)
>
> 
> Chris Mattmann
> chris.mattm...@gmail.com
>
>
>
>
> -Original Message-
> From: Christopher Warner 
> Reply-To: 
> Date: Wednesday, January 29, 2014 8:45 AM
> To: 
> Subject: Re: ApacheCon 2014
>
> >One of the more difficult things for me trying to sell OODT was espousing
> >why it's useful for structured processes whether they be government
> >initiated protocols for data handling or workflow handling of that data.
> >In
> >some cases I'm dealing with large data sets that exist in things like
> >excel, word, or some other unstructured crazy format. So part of the
> >problem is social and dealing with researchers outside of the science
> >paradigm whom aren't readily used to processing or extracting information
> >from their data in that way.
> >
> >I think it's a great idea that would give an immediate visually accessible
> >way of how it's useful for data sets not-related to science alone.
> >
> >From New York..
> >
> >
> >On Mon, Jan 27, 2014 at 2:08 PM, Tom Barber 
> >wrote:
> >
> >> So I saw the call go out that they were short of applications and it got
> >> me thinking once again.
> >>
> >> My issue with Apache OODT is the fact currently a load of science bods
> >>use
> >> it (no offence ;) ) and the stuff you see written about and presented
> >> whilst a) is cool as hell b) lacks day to day "real world" applications
> >>for
> >> non scientists because if people do use it for non science applications,
> >> its not publicised very well  which is where I want to try and advocate
> >> OODT
> >>
> >> But with my current clients don't have an obvious use case, but there
> >>must
> >> be a bunch if I sit down and think about them, so I was thinking about
> >> pitching a talk along the lines of
> >>
> >>"Utilising the power of Apache OODT with mainstream Business
> >>Intelligence tools"
> >>
> >>Where I can discuss a few real world scenarios and I was thinking I
> >>could demonstrate OODT integration with Pentaho Data
> >>Integration(Apache Licensed) using ETL to control ingestion and use
> >>Pentaho Data Integration to query and extract data from the Catalog.
> >>
> >>I could also use the Pentaho BI server to create a Dashboard
> >>framework to display relevant OODT information and as an alternative
> >>ingestion platform to the default File manager.
> >>
> >>
> >>
> >> Clearly non of this is yet written and I guess I'd write it if the talk
> >> got accepted and I got my fellow colleagues to help sponsor the trip,
> >>but i
> >> thought the idea was worth putting out there to garner feedback from the
> >> development community.
> >>
> >> Thoughts on a postcard please.
> >>
> >> Thanks
> >>
> >> Tom
> >>
> >>
> >> On 17/01/14 00:12, Chris Mattmann wrote:
> >>
> >>> Hi Tom,
> >>>
> >>> I was thinking of submitting one on the use of Apache OODT
> >>> in the Airborne Snow Observatory (ASO) project at JPL. or on
> >>> the Wicket-based web interfaces. Do you have any ideas of what
> >>> to submit?
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> 
> >>> Chris Mattmann
> >>> chris.mattm...@gmail.com
> >>>
> >>>
> >>>
> >>>
> >>> -Original Message-
> >>> From: Tom Barber 
> >>> Reply-To: 
> >>> Date: Tuesday, January 14, 2014 2:34 PM
> >>> To: "dev@oodt.apache.org" 
> >>> Subject: ApacheCon 2014
> >>>
> >>>  Hi folks,
> 
>  Just a quick query. I was wondering if any OODT developers were
> planning
>  to submit any talk proposals into ApacheCon?
> 
>  Thanks
> 
>  Tom
> 
>  --
>  *Tom Barber* | Technical Director
> 
>  meteorite bi
>  *T:* +44 20 8133 3730
>  *W:* www.meteorite.bi | *Skype:* meteorite.consulting
>  *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2
> 7YG,
>  UK
> 
> >>>
> >>>
> >>
> >> --
> >> *Tom Barber* | Technical Director
> >>
> >> meteorite bi
> >> *T:* +44 20 8133 3730
> >> *W:* www.meteorite.bi | *Skype:* meteorite.consulting
> >> *A:* Surrey Technology Centre, Surrey Research Park, Guildford, GU2
> >>7YG, UK
> >>
> >
> >
> >
> >--
> >Christopher Warner
> >http://cwarner.kernelcode.com
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: borked links

[Cameron Goodale]

Juha,

The URL will get updated soon. In the meantime the proper URL is:

http://www.apache.org/dyn/closer.cgi/oodt/apache-oodt-0.6-src.zip

Thanks for reporting this.

-Cameron


On Wed, Jan 29, 2014 at 3:19 PM, Juha Vierinen  wrote:

> Downloads -> 0.6 Release
>
> http://www.apache.org/dyn/closer.cgi/oodt/apache-oodt-0.6.zip
>
> juha
>



-- 

Sent from a Tin Can attached to a String

Re: Resource Manager 0.7-SNAPSHOT and Ganglia Monitor

[Cameron Goodale]

Thanks Rajith for the prompt reply. I will check into getting that
installed on our machine.

Cameron
On Dec 9, 2013 6:57 PM, "Rajith Siriwardana" 
wrote:

> hi Cameron,
>
> AFAIK there is no way to disable the Ganglia plugin, so you'll have to
> setup and run a Ganglia metadata daemon (Gmetad). I'll check that again.
>
> Cheers,
> Rajith
>
>
> On Tue, Dec 10, 2013 at 2:53 AM, Cameron Goodale 
> wrote:
>
> > Hey Rajith and Devs,
> >
> > I am upgrading my resource manager 0.3 to the latest version (currently
> > 0.7-SNAPSHOT) and I have run into the Ganglia Monitor Plugin with the
> > following errors:
> >
> > SEVERE: The required nodeId is not available: sneffels-11 :null
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73)
> > at
> >
> >
> org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105)
> > at
> >
> >
> org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232)
> > at
> >
> >
> org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134)
> > at
> >
> >
> org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108)
> > at java.lang.Thread.run(Thread.java:662)
> > Dec 9, 2013 1:17:05 PM
> > org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor
> getLoad
> > SEVERE: The required nodeId is not available: kush-11 :null
> > java.lang.NullPointerException
> > at
> >
> >
> org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73)
> > at
> >
> >
> org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105)
> > at
> >
> >
> org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232)
> > at
> >
> >
> org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134)
> > at
> >
> >
> org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108)
> > at java.lang.Thread.run(Thread.java:662)
> >
> >
> > I get this error message for all 44 of the nodes I have defined in my
> > nodes.xml config.
> >
> >
> > A couple questions:
> >
> > 1.  Can I simply run resource manager without the Plugin via a change in
> > config?  I tried to comment out the config lines, but it didn't work.
> >
> > 2.  If I cannot simply disconnect the Ganglia Dependency, then do I have
> to
> > setup and run a Ganglia metadata daemon?
> >
> >
> > I look forward to your reply,
> >
> >
> > Cameron
> >
> > --
> >
> > Sent from a Tin Can attached to a String
> >
>

Resource Manager 0.7-SNAPSHOT and Ganglia Monitor

[Cameron Goodale]

Hey Rajith and Devs,

I am upgrading my resource manager 0.3 to the latest version (currently
0.7-SNAPSHOT) and I have run into the Ganglia Monitor Plugin with the
following errors:

SEVERE: The required nodeId is not available: sneffels-11 :null
java.lang.NullPointerException
at
org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73)
at
org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105)
at
org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232)
at
org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134)
at
org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108)
at java.lang.Thread.run(Thread.java:662)
Dec 9, 2013 1:17:05 PM
org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor getLoad
SEVERE: The required nodeId is not available: kush-11 :null
java.lang.NullPointerException
at
org.apache.oodt.cas.resource.monitor.ganglia.GangliaResourceMonitor.getLoad(GangliaResourceMonitor.java:73)
at
org.apache.oodt.cas.resource.monitor.AssignmentMonitor.getLoad(AssignmentMonitor.java:105)
at
org.apache.oodt.cas.resource.scheduler.LRUScheduler.nodeAvailable(LRUScheduler.java:232)
at
org.apache.oodt.cas.resource.scheduler.LRUScheduler.schedule(LRUScheduler.java:134)
at
org.apache.oodt.cas.resource.scheduler.LRUScheduler.run(LRUScheduler.java:108)
at java.lang.Thread.run(Thread.java:662)


I get this error message for all 44 of the nodes I have defined in my
nodes.xml config.


A couple questions:

1.  Can I simply run resource manager without the Plugin via a change in
config?  I tried to comment out the config lines, but it didn't work.

2.  If I cannot simply disconnect the Ganglia Dependency, then do I have to
setup and run a Ganglia metadata daemon?


I look forward to your reply,


Cameron

-- 

Sent from a Tin Can attached to a String

Re: PushPull DaemonLauncher

[Cameron Goodale]

Hey Brian,

Thanks for looking into this for me.  Here is an excerpt from the logs
after I turned them up to FINE (trying to see anything I might be missing)
and I also added in some of my own log statements to track the Daemon
unregister method.  I added in a 3 star (***) prefix to note the log
messages I added.

One thing to note is after my log messages the process should have shutdown
but it would just hang there, so I think something outside the startDaemon
method is keeping the process alive.  Perhaps the missing notify() method
you mentioned can resolve it.


FINE: Should close connection in response to directive: close
Dec 3, 2013 9:46:13 AM org.apache.commons.httpclient.HttpConnection
releaseConnection
FINE: Releasing connection back to connection manager.
Dec 3, 2013 9:46:13 AM
org.apache.oodt.cas.pushpull.retrievalsystem.FileRetrievalSystem
addToDownloadQueue
WARNING: Skipping file {parent = 'null', path =
'/allData/1/MOD09GA/Recent/MOD09GA.A2013336.h25v05.005.NRT.hdf', isDir =
'false'} because it is already in staging area
Dec 3, 2013 9:46:16 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90111 on RMI registry port 9011 is shutting down
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon
unregister
***INFO: Unregistered the MBean
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon
unregister
***INFO: UnicastRemoteObject undone
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon
unregister
***INFO: After daemonListener
***Dec 3, 2013 9:46:16 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
***INFO: Unregistration Completed

To be complete, here is a Patch of where I added in the Log messages (just
in case I added them incorrectly):

Index: Daemon.java
===
--- Daemon.java (revision 1546755)
+++ Daemon.java (working copy)
@@ -239,6 +239,8 @@
 + Daemon.this.getDaemonID() + " on RMI registry
port "
 + Daemon.this.rmiRegPort + " is shutting down");
 Daemon.this.unregister();
+LOG.log(Level.INFO, "Unregistration Completed");
+
 }
 }).start();
 }
@@ -251,8 +253,11 @@
 this.mbs.unregisterMBean(new ObjectName(
 "org.apache.oodt.cas.pushpull.daemon:type=Daemon"
 + this.getDaemonID()));
+LOG.log(Level.INFO, "Unregistered the MBean");
 UnicastRemoteObject.unexportObject(this, true);
+LOG.log(Level.INFO, "UnicastRemoteObject undone");
 this.daemonListener.wasUnregisteredWith(this);
+LOG.log(Level.INFO, "After daemonListener");
 } catch (Exception e) {
 e.printStackTrace();
 }

-Cameron


On Tue, Dec 3, 2013 at 11:41 AM, Brian Foster  wrote:

> Hey Cameron,
>
> can you attach the logs when you come across another case where the
> pushpull gets stuck?
>
> i was doing a quick pass through the code... it looks like the
> DaemonController stays up all the time... wait() is called on it... i
> thought there was a notify() call when daemon queue became empty, which
> would cause the DaemonController to wake up and terminate, but i don't see
> it... i will see if i can get a fix in for that sometime this week.
>
> -Brian
>
>
> On Nov 30, 2013, at 03:08 PM, Cameron Goodale  wrote:
>
> Hey Brian and Jordan,
>
> I have a similar issue with trying to run pushpull within a cron because
> for some reason (i need to sort out) my pushpull daemon will just hang
> after running and sleeping successfully using
> the org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol. I haven't
> taken the time to find out how many sleep/wake cycles it takes to
> eventually hang, but I have seen it happen several times now.
>
> I looked into the code and it seems like updating the runInfo tag so
> period="" should short circuit the code from re-running, but when I have
> done this in testing I get a message like this:
>
> Nov 30, 2013 10:07:32 AM
> org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
> INFO: Disconnecting protocol
> org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
> Nov 30, 2013 10:07:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
> INFO: Daemon with ID = 90111 on RMI registry port 9011 is shutting down
>
> but the process doesn't actually stop. Looking at Daemon.java I see that:
> Daemon.this.unregister(); is called right after that log message, but it
> doesn't seem to quit the process. That c

Re: PushPull DaemonLauncher

[Cameron Goodale]

Hey Brian and Jordan,

I have a similar issue with trying to run pushpull within a cron because
for some reason (i need to sort out) my pushpull daemon will just hang
after running and sleeping successfully using
the org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol.  I haven't
taken the time to find out how many sleep/wake cycles it takes to
eventually hang, but I have seen it happen several times now.

I looked into the code and it seems like updating the runInfo tag so
period="" should short circuit the code from re-running, but when I have
done this in testing I get a message like this:

Nov 30, 2013 10:07:32 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol
org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocol
Nov 30, 2013 10:07:32 AM org.apache.oodt.cas.pushpull.daemon.Daemon$1 run
INFO: Daemon with ID = 90111 on RMI registry port 9011 is shutting down

but the process doesn't actually stop.  Looking at Daemon.java I see that:
  Daemon.this.unregister();  is called right after that log message, but it
doesn't seem to quit the process.  That command is run within a Java Thread
object, but nothing seems to stop the Thread, so the daemon will never exit.

If you guys have made any progress on this I would love to hear about it.
 I am also not an expert on Java Threads, so I could have missed something
in the code, but it seems like the Thread is started and never terminated.

Thanks for reading,



Cameron


On Thu, Nov 21, 2013 at 1:43 PM, Brian Foster  wrote:

> Hey Jordan,
>
> Try not specifying the  element for your daemons in
> RemoteSpecs.xml file.  It should just run once through for those sites.  I
> believe the pushpull shuts down automatically when all deamons have
> terminated, so after you run one pass it should just terminate.  It's been
> a while since i've worked with push-pull code, if that doesn't work i'll
> have to dig further into the code.
>
> if you looking to do just a on off download a given file, then you might
> consider just using cas-protocol directly.  Take a look at:
> https://svn.apache.org/repos/asf/oodt/trunk/protocol/api/src/main/resources/policy/cmd-line-actions.xml.
>  You can run cas-protocol via this bin script:
> https://svn.apache.org/repos/asf/oodt/trunk/protocol/api/src/main/bin/protocol.
>  Just build and deploy like all other cas products and that script will end
> up in the bin directory.  If you would like to add custom action to
> cas-protocol there is documentation for this at:
> https://svn.apache.org/repos/asf/oodt/trunk/cli/README.txt
>
> -Brian
>
> On Nov 07, 2013, at 05:31 PM, Jordan Padams 
> wrote:
>
> Hello all,
>
> I am trying to develop a pretty simple wrapper for the PushPull where I
> call the DaemonLauncher to run through the daemons in my config once and
> then shut down altogether. We currently don't have a need to have the
> software running at intervals.
>
> On that note, here are a few of the methods I've tried but haven't worked:
>
> 1. DaemonLauncher.main() - The software hangs because of the wait() waiting
> for a notify().
> 2. DaemonLauncher.viewDaemonWaitingList() - I've tried monitoring the
> daemon waiting list, and quit() the DaemonLauncher once it gets to 0, but
> then it misses the last daemon in my config.
>
> Is there a way for me to wait on the DaemonLauncher until all daemons have
> completed, then quit?
>
> I've implemented a quick workaround by creating the following method so I
> can see if daemons are still running:
>
> public boolean hasRunningDaemons() {
>
> return this.dm.hasRunningDaemons();
>
> }
> I don't have a lot of experience with daemon processes so this may just be
> something I'm blatantly missing. Appreciate the help.
>
> Thanks,
> Jordan
>
> --
> Jordan Padams
> Software Engineer
> NASA Jet Propulsion Laboratory
>
>


-- 

Sent from a Tin Can attached to a String

Re: How to process files in a sorted order

[Cameron Goodale]

Konstantinos,

My name is Cameron and I am a committer on the Apache OODT project.  I am
not familiar with the internals of crawler, but I don't believe there is a
way to accomplish your goal of enforcing a sorting algorithm within the
crawler config.  I think you will have to write your own crawler that will
implement your sorting logic.



Sincerely,


Cameron Goodale


On Thu, Nov 7, 2013 at 7:44 PM, Konstantinos Mavrommatis <
kmavromma...@celgene.com> wrote:

> Hi,
> In my environment I am using cas-crawler to process directories of 1000s
> of files. The metadata for these files are extracted automatically using
> the mimetypes definitions and small wrapper scripts.
> In these directories some of the files are derived from other files and
> metadata from the older files need to be transferred to the newer file.
> In order to achieve this I need to have the files processed by the
> cas-crawler starting from the older file to the newer file or in other
> cases in alphabetical order..
> Any ideas how this can be achieved?
>
> The crawler command I currently use is:
> ./crawler_launcher --operation --launchAutoCrawler --productPath $FILEPATH
> --filemgrUrl $FMURL --clientTransferer
> org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory
>  --mimeExtr
> actorRepo ../policy/mime-extractor-map.xml
>
> Thanks in advance for your help
> Konstantinos
>
> *
> THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS
> CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED
> INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL
> OR INDIVIDUALS NAMED ABOVE.
> If the reader is not the intended recipient, or the
> employee or agent responsible to deliver it to the
> intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this
> communication is strictly prohibited. If you have
> received this communication in error, please reply to the
> sender to notify us of the error and delete the original
> message. Thank You.
> *
>
>


-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-622) svn:ignore - request to add Eclipse configuration files to ignore list for several modules

[Cameron Goodale]

Thanks for taking the time to clean up the little things.

+1

-Cameron


On Sun, Jun 9, 2013 at 7:10 AM, Chris A. Mattmann (JIRA) wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13679068#comment-13679068]
>
> Chris A. Mattmann commented on OODT-622:
> 
>
> +1 commit away! :)
>
> > svn:ignore - request to add Eclipse configuration files to ignore list
> for several modules
> >
> --
> >
> > Key: OODT-622
> > URL: https://issues.apache.org/jira/browse/OODT-622
> > Project: OODT
> >  Issue Type: Wish
> >Affects Versions: 0.6
> >Reporter: Ross Laidlaw
> >Assignee: Ross Laidlaw
> >Priority: Trivial
> >  Labels: properties, repository, subversion
> > Fix For: 0.6
> >
> > Attachments: OODT-622.rlaidlaw.2013-06-09.patch.txt
> >
> >
> > For many modules in OODT, the svn:ignore lists already contain the
> Eclipse configuration files and directory (.classpath, .project and
> .settings).  But there are a few modules that don't yet have this setting.
>  I'd like to add the Eclipse config files to the ignore lists for the
> remaining modules.  I've produced a patch to do this and I'm happy to
> commit the changes if there are no objections.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>

Re: JIRA label for new users

[Cameron Goodale]

Mike,

Your list of tags + meaning works for me.

+1

-Cameron


On Tue, May 21, 2013 at 8:14 AM, Michael Joyce  wrote:

> +1 on this Gavin.
>
> Re. actual label names, how about a name followed by an (explanation) of
> the difficulty level:
>
> Don't Know
> Newcomer (Easy) - Everyone can do this level
> Committer (Medium) - This is for regular contributors/committers. Where the
> vast majority of issues would fall.
> Expert (Hard) or Guru or ChrisMattmann or whatever everyone likes for
> "hard" - Used sparingly when an issue requires expert component knowledge
>
> We could default the field to "Don't Know" so every issue always has some
> level?
>
> I like the CouchDB approach but I feel they have too many options.
> Easy
> Easy - Medium
> Medium - Hard
> Really Really Hard (Their Guru level)
>
> So, when something is "Medium," which category does it fall into? I think
> the most important part of this system is identifying the difficulty for
> newcomers. The other levels are just nice to have in my opinion. That being
> said, I think simpler is better here.
>
> What do you guys think?
>
>
> -- Joyce
>
>
> On Sun, May 19, 2013 at 8:30 AM, Rajith Siriwardana <
> rajithsiriward...@gmail.com> wrote:
>
> > +1 for having a label for issues that new users can help to resolve.
> >
> >
> > On Sun, May 19, 2013 at 8:51 PM, Mattmann, Chris A (398J) <
> > chris.a.mattm...@jpl.nasa.gov> wrote:
> >
> > > Thanks Gav. +1 to using a controlled value field to make the labels
> > > meaningful.
> > >
> > > Let's see what others think but I am for sure +1.
> > >
> > > Cheers,
> > > Chris
> > >
> > > ++
> > > Chris Mattmann, Ph.D.
> > > Senior Computer Scientist
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 171-266B, Mailstop: 171-246
> > > Email: chris.a.mattm...@nasa.gov
> > > WWW:  http://sunset.usc.edu/~mattmann/
> > > ++
> > > Adjunct Assistant Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Gavin McDonald 
> > > Organization: 16 degrees complete web solutions
> > > Reply-To: "dev@oodt.apache.org" ,
> > > "ga...@16degrees.com.au" 
> > > Date: Sunday, May 19, 2013 3:37 AM
> > > To: "dev@oodt.apache.org" 
> > > Subject: RE: JIRA label for new users
> > >
> > > >
> > > >
> > > >> -Original Message-
> > > >> From: Mattmann, Chris A (398J) [mailto:
> chris.a.mattm...@jpl.nasa.gov]
> > > >> Sent: Friday, 17 May 2013 1:37 AM
> > > >> To: d...@climate.incubator.apache.org
> > > >> Cc: dev@oodt.apache.org
> > > >> Subject: Re: JIRA label for new users
> > > >>
> > > >> Great idea, please implement on OODT too! (cc to dev@oodt for
> peeps)
> > :)
> > > >
> > > >We can start to use labels now for jira tickets, we just need an
> agreed
> > > >and
> > > >documented set
> > > >of labels. Why? Too many labels and we run the risk of there being too
> > > >many
> > > >and they sort of
> > > >Lose the impact and meaning. So a few well-chosen labels for attention
> > > >grabbing is needed.
> > > >
> > > >Every Jira ticket has a Labels section so this is available now.
> > > >
> > > >However, my main point of this mail, is that we can add custom fields
> to
> > > >our
> > > >issues. Some projects
> > > >have done such a thing, some, like the couchdb project, have
> > implemented a
> > > >drop down field in
> > > >their jiras so that anyone at a glance can determine if the ticket is
> > > >newcomer, contributor, committer,
> > > >hardcore project  guru or Greg Stein level. (OK so I made that last
> > level
> > > >up, but really, the choice of drop down
> > > >field name is ours to decide)
> > > >
> > > >Pop on over to the couchdb jira and take a look.
> > > >
> > > >I am happy to implement this in the OODT Jira if you agree it a good
> > > >idea.
> > > >
> > > >I'm a big +1 ftr
> > > >
> > > >Gav...
> > > >
> > > >>
> > > >> Cheers,
> > > >> Chris
> > > >>
> > > >> ++
> > > >> 
> > > >> Chris Mattmann, Ph.D.
> > > >> Senior Computer Scientist
> > > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > >> Office: 171-266B, Mailstop: 171-246
> > > >> Email: chris.a.mattm...@nasa.gov
> > > >> WWW:  http://sunset.usc.edu/~mattmann/
> > > >> ++
> > > >> 
> > > >> Adjunct Assistant Professor, Computer Science Department University
> of
> > > >> Southern California, Los Angeles, CA 90089 USA
> > > >> ++
> > > >> 
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> -Original Message-
> > > >> From: Michael Joyce 
> > > >> Reply-To: "d...@climate.incubator.apache.or

Re: New list ?

[Cameron Goodale]

+0 Gavin, it doesn't bother me. So if it goes either way, it's cool.

Cameron
On Apr 30, 2013 3:04 AM, "Gavin McDonald"  wrote:

> A personal peeve of mine, all the jira notifications going to the commits
> list, makes it hard to see whats going on.
>
> (Yes I could filter but..)
>
> How about an issues@ list where we can send the jira stuff ?
>
> Gav...
>
>
>

Re: Review Request: OODT-597: Improve theme/styling of OPSUI

[Cameron Goodale]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10535/#review19454
---

Ship it!


Rishi the style is exactly what you set out to achieve.  Simple, clean and high 
contrast.  Well done.

- Cameron Goodale


On April 19, 2013, 12:30 a.m., Rishi Verma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10535/
> ---
> 
> (Updated April 19, 2013, 12:30 a.m.)
> 
> 
> Review request for oodt, Chris Mattmann, Andrew Hart, Paul Ramirez, and 
> Cameron Goodale.
> 
> 
> Description
> ---
> 
> Hey all,
> 
> Per JIRA Issue: https://issues.apache.org/jira/browse/OODT-597, I proposed we 
> update some of OPSUI's look and feel to make it more readable.
> 
> The attached diff resolves all points addressed in OODT-597, so please take a 
> look at that for more details. I've attached some screenshots of what the new 
> OPSUI looks like - for your convenience. 
> 
> I know A LOT of work must have gone into the original OPSUI look and feel. So 
> I want to make sure my improvements to it (and in some cases changes) are 
> transparent and reviewed before we push. Like I said in OODT-597, the old 
> OPSUI was very slick. I feel it was a little bit to read though on 
> low-contrast screens, and that's why I proposed an improvement to the theme. 
> Also, I think its good if we keep some consistency between how we do OODT 
> logos, coloring, and themeing to make sure we are generating a consistent 
> product.
> 
> Anyways, let me know what you think! I'll push the commits when I get a 'ship 
> it'! :D
> 
> rishi
> 
> 
> This addresses bug OODT-597.
> https://issues.apache.org/jira/browse/OODT-597
> 
> 
> Diffs
> -
> 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.css
>  1468173 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.html
>  1468173 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage.java
>  1468173 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage_cleanwhite.css
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage_cleanwhite.html
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/BasePage_navyblue.css
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/DefaultSkin.css
>  1468173 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage.css
>  1468173 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage.java
>  1468173 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage_cleanwhite.css
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/HomePage_navyblue.css
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/OPSUIWebPage.java
>  PRE-CREATION 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/project_logo.png
>  UNKNOWN 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/java/org/apache/oodt/pcs/opsui/splash.png
>  UNKNOWN 
>   
> http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/src/main/webapp/META-INF/context.xml
>  1468173 
> 
> Diff: https://reviews.apache.org/r/10535/diff/
> 
> 
> Testing
> ---
> 
> Local testing. See screenshots.
> 
> 
> Screenshots
> ---
> 
> Splash page
>   https://reviews.apache.org/r/10535/s/20/
> Workflow monitor page
>   https://reviews.apache.org/r/10535/s/21/
> Workflow tasks diagram
>   https://reviews.apache.org/r/10535/s/22/
> Workflow task info
>   https://reviews.apache.org/r/10535/s/23/
> File Manager browser page
>   https://reviews.apache.org/r/10535/s/24/
> 
> 
> Thanks,
> 
> Rishi Verma
> 
>

Re: [ANNOUNCE] Welcome Gavin McDonald as an OODT PMC member and committer!

[Cameron Goodale]

Gavin,

First off welcome to OODT.  Secondly, thank you for your help with the svn
dump migration and finding the wrm-rcmes-project debacle and fixing it up.
 Monday I will be sure to scope the test svn repo you setup.

Cheers!


-Cameron


On Wed, Apr 17, 2013 at 8:24 AM, Mattmann, Chris A (398J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> -Original Message-
>
> From: Gavin McDonald 
> Organization: 16 degrees complete web solutions
> Reply-To: "dev@oodt.apache.org" ,
> "ga...@16degrees.com.au" 
> Date: Monday, April 15, 2013 5:55 PM
> To: "dev@oodt.apache.org" 
> Subject: RE: [ANNOUNCE] Welcome Gavin McDonald as an OODT PMC member and
> committer!
>
> >
> >
> >> -Original Message-
> >> From: Sean Kelly [mailto:ke...@apache.org]
> >> Sent: Wednesday, 10 April 2013 7:56 AM
> >> To: dev@oodt.apache.org
> >> Cc: gmcdon...@apache.org
> >> Subject: [ANNOUNCE] Welcome Gavin McDonald as an OODT PMC member
> >> and committer!
> >>
> >> Folks:
> >>
> >> I'm happy to report that Gavin McDonald has accepted our invitation to
> >> become a committer on Apache OODT and to join the OODT Project
> >> Management Committee. WOOT!
> >>
> >> Welcome, Gav. Please feel free to introduce yourself.
> >
> >Thanks Sean,
> >
> >Hi All,
> >
> >I hope to help out more in the near future, starting with making the PEAR
> >stuff more reliable/compatible/working.
> >Then onto other stuff.
>
> Dude you rock! Welcome Gav!
>
> >
> >I also do some ASF sysadmin type jobs around the place. :)
> >I look forward to working with you all .
>
> Me too dude!
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: Interested in participating GSoC 2013 with OODT

[Cameron Goodale]

Welcome to OODT Rajith,

It does sound like your skills will be a great asset to the project over
the summer.  I look forward to working with you soon.


Cheers,

Cameron


On Wed, Mar 27, 2013 at 8:44 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Rajith!
>
> Thank you for your interest in OODT-219! Would love to have you help out
> on the project.
>
> The mailing list doesn't accept attachments, so can you head over the
> OODT-219 and attach
> the diagram there?
>
> Thank you and looking forward to working with you on this!
>
> Cheers,
> Chris
>
> From: Rajith Siriwardana  rajithsiriward...@ieee.org>>
> Reply-To: "dev@oodt.apache.org" <
> dev@oodt.apache.org>
> Date: Wednesday, March 27, 2013 9:20 AM
> To: dev mailto:dev@oodt.apache.org>>
> Subject: Re: Interested in participating GSoC 2013 with OODT
>
> hi
> small mistake, forgot to attach the high level deployment architecture
> diagram which i was talking about. note that this architecture diagram
> changed to use mk_livestatus replacing Nagios XML engine to query stats. I
> can provide more details upon request  about the the project I carried out.
>
> Thank you,
>
> T.W.R.O. Siriwardana
> StMIEEE
> Undergraduate
> Department of Computer Science & Engineering
> Faculty of Engineering
> University of Moratuwa
> Sri Lanka.
>
>
> On Wed, Mar 27, 2013 at 9:38 PM, Rajith Siriwardana <
> rajithsiriward...@ieee.org> wrote:
> hi,
> I'm a 4th year (final year) undergraduate of Faculty of Engineering
> University of Moratuwa majoring in Computer Science and Engineering and I
> would like contribute to OODT.
> I did my internship last year which spanned for 9 months in WSO2[1]. which
> is a middle-ware company and I worked with their WSO2 Stratos product[2].
> Basically I was assigned for developing WSO2 Stratos system level health
> monitoring component and WSO2 Stratos heartbeat monitoring package.  For
> WSO2 Stratos health monitoring component[3] before finalizing Nagios[4] for
> monitoring with mk_livestatus[5] replacing NagiosXML engine for query
> stats, which is shown in the architecture image, I did some work with
> Ganglia (how to use it for WSO2 Stratos system level health monitoring).
> Furthermore I took part in test automation hackathon carried out for WSO2
> ESB[6] around 2 months.
> So I believe I have enough knowledge to complete the GSoC 2013 project
>  "Monitor that plugs into ganglia" OODT-219<
> https://issues.apache.org/jira/browse/OODT-219> [7] with some guidance.
> Please give me any advice, guidance you'd like to extend.
> You can access my Linkedin profile via [8]
>
> [1]http://wso2.com/
> [2]http://wso2.com/cloud/stratos/
> [3]http://mail.wso2.org/mailarchive/architecture/2012-June/008827.html
> [4]http://www.nagios.org/
> [5]http://mathias-kettner.de/checkmk_livestatus.html
> [6]http://wso2.com/products/enterprise-service-bus/
> [7]https://issues.apache.org/jira/browse/OODT-219
> [8]http://lk.linkedin.com/in/rajithsiriwardana/
>
> Thank you,
>
> T.W.R.O. Siriwardana
> StMIEEE
> Undergraduate
> Department of Computer Science & Engineering
> Faculty of Engineering
> University of Moratuwa
> Sri Lanka.
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: Min Pool Size in WorkflowManager Thread Pool (0.3)

[Cameron Goodale]

Gabe,

On Snow I raised the minPoolSize to the number of jobs I am able to run
concurrently on our servers via Resource Manager.

I am running minPoolSize=40 AND maxPoolSize=40 and everything works fine.
 I have noticed that if I flood the Workflow manager too quickly the jobs
tend to wait until all 40 jobs are ready before all of them will move from
STAGING to PGE_EXEC.

Hope that helps.


-Cam


On Mon, Mar 18, 2013 at 10:37 AM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Gabe,
>
> The best docs I can point you at are here:
>
> http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/package-s
> ummary.html#package_description
>
>
>
> I would recommend also reading:
>
> http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/Pooled
> Executor.html
>
>
> That really explains what the purpose of those diff properties are.
>
> Cheers,
> Chris
>
>
> On 3/14/13 1:57 PM, "Resneck, Gabriel M (388J)"
>  wrote:
>
> >Hi, guys!
> >I've run into a problem with the Workflow Manager in release 0.3.  The
> >number of active threads allowed by the pool seems to be dictated by the
> >minimum thread count when the queue supplied to the pool object is
> >unlimited.  The default number (6) is a bit low for our purposes, so I
> >was wondering how you guys have dealt with this issue in the past.  Did
> >you simply increase the minimum pool size or implement another solution?
> >If you increased the minimum pool size, what was the highest that you
> >have used and did you see any issues as a result?
> >Thanks!
> >
> >Gabe =)
> >
>
>


-- 

Sent from a Tin Can attached to a String

Re: FW: [GSoC Mentors] Google Summer of Code 2013

[Cameron Goodale]

Thanks Ross.  That sounds like the best way to approach the challenge of
have such a wide range of students and skill sets.

Best Regards,


Cameron


On Sun, Feb 17, 2013 at 3:18 PM, Ross Laidlaw wrote:

> Hi Cameron,
>
> I forgot to mention (apologies if you already know this) that GSoC is
> open to many types of student including undergraduates, postgraduates,
> mature students, full-time and part-time students, etc.  Consequently,
> it might be difficult to devise a project that presents the same level
> of challenge to every GSoC student.  The approach Chris took last year
> as my mentor was to have many interesting extensions lined up for me
> if I finished the original project.  I really liked this approach and
> I believe it's a good model for other GSoC projects to follow.
>
> Hope this is useful feedback,
>
> Ross
>
>
>
> On 14 February 2013 20:51, Cameron Goodale  wrote:
> > Ross,
> >
> > Thanks for the feedback and I am glad to hear that your GSoC experience
> was
> > good enough that you are considering doing it again. Is there anything we
> > can do differently to improve this summer?  (don't worry you won't hurt
> my
> > feelings)
> >
> > Cheers,
> >
> >
> > Cameron
> >
> >
> > On Wed, Feb 13, 2013 at 11:23 PM, Ross Laidlaw  >wrote:
> >
> >> Hi Chris & Cam,
> >>
> >> I'm listening in on the OODT and SIS lists and I've been following
> >> along with great interest.  My GSoC experience last summer was very
> >> positive.  I found OODT to be highly accessible because of its
> >> component-based structure.  This allowed me to focus on one area (File
> >> Manager) and start working on my project without having to learn
> >> everything upfront about each part of OODT.  The approach for my
> >> project was to break it down into smaller deliverable steps.  I liked
> >> this idea as it helped me to gain momentum.  I found the OODT and SIS
> >> communities to be very knowledgable, welcoming and encouraging.  All
> >> of these aspects gave me a lot of enthusiasm to contribute and to
> >> complete GSoC.
> >>
> >> I've tried to spread the word about GSoC to my classmates and
> >> colleagues.  Every year there are tons of applicants and there should
> >> be lots of healthy competition again this year for places.  I'm still
> >> a student and I'm very keen to apply this year if the OODT/SIS
> >> communities would be happy for me to contribute again via GSoC.
> >>
> >> Ross
> >>
> >>
> >>
> >> On 13 February 2013 22:09, Cameron Goodale  wrote:
> >> > Chris,
> >> >
> >> > I think it would be a good idea to check in with Ross who was our GSoC
> >> > student last summer, and ask him for feedback on his experience with
> >> OODT.
> >> >  I hope he is listening to the dev list still, but if not we should
> >> > check-in and see what he is doing now.
> >> >
> >> > My hope is that if Ross gives us feedback (good or bad or both) we can
> >> work
> >> > to improve this summer for our next GSoC student.  My apologies in
> >> advance
> >> > if you already did this when the GSoC last year was closed out.
> >> >
> >> > Just a few questions to get the feedback pump primed would be:
> >> >
> >> > Was the mini-project challenging enough?
> >> > Did the OODT Devs provide enough information and support?
> >> > Did you encounter anything that was frustrating or unnecessarily
> >> difficult?
> >> >
> >> > Any thoughts?
> >> >
> >> >
> >> > -Cam chee
> >> >
> >> >
> >> >
> >> > On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
> >> > chris.a.mattm...@jpl.nasa.gov> wrote:
> >> >
> >> >> [Sorry for cross posting]
> >> >>
> >> >> Guys,
> >> >>
> >> >> FYI please note that you can participate as a mentor from a PMC via
> >> Apache
> >> >> as they are a GSoC org. ComDev will coordinate our participation but
> >> start
> >> >> thinking about what projects we may want to do.
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >> From: Carol Smith mailto:car...@google.com>>
> >> >> Date: Monday, February 11, 2013 11:02 AM
> >

Re: FW: [GSoC Mentors] Google Summer of Code 2013

[Cameron Goodale]

Ross,

Thanks for the feedback and I am glad to hear that your GSoC experience was
good enough that you are considering doing it again. Is there anything we
can do differently to improve this summer?  (don't worry you won't hurt my
feelings)

Cheers,


Cameron


On Wed, Feb 13, 2013 at 11:23 PM, Ross Laidlaw wrote:

> Hi Chris & Cam,
>
> I'm listening in on the OODT and SIS lists and I've been following
> along with great interest.  My GSoC experience last summer was very
> positive.  I found OODT to be highly accessible because of its
> component-based structure.  This allowed me to focus on one area (File
> Manager) and start working on my project without having to learn
> everything upfront about each part of OODT.  The approach for my
> project was to break it down into smaller deliverable steps.  I liked
> this idea as it helped me to gain momentum.  I found the OODT and SIS
> communities to be very knowledgable, welcoming and encouraging.  All
> of these aspects gave me a lot of enthusiasm to contribute and to
> complete GSoC.
>
> I've tried to spread the word about GSoC to my classmates and
> colleagues.  Every year there are tons of applicants and there should
> be lots of healthy competition again this year for places.  I'm still
> a student and I'm very keen to apply this year if the OODT/SIS
> communities would be happy for me to contribute again via GSoC.
>
> Ross
>
>
>
> On 13 February 2013 22:09, Cameron Goodale  wrote:
> > Chris,
> >
> > I think it would be a good idea to check in with Ross who was our GSoC
> > student last summer, and ask him for feedback on his experience with
> OODT.
> >  I hope he is listening to the dev list still, but if not we should
> > check-in and see what he is doing now.
> >
> > My hope is that if Ross gives us feedback (good or bad or both) we can
> work
> > to improve this summer for our next GSoC student.  My apologies in
> advance
> > if you already did this when the GSoC last year was closed out.
> >
> > Just a few questions to get the feedback pump primed would be:
> >
> > Was the mini-project challenging enough?
> > Did the OODT Devs provide enough information and support?
> > Did you encounter anything that was frustrating or unnecessarily
> difficult?
> >
> > Any thoughts?
> >
> >
> > -Cam chee
> >
> >
> >
> > On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
> > chris.a.mattm...@jpl.nasa.gov> wrote:
> >
> >> [Sorry for cross posting]
> >>
> >> Guys,
> >>
> >> FYI please note that you can participate as a mentor from a PMC via
> Apache
> >> as they are a GSoC org. ComDev will coordinate our participation but
> start
> >> thinking about what projects we may want to do.
> >>
> >> Cheers,
> >> Chris
> >>
> >> From: Carol Smith mailto:car...@google.com>>
> >> Date: Monday, February 11, 2013 11:02 AM
> >> To: Google Summer of Code Mentors List <
> >> google-summer-of-code-mentors-l...@googlegroups.com >> google-summer-of-code-mentors-l...@googlegroups.com>>
> >> Subject: [GSoC Mentors] Google Summer of Code 2013
> >>
> >> Hi GSoC mentors and org admins,
> >>
> >> We've announced that we're doing Google Summer of Code 2013 [1]. Yay!
> >>
> >> If you would like to help spread the word about GSoC, we have
> >> presentations [2], logos [3], and flyers [4] for you to use. Please host
> >> meetups, tell your friends and colleagues about the program, go to
> >> conferences, talk to people about the program, and just generally do all
> >> the awesome word-of-mouth stuff you do every year to promote the
> program.
> >>
> >> The GSoC calendar, FAQ, and events timeline have all been updated with
> >> this year's important dates, so please refer to those for the milestones
> >> for this year's program. NB: the normal timeline for the program has
> been
> >> modified for this year. You'll probably want to examine the dates
> closely
> >> to make sure you know when important things are happening.
> >>
> >> Please consider translating the presentations and/or flyers into your
> >> native language and submitting them directly to me to post on the wiki.
> >> Localization for our material is integral to reaching the widest
> possible
> >> audience around the world. If you decide to translate a flyer, please
> fill
> >> out our form to request a thank you gift for your effo

Re: FW: [GSoC Mentors] Google Summer of Code 2013

[Cameron Goodale]

Chris,

I think it would be a good idea to check in with Ross who was our GSoC
student last summer, and ask him for feedback on his experience with OODT.
 I hope he is listening to the dev list still, but if not we should
check-in and see what he is doing now.

My hope is that if Ross gives us feedback (good or bad or both) we can work
to improve this summer for our next GSoC student.  My apologies in advance
if you already did this when the GSoC last year was closed out.

Just a few questions to get the feedback pump primed would be:

Was the mini-project challenging enough?
Did the OODT Devs provide enough information and support?
Did you encounter anything that was frustrating or unnecessarily difficult?

Any thoughts?


-Cam chee



On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> [Sorry for cross posting]
>
> Guys,
>
> FYI please note that you can participate as a mentor from a PMC via Apache
> as they are a GSoC org. ComDev will coordinate our participation but start
> thinking about what projects we may want to do.
>
> Cheers,
> Chris
>
> From: Carol Smith mailto:car...@google.com>>
> Date: Monday, February 11, 2013 11:02 AM
> To: Google Summer of Code Mentors List <
> google-summer-of-code-mentors-l...@googlegroups.com google-summer-of-code-mentors-l...@googlegroups.com>>
> Subject: [GSoC Mentors] Google Summer of Code 2013
>
> Hi GSoC mentors and org admins,
>
> We've announced that we're doing Google Summer of Code 2013 [1]. Yay!
>
> If you would like to help spread the word about GSoC, we have
> presentations [2], logos [3], and flyers [4] for you to use. Please host
> meetups, tell your friends and colleagues about the program, go to
> conferences, talk to people about the program, and just generally do all
> the awesome word-of-mouth stuff you do every year to promote the program.
>
> The GSoC calendar, FAQ, and events timeline have all been updated with
> this year's important dates, so please refer to those for the milestones
> for this year's program. NB: the normal timeline for the program has been
> modified for this year. You'll probably want to examine the dates closely
> to make sure you know when important things are happening.
>
> Please consider translating the presentations and/or flyers into your
> native language and submitting them directly to me to post on the wiki.
> Localization for our material is integral to reaching the widest possible
> audience around the world. If you decide to translate a flyer, please fill
> out our form to request a thank you gift for your effort. [5]
>
> If you decide to host a meetup, please email me to let me know the date,
> time, and location so I can put it on the GSoC calendar. Also, remember to
> take pictures at your meetup and write up a blog post for our blog using
> our provided template for formatting [6]. If you need promotional items for
> your attendees, please fill out our form [7] to request some; we're happy
> to send some along. We can provide up to about 25 pens, notebooks, or
> stickers and/or a few t-shirts. Please keep in mind, though, that shipping
> restrictions and timeline vary country-to-country; request items early to
> make sure they get there on time! If you have questions about hosting
> meetups, please see the section in our FAQ [8].
>
> Please consider applying to participate as an organization again this year
> or maybe joining as a mentor for your favorite organization if they are
> selected this year.
>
> We rely on you for your help for the success of this program, so thank you
> in advance for all the work you do!
>
> [1] -
> http://google-opensource.blogspot.com/2013/02/flip-bits-not-burgers-google-summer-of.html
> [2] -
> http://code.google.com/p/google-summer-of-code/wiki/ProgramPresentations
> [3] - http://code.google.com/p/google-summer-of-code/wiki/GsocLogos
> [4] - http://code.google.com/p/google-summer-of-code/wiki/GsocFlyers
> [5] - http://goo.gl/gEHDO
> [6] - http://goo.gl/wbZrt
> [7] - http://goo.gl/0BsR8
> [8] - http://goo.gl/2NGfp
>
> Cheers,
> Carol
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google Summer of Code Mentors List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-summer-of-code-mentors-list+unsubscr...@googlegroups.com
> .
> To post to this group, send email to
> google-summer-of-code-mentors-l...@googlegroups.com google-summer-of-code-mentors-l...@googlegroups.com>.
> Visit this group at
> http://groups.google.com/group/google-summer-of-code-mentors-list?hl=en-US
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

Re: [HEADS UP] Apache OODT now builds with Maven3 too

[Cameron Goodale]

Chris,

This is a huge win!  Now I can let Apple install Maven 3 with my next Java
update and I don't have to revert.

-Cam


On Sun, Feb 3, 2013 at 3:41 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hey Guys,
>
> With r1441987, OODT 0.6 trunk now builds with Maven3. Yay!
>
> Trunk still builds with Maven2 as well.
>
> [INFO] CAS Product Server  SUCCESS
> [2.791s]
> [INFO] CAS Workflow Manager Monitor Web App .. SUCCESS
> [2.341s]
> [INFO] Catalog and Archive File Management Browser ... SUCCESS
> [1.561s]
> [INFO] Catalog and Archive Workflow Management GUI Editor  SUCCESS
> [9.351s]
> [INFO] Process Control System Operator Interface Webapp .. SUCCESS
> [4.225s]
> [INFO] OODT Process Control System JAX-RS service layer .. SUCCESS
> [4.131s]
> [INFO] Apache OODT ... SUCCESS
> [37.794s]
> [INFO]
> 
> [INFO]
> 
> [INFO] BUILD SUCCESSFUL
> [INFO]
> 
> [INFO] Total time: 7 minutes 6 seconds
> [INFO] Finished at: Sun Feb 03 15:40:18 PST 2013
> [INFO] Final Memory: 100M/191M
> [INFO]
> 
> [chipotle:~/tmp/oodt0.6] mattmann%
>
>
> Cheers,
> Chris
>
>

Re: OODT 0.3 branch

[Cameron Goodale]

Chintu,

I see that your test data volume is 262GB, but I am curious about the make
up of the data.  On average what is your file size and how many files?

The reason I ask is because the process of extraction and ingestion can
vary wildly.  On the LMMP project I was ingesting 12GB DEMs over NFS and it
was a slow process.  It was basically serial with 1CR+1FM, but we didn't
have a requirement to push large volumes of data.

On our recent Snow Data System I am processing 160 workflow jobs in
parallel and OODT could handle the load, it turned out the filesystem was
our major bottleneck.  We used a SAN initially when doing development, but
when we increased the number of jobs in parallel the I/O became so bad we
moved to GlusterFS. GlusterFS had speed improvements over the SAN, but we
had to be careful about heavy writing, moving, deleting since the
clustering would try to replicate the data.  Turns out Gluster is great for
heavy writting OR heavy reading, but not both at the same time.  Finally we
are using NAS and it works great.

My point is the file system plays a major role in performance when
ingesting data.  The ultimate speed test would be if you could actually
write the data into the final archive directory and basically do an
ingestion in place (skip data transfer entirely), but I know that is rarely
possible.

This is an interesting challenge to see what configuration will yield the
best through put/performance.  I look forward to hearing more about your
progress on this.


Best Regards,



Cameron


On Wed, Dec 12, 2012 at 7:28 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Chintu,
>
> From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC]
> (GSFC-586.0)" mailto:chintu.mis...@nasa.gov>>
> Date: Wednesday, December 12, 2012 12:02 PM
> To: jpluser  chris.a.mattm...@jpl.nasa.gov>>, "dev@oodt.apache.org dev@oodt.apache.org>" mailto:dev@oodt.apache.org>>
> Subject: Re: OODT 0.3 branch
>
> If you are saying that FM can handle multiple connections at one time,
>
> Yep I'm saying that it can.
>
> then multiple crawlers pointing to same FM should increase performance
> significantly.
>
> Well that really depends to be honest. It sounds like you guys are hitting
> an IO bottleneck potentially in data transfer? What file sizes are you
> transferring? If you are IO bound on the data transfer part, the product
> isn't fully ingested until:
>
>
>   1.  it's entry is added to the catalog
>   2.  The data transfer finishes
>
> Are you checking the FM for status along the way? Also realize that the FM
> will never be faster than the file system, so if it takes the file system X
> minutes to transfer a file F1, Y to transfer F2, and Z to transfer F3, then
> you still have to wait at least the max(X,Y,Z) time, regardless for the 3
> ingestions to complete.
>
> But that’s not what we saw in our tests.
>
> For example,
> I saw barely 2 minutes performance difference between 2FM-6CR and 3FM-6CR.
>
> 1) 2 hour  6 minutes to process 262G   (1FM 3CR - 3CR to 1FM)
> 2) 1 hour 58 minutes to process 262G   (1FM 6CR - 6CR to 1FM)
> 3) 1 hour 39 minutes to process 262G   (2FM 6CR - 3CR to 1FM)
> 4) 1 hour 39 minutes to process 262G   (2FM 9CR - 4+CR to 1FM)
> 5) 1 hour 37 minutes to process 262G   (3FM 9CR - 3CR to 1FM)
> 6) 2 hourto process 262G   (3FM 20CR - 6+CR to 1FM)
> 7) 28 minutesto process 262G   (6FM 9CR - 1+CR to 1FM)   => This is my
> latest test and this is good number.
>
> What would be interesting is simply looking at the speed for how long it
> takes to cp the files (which I bet is what's happening) versus mv'ing the
> files by hand. If mv is faster, I'd:
>
>
>   1.  Implement a Data Transfer implementation that simply replaces the
> calls to FileUtils.copyFile or .moveFile with systemCalls (see ExecHelper
> from oodt-commons) to UNIX equivalents.
>   2.  Plug that data transfer in to your crawler invocations via the cmd
> line.
>
> HTH!
>
> Cheers,
> Chris
>
>
> From: , Chris A  chris.a.mattm...@jpl.nasa.gov>>
> Date: Wednesday, December 12, 2012 2:51 PM
> To: "Mistry, Chintu (GSFC-586.0)[COLUMBUS TECHNOLOGIES AND SERVICES INC]" <
> chintu.mis...@nasa.gov>, "
> dev@oodt.apache.org"  >
> Subject: Re: OODT 0.3 branch
>
> Hey Chintu,
>
> From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC]
> (GSFC-586.0)" mailto:chintu.mis...@nasa.gov>>
> Date: Tuesday, December 11, 2012 2:41 PM
> To: jpluser  chris.a.mattm...@jpl.nasa.gov>>, "dev@oodt.apache.org dev@oodt.apache.org>" mailto:dev@oodt.apache.org>>
> Subject: Re: OODT 0.3 branch
>
> Answers inline below.
>
> ---snip
>
> Gotcha, so you are using different product types. So, each crawler is
> crawling various product types in each one of the staging area dirs, that
> looks like e.g.,
>
> /STAGING_AREA_BASE
>   /dir1 – 1st crawler
>- file1 of product type 1
>- file2 of product type 3
>
>  /dir2 – 2nd crawler
>- file3 of p

Observation of the MetadataBasedProductMover

[Cameron Goodale]

For anyone who needs to re-organize your OODT Archive I found this little
tool to be quite useful.

You can check out the code here:
https://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataBasedProductMover.java


I have moved a lot of product using this tool, but I found a minor
inconsistency and I wanted to see if this is a bug.  You see in the index
output listed below the FileLocation doesn't match the newly updated
reference_data_store, but it does match the reference_orig path.

stored,indexed
stored,indexed
stored,indexed
*stored,indexed*
stored,indexed
stored,omitNorms
stored,omitNorms
*
stored,omitNorms
*
stored,omitNorms
stored,indexed
*
stored,omitNorms
*

So the question I am posing to the list is:  Should the FileLocation be
kept in-sync with the reference_data_store?  If this should happen, then I
think the MetadataBasedProductMover should be patched to make this update
at the time of the move.

Best Regards,


Cameron

Re: 1100+ Javadoc Warnings...

[Cameron Goodale]

Hey Brian,

I think the real issue is with the setup of the maven javadoc plugin.
 There is a way to ask the javadoc to search within the current project to
create the links between docs, I just haven't been able to get it working
for the maven plugin.  I agree that adding the full path will be incredibly
cumbersome so i hope with some maven magic 99% of the javadoc warnings will
go away.

Thanks for the reply to this.

-Cam

On Fri, Aug 24, 2012 at 10:52 PM, Brian Foster  wrote:

> hey cameron... sorry totally forgot about this email... i think you can
> just add an import statement for the {@link }... putting the
> full path in the {@link} will just make the javadoc messy looking... i will
> try to fix my eclipse setup so it complains about that and asks me to
> import so i remember to do it from now on
>
> -brian
>
> On Aug 13, 2012, at 7:02 AM, Cameron Goodale wrote:
>
> > Hey Brian, (I am addressing you because my focus for javadoc clean up is
> > the cli component)
> > and Paul (since you are the maven guru)
> >
> > 90% of the javadoc problems we (Mike Joyce and I) are finding are related
> > to the javadoc tool not searching within the java package to find and
> fill
> > in a complete reference.
> >
> > Here is an example of the DIFF I created from the cli module after
> cleaning
> > up over 100 javadoc errors.  2 or 3 were merely typos, but most of them
> > where like these:
> >
> > ===
> > --- src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
> > (revision
> > 1371572)
> > +++ src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
> > (working
> > copy)
> > @@ -17,7 +17,7 @@
> > package org.apache.oodt.cas.cli.option;
> >
> > /**
> > - * {@link CmdLineOption} for specifying {@link CmdLineAction}.
> > *+ * {@link CmdLineOption} for specifying {@link
> > org.apache.oodt.cas.cli.action.CmdLineAction}.*
> >  *
> >  * @author bfoster (Brian Foster)
> >  */
> > Index:
> >
> src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
> > ===
> > ---
> >
> src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
> > (revision
> > 1371572)
> > +++
> >
> src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
> > (working
> > copy)
> > @@ -20,7 +20,7 @@
> > import org.apache.oodt.cas.cli.option.CmdLineOptionInstance;
> >
> > /**
> > - * Validator for specified values of {@link CmdLineOption}s.
> > *+ * Validator for specified values of {@link
> > org.apache.oodt.cas.cli.option.CmdLineOption}s.*
> >  *
> >  * @author bfoster (Brian Foster)
> >  */
> >
> > So it seems like we have to use fully qualified package names
> > when referencing methods and objects within javadocs.  I was really
> hoping
> > that some kinda maven plugin magic would be able to search and resolve
> > these paths for us during the build process.
> >
> > I have tried a couple different things like using "mvn javadoc:fix" but
> > that doesn't seem to do much.
> >
> > So if anyone has any ideas to try on this I am all ears.
> >
> >
> >
> > Thanks,
> >
> >
> > Cameron
> >
> >
> > --
> >
> > Sent from a Tin Can attached to a String
>
>

1100+ Javadoc Warnings...

[Cameron Goodale]

Hey Brian, (I am addressing you because my focus for javadoc clean up is
the cli component)
and Paul (since you are the maven guru)

90% of the javadoc problems we (Mike Joyce and I) are finding are related
to the javadoc tool not searching within the java package to find and fill
in a complete reference.

Here is an example of the DIFF I created from the cli module after cleaning
up over 100 javadoc errors.  2 or 3 were merely typos, but most of them
where like these:

===
--- src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
(revision
1371572)
+++ src/main/java/org/apache/oodt/cas/cli/option/ActionCmdLineOption.java
(working
copy)
@@ -17,7 +17,7 @@
 package org.apache.oodt.cas.cli.option;

 /**
- * {@link CmdLineOption} for specifying {@link CmdLineAction}.
*+ * {@link CmdLineOption} for specifying {@link
org.apache.oodt.cas.cli.action.CmdLineAction}.*
  *
  * @author bfoster (Brian Foster)
  */
Index:
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
===
---
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
(revision
1371572)
+++
src/main/java/org/apache/oodt/cas/cli/option/validator/CmdLineOptionValidator.java
(working
copy)
@@ -20,7 +20,7 @@
 import org.apache.oodt.cas.cli.option.CmdLineOptionInstance;

 /**
- * Validator for specified values of {@link CmdLineOption}s.
*+ * Validator for specified values of {@link
org.apache.oodt.cas.cli.option.CmdLineOption}s.*
  *
  * @author bfoster (Brian Foster)
  */

So it seems like we have to use fully qualified package names
when referencing methods and objects within javadocs.  I was really hoping
that some kinda maven plugin magic would be able to search and resolve
these paths for us during the build process.

I have tried a couple different things like using "mvn javadoc:fix" but
that doesn't seem to do much.

So if anyone has any ideas to try on this I am all ears.



Thanks,


Cameron


-- 

Sent from a Tin Can attached to a String

Re: CSS issue with JIRA?

[Cameron Goodale]

Ross and Sheryl,

Thanks for double checking on this for me.  I wanted to make sure I didn't
have some odd Browser/Network settings on my machine.

I just jumped over to issues.apache.org/jira and saw "Maintenance in
Progress" so it looks like Ross guessed correctly.

In the future we can check out this page for details on the ASF
infrastructure status.

http://monitoring.apache.org/status/

I guess i will just keep checking up on this through out the day.


Cheers,

Cameron


On Fri, Aug 10, 2012 at 2:05 AM, Ross Laidlaw wrote:

> Hi Cam & Sheryl,
>
> I think JIRA might be down at the moment.  I can't access it here in the
> UK - I get a page saying 'Service Temporarily Unavailable'.  Perhaps some
> maintenance/upgrades going on I guess...
>
> Ross
>
>
>
> On 10 Aug 2012, at 07:11, Sheryl John  wrote:
>
> > Yep it does look like the css files are missing. Didn't get the
> screenshot.
> >
> > On Thu, Aug 9, 2012 at 11:06 PM, Cameron Goodale 
> wrote:
> >
> >> Hey OODTers,
> >>
> >> I have been trying to access JIRA and I can, but the css files seem to
> be
> >> missing.  Does anyone else see this same problem with:
> >> https://issues.apache.org/jira/browse/OODT-328 ?
> >>
> >>
> >> Screenshot of what I see is attached.
> >>
> >> Thanks.
> >>
> >>
> >> -Cameron
> >>
> >
> >
> >
> > --
> > -Sheryl
>

CSS issue with JIRA?

[Cameron Goodale]

Hey OODTers,

I have been trying to access JIRA and I can, but the css files seem to be
missing.  Does anyone else see this same problem with:
https://issues.apache.org/jira/browse/OODT-328 ?


Screenshot of what I see is attached.

Thanks.


-Cameron

Re: [jira] [Commented] (OODT-328) Remove jpl.nasa.gov references from code

[Cameron Goodale]

Mike and Chris,

I just scope this svn url in the file and I wonder if the CAS-SSO code has
been pushed to Apache.  If the code has not been migrated then it will be a
blocker to this issue.

I did find an SSO package in the OODT Pear Channel here:
http://pear.apache.org/oodt/

Assuming the OODT Pear Channel has all the code in it:

Since the edit is focused on a comment block / docstring change, it would
seem that the checkout/build process needs to be re-written in this file,
or omitted entirely if no longer necessary.


Mike if you are unfamiliar with the PHP build/install process check with
Andrew Hart or Shakeh Khudikyan (both on this mailing-list) for help.


Keep up the great work,


Cam

On Tue, Aug 7, 2012 at 10:09 PM, Chris A. Mattmann (JIRA)
wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430872#comment-13430872]
>
> Chris A. Mattmann commented on OODT-328:
> 
>
> Mike: I would focus on the SVN lines since the DTD stuff and XML schema NS
> involves other discussions (for example, see Ross's recent Geospatial stuff
> e.g., in OODT-469 and OODT-471).
>
> If you fix files in SVN refs, I think we can call this done!
>
>
> > Remove jpl.nasa.gov references from code
> > 
> >
> > Key: OODT-328
> > URL: https://issues.apache.org/jira/browse/OODT-328
> > Project: OODT
> >  Issue Type: Bug
> >  Components: build proces
> >Reporter: Chris A. Mattmann
> > Fix For: 0.5
> >
> > Attachments: grepoutput.txt
> >
> >
> > We need to remove all references to jpl.nasa.gov in our code (for
> example I just saw one in:
> http://svn.apache.org/repos/asf/oodt/tags/0.3/balance/lib/pear/Providers/Auth/LDAPAuthenticationProvider.class.phpwhich
>  made me think that we need to code review before we roll 0.4).
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-310) Port WEngine to trunk

[Cameron Goodale]

That is great news.  At JPL my issue was using a shortcut url for our svn
repo.  http://oodt/repo  vs. http://oodt.jpl.nasa.gov/repo but the same
basic deal.


-Cameron

On Mon, Aug 6, 2012 at 8:51 PM, Sheryl John  wrote:

> Thanks Cameron!
> It worked after changing the url to "http".
>
> On Mon, Aug 6, 2012 at 7:43 PM, Cameron Goodale 
> wrote:
>
> > Sheryl,
> >
> > I get the same error on another project using review board. The issue is
> > caused by filling out too much or too little of the base repository url.
> >
> > If/when you get the right combination to work, make note of it or grab a
> > screen shot. It'll help keep your sanity.
> >
> > Cameron
> > On Aug 6, 2012 11:34 AM, "Brian Foster (JIRA)"  wrote:
> >
> > >
> > > [
> > >
> >
> https://issues.apache.org/jira/browse/OODT-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429325#comment-13429325
> > ]
> > >
> > > Brian Foster commented on OODT-310:
> > > ---
> > >
> > > EngineRunnerFactory additions fixed in r1369914
> > >
> > > > Port WEngine to trunk
> > > > -
> > > >
> > > > Key: OODT-310
> > > > URL: https://issues.apache.org/jira/browse/OODT-310
> > > > Project: OODT
> > > >  Issue Type: Sub-task
> > > >  Components: workflow manager
> > > >Reporter: Chris A. Mattmann
> > > >Assignee: Chris A. Mattmann
> > > > Fix For: 0.5
> > > >
> > > > Attachments: OODT-310.2012-08-03.bfoster.patch.txt,
> > > OODT-310.Mattmann.082311.patch.txt, OODT-310.Mattmann.100911.patch.txt,
> > > OODT-310.sherylj.101711.patch.txt
> > > >
> > > >
> > > > This issue will track the porting of the wengine-branch
> WorkflowEngine
> > > interface and associated classes to trunk. This involves changing to
> the
> > > Processor model (Sequential, Condition, etc.) which has already begun
> (as
> > > of OODT-70), and also the actual Engine implementation itself, that
> > doesn't
> > > block and that includes a queue-based model developed by [~bfoster].
> > >
> > > --
> > > This message is automatically generated by JIRA.
> > > If you think it was sent incorrectly, please contact your JIRA
> > > administrators:
> > >
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> > > For more information on JIRA, see:
> > http://www.atlassian.com/software/jira
> > >
> > >
> > >
> >
>
>
>
> --
> -Sheryl
>



-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-310) Port WEngine to trunk

[Cameron Goodale]

Sheryl,

I get the same error on another project using review board. The issue is
caused by filling out too much or too little of the base repository url.

If/when you get the right combination to work, make note of it or grab a
screen shot. It'll help keep your sanity.

Cameron
On Aug 6, 2012 11:34 AM, "Brian Foster (JIRA)"  wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429325#comment-13429325]
>
> Brian Foster commented on OODT-310:
> ---
>
> EngineRunnerFactory additions fixed in r1369914
>
> > Port WEngine to trunk
> > -
> >
> > Key: OODT-310
> > URL: https://issues.apache.org/jira/browse/OODT-310
> > Project: OODT
> >  Issue Type: Sub-task
> >  Components: workflow manager
> >Reporter: Chris A. Mattmann
> >Assignee: Chris A. Mattmann
> > Fix For: 0.5
> >
> > Attachments: OODT-310.2012-08-03.bfoster.patch.txt,
> OODT-310.Mattmann.082311.patch.txt, OODT-310.Mattmann.100911.patch.txt,
> OODT-310.sherylj.101711.patch.txt
> >
> >
> > This issue will track the porting of the wengine-branch WorkflowEngine
> interface and associated classes to trunk. This involves changing to the
> Processor model (Sequential, Condition, etc.) which has already begun (as
> of OODT-70), and also the actual Engine implementation itself, that doesn't
> block and that includes a queue-based model developed by [~bfoster].
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

Re: Problem when running pushpull

[Cameron Goodale]

YunHee,

Thank you for testing out the PushPull User Guide.  I have been waiting for
someone to run through it and have success (which it appears you have).  I
just added in a small block about how to Launch the PushPull Daemon, but it
seems that you got it running already.

If there is anything that you think should be added to the User Guide, or
if any part was confusing I would appreciate hearing from you.

Cheers!


Cameron

On Sat, Jul 21, 2012 at 10:15 PM, YunHee Kang  wrote:

> HI chris,
>
> > Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
> >
> > 1. Create yourself an account at our JIRA issue tracker:
> > https://issues.apache.org/jira/browse/OODT
> >
> > 2. File an issue (create a new issue/bug) against oodt 0.4 (set the
> affects
> > version to 0.4)
> >
> I just created  the isse named "OODT-474"  related with config file
> bug in oodt0.4.
> It is the first time to write an issue report.  Can you check the
> issue that I wrote ?
>
>
> > Ahh, what FTP site are you trying to access? That way I can try and
> > test this myself in my own local checkout.
>
> I tried to test downloading a file  based on WIKI site
> "https://cwiki.apache.org/OODT/oodt-push-pull-user-guide.html";. There
> is no problem to download the file  as follows:
>  $ ftp l4ftl01.larc.nasa.gov
> Connected to l4ftl01.larc.nasa.gov.
> 220-
> 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 220-
> 220- This US Government computer is for authorized users only.
> 220- By accessing this system you are consenting to complete
> 220- monitoring with no expectation of privacy. Unauthorized
> 220- access or use may subject you to disciplinary action and
> 220- criminal prosecution.
> 220-
> 220--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 220-
> 220-
> 220-
> 220 FTP server ready.
> Name (l4ftl01.larc.nasa.gov:yhkang): anonymous
> 331 Guest login ok, send your complete e-mail address as password.
> Password:
> 230 Guest login ok, access restrictions apply.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> cd /TES/TL2CO2N.005
> 250 CWD command successful.
> ftp> cd 2004.09.20
> 250 CWD command successful.
> ftp> get TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5
> local: TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5 remote:
> TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5
> 200 PORT command successful.
> 150 Opening BINARY mode data connection for
> TES-Aura_L2-CO2-Nadir_r002147_F06_09.he5 (92150280 bytes).
> 226 Transfer complete. Checksum matched.
> 92150280 bytes received in 171.84 secs (523.7 kB/s)
>
>
> Thanks,
>
> Yunhee
>
> 2012/7/21 Mattmann, Chris A (388J) :
> > Hi YunHee,
> >
> > On Jul 19, 2012, at 11:19 AM, YunHee Kang wrote:
> >
> >> I just found a problem that is in a file
> >> "$CAS_PP_HOME/policy/ProtocolFactoryInfo.xml" in  oodt 0.4.
> >>
> >> The problem is caused by missing classes related with an element
> >> "protocolFactory" in the file described above.
> >> To fix this problem,  It needs to replace a wrong class name with  a
> >> correct class name.
> >> For example  there is no class named
> >> org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpClientFactory in any jar
> >> file in the directory,  $CAS_PP_HOME/lib.  To run the script
> >> $CAS_PP_HOME/bin/pushpull,   CogJGlobusFtpClientFactory  needs to be
> >> replaced with CogJGlobusFtpProtocolFactory.
> >
> > Ahh gotcha. You've found a config file bug in oodt0.4. Can you please:
> >
> > 1. Create yourself an account at our JIRA issue tracker:
> > https://issues.apache.org/jira/browse/OODT
> >
> > 2. File an issue (create a new issue/bug) against oodt 0.4 (set the
> affects
> > version to 0.4)
> >
> > If you are willing to, we'd love to have a patch file from you that fixes
> > this config file. The patch process would be something like:
> >
> > 1. svn co http://svn.apache.org/repos/asf/oodt/trunk oodt0.5
> > 2. edit
> odot0.5/pushpull/src/main/resources/policy/ProtocolFactoryInfo.xml and
> > fix XML policy file.
> > 3. svn diff > OODT-xxx.ykang.yyMMdd.patch.txt
> >
> > where xxx is the JIRA issue ID from #2 above.
> > 4. attach your patch to the JIRA issue from #2.
> >
> >>
> >> But I don't know how to replace
> >> org.apache.oodt.cas.protocol.local.LocalClientFactory.
> >
> > I'm not sure you have to -- this is probably OK.
> >
> >> To be honest, I am not familiar with OODT.
> >>
> >>
> >> 
> >>
> >>
> >> >> class="org.apache.oodt.cas.protocol.ftp.CogJGlobusFtpProtocolFactory"/>
> >>
> >> >> class="org.apache.oodt.cas.protocol.ftp.CommonsNetFtpProtocolFactory"/>
> >>
> >>
> >> 
> >>
> >>
> >> >> class="org.apache.oodt.cas.protocol.http.HttpProtocolFactory"/>
> >>
> >>
> >>
> >>
> >>
> >> >> class="org.apache.oodt.cas.protocol.sftp.JschSftpProtocolFactory"/>
> >>
> >>
> >>
> >>
> >>
> >> >> class="org.apache.oodt.cas.protocol.imaps.ImapsProtocolFactory"/>
> >>

Re: OODT SoCal Meetup/Hackathon

[Cameron Goodale]

I think the pool/house idea sounds great.  If we start early enough, then
we can hack for like 4 - 6 hours, then use the pool as a reward for a job
well done.

My hope is that we can get some long stretches of time to work on OODT,
without email, twitter, IM, or facebook distractions...well the exception
being the Google Hangout (of course).

-Cam

On Thu, May 17, 2012 at 7:06 PM, Ramirez, Paul M (388J) <
paul.m.rami...@jpl.nasa.gov> wrote:

> Your place sounds good to me. Now we just need to lock down a date. 1st or
>2nd Saturday of June work?
>
> +1 to Google hangout.
>
> Probably best if people do a SVN checkout before we meetup.
>
> --Paul
>
> On May 17, 2012, at 10:45 AM, "Mattmann, Chris A (388J)" <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > Now *that* is a great idea bfost. We can set up a Google Hangout for
> this.
> >
> > OK, then I am going to throw out my pool/house for the meetup location,
> > depending on what people think!
> >
> > Cheers,
> > Chris
> >
> > On May 16, 2012, at 11:32 PM, Brian Foster wrote:
> >
> >> google+ hangout 4 those out of state!
> >>
> >> On May 16, 2012, at 11:54 AM, "Mattmann, Chris A (388J)" <
> chris.a.mattm...@jpl.nasa.gov> wrote:
> >>
> >>> Hey Paul,
> >>>
> >>> This sounds like a great idea and I'm in (of course!)
> >>>
> >>> One thing we'll need to do is make sure that we do a really good job
> of trying
> >>> to include everyone, even folks that are not in the SoCal area. That
> means
> >>> potentially using the Wiki, and other forms of communication (JIRA,
> ReviewBoard, etc.)
> >>> to make sure everyone (including those not able to attend) feels
> included.
> >>>
> >>> Great idea!
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On May 15, 2012, at 2:21 PM, Ramirez, Paul M (388J) wrote:
> >>>
>  Hi All,
> 
>  Anyone interested in doing a meetup/hackathon at the beginning of
> June. The goal would be to get some updates to our code base but really
> focused on getting 0.4 out the door. If there is enough interest we would
> need a place to host it. Anyone and everyone would be welcome as it would
> be a good way to understand the ways to contribute which go well beyond
> just code. If enough people seem interested  I will work out the details.
> My initial thought is this would be on a weekend.
> 
>  --Paul
> >>>
> >>>
> >>> ++
> >>> Chris Mattmann, Ph.D.
> >>> Senior Computer Scientist
> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> Office: 171-266B, Mailstop: 171-246
> >>> Email: chris.a.mattm...@nasa.gov
> >>> WWW:   http://sunset.usc.edu/~mattmann/
> >>> ++
> >>> Adjunct Assistant Professor, Computer Science Department
> >>> University of Southern California, Los Angeles, CA 90089 USA
> >>> ++
> >>>
> >
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattm...@nasa.gov
> > WWW:   http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
>

Re: OODT SoCal Meetup/Hackathon

[Cameron Goodale]

If you schedule it, I will try to be there.  June I might be in Paris, or
on Holiday (no these are no the same thing).

Just need some good wiFi, so hopefully we can grab some from the neighbors.

_C

On Tue, May 15, 2012 at 9:08 PM, David Woollard  wrote:

> I think I could make a weekend in June work. We should do a Balmer
> Peak-a-thon (http://xkcd.com/323/).
>
> -Dave
>
> On May 15, 2012, at 5:21 PM, Ramirez, Paul M (388J) wrote:
>
> > Hi All,
> >
> > Anyone interested in doing a meetup/hackathon at the beginning of June.
> The goal would be to get some updates to our code base but really focused
> on getting 0.4 out the door. If there is enough interest we would need a
> place to host it. Anyone and everyone would be welcome as it would be a
> good way to understand the ways to contribute which go well beyond just
> code. If enough people seem interested  I will work out the details. My
> initial thought is this would be on a weekend.
> >
> > --Paul
>
>

Re: Resource Manager client question

[Cameron Goodale]

Hey Mike and Sheryl,

Mike was asking me for some similar advice and I plain ran outta talent on
this topic.  From what I can tell Mike would like to run his python scripts
on Resource Manager without the need for setting up Workflow or PGE.

At the time I hadn't really thought through all the configuration files
needed, but having stewed on it I thought I should reply.  Now my current
 SnowDS implementation is to have the Workflow Task reference a CAS-PGE
(which contains the execution block for my python program i want to run).
 Then the Workflow is merely configured to farm the jobs out to the
Resource Manager.

Here is a list of questions that I have started to wonder about with Mike's
help, any answers would be appreciated:

1.  Can Resource Manager + Batchstubs be used without any additional OODT
components?

2.  Is PGE required to run/wrap non-Java programs so they can run within
Resource Manager?

Closing comments to Mike:

If you are planning to use OODT for data management, it
is initially very tempting to only setup and configure the minimal set of
components because you will feel productive and it feels like progress is
being made.  Trust me I know since I was in your shoes about 6 months ago
when trying to get some image processing IDL code to run and I bably needed
to see progress (notice I didn't use the works "make progress").  Because I
wanted to use (what I thought was) the "easier" solution I ended up
hardcoding paths to resources my python code needed in the code instead of
passing the parameters into the code in the first place.  This worked
reasonably well as long as everything stayed the samebut then it didn't
so I had to re-visit my "easier" setup and fix it.
Recently I have been working to undo my mistakes and python has been
very forgiving, but the best part was that  all the strange and mystic
Workflow setups and PGEConfig.xml files actually started to make a whole
lot more sense.  I am now able to configure and stand up a complete
workflow config, then jump into PGEConfig and get the input parameters to
my python code.  This means if the input files i need to process changes I
don't need to change my python code, instead I can merely pass in a
different set of parameters into the workflow and they will persist to my
wrapped python.
In short I know that combing through all the xml config is tough,
especially when things are not working as quickly as you would like.  I
understand how defeated and frustrating it can be to have the component
fail and just feel lost, not knowing what is causing the problem.  I know
the documentation isn't perfect and sometimes it is missing altogether, but
the people that are on this list will bend over backwards to help you
understand (some will even share their config files with line-by-line
comments included at no extra charge ;)

Thank you Sheryl for being awesome and helpful (you always are).  Mike keep
the questions coming and I will be sure to add in my $0.02 when I am able
to.

Best Regards,


Cameron

On Mon, May 7, 2012 at 5:09 PM, Sheryl John  wrote:

> Hi Mike,
>
> Yup, you can run your python scripts, java programs etc. from CAS-PGE which
> is used with the Workflow Manager. Check out this cas-pge guide [1] and the
> other wiki pages related to workflow.
>
> You can use Resource Manager to run tasks sent from the Workflow Manager.
> I've recently started testing this but there are others on the list who can
> guide you more on the Resource Manager.
>
> HTH!
>
> Sheryl
>
> [1] https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>
>
> On Mon, May 7, 2012 at 3:43 PM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] <
> michael.iwu...@nasa.gov> wrote:
>
> >
> > I have two questions, I am able to run the Resource Manager with no
> > issues. I have some python scripts and possibly some other programs  I
> > would like to run using the Resource Manager. From what I know so far I
> > believe the cas-pge component needs to be used in conjunction with the
> > Resource Manager and is used as a wrapper program for running my scripts.
> > Can someone give me more information on how this can be accomplished or
> are
> > there any examples to view?
> >
> >  I would also like to be able to utilize the Job Scheduler, Monitor and
> > Job queue classes that are part of the Resource Manager. I can't find any
> > examples of how they are used anywhere. And if examples do exist can
> > someone point me in the right direction or give me more information on
> this?
> >
> > Thanks
> >
> > Mike
>
>
>
>
> --
> -Sheryl
>

Added a Resource Manager FAQ Page to the Wiki

[Cameron Goodale]

Hey OODT Community,

Last night I was pulling my hair out trying to figure out why Resource
Manager was falling over on my project.  This morning I finally found the
error in my policy setups.  With solution in place I was encouraged to
start an FAQ (hat tip: Chris Mattmann) page on the Wiki to help others
avoid repeating my mistake.

The page is here:
https://cwiki.apache.org/confluence/display/OODT/FAQ+-+Resource+Manager

It has been linked from the Main Wiki Page as well.  If you have other
gotcha's or anything else to add, please feel free to do so.  I tried to
include the full error message in hopes that Google would index it and
direct others to the page.

Prost!


Cameron

Re: Resource Manager Info.

[Cameron Goodale]

Hey Mike,

I don't have any formal documentation on the Resource Manager, but I can
try to explain how the Snow Data System team is using it to process large
amounts of MODIS MOD09GA tile data.  From there it might be easier to just
ask additional questions since the Components are a lot like legos and you
can swap them around to make several different systems in the end.

On Snow (Snow Data System Project) we have 5 Servers (1 VM, and 4 Work
Horses with 48 Cores Each).  The Workflow Manager and Resource Manager are
installed on the VM and it is sort of like a head node where it does all
the Job Creation and Submission, but it does none of the tile processing.
 Here is a list of high-level setups/config that enable us to run the
MOSCAG algorithm across 3 of the 4 Work Horse Machines (the 4th Machine is
used for another processing pipeline, so we will ignore it for now).


   1. On the Head Node (VM) we have a Workflow Manager (WM) configured to
   Create MODSCAG Workflows.
   2. On the Head Node we also have a Resource Manager (RM) configured to
   send jobs out to BatchStubs on the 3 Work Horse Machines.
   3. On the Head Node the WM is configured to USE the RM.  This means the
   WM will submit jobs to the RM until the RM has reached Capacity, then the
   Workflow Manager will queue any additional jobs and wait for the RM to have
   additional Capacity by completing a Job.
   4. Our MODSCAG Jobs run in a Quad Core Process, so we have configured
   the RM to only submit 10 Jobs to each Work Horse machine (hence using 40 of
   the 48 cores).  This is unique to the image processing code base, your
   mileage (and core usage) may vary.

So I hope this helps.  Again your configuration might be that you have 100
4-core machines that you want the RM to use for job processing, or you may
want to stand up BatchStubs on your localhost for testing purposes.  At
that point it really is just a matter of getting the configuration setup
and restarting the components.

If anyone else in the community has anything to add or comment, that would
be great.  And Mike if you have any questions at all, don't hesitate to ask.

Best Regards,


Cameron


On Tue, Apr 17, 2012 at 9:52 AM, Iwunze, Michael C (GSFC-4700)[NOAA-JPSS] <
michael.iwu...@nasa.gov> wrote:

> Hello,
>
> I am one of the GRAVITE developers that is new to OODT. Currently
> researching the resource manager component, I saw the user guide online. Do
> you know of any other locations were I can get more information on this
> component as well as more examples and how it can be used?
>
> Thanks
>
> Mike
>



-- 

Sent from a Tin Can attached to a String

Re: OODT-402 progress update

[Cameron Goodale]

Ross,

I agree with Chris that you are doing a great job on this.  Thank you for
all the hard work in setting this up.  Having a standard geospatial
interface from the FileManager to SIS will prove extremely helpful to
several JPL/NASA Earth Science Missions in the future.

Just wanted to say thanks!



Cameron

On Sat, Mar 31, 2012 at 2:47 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> (sorry for the cross post, but of interest to both communities)
>
> Hey Ross,
>
> That is awesome to hear! Comments below:
>
> On Mar 31, 2012, at 4:04 AM, Ross Laidlaw wrote:
>
> > Hi Chris,
> >
> > Many thanks for your comments and advice.  I have now installed SIS on
> > my system and I have the demo up and running.  I've also been
> > experimenting with the connection between File Manager and SIS and I
> > have a basic connection working.  Here are the steps I have carried so
> > far:
> >
> > Firstly, I added elements for 'Latitude' and 'Longitude' to the
> > GenericFile policy.  Then I created a set of test files (simple text
> > files) with corresponding '.met' metadata files.  I added 'Latitude'
> > and 'Longitude' metadata to the '.met' files.  I then ingested these
> > into my File Manager repository.
>
> That's perfect.
>
> >
> > After that, I enabled the 'georss' tags in the 'rssconf.xml'
> > configuration file (I added tags for geo:lat and geo:long to start
> > with) and viewed the RSS output from the CAS REST API in my browser.
> >
> > I then added the URL
> > ('http://localhost:8080/fmprod/viewRecent?channel=ALL') as an RSS feed
> > to the 'sis-location-config.xml' config in my SIS webapp.
> >
> > Next, I ran a bounding box query from the SIS demo.jsp and my file
> > manager files appeared on the map!  Awesome!
>
> That is SWEET!
>
> >
> > I have uploaded a set of screenshots showing my progress, hopefully
> > they're accessible and make sense (I'm new to flickr, so apologies if
> > things look a bit disorganised!):
> >
> >
> http://www.flickr.com/photos/rlaidlaw/sets/72157629342438940/with/7031581313/
>
> They look great, I checked them out. Would you be willing to do the
> following?
>
> 1. Log onto the SIS wiki and add a little tutorial (small) based on the
> above email
> and screenshots?
>
> wiki here: https://cwiki.apache.org/confluence/display/SIS/Home
>
> 2. Link the SIS tutorial via the OODT wiki, here:
>
> https://cwiki.apache.org/confluence/display/OODT/Home
>
> >
> > I tried to order the screenshots to follow the steps I outlined above,
> > with the final screenshot being the output from the SIS demo showing
> > the File Manager files on the map.
>
> That's awesome.
>
> >
> > For next steps, I could start to formalize some of my experimental
> > code and begin to address the key points you have outlined on the
> > OODT-402 jira page (i.e. writing a default GeoRSS config and a default
> > file manager policy for a 'LocationAwareProduct' type).
>
> That would be perfect!
>
> >  I will follow
> > up with specific questions about both of these, but for now I just
> > wanted to give you an update on my progress and to check that I'm
> > heading in the right direction.
>
> You are definitely heading in the right direction that's precisely what I
> was
> hoping you would figure out how to do. Thank you Ross for the update
> and your results look VERY promising, great job so far!
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>

Re: question on push pull

[Cameron Goodale]

Luca,

Good catch.  I was typing faster than my brain was working.  I meant to say
Push Pull instead of Crawler.

Sorry for the confusion.  Chris laid it all out really well.  Guess I was
just a little to excited to answer you and goofed in my haste.

To answer your question, I am not sure where the code is in PUSHPULL (got
it right this time ;) since I haven't had reason to use it yet.

I did check the etc/examples directory in PushPull and I only found file,
ftp, and sftp examples.  If you do sort out the scp and scp -r versions it
would be great to add them to the etc/examples area.

It could be part of the Apache Extras that Chris mentioned:
http://code.google.com/a/apache-extras.org/p/oodt-pushpull-plugins/

Good Luck.



-Cameron

On Wed, Feb 29, 2012 at 10:23 AM, Cinquini, Luca (3880) <
luca.cinqu...@jpl.nasa.gov> wrote:

> Hi Cameron,
>Maybe I am confused, but I was actually asking about the push-pull
> capabilities - does the crawler plug into the push-pull framework ? (sorry
> about my ignorance here). If push-pull supports scp, would you know the
> name of the protocol transfer factory to use - I haven't found one.
> thanks a lot,
> Luca
>
> On Feb 28, 2012, at 8:40 AM, Cameron Goodale wrote:
>
> > Luca,
> >
> > I haven't tried this exact use case within Crawler, but Crawler does
> > support scp and I have used 'scp -r' to recursively download a folder and
> > all content housed within.  I can only imagine ftp has a similar
> recursive
> > option as well.
> >
> > Maybe another more Crawler Savy dev can shine some light on the recursion
> > use case when using Crawler.
> >
> > -Cameron
> >
> > P.S. When we get a final answer let's add this to the Crawler User Guide
> > Wiki too as an example use case.  Glad you found the Crawler Wiki page
> > useful.
> >
> > On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) <
> > luca.cinqu...@jpl.nasa.gov> wrote:
> >
> >> Hi all,
> >>   I have a quick question concerning the pushpull framework : is
> >> there any way to transfer full directory trees, as opposed to single
> files
> >> ? And which of the currently implemented transfer protocols would allow
> >> that ? I haven't see any examples on that, though I might have missed
> it.
> >>
> >> thanks a lot,
> >> Luca
> >>
> >> P.S.: Cameron, thanks for writing the push-pull user guide - it's great.
> >>
> >>
> >
> >
> > --
> >
> > Sent from a Tin Can attached to a String
>
>


-- 

Sent from a Tin Can attached to a String

Re: question on push pull

[Cameron Goodale]

Luca,

I haven't tried this exact use case within Crawler, but Crawler does
support scp and I have used 'scp -r' to recursively download a folder and
all content housed within.  I can only imagine ftp has a similar recursive
option as well.

Maybe another more Crawler Savy dev can shine some light on the recursion
use case when using Crawler.

-Cameron

P.S. When we get a final answer let's add this to the Crawler User Guide
Wiki too as an example use case.  Glad you found the Crawler Wiki page
useful.

On Tue, Feb 28, 2012 at 7:01 AM, Cinquini, Luca (3880) <
luca.cinqu...@jpl.nasa.gov> wrote:

> Hi all,
>I have a quick question concerning the pushpull framework : is
> there any way to transfer full directory trees, as opposed to single files
> ? And which of the currently implemented transfer protocols would allow
> that ? I haven't see any examples on that, though I might have missed it.
>
> thanks a lot,
> Luca
>
> P.S.: Cameron, thanks for writing the push-pull user guide - it's great.
>
>


-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-393) Column Based Catalog for Filemanager

[Cameron Goodale]

I think this is a great idea to help bring more JPL code to Apache so
everyone can benefit.

Cameron
On Feb 27, 2012 8:20 PM, "Chris A. Mattmann (Commented) (JIRA)" <
j...@apache.org> wrote:

>
>[
> https://issues.apache.org/jira/browse/OODT-393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217879#comment-13217879]
>
> Chris A. Mattmann commented on OODT-393:
> 
>
> Hey Michael, I'll be happy to work with you to shepherd this in.
>
> > Column Based Catalog for Filemanager
> > 
> >
> > Key: OODT-393
> > URL: https://issues.apache.org/jira/browse/OODT-393
> > Project: OODT
> >  Issue Type: New Feature
> >  Components: file manager
> >Affects Versions: 0.4
> >Reporter: Michael Starch
> >Assignee: Chris A. Mattmann
> >Priority: Minor
> >
> > This issue is to port the ColumnedBasedCatalog back-end that works with
> the JPL filemanager to a place at Apache.
> > It uses a column-based table structure in the database back-end, which
> is more efficient for most databases as it is a true relational structure.
>  This will also add in the "ComplexQuery" feature that exists in the JPL
> branch.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

Re: Question about xmlrpc

[Cameron Goodale]

Rough python code is now attached so others can download and see how I am
throttling the connections now.

-Cam

On Thu, Feb 23, 2012 at 8:37 AM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hey Cam,
>
> I'd consider posting your script as-is to the JIRA issue simply as a
> patch. That doesn't mean
> you'll commit the patch outright, but JIRA is a good way to share code in
> the open (read: not
> over email) where everyone else can benefit from your ability to pass a
> URL around.
>
> Cheers,
> Chris
>
> On Feb 22, 2012, at 5:30 PM, Cameron Goodale wrote:
>
> > Chris,
> >
> > I think the lsof guard could probably be added into the existing
> wmgr-client shell script and possibly activated via some option like number
> of connections.  The Python I have attached is extremely closely tied to
> running MODSCAG jobs for snow.  I will create a JIRA issue for the
> enhancement to check for connections within the wmgr-client script.
> >
> > Michael,
> >
> > A few key areas in the script you might need to tweak are as follows:
> >
> > Here you will want to swap out your lsof exec for mine, and we run
> workflow mgr on port 9001
> > 37   cmd = '/usr/sbin/lsof -i :9001 | grep ESTABLISHED | wc'
> >
> > Line 48 - That is our implementation specific wmgr-client command string
> and pass in out specific Metadata Key/Values to kick off the modscagv2
> event.
> >
> > One last note:
> > I miss spoke in my previous email since i was working from memory.  In
> this code I am checking the number of connections to the workflow manager
> (localhost:9001) after 8 jobs are submitted, and when I find over 30
> connections we sleep for 60 seconds.  Since we are using the resource
> manager as well, the number of connections to the workflow manager can
> climb pretty quickly, so this was a pretty conservative setup.  Feel free
> to tweak it to your liking, and if you have any questions let me know.
> >
> > You could probably shorten up lines 59 - 76 based on your use case (and
> looking back at the code again I could clean mine up as well).
> >
> > Hope this helps.
> >
> >
> > -Cameron
> >
> >
> > On Wed, Feb 22, 2012 at 2:18 PM, Mattmann, Chris A (388J) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
> > Hey Cam,
> >
> > On Feb 22, 2012, at 8:02 AM, Cameron Goodale wrote:
> >
> > > Irina et al,
> > >
> > > I have run into this same issue with exceeding the 100 connections to
> > > xml-rpc filemanager, and workflow manager and here were the steps I
> took to
> > > try and avoid hitting the limit ever.
> > >
> > > 1.  Optimize the Lucene Index (if you are using the Lucene Catalog and
> have
> > > 100,000's of entries this can help improve how quickly your requests
> are
> > > handled and this will free up used connections faster).
> > > 2.  I used lsof to detect how many active connections where made to the
> > > FileManager, if the number exceeded 85 (to be safe) I would have my
> > > submission code sleep for 10 seconds and try again.  Not the most
> > > performant fix, but I never lose a job submission.
> > >
> > > Option 2 was written in Python, and if you want a copy of it just let
> me
> > > know and will be happy to fwd it along.
> >
> > How about a JIRA issue in Apache OODT ville and a devops script,
> > committed to  trunk/workflow/src/main/bin?
> >
> > :)
> >
> > Cheers,
> > Chris
> >
> > >
> > > On Tue, Feb 21, 2012 at 2:21 PM, holenoter  wrote:
> > >
> > >>
> > >> hey irina,
> > >>
> > >> how many retries do you have set for each task and how long do is your
> > >> interval between retries?
> > >>
> > >> -brian
> > >>
> > >> On Feb 21, 2012, at 09:56 AM, "Tkatcheva, Irina N (388D)" <
> > >> irina.n.tkatch...@jpl.nasa.gov> wrote:
> > >>
> > >> Hi Brian and all,
> > >>
> > >> I have noticed that the system does recover after the "System
> overload:
> > >> Maximum number of concurren trequests (100) exceeded" message, but
> usually
> > >> some jobs stay in 'Waiting on resource (executing)' condition and
> never
> > >> proceed further. I have seen it every time after the overload
> messages. I
> > >> usually run a test that runs a bunch of jobs overnight. If there is no
> > >> overload messages, all j

Re: Question about xmlrpc

[Cameron Goodale]

Irina,

I just sent the python script to the dev listbut I am now thinking it
was stripped off.  I will email a copy directly to you and Michael as a
back up.

-Cameron

On Wed, Feb 22, 2012 at 4:35 PM, Tkatcheva, Irina N (388D) <
irina.n.tkatch...@jpl.nasa.gov> wrote:

> Hi Cameron,
>
> Thank you for your input.
>
> I think your script could be useful in the operations (but it is for
> Michael (Starch) to decide). In any case, please forward it to us.
>
> Thanks,
>
> Irina
>
> On Feb 22, 2012, at 7:02 AM, Cameron Goodale wrote:
>
> > Irina et al,
> >
> > I have run into this same issue with exceeding the 100 connections to
> > xml-rpc filemanager, and workflow manager and here were the steps I took
> to
> > try and avoid hitting the limit ever.
> >
> > 1.  Optimize the Lucene Index (if you are using the Lucene Catalog and
> have
> > 100,000's of entries this can help improve how quickly your requests are
> > handled and this will free up used connections faster).
> > 2.  I used lsof to detect how many active connections where made to the
> > FileManager, if the number exceeded 85 (to be safe) I would have my
> > submission code sleep for 10 seconds and try again.  Not the most
> > performant fix, but I never lose a job submission.
> >
> > Option 2 was written in Python, and if you want a copy of it just let me
> > know and will be happy to fwd it along.
> >
> > Good luck.
> >
> >
> > -Cameron
> >
> > On Tue, Feb 21, 2012 at 2:21 PM, holenoter  wrote:
> >
> >>
> >> hey irina,
> >>
> >> how many retries do you have set for each task and how long do is your
> >> interval between retries?
> >>
> >> -brian
> >>
> >> On Feb 21, 2012, at 09:56 AM, "Tkatcheva, Irina N (388D)" <
> >> irina.n.tkatch...@jpl.nasa.gov> wrote:
> >>
> >> Hi Brian and all,
> >>
> >> I have noticed that the system does recover after the "System overload:
> >> Maximum number of concurren trequests (100) exceeded" message, but
> usually
> >> some jobs stay in 'Waiting on resource (executing)' condition and never
> >> proceed further. I have seen it every time after the overload messages.
> I
> >> usually run a test that runs a bunch of jobs overnight. If there is no
> >> overload messages, all jobs are completed; if there are overload
> messages,
> >> usually in the morning some jobs are stuck in 'Waiting on resource
> >> (executing)' state. So it looks to me that the system does not recover
> >> completely.
> >>
> >> Irina
> >>
> >>
> >>
> >> On Feb 17, 2012, at 9:17 AM, Brian Foster wrote:
> >>
> >> Hey Chris,
> >>
> >> ya I'm in favor of adding the property but let's make it use 100 by
> >> default if the property is not set and I would even say let's add it to
> the
> >> properties file but comment it out or something.. that's a really
> advanced
> >> flag which only needs to be changed to get rid of that logging
> message...
> >> CAS works fine even when that message is being thrown... I think it
> prints
> >> to sndout, otherwise I would have just turned the logging for that off
> back
> >> when I added the client retry handlers that fixed the issue... oh and
> this
> >> is another thing your probably gonna want to port to trunk workflow :)
> >>
> >> -Brian
> >>
> >> "Mattmann, Chris A (388J)"  >> chris.a.mattm...@jpl.nasa.gov>> wrote:
> >>
> >> Thanks Brian, I was thinking this too, +1, which is why I cautioned
> >> against any number greater than 256
> >> in terms of thread count in my reply email too, since the risk is either
> >> that (a) you have to increase the
> >> ulimit (which extends the boundaries from devops oriented updates to
> >> sysops on the sysadmin side);
> >> and (b) the JVM will likely start trashing unless there is an inordinate
> >> amount of RAM, or swap space, etc.
> >>
> >> I think the best solution here is to simply make it a configurable
> >> property and then encourage projects
> >> to use a sensible default that's not too large...
> >>
> >> Cheers,
> >> Chris
> >>
> >> On Feb 16, 2012, at 12:52 AM, Brian Foster wrote:
> >>
> >> You have to be careful with the number you set that too because you are
> >> basically tellin

Re: Question about xmlrpc

[Cameron Goodale]

Chris,

I think the lsof guard could probably be added into the existing
wmgr-client shell script and possibly activated via some option like number
of connections.  The Python I have attached is extremely closely tied to
running MODSCAG jobs for snow.  I will create a JIRA issue for the
enhancement to check for connections within the wmgr-client script.

Michael,

A few key areas in the script you might need to tweak are as follows:

Here you will want to swap out your lsof exec for mine, and we run workflow
mgr on port 9001
37   cmd = '/usr/sbin/lsof -i :9001 | grep ESTABLISHED | wc'

Line 48 - That is our implementation specific wmgr-client command string
and pass in out specific Metadata Key/Values to kick off the modscagv2
event.

One last note:
I miss spoke in my previous email since i was working from memory.  In this
code I am checking the number of connections to the workflow manager
(localhost:9001) after 8 jobs are submitted, and when I find over 30
connections we sleep for 60 seconds.  Since we are using the resource
manager as well, the number of connections to the workflow manager can
climb pretty quickly, so this was a pretty conservative setup.  Feel free
to tweak it to your liking, and if you have any questions let me know.

You could probably shorten up lines 59 - 76 based on your use case (and
looking back at the code again I could clean mine up as well).

Hope this helps.


-Cameron


On Wed, Feb 22, 2012 at 2:18 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hey Cam,
>
> On Feb 22, 2012, at 8:02 AM, Cameron Goodale wrote:
>
> > Irina et al,
> >
> > I have run into this same issue with exceeding the 100 connections to
> > xml-rpc filemanager, and workflow manager and here were the steps I took
> to
> > try and avoid hitting the limit ever.
> >
> > 1.  Optimize the Lucene Index (if you are using the Lucene Catalog and
> have
> > 100,000's of entries this can help improve how quickly your requests are
> > handled and this will free up used connections faster).
> > 2.  I used lsof to detect how many active connections where made to the
> > FileManager, if the number exceeded 85 (to be safe) I would have my
> > submission code sleep for 10 seconds and try again.  Not the most
> > performant fix, but I never lose a job submission.
> >
> > Option 2 was written in Python, and if you want a copy of it just let me
> > know and will be happy to fwd it along.
>
> How about a JIRA issue in Apache OODT ville and a devops script,
> committed to  trunk/workflow/src/main/bin?
>
> :)
>
> Cheers,
> Chris
>
> >
> > On Tue, Feb 21, 2012 at 2:21 PM, holenoter  wrote:
> >
> >>
> >> hey irina,
> >>
> >> how many retries do you have set for each task and how long do is your
> >> interval between retries?
> >>
> >> -brian
> >>
> >> On Feb 21, 2012, at 09:56 AM, "Tkatcheva, Irina N (388D)" <
> >> irina.n.tkatch...@jpl.nasa.gov> wrote:
> >>
> >> Hi Brian and all,
> >>
> >> I have noticed that the system does recover after the "System overload:
> >> Maximum number of concurren trequests (100) exceeded" message, but
> usually
> >> some jobs stay in 'Waiting on resource (executing)' condition and never
> >> proceed further. I have seen it every time after the overload messages.
> I
> >> usually run a test that runs a bunch of jobs overnight. If there is no
> >> overload messages, all jobs are completed; if there are overload
> messages,
> >> usually in the morning some jobs are stuck in 'Waiting on resource
> >> (executing)' state. So it looks to me that the system does not recover
> >> completely.
> >>
> >> Irina
> >>
> >>
> >>
> >> On Feb 17, 2012, at 9:17 AM, Brian Foster wrote:
> >>
> >> Hey Chris,
> >>
> >> ya I'm in favor of adding the property but let's make it use 100 by
> >> default if the property is not set and I would even say let's add it to
> the
> >> properties file but comment it out or something.. that's a really
> advanced
> >> flag which only needs to be changed to get rid of that logging
> message...
> >> CAS works fine even when that message is being thrown... I think it
> prints
> >> to sndout, otherwise I would have just turned the logging for that off
> back
> >> when I added the client retry handlers that fixed the issue... oh and
> this
> >> is another thing your probably gonna want to port to trunk workflow :)
> >>
> >> -Brian

Re: Question about xmlrpc

[Cameron Goodale]

Irina et al,

I have run into this same issue with exceeding the 100 connections to
xml-rpc filemanager, and workflow manager and here were the steps I took to
try and avoid hitting the limit ever.

1.  Optimize the Lucene Index (if you are using the Lucene Catalog and have
100,000's of entries this can help improve how quickly your requests are
handled and this will free up used connections faster).
2.  I used lsof to detect how many active connections where made to the
FileManager, if the number exceeded 85 (to be safe) I would have my
submission code sleep for 10 seconds and try again.  Not the most
performant fix, but I never lose a job submission.

Option 2 was written in Python, and if you want a copy of it just let me
know and will be happy to fwd it along.

Good luck.


-Cameron

On Tue, Feb 21, 2012 at 2:21 PM, holenoter  wrote:

>
> hey irina,
>
> how many retries do you have set for each task and how long do is your
> interval between retries?
>
> -brian
>
> On Feb 21, 2012, at 09:56 AM, "Tkatcheva, Irina N (388D)" <
> irina.n.tkatch...@jpl.nasa.gov> wrote:
>
> Hi Brian and all,
>
> I have noticed that the system does recover after the "System overload:
> Maximum number of concurren trequests (100) exceeded" message, but usually
> some jobs stay in 'Waiting on resource (executing)' condition and never
> proceed further. I have seen it every time after the overload messages. I
> usually run a test that runs a bunch of jobs overnight. If there is no
> overload messages, all jobs are completed; if there are overload messages,
> usually in the morning some jobs are stuck in 'Waiting on resource
> (executing)' state. So it looks to me that the system does not recover
> completely.
>
> Irina
>
>
>
> On Feb 17, 2012, at 9:17 AM, Brian Foster wrote:
>
> Hey Chris,
>
> ya I'm in favor of adding the property but let's make it use 100 by
> default if the property is not set and I would even say let's add it to the
> properties file but comment it out or something.. that's a really advanced
> flag which only needs to be changed to get rid of that logging message...
> CAS works fine even when that message is being thrown... I think it prints
> to sndout, otherwise I would have just turned the logging for that off back
> when I added the client retry handlers that fixed the issue... oh and this
> is another thing your probably gonna want to port to trunk workflow :)
>
> -Brian
>
> "Mattmann, Chris A (388J)"  chris.a.mattm...@jpl.nasa.gov>> wrote:
>
> Thanks Brian, I was thinking this too, +1, which is why I cautioned
> against any number greater than 256
> in terms of thread count in my reply email too, since the risk is either
> that (a) you have to increase the
> ulimit (which extends the boundaries from devops oriented updates to
> sysops on the sysadmin side);
> and (b) the JVM will likely start trashing unless there is an inordinate
> amount of RAM, or swap space, etc.
>
> I think the best solution here is to simply make it a configurable
> property and then encourage projects
> to use a sensible default that's not too large...
>
> Cheers,
> Chris
>
> On Feb 16, 2012, at 12:52 AM, Brian Foster wrote:
>
> You have to be careful with the number you set that too because you are
> basically telling XML-RPC that it is now allowed to create 2000 threads in
> the same JVM... not a good practice... I don't remember the exact number
> but the JVM will crash if it creates a certain number of threads because
> there is a limit to the number of threads one process can create and I
> believe this is restricted at the operating system level... and i believe
> this number is less than 2000... The trunk filemgr and wengine already have
> built-in client retry handling support and are configurable via java
> properties (i.e.
> org.apache.oodt.cas.filemgr.system.xmlrpc.connection.retries and
> o.a.o.c.filemger.system.connection.retry.interval.seconds and there are
> similar ones for wengine)... The message you are seeing is XML-RPC server
> logging that it already using a 100 worker threads... you will see this
> message if you create a 100+ jobs in the RM (e.g. Workflow Conditions and
> Tasks) and they all start talking to the workflow manager or file manger at
> the same time... the client retry handlers will catch this error and just
> wait and retry again... you shouldn't be loosing any data... the only
> inconvenience I guess is that message is cluttering the logs
>
> -Brian
>
> On Feb 15, 2012, at 10:42 PM, "Cheng, Cecilia S (388K)" <
> cecilia.s.ch...@jpl.nasa.gov> wrote:
>
>
> Hi Chris,
>
> Sure we can discuss this in dev@oodt.apache.org >.
>
> If you feel comfortable w/ the 2000 number, of course I can push the patch
> upstream into Apache OODT. But what kind of tests, if any, should we do
> before we deliver the patch? Our projects are concerned that if we
> arbitrarily set a number, we don't know what other problems it might cause.
>
> Thanks,
> 

Re: [jira] [Created] (OODT-380) Pypes UX

[Cameron Goodale]

Adam,

I am digging this tool, especially since right now I am in working in SNOW
workflow heaven and i think a visual tool would really help everyone see
what is happening in the pipeline.

The License is ALv2 so that is a good start, and it is written in Python ;)

I would be up for throwing a couple hours at trying to set it up and
install in on my machine and test it out.

-Cam

On Wed, Feb 15, 2012 at 5:01 PM, Adam Estrada (Created) (JIRA) <
j...@apache.org> wrote:

> Pypes UX
> 
>
> Key: OODT-380
> URL: https://issues.apache.org/jira/browse/OODT-380
> Project: OODT
>  Issue Type: New Feature
>  Components: website
>Affects Versions: 0.3
> Environment: All
>Reporter: Adam Estrada
>Priority: Minor
> Fix For: 0.4
>
>
> What would you think about integrating http://www.pypes.org/ as the UI
> for the workflow manager? Seems like a decent idea as most of OODT is about
> building workflows, right?
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators:
> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>

OODT Meet Up at PyCon 2012

[Cameron Goodale]

Hey OODTers,

Paul R and I will be attending PyCon in Santa Clarita from March 7, 2012 to
March 11, 2012.

We are planning to organize an OODT/ASF meet up with anyone else on this
mailing list who will also be attending PyCon this year.  The tentative
meet-up will happen on Friday March 10th after the first day of sessions.
 The idea is to select a venue that is within walking distance of the
conference so everyone can attend without a lot of logistic head-aches.

If you have any questions, comments, or want to confirm you are attending
PyCon please reply-all to this message.

Details on PyCon:
https://us.pycon.org/2012/about/

Thanks for your Time,


-Cameron

Re: SOPA & PIPA for OODT and other 4-letter acronyms

[Cameron Goodale]

Sean,

You have my +1.  We only have 4 days left if we are gonna do this.  Do you
have any idea's how we can make the site updates needed?

I would imagine this would require majority support from the OODT PMC as
well.

-Cameron

On Sat, Jan 14, 2012 at 1:15 AM, Mark Struberg  wrote:

> basically +1
>
> we could do this with CSS tricks, to overlay a png applied to  with
> a 'black hole' graphic which gets a higher Z axis or kind of trick.
> We should not need to change each and every page if possible.
>
> LieGrue,
> strub
>
>
> - Original Message -
> > From: "Mattmann, Chris A (388J)" 
> > To: "dev@oodt.apache.org" 
> > Cc: "memb...@apache.org" 
> > Sent: Saturday, January 14, 2012 4:57 AM
> > Subject: Re: SOPA & PIPA for OODT and other 4-letter acronyms
> >
> > +1 from me.
> >
> > Cheers,
> > Chris
> >
> > On Jan 13, 2012, at 6:43 PM, Sean Kelly wrote:
> >
> >>  Colleagues:
> >>
> >>  The Internet is a global resource and I'm thankful that the majority of
> > our Apache and OODT users needn't put up with any of the shenanigans that
> > the governments of Syria, Iran, and China employ routinely to throttle
> the free
> > distribution of software, open exchange of ideas, and unmoderated dialog
> of
> > critique deployed against their respective peoples.
> >>
> >>  Sadly, the United States of America is considering legislation [1],
> [2] in
> > its dual legislative houses of government that is on par with these
> oppressive
> > regimes[3]—all in a misguided attempt to stop piracy of both durable and
> logical
> > goods.
> >>
> >>  While we are an international organization that freely distributes
> software
> > beyond national borders, we've got to admit that the practices employed
> > within the United States has global ramifications, especially since our
> > software, being free and open source, enjoys a worldwide audience and
> runs the
> > majority of the web's infrastructure.
> >>
> >>  Other efforts within the Apache Software Foundation have already
> stepped
> > forward with visible, tangible displays of their commitment against the
> SOPA and
> > PIPA acts currently under consideration by the USA House of
> Representatives and
> > Senate, respectively.
> >>
> >>  Some of these efforts have elected to participate in an "internet
> > strike" by displaying banners and/or "going dark" on the 18th of
> > January, 2012.
> >>
> >>  I would strongly urge the Object Oriented Data Technology (OODT)
> project of
> > the Apache Software Foundation (ASF) to join in this solidarity and
> demonstrate
> > a commitment towards the unregulated and open exchange of ideas that we
> expect
> > on the internet.
> >>
> >>  Towards that end, I propose we shutter the OODT website at
> > http://oodt.apache.org/ and replace every page accessible under that
> domain with
> > a singular white-text-on-black-background that conveys, without any
> doubt, our
> > commitment to an open, un-regulated, free-of-government control internet
> that
> > eschews the ideas of both SOPA and PIPA.
> >>
> >>  I hope OODT will join in this protest. And, if it is at all possible,
> ALL
> > of apache.org will follow suit. The Internet is at a vital nexus point,
> and we
> > face an opportunity for change we dare not squander.
> >>
> >>  Yours faithfully,
> >>  Sean.
> >>
> >>
> >>  [1] http://en.wikipedia.org/wiki/Stop_Online_Piracy_Act
> >>  [2] http://edrn.me/S
> >>  [3] http://edrn.me/t
> >>
> >
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattm...@nasa.gov
> > WWW:   http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
>

Re: PushPull Question....

[Cameron Goodale]

Brian,

Thanks for the info.  I don't have my eclipse setup properly to run that
java file as an application so I will have to try that with some help from
Paul or Chris next week perhaps.  I looked up the regex for Java and the \w
is supposed to be a word character matching alphanumeric and underscore.

I ran my regex against this online tool I found and it seems to be a valid
match.
http://www.regexplanet.com/advanced/java/index.html
Regex:  TES-Aura_L2-CO2-Nadir_r\d{10}\w{2}\d{2}\w\d{2}\.he5

Input:  TES-Aura_L2-CO2-Nadir_r002931_F06_08.he5


Thanks for the feedback and help on this.  I have tried so many different
tweaks with backslashes, etc.  I am going to just walk away now and look at
it later.


-Cam


On Tue, Dec 27, 2011 at 2:58 PM, Brian Foster  wrote:

> Hey cameron... I think the problem is with ur
> DirStructXmlParserFiles/TESL2CO2.xml file... for the file name regExp u are
> specifying whitespace (i.e. \w instead of underscore)... pushpull looks
> like it is behaving normally... it is acting like it is not finding any
> files that match ur regExps... I wrote an app which may help u with your
> regular expressions... in the pushpull util package in eclipse right click
> on ExpressionValidator.java and Run As: Java Application... you can test
> your regExp in the GUI that pops up.
>
> -brian
>
> Cameron Goodale  wrote:
>
> >Hey Devs,
> >
> >I have been working on building an OODT PushPull User Guide from an
> Example
> >I am trying to get working for a project, and I have encountered a little
> >issue (at least I hope it is little).
> >
> >Here is the guide (work in progress):
> >
> https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide
> >
> >The setups listed in the User Guide are the same ones I am working
> against.
> > Now here is the log messages I am seeing.
> >
> >Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
> >setAsRunning
> >INFO: Daemon with ID = 91131 was given permission to run
> >org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
> >Dec 27, 2011 10:59:00 AM
> >org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
> >INFO: Testing protocol
> >org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient . . . this
> >may take a few minutes . . .
> >Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.Protocol
> >cdToHOME
> >INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/isDir=true'
> >Dec 27, 2011 10:59:00 AM
> >org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
> >SEVERE: Protocol
> >org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient failed
> >compatibility test : [CogJGlobusFtpClient] Failed to get list of files :
> >Server refused performing the request. Custom message:  (error code 1)
> >[Nested exception message:  Custom message: Unexpected reply: 500 'ORT
> >97,93,139,127,210,161': command not understood.]
> >Dec 27, 2011 10:59:00 AM
> >org.apache.oodt.cas.pushpull.protocol.ProtocolHandler
> >getAppropriateProtocolBySite
> >WARNING: ProtocolFactory
> >org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClientFactory is
> not
> >compatible with server at ftp://l4ftl01.larc.nasa.gov
> >Dec 27, 2011 10:59:01 AM
> >org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
> >INFO: Testing protocol
> >org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient . . . this
> >may take a few minutes . . .
> >Dec 27, 2011 10:59:01 AM org.apache.oodt.cas.pushpull.protocol.Protocol
> >cdToHOME
> >INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/isDir=true'
> >Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol
> >cdToROOT
> >INFO: Changing to ROOT directory '/'
> >Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol
> >cdToHOME
> >INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/isDir=true'
> >Dec 27, 2011 10:59:02 AM
> >org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
> >INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with protocol
> >'org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient' and
> >username 'anonymous'
> >Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol cd
> >INFO: Changing to directory
> >'ftp://l4ftl01.larc.nasa.gov/TES/TL2CO2N.005isDir=true'
> >PageSize: 8 PageLoc: 0
> >FileList size: 491
> >Dec 27, 2011 10:59:03 AM
> >org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconn

PushPull Question....

[Cameron Goodale]

Hey Devs,

I have been working on building an OODT PushPull User Guide from an Example
I am trying to get working for a project, and I have encountered a little
issue (at least I hope it is little).

Here is the guide (work in progress):
https://cwiki.apache.org/confluence/display/OODT/OODT+Push-Pull+User+Guide

The setups listed in the User Guide are the same ones I am working against.
 Now here is the log messages I am seeing.

Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.daemon.DaemonManager
setAsRunning
INFO: Daemon with ID = 91131 was given permission to run
org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructXmlParser
Dec 27, 2011 10:59:00 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
INFO: Testing protocol
org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient . . . this
may take a few minutes . . .
Dec 27, 2011 10:59:00 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToHOME
INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/ isDir=true'
Dec 27, 2011 10:59:00 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
SEVERE: Protocol
org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClient failed
compatibility test : [CogJGlobusFtpClient] Failed to get list of files :
Server refused performing the request. Custom message:  (error code 1)
[Nested exception message:  Custom message: Unexpected reply: 500 'ORT
97,93,139,127,210,161': command not understood.]
Dec 27, 2011 10:59:00 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler
getAppropriateProtocolBySite
WARNING: ProtocolFactory
org.apache.oodt.cas.pushpull.protocol.ftp.CogJGlobusFtpClientFactory is not
compatible with server at ftp://l4ftl01.larc.nasa.gov
Dec 27, 2011 10:59:01 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler isOkProtocol
INFO: Testing protocol
org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient . . . this
may take a few minutes . . .
Dec 27, 2011 10:59:01 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToHOME
INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/ isDir=true'
Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToROOT
INFO: Changing to ROOT directory '/'
Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol
cdToHOME
INFO: Changing to HOME directory 'ftp://l4ftl01.larc.nasa.gov/ isDir=true'
Dec 27, 2011 10:59:02 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler connect
INFO: Successfully connected to ftp://l4ftl01.larc.nasa.gov with protocol
'org.apache.oodt.cas.pushpull.protocol.ftp.CommonsNetFtpClient' and
username 'anonymous'
Dec 27, 2011 10:59:02 AM org.apache.oodt.cas.pushpull.protocol.Protocol cd
INFO: Changing to directory
'ftp://l4ftl01.larc.nasa.gov/TES/TL2CO2N.005isDir=true'
PageSize: 8 PageLoc: 0
FileList size: 491
Dec 27, 2011 10:59:03 AM
org.apache.oodt.cas.pushpull.protocol.ProtocolHandler disconnect
INFO: Disconnecting protocol from ftp://l4ftl01.larc.nasa.gov
Dec 27, 2011 10:59:03 AM org.apache.oodt.cas.pushpull.daemon.Daemon sleep
INFO: Daemon with ID = 91131 on RMI registry port 9113 is going to sleep
until Tue Dec 27 11:00:00 PST 2011


First PushPull tries to use the CogJGlobusFtpClient and it encounters an
error so it switches to CommonsNetFtpClient does a couple cd commands,
connects successfully then disconnects and goes back to sleep.  It will
then wake up after the 3 minutes and repeat the same process.  Has anyone
seen this issue or have any suggestions for a fix?

Thank you in advance.

-Cameron

P.S.  If you have any feedback on the PushPull User Guide be sure to leave
comments or make changes to the wiki.

Re: [ANNOUNCE] Welcome Ricky Nguyen to the Apache OODT PMC and as a committer

[Cameron Goodale]

Welcome Ricky!

ApacheCon 2011 will go down in the History Books as a great time...from
pizza to Irish pubs+car-bombs.

Cheers!

-Cameron

On Thu, Dec 15, 2011 at 12:10 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> #YouAreFullOfWin RICKY!
>
> Glad to have you on board!
>
> On Dec 15, 2011, at 11:38 AM, Nguyen, Ricky wrote:
>
> > Hi all!
> >
> > Thanks for welcoming me into the OODT community so warmly and so
> quickly. It's been great working with Chris, Andrew, Cameron and Dan these
> past few months as they've helped me with the CHLA VPICU project (@vpicu)
> (along with Sheryl, Paul Vee, and Dave). I also met Paul Ramirez at
> ApacheCon in Vancouver, where my eyes were opened to the amazing Apache
> community. Yesterday I just met Steve Hughes who offered great insight into
> ontologies. Brian, Sean, Rishi, and Thomas,  I've seen your great work, on
> the mailing lists, in the code, and on the wikis. I look forward to working
> with you all and the others on this list.
> >
> > A bit about myself… I left the beaches of Hawaii for the snow of Boston
> to get my degree from MIT. Without professional sports in Hawaii, I became
> an adopted Red Sox fan and fond of all things Boston. My sister and parents
> moved to the Bay Area, so I joined them and worked for Oracle for 4 years.
> My girlfriend earned a pediatric residency at CHLA this fall, so I moved
> down and coincidentally the VPICU at CHLA had an amazing position open for
> me! I've been working with OODT ever since. I am sad that Bay Area football
> decided to suddenly become awesome as soon as I left, happy that the NBA
> lockout is over and CP3+Griffin=WIN, and excited to contribute to and learn
> from the OODT community! :D
> >
> > Ricky
> >
> > On Dec 14, 2011, at 1:20 PM, Mattmann, Chris A (388J) wrote:
> >
> >> Hi Folks,
> >>
> >> Ricky Nguyen has recently been elected to join our ranks as an Apache
> OODT PMC and
> >> committer. Welcome Ricky! Please feel free to share a bit about
> yourself.
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >> ++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattm...@nasa.gov
> >> WWW:   http://sunset.usc.edu/~mattmann/
> >> ++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++
> >>
> >
> >
> > 
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>

Re: CAS-CLI option standardiztion...

[Cameron Goodale]

I say as long as --help can tell me what I need to know to get it working
then we all win.

No matter which you choose, please keep up the great docs. Great docs are
always in fashion.

:-D
Cameron
On Nov 18, 2011 6:00 PM, "Mattmann, Chris A (388J)" <
chris.a.mattm...@jpl.nasa.gov> wrote:

> You rox!
>
> Cheers,
> Chris
>
> On Nov 18, 2011, at 5:23 PM, holenoter wrote:
>
> > oh that's right... filemgr... how could i forget about the filemgr???..
> i'll do that one next... i'll integrate all the random tools into the
> client's supported actions as well... such as the delete and query tools..
> then in the mean time we can talk about how we want the crawler
> command-line to look... i'll create an issue for both the filemgr and
> crawler with cas-cli so we have a place to keep this conversation going...
> onto the filemgr conversion then...
> >
> > -brian
> >
> > On Nov 18, 2011, at 03:01 PM, "Mattmann, Chris A (388J)" <
> chris.a.mattm...@jpl.nasa.gov> wrote:
> >
> >> Hey duder,
> >>
> >> On Nov 18, 2011, at 2:28 PM, holenoter wrote:
> >>
> >> >
> >> > hey guys,
> >> >
> >> > okay so the workflow and resource manager have been upgraded to using
> cas-cli... they maintained the same look as they had before...
> >>
> >> They look amazing. A true testament to the great generic cli framework
> you've made.
> >>
> >> > now the question i have is: the command-line look they have now, is
> that the look we want going forward or do we want to give the command-line
> a new look... if we decide to change the command-line, backward
> compatibility can be easily maintained by keeping the current cas-cli
> spring xml files around and all any user has to do is point at those spring
> files and workflow and resource manager command-line will work and look
> like it always has... i don't really have an opinion one way of the other,
> >>
> >> Yeah I'm kind of the same way. I'm used to the FM WM and RM working
> that way (with the --operation  step), so I'm not necessarily
> looking for them to change much. We also haven't had a lot of complaints.
> I've heard the occasional complaints about --operation, but I haven't seen
> a lot of folks willing to put in the effort that you have to get them all
> integrated and unified together, and/or changed.
> >>
> >> Because of that, I'm tentative to actually suggest they should be
> changed. Furthermore, I guess in a way we could consider the FM, WM and RM
> to
> >> be different beasts than e.g., crawler and/or pushpull and cas-pge
> since they are the core *services* and the crawler/pushpull and cas-pge are
> really
> >> client-side frameworks. See more below.
> >>
> >> > but my next component which is going to get cas-cli is the crawler,
> and it's command-line doesn't look like the workflow or resource manager
> (probably my fault)
> >>
> >> Nah, you were just ahead of your time per usual :-)
> >>
> >> > ... so i would like to make it either look like the workflow and
> resource manager (i.e. --operation --) or make the workflow and
> resource manager look more like the crawler... or given them all a "new"
> look... either way i would like to make all the components' command-line
> work in a way such that if you learn one, you learn them all... let me know
> what you guys think.
> >>
> >> I'm +1 for your statement about knowing one of them, and then knowing
> them all. However, I think you've done that from the perspective of having
> >> your statement now be true with FM, WM and RM. And I think you're going
> to do that for the other client frameworks and they will look the same.
> >>
> >> Having said that, the differences are pretty small (whether or not an
> --operation flag has to be passed to fire off the action with name
> , compared
> >> to baking in the  name into the --flag. That's not too
> different. I think being on cas-cli will help bring them all together in
> the long run
> >> anyways, but I don't see a huge value added from changing the CLI
> contracts as they have existed for a while. On the other hand, I see a HUGE
> advantage
> >> for standardizing the software framework with a flexible CLI framework
> like you've created to help that.
> >>
> >> My 2 cents.
> >>
> >> Cheers,
> >> Chris
> >>
> >> ++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattm...@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++
> >>
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-26

Re: Two File Manager Questions

[Cameron Goodale]

Tim,

Mattmann and I worked out the issue regarding underscores and hyphens.  I
logged a JIRA issue here:

https://issues.apache.org/jira/browse/OODT-350

and uploaded a patch, so you can patch your instance and get back to
running queries.  This bug fix will be added into the next release of
Apache OODT.

Thanks!


Cameron

P.S.  The reason Cecilia's code works is because it was patched by Brian
Foster.  In fact this patch is just a port of his code from branch back to
trunk.  So thanks to B.Foster and Enjoy!

On Thu, Nov 17, 2011 at 8:18 PM, Cameron Goodale  wrote:

> Tim,
>
> I just encountered the same issue when querying for a
> 'ProductType:TCCON-garmisch'.  This is throwing the same errors you got:
>
> WARNING: Query: [q=ProductType:TCCON AND ProductType:garmisch] for Product
> Type: [urn:co2:TCCON-garmisch] returned no results
> Nov 17, 2011 7:46:45 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
> paginateQuery
>
> As best I can figure the hyphen and underscore must be some kind of
> reserved character in Lucene.  We are running an older
> cas-filemgr-0.3-SNAPSHOT.jar on Linux CentOS 5.7
>
> Has anyone else encountered this issue besides Tim?  Also Tim what is the
> OS you are running on with your problem?
>
> Cecilia,
>
> What OS version are you running on where this is not an issue?  Also what
> version of fileManager are you using??
>
> Thanks!
>
>
>
> Cameron
>
>
>
> On Wed, Nov 9, 2011 at 10:40 AM, Stough, Timothy M (388F) <
> timothy.m.sto...@jpl.nasa.gov> wrote:
>
>> Cecilia,
>>
>> Thanks for the reply.  I tried your suggested query and got similar
>> results as before.  Here's what I tried...
>>
>> In preparation, I ingested a blank text file with the same product name
>> as in your example:
>>
>> [stough@aria1 bin]$ ./filemgr-client --url http://localhost:9000--operation 
>> --ingestProduct --productName
>> acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt
>> --productStructure Flat --productTypeName GenericFile --metadataFile
>> file:///proj/aria/cas/data-system/bin/blah.txt.met --refs
>> file:///proj/aria/cas/data-system/bin/blah.txt
>> Nov 9, 2011 10:32:35 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
>> INFO: Loading File Manager Configuration Properties from:
>> [../etc/filemgr.properties]
>> ingestProduct: Result: 31f26edf-0b01-11e1-b910-4da260ad04a8
>>
>> It was properly ingested as indicated in the logs:
>>
>> Nov 9, 2011 10:32:35 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
>> INFO: Running Met Extractor:
>> [org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor] for
>> product type: [GenericFile]
>> Nov 9, 2011 10:32:35 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
>> INFO: Running Met Extractor:
>> [org.apache.oodt.cas.filemgr.metadata.extractors.examples.MimeTypeExtractor]
>> for product type: [GenericFile]
>> Nov 9, 2011 10:32:35 AM
>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
>> INFO: Running Met Extractor:
>> [org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor]
>> for product type: [GenericFile]
>> Nov 9, 2011 10:32:35 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>> toDoc
>> WARNING: No Metadata specified for product
>> [acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt] for
>> required field [DataVersion]: Attempting to continue processing metadata
>> Nov 9, 2011 10:32:35 AM
>> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferer moveFile
>> INFO: LocalDataTransfer: Moving File:
>> file:///proj/aria/cas/data-system/bin/blah.txt to
>> file:/proj/aria/repository/files/acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt/blah.txt
>> Nov 9, 2011 10:32:35 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>> toDoc
>> WARNING: No Metadata specified for product
>> [acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt] for
>> required field [DataVersion]: Attempting to continue processing metadata
>>
>> I tried the query that you suggested with a few minor mods for my install:
>>
>> ./query_tool --url http://localhost:9000 --lucene -query
>> CAS.ProductName:acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt
>>
>> and got the following in my File Manager logs:
>>
>> datacasting 3.1.0cWARNING: Query: [q=CAS.ProductName:acos AND
>> CAS.ProductName:qSnd_090627_04 AND
>> CAS.ProductName:Production_v050050_Sel2900_r05_09172137i AND
>&g

Re: Two File Manager Questions

[Cameron Goodale]

Tim,

I just encountered the same issue when querying for a
'ProductType:TCCON-garmisch'.  This is throwing the same errors you got:

WARNING: Query: [q=ProductType:TCCON AND ProductType:garmisch] for Product
Type: [urn:co2:TCCON-garmisch] returned no results
Nov 17, 2011 7:46:45 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
paginateQuery

As best I can figure the hyphen and underscore must be some kind of
reserved character in Lucene.  We are running an older
cas-filemgr-0.3-SNAPSHOT.jar on Linux CentOS 5.7

Has anyone else encountered this issue besides Tim?  Also Tim what is the
OS you are running on with your problem?

Cecilia,

What OS version are you running on where this is not an issue?  Also what
version of fileManager are you using??

Thanks!



Cameron



On Wed, Nov 9, 2011 at 10:40 AM, Stough, Timothy M (388F) <
timothy.m.sto...@jpl.nasa.gov> wrote:

> Cecilia,
>
> Thanks for the reply.  I tried your suggested query and got similar
> results as before.  Here's what I tried...
>
> In preparation, I ingested a blank text file with the same product name as
> in your example:
>
> [stough@aria1 bin]$ ./filemgr-client --url http://localhost:9000--operation 
> --ingestProduct --productName
> acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt
> --productStructure Flat --productTypeName GenericFile --metadataFile
> file:///proj/aria/cas/data-system/bin/blah.txt.met --refs
> file:///proj/aria/cas/data-system/bin/blah.txt
> Nov 9, 2011 10:32:35 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient 
> INFO: Loading File Manager Configuration Properties from:
> [../etc/filemgr.properties]
> ingestProduct: Result: 31f26edf-0b01-11e1-b910-4da260ad04a8
>
> It was properly ingested as indicated in the logs:
>
> Nov 9, 2011 10:32:35 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
> INFO: Running Met Extractor:
> [org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor] for
> product type: [GenericFile]
> Nov 9, 2011 10:32:35 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
> INFO: Running Met Extractor:
> [org.apache.oodt.cas.filemgr.metadata.extractors.examples.MimeTypeExtractor]
> for product type: [GenericFile]
> Nov 9, 2011 10:32:35 AM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
> INFO: Running Met Extractor:
> [org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor]
> for product type: [GenericFile]
> Nov 9, 2011 10:32:35 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
> toDoc
> WARNING: No Metadata specified for product
> [acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt] for
> required field [DataVersion]: Attempting to continue processing metadata
> Nov 9, 2011 10:32:35 AM
> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferer moveFile
> INFO: LocalDataTransfer: Moving File:
> file:///proj/aria/cas/data-system/bin/blah.txt to
> file:/proj/aria/repository/files/acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt/blah.txt
> Nov 9, 2011 10:32:35 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
> toDoc
> WARNING: No Metadata specified for product
> [acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt] for
> required field [DataVersion]: Attempting to continue processing metadata
>
> I tried the query that you suggested with a few minor mods for my install:
>
> ./query_tool --url http://localhost:9000 --lucene -query
> CAS.ProductName:acos_qSnd_090627_04_Production_v050050_Sel2900_r05_09172137i.txt
>
> and got the following in my File Manager logs:
>
> datacasting 3.1.0cWARNING: Query: [q=CAS.ProductName:acos AND
> CAS.ProductName:qSnd_090627_04 AND
> CAS.ProductName:Production_v050050_Sel2900_r05_09172137i AND
> CAS.ProductName:txt] for Product Type: [urn:oodt:GenericFile] returned no
> results
>
> The addition of single, or double quotes had no effect.
>
> Anymore thoughts?
>
> Thanks,
> Tim.
>
> On Nov 9, 2011, at 9:52 AM, Cheng, Cecilia S (388K) wrote:
> > Looks like emails to u...@oodt.apache.org take longer to arrive, so I’m
> sending this directly to you. Hope it helps.
> >
> > -- cecilia
> >
> > -- Forwarded Message
> > From: "Cheng, Cecilia S (388K)" 
> > Date: Wed, 9 Nov 2011 09:30:37 -0800
> > To: "u...@oodt.apache.org" 
> > Subject: Re: Two File Manager Questions
> >
> > Hi Tim,
> >
> > > B) Disallowed characters in product names?
> > >
> > >  My product names look like this:  "ALPSRP225250610-H1.0__A"  If I try
> to
> > > search for a product by name using a Lucene query through query_tool,
> the
> > > query breaks across the "_".  What I see in the log is:
> > >
> > > WARNING: Query: [q=Filename:ALPSRP154650650-H1.0 AND Filename:A] for
> Product
> > > Type: [urn:oodt:GenericFile] returned no results
> > >
> > > So it looks like the __ gets turned into an "AND".  What's the deal
> and how do
> > > I fix it?
> >
> > The '_' in the product name works for us. See:
> >
> >> $

Re: [ANNOUNCE] Welcome Sheryl John as new Apache OODT committer and PMC member

[Cameron Goodale]

Welcome to OODT Sheryl!  I am glad to have you on the team.


-Cameron

On Sat, Nov 12, 2011 at 1:01 PM, Sheryl John  wrote:

> Hello everyone!
>
> First of all, thanks to all from the OODT lists and the JPL team who've
> helped me in the past couple of months. And, a special shout out to Prof
> Mattmann for mentoring me all along. :)
>
> About myself:
> A native of south India, grew up in Dubai, and then completed my Bachelors
> in Computer Science and Engineering from National Institute of
> Technology,Trichy (India) on May 2010.
> I joined University of Southern California last Fall for MS in CS.
> I was introduced to Apache OODT world by Prof. Mattmann during his Software
> Architecture class last Spring and, since this summer, I've been working as
> an intern at Children's Hospital Los Angeles for the Virtual Pediatric
> Intensive Care Unit (VPICU). At VPICU, I'm learning how to leverage OODT's
> components for clinical data workflows and medical data archiving and
> management.
>
> It's been awesome working with OODT and I hope to contribute more to the
> project and the community.
>
> Go OODT! :-D
>
> Thanks again for having me aboard!
> Sheryl
>
>
>
>
> On Sat, Nov 12, 2011 at 12:21 PM, Mattmann, Chris A (388J) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > Hi Folks,
> >
> > A while back I nominated Sheryl John for OODT PMC and committership. The
> > VOTE results are back
> > and I'm happy to report that Sheryl is now a member of the OODT PMC and a
> > committer!
> >
> > Welcome, Sheryl. Please feel free to say a bit about yourself.
> >
> > Cheers,
> > Chris
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattm...@nasa.gov
> > WWW:   http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
> >
>
>
> --
> -Sheryl
>



-- 

Sent from a Tin Can attached to a String

Re: [jira] [Updated] (OODT-345) Integration CAS-Workflow with CAS-CLI

[Cameron Goodale]

Brian,

This isn't really technically relevant, but I must comment that you are a
MACHINE!  I am actually hoping that I have a project that will use your new
CLI interface.  Just wanted to let you know that your work is appreciated.

-Cam

On Mon, Nov 7, 2011 at 1:32 PM, Brian Foster (Updated) (JIRA) <
j...@apache.org> wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Brian Foster updated OODT-345:
> --
>
>Attachment: SwitchWorkflowToUseCasCli.patch.txt
>
> - attached patch
>
> - code can be reviewed here: https://reviews.apache.org/r/2755/
>
> > Integration CAS-Workflow with CAS-CLI
> > -
> >
> > Key: OODT-345
> > URL: https://issues.apache.org/jira/browse/OODT-345
> > Project: OODT
> >  Issue Type: Improvement
> >  Components: workflow manager
> >Affects Versions: 0.4
> > Environment: none
> >Reporter: Brian Foster
> >Assignee: Brian Foster
> >Priority: Minor
> > Fix For: 0.4
> >
> > Attachments: SwitchWorkflowToUseCasCli.patch.txt
> >
> >
> > Here are some example outputs of what the new command line will look
> like:
> > {code}
> > $ ./wmgr-client -h
> >
> -
> > | Short | Long | Description
> >
> -
> >  -u, --url Workflow
> Manager URL
> >   Handler:
> >Sets the
> following Java Properties:
> >
>  [org.apache.oodt.cas.workflow.url]
> >  -op,--operationDeclare that
> you wish to present an operation
> >  -p, --pageNumPage Number
> >
> Requirement Rules:
> >
>  [GetNextPage : REQUIRED, GetPrevPage : REQUIRED]
> >   Handler:
> >Will
> invoke 'setPageNum' on action selected,
> >except
> for the following actions: []
> >  -m, --metaData Declare that
> you wish to present metadata
> >
> Requirement Rules:
> >
>  [SendEvent : OPTIONAL, DynMetadata : OPTIONAL]
> >  -en,--eventNameName of the
> Event to trigger
> >
> Requirement Rules:
> >
>  [SendEvent : REQUIRED, GetWorkflowsByEvent :
> >REQUIRED]
> >   Handler:
> >Will
> invoke 'setEventName' on action selected,
> >except
> for the following actions: []
> >  -s, --status   Workflow
> status
> >
> Requirement Rules:
> >
>  [GetFirstPage : OPTIONAL, GetNextPage : OPTIONAL,
> >
>  GetPrevPage : OPTIONAL, GetLastPage : OPTIONAL]
> >   Handler:
> >Will
> invoke 'setStatus' on action selected, except
> >for the
> following actions: []
> >  -tids,  --taskIdsList of
> workflow task ids
> >
> Requirement Rules:
> >
>  [DynWorkflow : REQUIRED]
> >   Handler:
> >Will
> invoke 'setTaskIds' on action selected,
> >except
> for the following actions: []
> >  -id,--id   ID
> >
> Requirement Rules:
> >
>  [GetTaskById : REQUIRED, GetConditionById :
> >REQUIRED,
> GetWorkflowById : REQUIRED,
> >
>  GetWorkflowInst : REQUIRED, GetWallClockTime :
> >REQUIRED,
> GetTaskWallClockTime : REQUIRED,
> >
>  StopWorkflowInst : REQUIRED, PauseWorkflowInst :
> >REQUIRED,
> ResumeWorkflowInst : REQUIRED,
> >
>  GetWorkflowInstMet : REQUIRED]
> >   Handler:
> >Will
> invoke 'setId' on action selected, excep

Re: Getting started

[Cameron Goodale]

Thien,

Thanks for the feedback on the site.  Below is my take on OODT, and I am
sure others can chime in with maybe some publications you can read.  I hope
you find this helpful.

OODT is not a turnkey solution.  (Just wanted to get that out of the way,
since many people see OODT as a product and not a framework)

OODT is a framework for data management (this means ANY data, from Space to
Climate to Medical to Images can be archive and processed with OODT)

The power of OODT is the flexibility of the framework, but this also makes
it challenging to grasp and understand everything it can do.

The power of OODT is in the community of Devs and Users.  Chances are
someone else in the community has encountered the same challenge or a
similar one.

Keep the Questions and Comments coming.


-Cameron




On Fri, Oct 28, 2011 at 9:27 AM, Nguyen, Thien  wrote:

> Hello,
>
> I'm a medical informatics software developer at the Boston Department of
> Veteran Affairs. We are reviewing tools to prototype for use in our systems
> that would help manage, share, and track research information. We have
> genomic data, health records, phenotypic data, and results of nlp/machine
> learning techniques.
>
> OODT sounds like a nice package that we would like to try out, but,
> honestly, the website says very little. I'm not even sure where to begin to
> even try interacting with the downloaded source. Any direction on how to get
> started or lower-level resources to read up on would be greatly appreciated.
>
> Thanks,
> Thien
>

Re: [DISCUSS] OODT 0.4

[Cameron Goodale]

+1 from me.  I think Thomas Bennett and I have put in enough work between
xdoc updates and wiki creation to mark OODT-52 complete this week.

I also plan to keep up on the RADiX [OODT-120] effort and help with docs and
testing so it will be ready for ApacheCon.

-Cameron

On Wed, Oct 5, 2011 at 6:36 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Paul,
>
> +1 for you to RM. I was originally intending to do it, but am happy to cede
> it to you
> in favor of spending my cycles wrapping up OODT-215.
>
> My issues I'd like to get wrapped up:
>
> OODT-215 -- Workflow 2 architecture
> OODT-229 -- Refactor all o.a.oodt.cas.filemgr.catalog.Catalog
> implementations and co-locate their shared code
> OODT-154 -- Flexible response handling for XMLPS
> OODT-323 -- Add new command line option in Workflow manager client to get
> Workflow Instance Metadata
>
> Cheers,
> Chris
>
> On Oct 5, 2011, at 9:32 AM, Ramirez, Paul M (388J) wrote:
>
>
> > Hey All,
> >
> > I think it would be great if we could get a 0.4 release out before
> ApacheCon 2011. To accomplish this I would like to be able to roll a release
> candidate on or before October 30th. There is already quite a few issues
> closed for the release but I think we should push to get the workflow
> updates in and the radix archetype finished off. I can commit having the
> latter done in time. Is there anything else that we should try to get into
> this release? If so, does the timeframe seem reasonable. If not, lets
> discuss and consider whether items can be pushed to 0.5. I think it would be
> a big win for us to get a 0.4 release out in time for ApacheCon.
> >
> > Thanks,
> > Paul Ramirez
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>

Re: [jira] [Commented] (OODT-52) Update the CAS File Manager User Guide

[Cameron Goodale]

Great catch and solution on the Maven 3 vs. 2 issue on Mac.

The filemanager xdocs used to tell you how to first install oodt-core, then
ONLY install filemanager.  I tried this last night and it would not build
properly due to missing jars, etc...

My plan is to update the xdoc for File Manager to instruct the user to
checkout and built OODT (the complete suite) it will just be easier.  Once
they get the build success message I will send them to the wiki.

I will also be sure to include the warning message for Mac OS X users.  Even
in 10.5.x the Mac Java updates also update to Maven 3.

Getting closer to closing this issue out.

-Cam

On Tue, Sep 27, 2011 at 4:44 PM, Thomas Bennett  wrote:

> Hi Cameron,
>
> I previously built another wiki page that is dedicated to File Manager
> > Policy.  If you ever decide to tackle that part of File Manager I have a
> > pretty good start you can fold back into the User Guide.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/OODT/Everything+you+want+to+know+about+File+Manger+Policy
> >
>
> Excellent! Thank you.  I'll work it into my current wiki.
>
>
> > I have a some free cycles this weekend so I will start updating the xdoc
> > page for the Basic User Guide on the site.  I will use the page to
> explain
> > the maven install and build process.  Then point the user off to your
> great
> > wiki page.
> >
>
> Great! Thanks. This is one aspect that would be extremely useful but that
> I've been avoiding - so thank you.
>
> I did however make some brief notes on getting the environment working on a
> Mac. Please feel free to use.
>
> *Java and Maven on OS X 10.7*
>
> *Maven*
>
> Mac's ship with maven 3 by default. OODT needs maven 2. Should you wish to
> get maven 2 set up as your default, you can use the following steps:
>
>
>   1. Download apache-maven-2.2.1-bin tar file.
>   2. Install it to /usr/share/ by running sudo tar
>   xzvf apache-maven-2.2.1-bin.tar.gz -C /usr/share
>   3. Now hook it in as the default maven.
>
> $ sudo rm /usr/bin/mvn
> $ sudo rm /usr/share/maven
> $ sudo ln -s /usr/share/apache-maven-2.2.1 /usr/share/maven
> $ sudo ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
>
> *Java VM*
>
> To get the java environment working, you need to get the java_home command
> into your *$PATH.*
> $ ln -s
>
> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home
> /usr/bin/java_home
>
> Now set up the environment variable. Add this line to your ~/.profile or
> file.
> JAVA_HOME=$( java_home )
>
> Or source it directly:
> JAVA_HOME=$(
> System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home
> )
>
> Cheers,
> Tom
>



-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-52) Update the CAS File Manager User Guide

[Cameron Goodale]

Tom,

I previously built another wiki page that is dedicated to File Manager
Policy.  If you ever decide to tackle that part of File Manager I have a
pretty good start you can fold back into the User Guide.

https://cwiki.apache.org/confluence/display/OODT/Everything+you+want+to+know+about+File+Manger+Policy
*
*
*
*
I have a some free cycles this weekend so I will start updating the xdoc
page for the Basic User Guide on the site.  I will use the page to explain
the maven install and build process.  Then point the user off to your great
wiki page.

-Cameron

On Sat, Sep 24, 2011 at 2:27 AM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> WOW Tom, this is fantastic!
>
> I will try and add a link from the main file manager page to this guide
> since it will likely be more up-to-date
> because more folks can work on it in the Wiki.
>
> Cheers,
> Chris
>
> On Sep 22, 2011, at 6:48 AM, Thomas Bennett wrote:
>
> > Hi Cameron,
> >
> > Thanks for your reply. I've start a wiki page.
> >
> > https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
> >
> > Regards,
> > Tom
> >
> > On 22 September 2011 06:20, Cameron Goodale (JIRA) 
> wrote:
> >
> >>
> >>   [
> >>
> https://issues.apache.org/jira/browse/OODT-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112315#comment-13112315
> ]
> >>
> >> Cameron Goodale commented on OODT-52:
> >> 
> >>
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>


-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-52) Update the CAS File Manager User Guide

[Cameron Goodale]

Chris,

I will have a couple hours while in a plane to Denver, then Korea, so I
think I can wrap this up in the next couple weeks.

-Cam

On Fri, Sep 9, 2011 at 10:07 AM, Chris A. Mattmann (JIRA)
wrote:

>
>[
> https://issues.apache.org/jira/browse/OODT-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101342#comment-13101342]
>
> Chris A. Mattmann commented on OODT-52:
> ---
>
> Cam, got time for any action here?
>
> > Update the CAS File Manager User Guide
> > --
> >
> > Key: OODT-52
> > URL: https://issues.apache.org/jira/browse/OODT-52
> > Project: OODT
> >  Issue Type: Task
> >  Components: file manager
> >Affects Versions: 0.1-incubating
> >Reporter: Cameron Goodale
> >Assignee: Cameron Goodale
> >Priority: Trivial
> > Fix For: 0.4
> >
> >   Original Estimate: 4h
> >  Remaining Estimate: 4h
> >
> > The current CAS File Manager User Guide that is packaged with the file
> manager contains download and build instructions that are really out of
> date.  If someone outside of the OODT team tried to follow the directions
> they would have a really hard time.
> > This issue is to bring the document current to reflect the current
> build/deploy process.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: Deploying CAS-FileManager within Windows

[Cameron Goodale]

Rishi,

I tried to get filemgr started on a windows machine about 2 years ago and I
ran into a lot of issues.  It is definitely made for a *nix type OS.  If you
do figure out the main sticking points I hope we can get some docs or issues
setup to guide other Win users.

OODT-187: https://issues.apache.org/jira/browse/OODT-187   Deals with the
file pathing within the properties file.  This could be an issue that might
solve the problem.  I would hope that file:/// would be supported by Win and
*nix systems, but I haven't done the research to confirm.

Good luck,


Cameron



On Fri, Sep 2, 2011 at 4:45 PM, Verma, Rishi (317I) <
rishi.ve...@jpl.nasa.gov> wrote:

> Hello fellow developers,
>
> I'm trying to deploy CAS-Filemanager (0.4-SNAPSHOT) to a Windows XP
> machine, but am witnessing some warnings due to paths being set for a
> windows environment.
>
> Two questions I want to ask:
> [1] Does filemgr.properties ASSUME a UNIX-like path for the following two
> props (org.apache.oodt.cas.filemgr.repositorymgr.dirs,
> org.apache.oodt.cas.filemgr.validation.dirs)?
> [2] Is the prefix "file://" for the above two properties applicable only to
> a unix environment?
>
> I say this because I have tried (on the Windows machine) to place paths
> using Windows-style forward-slash "\" paths, as well as using Cygwin style
> paths to the same corresponding directories; however, neither method
> eliminates the below warnings when attempting to start filemgr within
> Windows (running within Cygwin I might add).
>
> Thanks!
> Rishi
>
> ---
> Below is an attempt to start filemgr:
>
> $ ./filemgr start
> Starting cas file manager: OK
> Sep 2, 2011 2:47:30 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
> load
> Configuration
> INFO: Loading File Manager Configuration Properties from:
> [../etc/filemgr.proper
> ties]
> Sep 2, 2011 2:47:30 PM
> org.apache.oodt.cas.filemgr.validation.XMLValidationLayer
>  loadElements
> WARNING: Element directory:
> file:///cygdrive/c/proteome/proteome-filemgr-win/pol
> icy is not a directory: skipping element loading from it.
> Sep 2, 2011 2:47:30 PM
> org.apache.oodt.cas.filemgr.validation.XMLValidationLayer
>  loadProductTypeMap
> WARNING: Element directory:
> file:///cygdrive/c/proteome/proteome-filemgr-win/pol
> icy is not a directory: skipping product type element map loading from it.
> Sep 2, 2011 2:47:31 PM
> org.apache.oodt.cas.filemgr.repository.XMLRepositoryManag
> er loadProductTypes
> WARNING: Product type directory:
> file:///cygdrive/c/proteome/proteome-filemgr-wi
> n/policy is not a directory: skipping product type loading from it.
> Sep 2, 2011 2:47:31 PM
> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClien
> t 
> INFO: Loading File Manager Configuration Properties from:
> [../etc/filemgr.proper
> ties]
> Sep 2, 2011 2:47:31 PM
> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransfe
> rer setFileManagerUrl
> INFO: Local Data Transfer to: [http://localhost:9000] enabled
> Sep 2, 2011 2:47:31 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager
>  t>
> INFO: File Manager started
>



-- 

Sent from a Tin Can attached to a String

Mac Update HOSED Maven...AGAIN (auto update to Maven 3)

[Cameron Goodale]

Hey Everyone,

It has been a while since I last built OODT from source, but I was testing
some of my open issues and stumbled upon this issue AGAIN.  The latest Mac
OSX updates (I am on Leopard) have once again force installed Maven 3 (this
time 3.0.3).

If you cannot even run mvn -version, you can pretty much guarantee you have
the same issue I have.  To save you sometime here is the list of commands I
used to get back to Maven 2.2.1.


cd /usr/share/
ls -la maven
lrwxr-xr-x  1 root  wheel  16 Jul 12 06:38 maven ->* java/maven-3.0.3*

sudo rm /usr/bin/mvn
sudo ln -s /usr/local/apache-maven/apache-maven-2.2.1/bin/mvn /usr/bin/mvn
mvn -version

Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700)
Java version: 1.6.0_26
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x" version: "10.5.8" arch: "x86_64" Family: "mac"

All Fixed again.  Hope this helps someone else get back the 20 minutes I
just lost.

-Cameron

Re: [jira] [Updated] (OODT-306) File Manager Tools for BASH and TCSH

[Cameron Goodale]

So I have been mulling this over since I wrote this JIRA issue and I think I
will break fmquery into:

lucene_query and sql_query since the query_tool that is being called under
the code is able to query both, and prefixing fm seems a bit redundant.


-Cameron


On Fri, Jul 22, 2011 at 6:03 PM, Cameron Goodale (JIRA) wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Cameron Goodale updated OODT-306:
> -
>
>Description:
> *Chris Mattmann* has been writing some really useful shell alias commands
> to help with some of the really repetitive tasks that tend to appear when
> using the File Manager component.  This issue will be used to share and
> commit 2 files that contain environment settings that can be easily sourced,
> or just copied into a users .bashrc or .tcshrc startup file.
>
> The files will contain the scripting commands and some basic documentation.
>
> I plan to drop the files into oodt/filemgr/src/main/resources   since they
> really only apply to the filemanager.
>
> Here is my initial working Docu-Code (for BASH) I plan to clean up and
> commit.
>
> {code}
> #  Copy and Paste this Block into the .bashrc of your deployment user
> account ##
> #
> # The following aliases must be used within a filemgr installation's
> # bin directory since relative pathing is being used.  This block also
> # assumes that the filemgr is running on port 9000 (the default port of
> filemgr)
> #
> alias fmquery="java
> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url
> http://localhost:9000 --lucene -query "
> #
> alias fmdel="java
> -Dorg.apache.oodt.cas.filemgr.properties=../etc/filemgr.properties
> -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.DeleteProduct
> --fileManagerUrl http://localhost:9000 --read"
> #
> alias metdump="java -Djava.ext.dirs=../lib
> org.apache.oodt.cas.filemgr.tools.MetadataDumper --url $FILEMGR_URL --out .
> --productId "
> #
>  END OF BLOCK ###
> {code}
> h2. Previous Commands Explained
>
> h3. fmquery
> This will execute a Lucene Query against the filemanager catalog and return
> a list of ProductIDs.  These Product IDs can then be piped into the fmdel
> alias to remove all products that match the query criteria.
>
> *Usage:*
> - Return the ProductIds to the terminal window -
> fmquery "ProductType:ISMRawData"
>
> - Pipe the ProductIds to a file -
> fmquery "ProductType:ISMRawData" > ProductID_List.txt
>
>
> h3. fmdel
> Use this command to delete a single product from the catalog.  This will
> remove the lucene index entry from the catalog and will remove the archived
> product from disk.  (If you just want to see what products will be removed
> from the archive, then you will need to add the --nocommit parameter to the
> fmdel command alias after the --read param)
>
> *Usage:*
> - Remove one product from the catalog
> fmdel 0691ee17-96ec-11e0-8556-3b3ca12e4f64
>
> - Remove a set of products from the catalog using the fmquery command
> [fmquery Metadata_Key:"Matching_Key_Value" | fmdel]
> For example: To remove all Products with a GenericFile ProductType use
> fmquery ProductType:"GenericFile" | fmdel
>
> h3. metdump
> Just added to the list.  This will take in a product_id and dump out the
> product's metadata into the current directory.  *This one needs a little
> more testing and docs.*
>
> Comments are welcome.
>
>   was:
> Chris Mattmann has been writing some really useful shell alias commands to
> help with some of the really repetitive tasks that tend to appear when using
> the File Manager component.  This issue will be used to share and commit 2
> files that contain environment settings that can be easily sourced, or just
> copied into a users .bashrc or .tcshrc startup file.
>
> The files will contain the scripting commands and some basic documentation.
>
> I plan to drop the files into oodt/filemgr/src/main/resources   since they
> really only apply to the filemanager.
>
> Here is my initial working Docu-Code (for BASH) I plan to clean up and
> commit.
>
>
> #  Copy and Paste this Block into the .bashrc of your deployment user
> account ##
> #
> # The following aliases must be used within a filemgr installation's
> # bin directory since relative pathing is being used.  This block also
> # assumes that the filemgr is running on port 9000 (the default port of
> filemgr)
> #
> a

Re: [jira] [Updated] (OODT-296) Workflow Editor GUI

[Cameron Goodale]

Chris,

I saw the screen shot you attached and it looks really good. Hats off to you
and bfoster on this one.

Cameron
On Jul 6, 2011 9:41 AM, "Chris A. Mattmann (JIRA)"  wrote:
>
> [
https://issues.apache.org/jira/browse/OODT-296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Chris A. Mattmann updated OODT-296:
> ---
>
> Attachment: OODT-296.070611.Mattmann.patch.txt
>
> - latest patch. I'm trying to figure out how to make Jgraph and JUNG play
nicely together. I *sort* of have it down but am still learning.
Consequently, the editor doesn't load the workspace anymore, NPE, but you
can see where I'm going.
>
>> Workflow Editor GUI
>> ---
>>
>> Key: OODT-296
>> URL: https://issues.apache.org/jira/browse/OODT-296
>> Project: OODT
>> Issue Type: Sub-task
>> Components: workflow manager
>> Reporter: Chris A. Mattmann
>> Assignee: Chris A. Mattmann
>> Fix For: 0.4
>>
>> Attachments: OODT-296.070611.Mattmann.patch.txt,
OODT-296.Mattmann.070611.patch.txt, Screen shot 2011-07-05 at 2.28.51 PM.png
>>
>>
>> Internally at JPL, we built a Workflow 2 GUI (thanks [~bfoster]!) :-) I'm
working on preparing a patch to port it from the old internal JPL workflow
manager (wengine) code to Apache OODT. I'll use this patch to track the
progress. I need to migrate:
>> * namespaces from g.j.n.oodt to o.a.oodt
>> * use of jgraph library to http://jung.sourceforge.net/doc/api/index.html
>> * some old deps on wengine-branch internal to JPL
>> I'll attach a screenshot to show what it looks like (and will look like)
shortly.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>

Re: [jira] [Closed] (INFRA-3707) Enable svnpubsub on https://svn.apache.org/repos/asf/oodt/site

[Cameron Goodale]

Chris,

This is great news.  I need to push out updates to the website, so I will
try it without an svn up at the end and see what happens.  If all goes well
I will update the Wiki on how to build the website to remove that step.

Thanks for setting this up.

-Cameron

On Wed, Jun 29, 2011 at 4:51 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> All, FYI, svnpubsub now enabled on the oodt/site directory. Cam, when you
> check in site updates, now it will automatically push it out to the mirrors,
> no need to SVN up anymore...
>
> Cheers,
> Chris
>
> Begin forwarded message:
>
> > From: "Gavin (JIRA)" 
> > Date: June 29, 2011 4:43:28 PM PDT
> > To: "Mattmann, Chris A (388J)" 
> > Subject: [jira] [Closed] (INFRA-3707) Enable svnpubsub on
> https://svn.apache.org/repos/asf/oodt/site
> >
> >
> > [
> https://issues.apache.org/jira/browse/INFRA-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
> >
> > Gavin closed INFRA-3707.
> > 
> >
> >Resolution: Fixed
> >  Assignee: (was: Gavin)
> >
> > All done, apologies for delay
> >
> >> Enable svnpubsub on https://svn.apache.org/repos/asf/oodt/site
> >> --
> >>
> >>Key: INFRA-3707
> >>URL: https://issues.apache.org/jira/browse/INFRA-3707
> >>Project: Infrastructure
> >> Issue Type: Task
> >> Security Level: public(Regular issues)
> >> Components: SvnPubSub
> >>Environment: OODT site
> >>   Reporter: Chris A. Mattmann
> >>   Priority: Minor
> >>
> >> Hi Guys,
> >> Can you please enable svnpubsub on
> https://svn.apache.org/repos/asf/oodt/site/?
> >> Thanks!
> >> Cheers,
> >> Chris
> >
> > --
> > This message is automatically generated by JIRA.
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
> >
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>


-- 

Sent from a Tin Can attached to a String

Re: [VOTE] Apache OODT 0.3 Release

[Cameron Goodale]

Team,

Here is my check list:

mvn package - SUCCESS


 mvn site - FAIL


[INFO]


[ERROR] BUILD ERROR

[INFO]


[INFO] Failed to resolve dependencies for one or more projects in the
reactor. Reason: Missing:

--

1) org.apache.oodt:oodt-commons:jar:0.3


...


  Path to dependency:

  1) org.apache.oodt:pcs-input:jar:0.3

  2) org.apache.oodt:oodt-commons:jar:0.3


--

1 required artifact is missing.


for artifact:

  org.apache.oodt:pcs-input:jar:0.3


from the specified remote repositories:

  javax-transaction (http://download.java.net/maven/2),

  apache.snapshots (http://repository.apache.org/snapshots/),

  globus-repo (
http://projects.iplantcollaborative.org/archiva/repository/internal),

  central (http://repo1.maven.org/maven2),

  oodt-m2-repo (http://oodt.jpl.nasa.gov/download/private),

  apache.org (http://people.apache.org/repo/m2-snapshot-repository),

  maven.topazproject.org (http://maven.topazproject.org/maven2),

  hadoop.apache.org (http://people.apache.org/~kalle/mahout/maven2/)


 startup FileManager - SUCCESS

 Ingest the blah.txt - SUCCESS

 start up Workflow - SUCCESS

 run the test workflow - SUCCESS


Running mvn site is not a blocker in my opinion, I will just sort it out
post-release when I update the website.


+1 From me.



-Cameron




On Fri, Jun 24, 2011 at 7:06 AM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> On Jun 24, 2011, at 6:06 AM, Sean Kelly wrote:
>
> >> Signatures and checksums are good. I confirmed the Maven build as well.
> >
> > How did the Python build go?
>
> Good question. I didn't run the python tests but looks like woollard ran
> them and they look good.
>
> I just tried them out via:
>
> python setup.py test in the agility/ folder and...
>
> Test the `Documentable` and `DocumentableField` classes. ... ok
> Doctest: oodt.xmlutils.add ... ok
> Doctest: oodt.xmlutils.text ... ok
>
> --
> Ran 34 tests in 0.070s
>
> OK
>
> All is well.
>
> Did they run on your machine?
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>

Re: A Question About OODT Documentation

[Cameron Goodale]

Phil,

Consider me inspired by your drive to learn about OODT.  It wasn't that long
ago that I didn't understand what a FileManager was, or when to use the
Crawler, but in working with OODT I have lost sight of the steep learning
curve as I have started to climb it.  So to help you (and anyone else that
has questions and wants answers) I created a wiki page that we can use to
collaborate.  It is specifically focused on New User Questions.  Head on
over and check it out.  You should be able to create a new account (if you
don't have one already) and start making updates, adding in questions you
want answered (even if they are related to the 'old JPL docs') and we will
do our best to answer and explain and get a New User Overview out of the
wiki and published to the oodt website.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27361963

Bookmark it, and jot down questions and comments as you explore our project.
 I encourage the rest of the community to also follow/Watch the page and see
how you can help grow it since we all have certain components we know better
than others.


I am looking forward to working with you Phil,


Cameron



On Fri, Jun 24, 2011 at 7:51 AM, Phillip Rhodes
wrote:

> Regarding the old docs:  I kinda think it would be good to link to pretty
> much all of those same papers (assuming they're
> still available and that it's legal, blah, blah) that were up on the JPL
> site.  Even if some of them are dated now, they provide historical
> context and something of an overview.
>
> Otherwise, the one thing that I think would be OODT n00bs (like myself)
> would be a nice, up to date, overview that lays out "What are
> all these pieces, how do they fit together, and how does the whole system
> work?"   Just jumping into what's at http://oodt.apache.org, it's
> not necessarily real apparent which piece (or pieces) are needed, which one
> to focus on first, etc.  Maybe something of a "New User Overview"?
>
> I'd be happy to help with such a thing, but I currently lack the knowledge
> to contribute one myself.  I'm still just going through the existing
> docs and muddling through, trying to get my head around it all.  :-)
>
>
> Phil
>
> On Fri, Jun 24, 2011 at 10:44 AM, Cameron Goodale  >wrote:
>
> > Phil,
> >
> > Thanks for the input.  Documentation is always important and something I
> > have been trying to address.  Currently our documentation is grouped by
> > component, and some of the more mature components have more
> documentation.
> >
> > For Example FileManager:
> > http://oodt.apache.org/components/maven/filemgr/
> >
> > Has 3 Software Documentation Pages (one is still under construction)
> >
> > If you need more information on a specific component, or have suggestions
> > for including JPL Docs @Apache I welcome the input.  I appreciate that
> you
> > took the time to dig around looking for our old docs.
> >
> > Best Regards,
> >
> >
> > Cameron
> >
> >
> >
> >
> >
> > On Fri, Jun 24, 2011 at 6:53 AM, Phillip Rhodes
> > wrote:
> >
> > > Hey guys, quick question... I'm interested in getting more familiar
> with
> > > OODT, so I started poking around the website looking for docs.  Then I
> > > started
> > > Googling for stuff, and at one time had found the old JPL page which
> had
> > > links to a lot of stuff on it.  This is even mentioned in the OODT
> > > Incubator
> > > Proposal
> > >
> > > "There is a wealth of documentation available on OODT. The best
> starting
> > > point is the existing OODT JPL website (which will be ported to be
> > sync'ed
> > > or just a pointer to the Apache website)http://oodt.jpl.nasa.gov";
> > >
> > > But as far as I can tell, since the oodt.jpl.nasa.gov page was
> > re-pointed
> > > to
> > > oodt.apache.org, no corresponding documentation page was ever created
> > that
> > > links to that stuff.  I was able to go through the Wayback Machine and
> > find
> > > the docs from the old site, but for the benefit of new-comers, it might
> > be
> > > good to
> > > recreate that here?   Or if it has already been recreated, maybe the
> link
> > > should be more prominent? (I didn't find one, but maybe I just missed
> > it).
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Phil
> > >
> >
>

Re: A Question About OODT Documentation

[Cameron Goodale]

Phil,

Thanks for the input.  Documentation is always important and something I
have been trying to address.  Currently our documentation is grouped by
component, and some of the more mature components have more documentation.

For Example FileManager:
http://oodt.apache.org/components/maven/filemgr/

Has 3 Software Documentation Pages (one is still under construction)

If you need more information on a specific component, or have suggestions
for including JPL Docs @Apache I welcome the input.  I appreciate that you
took the time to dig around looking for our old docs.

Best Regards,


Cameron





On Fri, Jun 24, 2011 at 6:53 AM, Phillip Rhodes
wrote:

> Hey guys, quick question... I'm interested in getting more familiar with
> OODT, so I started poking around the website looking for docs.  Then I
> started
> Googling for stuff, and at one time had found the old JPL page which had
> links to a lot of stuff on it.  This is even mentioned in the OODT
> Incubator
> Proposal
>
> "There is a wealth of documentation available on OODT. The best starting
> point is the existing OODT JPL website (which will be ported to be sync'ed
> or just a pointer to the Apache website)http://oodt.jpl.nasa.gov";
>
> But as far as I can tell, since the oodt.jpl.nasa.gov page was re-pointed
> to
> oodt.apache.org, no corresponding documentation page was ever created that
> links to that stuff.  I was able to go through the Wayback Machine and find
> the docs from the old site, but for the benefit of new-comers, it might be
> good to
> recreate that here?   Or if it has already been recreated, maybe the link
> should be more prominent? (I didn't find one, but maybe I just missed it).
>
>
> Thanks,
>
>
> Phil
>

Re: svn commit: r1138739 - in /oodt/trunk: core/pom.xml pom.xml

[Cameron Goodale]

Paul,

Great job tracking this down.  You are one of a small few who would track
this down and fix it, so thank you for that.  I am sure this will benefit
all future Releases and Release Managers.

-Cam

On Wed, Jun 22, 2011 at 10:22 PM,  wrote:

> Author: pramirez
> Date: Thu Jun 23 05:22:53 2011
> New Revision: 1138739
>
> URL: http://svn.apache.org/viewvc?rev=1138739&view=rev
> Log:
> The maven release plugin is real picky. Having the scm parameters
> configured in the common parent pom confused the generation of the scm.url
> during the release:prepare. The scm parameters were simply moved from
> core/pom.xml to pom.xml. Unfortunately this can't be fully tested until it
> is committed as the release:prepare goal bails unless everything is commited
> in the current check out. It does however get far enough to see that the
> scm.url in the release.properties is corrected.
>
>
> Modified:
>oodt/trunk/core/pom.xml
>oodt/trunk/pom.xml
>
> Modified: oodt/trunk/core/pom.xml
> URL:
> http://svn.apache.org/viewvc/oodt/trunk/core/pom.xml?rev=1138739&r1=1138738&r2=1138739&view=diff
>
> ==
> --- oodt/trunk/core/pom.xml (original)
> +++ oodt/trunk/core/pom.xml Thu Jun 23 05:22:53 2011
> @@ -255,12 +255,6 @@ the License.
> manual
> 
> 
> -
> -scm:svn:https://svn.apache.org/repos/asf/oodt/trunk
> 
> -scm:svn:
> https://svn.apache.org/repos/asf/oodt/trunk
> -http://svn.apache.org/viewvc/oodt/trunk
> -HEAD
> -
> 
> Apache OODT
> http://oodt.apache.org/
>
> Modified: oodt/trunk/pom.xml
> URL:
> http://svn.apache.org/viewvc/oodt/trunk/pom.xml?rev=1138739&r1=1138738&r2=1138739&view=diff
>
> ==
> --- oodt/trunk/pom.xml (original)
> +++ oodt/trunk/pom.xml Thu Jun 23 05:22:53 2011
> @@ -34,7 +34,14 @@ the License.
> 
> Apache OODT is middleware for metadata.
> 
> -
> +
> +
> +scm:svn:https://svn.apache.org/repos/asf/oodt/trunk
> 
> +scm:svn:
> https://svn.apache.org/repos/asf/oodt/trunk
> +http://svn.apache.org/viewvc/oodt/trunk
> +HEAD
> +
> +
> 
> core
> commons
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: Updated RADIX web page data dir

[Cameron Goodale]

Guys,

Sorry for being largely absent from the discussion, but I think you guys
have it well under control.  My vision for RADiX was an 80% solution so
Chris if you want to be in the other 20% that is cool with me ;)

I think we should make it clear that RADiX is an OPTION, and not the ONLY
way to build and deploy OODT.  Our desire is to help OODT Devs and Users by
giving them a sort of 'default' development and deployment structure.  I
think the Ruby on Rails guys said it best, "Convention over configuration"
since RADiX will already be configured to use it's convention.

Great input and feedback Chris.  I wonder if anyone else has given the Wiki
a look and wants to give their input, we are open to it.

https://cwiki.apache.org/OODT/radix-powered-by-oodt.html


Prost!

Cameron

On Thu, May 19, 2011 at 10:13 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> +1 to /failure/ingest and /failure/jobs, etc.
>
> That makes sense to me...though unfortunately will cause grumpy people like
> me to have to change to take advantage of the power of RADIX. I'll learn
> though (slowly) over time :)
>
> Cheers,
> Chris
>
> On May 19, 2011, at 7:40 AM, Ramirez, Paul M (388J) wrote:
>
> > Hey Chris,
> >
> > The failure was just meant to be more generic as there could be other
> items, from say workflow, that would also go in there. How about having
> "/failure/ingest"? That still does remove the files word though so
> "/failure/ingest_files" is a possibility. The reasoning behind not having
> files in the directory name is that at the highest level we talk about
> ingestion of products and not files.
> >
> > Thanks,
> > Paul
> >
> >
> >
> >
> > On May 18, 2011, at 6:21 PM, Mattmann, Chris A (388J) wrote:
> >
> >> Hi Paul,
> >>
> >> met could be for either the crawler or for the curator.
> >>
> >> As for failed_ingest_files, why the rename? Just to be shorter? I think
> failed_ingest_files, though more verbose, conveys more meaning than
> failure...
> >>
> >> Cheers,
> >> Chris
> >>
> >> On May 18, 2011, at 7:50 AM, Ramirez, Paul M (388J) wrote:
> >>
> >>> Hey Chris,
> >>>
> >>> The met directory was for generated metadata by the crawler or was it
> for the curator? I updated the page with a description of the paths and
> renamed failed_ingest_files to failure. On the next pass I'll start posting
> a description of the source structure that will be available as output from
> the RADiX archetype.
> >>>
> >>> Thanks,
> >>> Paul
> >>>
> >>> On May 17, 2011, at 7:45 PM, Mattmann, Chris A (388J) wrote:
> >>>
>  Hey Guys,
> 
>  FYI:
> https://cwiki.apache.org/confluence/display/OODT/RADiX+Powered+By+OODT
> 
>  I updated the page to propose a:
> 
>  /data
>  /archive
>  /staging
>  /work
>  /met
>  /failed_ingest_files
> 
>  Dir structure for RADIX to more closely match our existing OODT
> deployments.
> 
>  Cheers,
>  Chris
> 
>  ++
>  Chris Mattmann, Ph.D.
>  Senior Computer Scientist
>  NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>  Office: 171-266B, Mailstop: 171-246
>  Email: chris.a.mattm...@nasa.gov
>  WWW:   http://sunset.usc.edu/~mattmann/
>  ++
>  Adjunct Assistant Professor, Computer Science Department
>  University of Southern California, Los Angeles, CA 90089 USA
>  ++
> 
> >>>
> >>
> >>
> >> ++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattm...@nasa.gov
> >> WWW:   http://sunset.usc.edu/~mattmann/
> >> ++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++
> >>
> >
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>


-- 

Sent from a Tin Can attached to a String

Re: [jira] [Resolved] (OODT-156) Rewrite Workflow Monitor webapp using Apache Wicket

[Cameron Goodale]

Chris,

I have seen this webapp in action and it really is impressive.  Good job
learning wicket.  I am looking forward to a 3rd wicket app...*hint *hint.

For anyone on the list who is wondering what the workflow monitor looks like
in action, take a look at this:

http://min.us/mvfGVMn

HTH

Cameron

On Fri, May 13, 2011 at 8:43 AM, Chris A. Mattmann (JIRA)
wrote:

>
> [
> https://issues.apache.org/jira/browse/OODT-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Chris A. Mattmann resolved OODT-156.
> 
>
>Resolution: Fixed
>
> - fixed in r1102794. Wow, this was a massive undertaking!
>
> I removed the old workflow JSP webapp as part of r1102794. Also, this is my
> 2nd Wicket webapp so I'm getting better, but tons of room for improvement!
> Comments/updates welcome!
>
> > Rewrite Workflow Monitor webapp using Apache Wicket
> > ---
> >
> > Key: OODT-156
> > URL: https://issues.apache.org/jira/browse/OODT-156
> > Project: OODT
> >  Issue Type: Improvement
> >  Components: workflow manager
> >Reporter: Chris A. Mattmann
> >Assignee: Chris A. Mattmann
> >  Labels: java, monitor, webapp, wicket, workflow
> > Fix For: 0.3
> >
> > Attachments: OODT-156.Mattmann.050411.patch.txt,
> OODT-156.Mattmann.050911.patch.txt
> >
> >
> > Similar to OODT-155, except for the Workflow Monitor web application.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>



-- 

Sent from a Tin Can attached to a String

Re: [jira] [Commented] (OODT-187) Standardize File Pathing within filemgr.properties

[Cameron Goodale]

Chris,

I agree that we really should support BOTH file:/// and / for pathing within
the properties files.  My goal in doing this exercise is to also update the
pre-configured properties so they follow the file:/// convention.  Guess I
am just a little OCD about my properties files...especially when my
filemanager will not restart properly because I was using a '/' instead of
'file:///'.

This is also something we came across in building out RADiX.

-Cam

On Wed, May 4, 2011 at 11:16 AM, Chris A. Mattmann (JIRA)
wrote:

>
>[
> https://issues.apache.org/jira/browse/OODT-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028872#comment-13028872]
>
> Chris A. Mattmann commented on OODT-187:
> 
>
> Hi Cameron,
>
> I'm OK with the general principle of your issue -- however, I'd favor
> either approach. That is, let's allow folks to use *either* file:// or just
> plain absolute paths in each of the pathing parameters.
>
> Thoughts?
>
>
> > Standardize File Pathing within filemgr.properties
> > --
> >
> > Key: OODT-187
> > URL: https://issues.apache.org/jira/browse/OODT-187
> > Project: OODT
> >  Issue Type: Improvement
> >      Components: file manager
> >Affects Versions: 0.2
> >Reporter: Cameron Goodale
> >Assignee: Cameron Goodale
> >Priority: Minor
> > Fix For: 0.3
> >
> >
> > At the moment some of the file manager properties are set with
> > file:///path/to/file/or/directory
> > others are set with
> > /path/to/file/or/directory
> > Excerpt from filemgr.properties (properties that need to be addressed)
> >
> org.apache.oodt.cas.filemgr.catalog.mappeddatasource.mapFile=/path/to/ops.catalog.typemap.properties
> >
> org.apache.oodt.cas.filemgr.catalog.lucene.idxPath=/path/to/your/lucene/index
> > org.apache.oodt.cas.filemgr.mime.type.repository=/path/to/mime-types.xml
> > It would really help to standardize on one.  So I propose we stick with
> the file:/// version since this is the URI syntax.  Of course we will have
> to fix this and remain backwards compatible so we will need to support both
> versions.  Once this issue has been addressed we also need to update the
> default filemgr.properties file to reflect this change to file:/// moving
> forward.
> > This will help new OODT users follow a standard way to set properties.
>  Eventually I'd like to see this standard followed across all OODT
> components. (crawler, workflow, etc)
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>



-- 

Sent from a Tin Can attached to a String

Re: Obtaining workflow instance unique ID

[Cameron Goodale]

Chris,

That is a huge help!  My term serialize (which i might be using in the wrong
context = probably should be replaced with SEQUENTIAL instead) means running
'things' (workflow instances, task instances) in a row instead of in
parallel.  So with that explained let me copy those questions to the top of
this email, to see what your answers are:

>   - You cannot use standard configuration to Serialize Workflows
spawned from a single Event
(So if an event spawns 4 workflows they will try to run in parallel by
default, but can the workflow models be configured to run in a specific
order?)

>   - [Can tasks be parallelized/serialized using config?]
(Same use of serialize here, meaning if a Workflow Model has 3 tasks, will
the workflow instance try to run all 3 tasks at once, or in sequence.  What
is the default behavior?  And can we switch between the two using
configuration?)


Sure -- by creating a workflow with a single task. I'm not sure what
serialized means? Are you talking about the functionality of the Workflow
Instance repository, which (de-)serializes the Workflow Metadata, Task
configuration and current status?


Thanks Guys.

-Cameron



On Thu, Apr 28, 2011 at 8:11 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Cameron,
>
> Looking good! Comments inline below:
>
> On Apr 26, 2011, at 8:56 PM, Cameron Goodale wrote:
>
> > Ok OODT List I am I think I am finally getting a formal OODT-WORKFLOW
> > Education here.  But I still have some questions related to Workflow in
> > general. I hope to get some understanding and to also NAIL DOWN the
> > vocabulary surrounding Workflow.  I read the excellent Developer Guide
> here:
> >
> http://oodt.apache.org/components/maven/workflow/development/developer.html
> > but I still have some question I hope I can get clarified.
>
> Thanks for actually checking it out rather than first declaring that you
> don't know what's going on! :) There are a lot of documentation like that on
> the website and glad you found it useful.
>
> >
> > Vocabulary (and a sort of TRUE/FALSE, so please correct the list I am
> > making)
> >
> >
> > Workflow Heirarchy:
> >
> >   - Event
> >  - Workflow
> > - Task = java class
> >
> >
> > Event
> >
> >   - This is where [ALL] workflows must begin.
> >   - An Event initializes 1 or more workflows
> >   - IF an Event initializes more than 1 workflow the workflows will run
> in
> >   parallel
>
> Yep.
>
> >   - You cannot use standard configuration to Serialize Workflows spawned
> >   from a single Event
>
> Not sure what this means? What do you mean by "Serialize"?
>
>
> >   - Events DO NOT generate an INSTANCE_ID
>
> Events don't -- Workflows that are turned into Workflow Instances do. The
> Workflow Engine is responsible for setting and returning this ID as part of
> creating a Workflow Instance from a Workflow (model).
>
> >
> >
> > Workflow
> >
> >   - Workflows DO generate an INSTANCE_ID
>
> (Workflow Instances do -- Workflows are just models).
>
> >   - Workflows must be used to initialize tasks
> >   - Workflows are a way to logically GROUP Tasks together
> >   - Workflows spawn 1 or more tasks
>
> Yep.
>
> >
> > Task
> >
> >   - Lowest atomic level
> >   - Tasks map to a single Java Class
> >   - [Can tasks launch non-Java Code?]
>
> Yep via system calls, via CAS-PGE, via the Script task, etc.
>
> >   - [Can tasks be parallelized/serialized using config?]
>
> Sure -- by creating a workflow with a single task. I'm not sure what
> serialized means? Are you talking about the functionality of the Workflow
> Instance repository, which (de-)serializes the Workflow Metadata, Task
> configuration and current status?
>
>
> >   - [Do tasks generate an INSTANCE_ID?]
>
> Nope, but they generate a TaskId, which is unique.
>
> >   - [Can tasks pass input/output to each other?]
>
> Sure, via the common Metadata context.
>
> >
> >
> > I think that is enough questions/statements for now.  I am eager to read
> > your replies.
>
> Let me know what you think.
>
> Cheers,
> Chris
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>


-- 

Sent from a Tin Can attached to a String

Re: Obtaining workflow instance unique ID

[Cameron Goodale]

Ok OODT List I am I think I am finally getting a formal OODT-WORKFLOW
Education here.  But I still have some questions related to Workflow in
general. I hope to get some understanding and to also NAIL DOWN the
vocabulary surrounding Workflow.  I read the excellent Developer Guide here:
 http://oodt.apache.org/components/maven/workflow/development/developer.html
 but I still have some question I hope I can get clarified.

Vocabulary (and a sort of TRUE/FALSE, so please correct the list I am
making)


Workflow Heirarchy:

   - Event
  - Workflow
 - Task = java class


Event

   - This is where [ALL] workflows must begin.
   - An Event initializes 1 or more workflows
   - IF an Event initializes more than 1 workflow the workflows will run in
   parallel
   - You cannot use standard configuration to Serialize Workflows spawned
   from a single Event
   - Events DO NOT generate an INSTANCE_ID


Workflow

   - Workflows DO generate an INSTANCE_ID
   - Workflows must be used to initialize tasks
   - Workflows are a way to logically GROUP Tasks together
   - Workflows spawn 1 or more tasks

Task

   - Lowest atomic level
   - Tasks map to a single Java Class
   - [Can tasks launch non-Java Code?]
   - [Can tasks be parallelized/serialized using config?]
   - [Do tasks generate an INSTANCE_ID?]
   - [Can tasks pass input/output to each other?]


I think that is enough questions/statements for now.  I am eager to read
your replies.

-Cameron




On Tue, Apr 26, 2011 at 4:47 AM, Bruce Barkstrom wrote:

> As an outside thought, you might want to think of a workflow as
> a kind of graph whose vertices are either processes or "files" (metadata
> perhaps being one kind of file - if you want to think of database
> transactions
> as "files", you can add those to the category of "file").  The edges in the
> graph are events.  Maybe you want to have an inventory of vertices or
> an inventory of edges.  If you want to think about a hierarchical graph,
> that might be useful as well, although the data structure and access
> methods might be a bit more complex.
>
> If you want a graph (or collection of graphs - one graph per workflow),
> you might use hashing on the ID's to access either vertices or edges.
>
> Interesting problem.
>
> Bruce R. Barkstrom
>
> On Mon, Apr 25, 2011 at 11:00 PM, Verma, Rishi (317I)
>  wrote:
> > Hi Paul,
> >
> > Thanks a lot for your suggestions.
> >
> > I agree, the second approach is probably the way to go. Although, would
> it be more beneficial to save and return a single 'event' ID instead of
> multiple workflow instance IDs? I'm under the assumption that workflow
> behaves something akin to a job submission system. This assumption may be
> incorrect, but it might be beneficial to just get back a single unique ID
> for the launch of a workflow top-level event as opposed to getting back a
> list of IDs...
> >
> > So perhaps within XmlRpcWorkflowManager.handleEvent, we could generate
> and save a unique ID for the single event? This ID could be queried by a
> client.
> >
> > Thanks!
> > rishi
> >
> > On Apr 25, 2011, at 1:25 PM, Ramirez, Paul M (388J) wrote:
> >
> >> Hey All,
> >>
> >> Thought this might be useful to the larger OODT community.
> >>
> >>
> >>
> >> Hey Rishi,
> >>
> >> I talked with Mike about this last week but never got back to him with a
> concrete answer. However, after putting some more thought into it there are
> 2 approaches I can think of.
> >>
> >> First Approach:
> >> 1) Put some type of user generated id in the metadata object that gets
> into the workflow instance metadata
> >> 2) Page through the lists of workflow instances that the workflow
> manager has and identify those with your tag. Remember an event can kick off
> more than one workflow.
> >> 3) Any that have metadata that match your user generated id are your
> workflow instances you're interested in.
> >>
> >> Second Approach:
> >> 1) Add a new method to the client and server so that list of workflow
> instance ids is returned
> >>
> >> Looking at XmlRpcWorkflowManager.java
> >>
> >>
> >>public boolean handleEvent(String eventName, Hashtable metadata)
> >>throws RepositoryException, EngineException {
> >>LOG.log(Level.INFO, "WorkflowManager:
> Received event: " + eventName);
> >>
> >>List workflows = null;
> >>
> >>try {
> >>workflows = repo.getWorkflowsForEvent(eventName);
> >>} catch (Exception e) {
> >>e.printStackTrace();
> >>throw new RepositoryException(
> >>"Exception getting workflows associated with event: "
> >>+ eventName + ": Message: " +
> e.getMessage());
> >>}
> >>
> >>if (workflows != null) {
> >>for (Iterator i = workflows.iterator(); i.hasNext();) {
> >>Workflow w = (Workflow) i.next();
> >>LOG.log(Level.INFO,
> "WorkflowManager: Workflow " + w.getName(

Re: CAS_PGE's ExternExtractorMetWriter config file

[Cameron Goodale]

Faranak,

I just dug into this a little deeper and you just need to add one line of
code to the Python Extractor.  Make the FIRST line:

#!/usr/bin/python


Then retry running the Extractor as ./CARVE_KML_Extractor etc
With that line first, the shell will realize the script is a python
program and run it properly.


It will send a lot of messages to the Terminal, but it will run.  I
designed the script to be run using python.


Hope that helps, and sorry to hear about your eyes.


-Cameron






On Sat, Apr 16, 2011 at 1:25 PM, Cameron Goodale  wrote:

> Faranak,
>
> I haven't tried the code yet, but I think the problem is you call python
> code like this:
>
> python CARVE_KML_Extractor.py
>
> I think using ./ tells the she'll to run it. The shell doesn't understand
> how to import modules.
>
> Try using python instead of the ./ and let me know how it works.
>
> Cameron
> On Apr 15, 2011 9:24 PM, "Davoodi, Faranak (388J)" <
> faranak.davo...@jpl.nasa.gov> wrote:
> > Due to having a problem with my eyes, I was hesitant to go through this
> > documentation. I finally endured the pain(literally) and went through it.
> > Apparently those lines are for running that mp3 java code. Please answer
> > my question below:
> >
> > 1.
> > The python extractor I have generates the file bellow:
> >
> > 
> > http://oodt.jpl.nasa.gov/1.0/cas";>
> > 
> > MasterQualityFlag
> > Good
> > 
> > 
> > ProcessingLevel
> > L1
> > 
> > 
> >
> >
> > I do: chmod +x script.py however, it when I try to run it as ./script.py.
> > It wouldn't and complains about :
> > ./CARVE_KML_Extractor.py: line 20: import: command not found
> > ./CARVE_KML_Extractor.py: line 21: import: command not found
> > ./CARVE_KML_Extractor.py: line 22: import: command not found
> > from: can't read /var/mail/xml.dom.minidom
> > However if I simply run like: python script.py [inputProduct][outputdir],
> > it works fine. Do I need to be able to run that python script like
> > ./script.py to be able to use it as an metadata extractor.
> >
> > Cameron, since this is your code. Would you help me and see what you can
> > do to make it run like: ./script.py.
> >
> > 2. I noticed that in the link
> > http://oodt.apache.org/components/maven/curator/user/basic.html, the
> > python code you just specify the input file:
> > ./mp3PythonExtractor.py
> /usr/local/staging/products/mp3/Bach-SuiteNo2.mp3.
> > Since I need to integrate this to my cas-pge crawler, and it needs to
> > perform this metadata extraction on the product just generated, what
> > exactly I need to pass as argument in terms of [inputFile][outmetpath]?
> >
> > Thanks,
> > Faranak
> >
> >
> >
> >
> > On 4/15/11 7:51 PM, "Davoodi, Faranak (388J)"
> >  wrote:
> >
> >>Based on the document you have sent me, for the simple python script that
> >>runs like this: kml.python [input][outputmet]. I need to have the config
> >>file like:
> >>
> >>
> >>
> >>http://oodt.jpl.nasa.gov/1.0/cas";>
> >> 
> >>
>
> >>/usr/local/extractors/mp3extractor/mp3PythonExtractor.py
> >><
> >>/extractorBinPath>
> >> 
> >> 
> >> 
> >> 
> >>
> >>
> >>
> >>Do I have to specify the working directory like the first line:  >>workingDir="">. Because I don't see that line in the peate's sample file.
> >>Also I see the python extractor has these lines that mines doesn't. Is
> >>this the reason my python script doesn't get run? The python file I have
> >>simply parses the file and generates the output met.
> >>
> >>
> >>cmd = "java -jar /Users/woollard/Desktop/extractors/mp3extractor/"
> >>cmd += "tika-app-0.5-SNAPSHOT.jar -m "+fullPath+" | awk -F:"
> >>cmd += " 'BEGIN {print \"http://oodt.jpl.nasa.gov/1.0/cas\\\";>\"}"
> >>cmd += " {print \"\"$1\"\"substr($2,2)\""
> >>cmd += "\"}' > "+fileName+".met"
> >>
> >>
> >>
> >>
> >>
> >>
> >>#!/usr/bin/python
> >>
> >>import os
> >>import sys
> >>
> >>fullPath = sys.argv[1]
> >>pathElements = fullPath.split("/");
> >>fileName = pathElements[len(pathElements)-1]
> >>fileLocation = fullPath[:(len(

Re: CAS_PGE's ExternExtractorMetWriter config file

[Cameron Goodale]

Faranak,

I haven't tried the code yet, but I think the problem is you call python
code like this:

python CARVE_KML_Extractor.py

I think using ./ tells the she'll to run it. The shell doesn't understand
how to import modules.

Try using python instead of the ./ and let me know how it works.

Cameron
On Apr 15, 2011 9:24 PM, "Davoodi, Faranak (388J)" <
faranak.davo...@jpl.nasa.gov> wrote:
> Due to having a problem with my eyes, I was hesitant to go through this
> documentation. I finally endured the pain(literally) and went through it.
> Apparently those lines are for running that mp3 java code. Please answer
> my question below:
>
> 1.
> The python extractor I have generates the file bellow:
>
> 
> http://oodt.jpl.nasa.gov/1.0/cas";>
> 
> MasterQualityFlag
> Good
> 
> 
> ProcessingLevel
> L1
> 
> 
>
>
> I do: chmod +x script.py however, it when I try to run it as ./script.py.
> It wouldn't and complains about :
> ./CARVE_KML_Extractor.py: line 20: import: command not found
> ./CARVE_KML_Extractor.py: line 21: import: command not found
> ./CARVE_KML_Extractor.py: line 22: import: command not found
> from: can't read /var/mail/xml.dom.minidom
> However if I simply run like: python script.py [inputProduct][outputdir],
> it works fine. Do I need to be able to run that python script like
> ./script.py to be able to use it as an metadata extractor.
>
> Cameron, since this is your code. Would you help me and see what you can
> do to make it run like: ./script.py.
>
> 2. I noticed that in the link
> http://oodt.apache.org/components/maven/curator/user/basic.html, the
> python code you just specify the input file:
> ./mp3PythonExtractor.py /usr/local/staging/products/mp3/Bach-SuiteNo2.mp3.
> Since I need to integrate this to my cas-pge crawler, and it needs to
> perform this metadata extraction on the product just generated, what
> exactly I need to pass as argument in terms of [inputFile][outmetpath]?
>
> Thanks,
> Faranak
>
>
>
>
> On 4/15/11 7:51 PM, "Davoodi, Faranak (388J)"
>  wrote:
>
>>Based on the document you have sent me, for the simple python script that
>>runs like this: kml.python [input][outputmet]. I need to have the config
>>file like:
>>
>>
>>
>>http://oodt.jpl.nasa.gov/1.0/cas";>
>> 
>>
>>/usr/local/extractors/mp3extractor/mp3PythonExtractor.py
>><
>>/extractorBinPath>
>> 
>> 
>> 
>> 
>>
>>
>>
>>Do I have to specify the working directory like the first line: >workingDir="">. Because I don't see that line in the peate's sample file.
>>Also I see the python extractor has these lines that mines doesn't. Is
>>this the reason my python script doesn't get run? The python file I have
>>simply parses the file and generates the output met.
>>
>>
>>cmd = "java -jar /Users/woollard/Desktop/extractors/mp3extractor/"
>>cmd += "tika-app-0.5-SNAPSHOT.jar -m "+fullPath+" | awk -F:"
>>cmd += " 'BEGIN {print \"http://oodt.jpl.nasa.gov/1.0/cas\\\";>\"}"
>>cmd += " {print \"\"$1\"\"substr($2,2)\""
>>cmd += "\"}' > "+fileName+".met"
>>
>>
>>
>>
>>
>>
>>#!/usr/bin/python
>>
>>import os
>>import sys
>>
>>fullPath = sys.argv[1]
>>pathElements = fullPath.split("/");
>>fileName = pathElements[len(pathElements)-1]
>>fileLocation = fullPath[:(len(fullPath)-len(fileName))]
>>productType = "MP3"
>>
>>cmd = "java -jar /Users/woollard/Desktop/extractors/mp3extractor/"
>>cmd += "tika-app-0.5-SNAPSHOT.jar -m "+fullPath+" | awk -F:"
>>cmd += " 'BEGIN {print \"http://oodt.jpl.nasa.gov/1.0/cas\\\";>\"}"
>>cmd += " {print \"\"$1\"\"substr($2,2)\""
>>cmd += "\"}' > "+fileName+".met"
>>
>>os.system(cmd)
>>
>>f = open(fileName+".met", 'a')
>>f.write('ProductType'+productType)
>>f.write('\nFilename')
>>f.write(fileName+'\n'FileLocation')
>>f.write(''+fileLocation+'\n')
>>f.write('')
>>f.close()
>>
>>
>>
>>
>>On 4/15/11 6:22 PM, "Davoodi, Faranak (388J)"
>> wrote:
>>
>>>Thanks Brian. The document was actually very helpful.
>>>
>>>--Faranak
>>>
>>>From: holenoter mailto:holeno...@me.com>>
>>>Reply-To: "dev@oodt.apache.org"
>>>mailto:dev@oodt.apache.org>>
>>>Date: Fri, 15 Apr 2011 14:19:04 -0700
>>>To: "dev@oodt.apache.org"
>>>mailto:dev@oodt.apache.org>>
>>>Cc: "dev@oodt.apache.org"
>>>mailto:dev@oodt.apache.org>>
>>>Subject: Re: CAS_PGE's ExternExtractorMetWriter config file
>>>
>>>http://oodt.apache.org/components/maven/metadata/user/basic.html
>>>
>>>On Apr 15, 2011, at 02:09 PM, "Davoodi, Faranak (388J)"
>>>mailto:faranak.davo...@jpl.nasa.gov>>
>>>wrote:
>>>
>>>I have a couple of out put products that I am trying to extract extra
>>>metadata and add them to the final .met files. Here is how I run my
>>>files:
>>>
>>>Python [someBinDir]/ncdump [PathToPythonExtractor]/extractor1.py
>>>[PathToOutputProduct]/productName [PathToOutPutMet]
>>>
>>>How should I write the extern extractor config file:
>>>
>>>
>>>
>>>
>>>http://oodt.jpl.nasa.gov/1.0/cas";>
>>>
>>>
>envReplace="true">[PathToPythonExtractor]
>>>
>>>python
>>>[someBinDir]/ncdump
>>>
>>>extracto

Re: cas-wm-webapp javadoc breaking build . . .

[Cameron Goodale]

Brian,

I just ran:

svn up
mvn install   (in the top level of oodt just to be sure)
cd webapp/workflow  (i have the trunk checked out into it's own project)
mvn package

Got the same problem.  Time to log a JIRA issue on this I think.

Here is what my system is setup as:

cgoodale-mac:workflow cgoodale$ mvn -version
Apache Maven 2.2.1 (r801777; 2009-08-06 12:16:01-0700)
Java version: 1.6.0_24
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x" version: "10.5.8" arch: "x86_64" Family: "mac"

BUILD ERROR BELOW:

[INFO]

[ERROR] BUILD ERROR
[INFO]

[INFO] An error has occurred in JavaDocs report generation:
Exit code: 1 - javadoc: error - No public or protected classes found to
document.

Command line was:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/javadoc
@options @packages

Refer to the generated Javadoc files in
'/Users/cgoodale/Documents/workspace/apache-oodt/oodt/webapp/workflow/target/site/apidocs'
dir.


[INFO]

[INFO] Trace
org.apache.maven.lifecycle.LifecycleExecutionException: An error has
occurred in JavaDocs report generation:
Exit code: 1 - javadoc: error - No public or protected classes found to
document.

Command line was:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/javadoc
@options @packages

Refer to the generated Javadoc files in
'/Users/cgoodale/Documents/workspace/apache-oodt/oodt/webapp/workflow/target/site/apidocs'
dir.

at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:719)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(DefaultLifecycleExecutor.java:556)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:535)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
Caused by: org.apache.maven.plugin.MojoExecutionException: An error has
occurred in JavaDocs report generation:
Exit code: 1 - javadoc: error - No public or protected classes found to
document.

Command line was:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/javadoc
@options @packages

Refer to the generated Javadoc files in
'/Users/cgoodale/Documents/workspace/apache-oodt/oodt/webapp/workflow/target/site/apidocs'
dir.

at
org.apache.maven.plugin.javadoc.JavadocReport.execute(JavadocReport.java:298)
at
org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
at
org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
... 17 more
Caused by: org.apache.maven.reporting.MavenReportException:
Exit code: 1 - javadoc: error - No public or protected classes found to
document.

Command line was:
/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/javadoc
@options @packages

Refer to the generated Javadoc files in
'/Users/cgoodale/Documents/workspace/apache-oodt/oodt/webapp/workflow/target/site/apidocs'
dir.

at
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5021)
at
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeReport(AbstractJavadocMojo.java:1964)
at
org.apache.maven.plugin.javadoc.JavadocReport.generate(JavadocReport.java:124)
at
org.apache.maven.plugin.javadoc.JavadocReport.execute(JavadocReport.java:292)
... 19 more
[INFO]

[INFO] Total time: 10 seconds
[INFO] Finished at: Tue Apr 12 17:45:31 PDT 2011
[INFO] Final Memory: 33M/103M
[INFO]













On Tue, Apr 12,

[DISCUSS] - PGE Documentation

[Cameron Goodale]

Hey Team,

I opened OODT-171  to track
the progress of making documentation for the PGE component.  The goal is to
get a Basic User Guide out the door in the next 2 weeks.  I know that
several of you have used PGE with great success, so if you can contribute to
the documentation it would really be appreciated.

Right now I am just collecting thoughts, ideas and brainstorming on this
wiki page:
https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Documentation

Once we have enough content to warrant a Basic User Guide I will capture the
content and feed it back into the trunk and site.  If you want to add in
Advanced User Guide material, that is great too, since we will need one of
those as well, I just wanted to cover the basics during this first
iteration.

Thanks to everyone and I look forward to your contributions.

Best Regards,


Cameron

-- 

Sent from a Tin Can attached to a String

Re: NASA Open Source Summit

[Cameron Goodale]

I think since the presentation was on slideshare.net, there is no recorded
voice over.  I think the play button in the middle is a tad misleading as it
just advances the slides at a set interval like every 7 seconds.

I like it Chris.  I hope the feedback was positive and the breakout sessions
productive.  Cannot wait to hear back on how the presentation was received.

-Cameron

On Tue, Mar 29, 2011 at 4:56 PM, Davoodi, Faranak (388J) <
faranak.davo...@jpl.nasa.gov> wrote:

> Thanks Chris for sending us the link to your talk. However, when I try to
> play it, there is no sound. Should I download the whole file first?
>
>
> --Faranak
>
> On 3/29/11 4:37 PM, "Mattmann, Chris A (388J)"
>  wrote:
>
> >Hey OODT Community,
> >
> >I just gave a talk on Open Source, Apache, and OODT at the NASA Open
> >Source Summit.
> >
> >The link to the talk is here:
> >http://www.slideshare.net/ckleclerc/nasaoss-mattmann
> >
> >Comments welcome!
> >
> >Cheers,
> >Chris
> >
> >++
> >Chris Mattmann, Ph.D.
> >Senior Computer Scientist
> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >Office: 171-266B, Mailstop: 171-246
> >Email: chris.a.mattm...@nasa.gov
> >WWW:   http://sunset.usc.edu/~mattmann/
> >++
> >Adjunct Assistant Professor, Computer Science Department
> >University of Southern California, Los Angeles, CA 90089 USA
> >++
> >
>
>


-- 

Sent from a Tin Can attached to a String

Maven3 Update on OS X...and how to get back to Maven2

[Cameron Goodale]

For everyone on this list who has been building OODT on a Mac, be sure to
star,flag,save this email for later.  If you haven't encountered the Maven 2
to Maven 3 gremlin yet, it will show up eventually.

Here is a great blog post on the errors you might see and a really simple
and straight forward way to revert back to Maven2.

http://cyntech.wordpress.com/2011/03/09/maven-2-error/

Hope this helps.


-Cameron

-- 

Sent from a Tin Can attached to a String

Re: cannot install OODT, help!

[Cameron Goodale]

Derek, that seems like a lot of errors. Just to cover the basics, what
version of maven are you using? OODT only builds on maven 2. This causes a
lot of issues when trying to build.
Check the version and let me know what you find.

Cameron
On Mar 26, 2011 10:34 PM, "Derek"  wrote:
> Hi,
>
> I cannot install OODT in my system neither using Maven Eclipse plugin nor
by
> mvn -install command.
>
> My OS is Win 7 64bit and the Following is the whole stack trace:
>
> Would you please help me to figure out what's the problem? I will really
> appreciate it.
>
> Derek
>
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:oodt-core:pom:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 257,
> column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-plugin is missing. @ line 276,
> column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @ line 264, column
21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:oodt-commons:jar:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 257, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 264, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:pcs-input:jar:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 257, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 264, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:cas-metadata:jar:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 257, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-plugin is missing. @ line 70,
column
> 15
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 264, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:oodt-xmlquery:jar:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 257, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-plugin is missing. @ line 53,
column
> 15
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 264, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:oodt-sso:jar:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 257, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 276, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 264, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective
model
> for org.apache.oodt:cas-filemgr:jar:0.2
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-compiler-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 257, column 21
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.oodt:oodt-core:0.2,
> E:\workspace\578\apache-oodt-0.2-src\core\pom.xml, line 264, column 21
> [WARNING]
> [WARNING] Some problems were enco

Re: FileManager and RepositoryManager

[Cameron Goodale]

Niloofar,
I think you are looking for a versioner. A versioner determines where a
product is archived after ingestion. The versioner is declared in the
product-types.xml file.
Hope that helps.

Cameron
On Mar 17, 2011 6:24 PM, "niloofar montazeri" 
wrote:
> Hi!
>
> I am having trouble with trackimg a design decision about the FileManager
> component back into the code.
>
> The decision is: "The system Interface uses the information about Product
> Type policy made available by the Repository Manager in order to
understand
> whether or not the product should be transferred, where its root
repository
> path should be, and so on ."
>
> Looking at the code of ingestProduct method and examining all the methods
> that access the FileManager's repositoryManager instance, I found that the
> only role the repositoryManager plays in ingestProduct method is to assign
> meta-data to the product. After that, there is no such decision point for
> sending or not sending the product (the product is always transferred):
>
> public String ingestProduct(Hashtable productHash,
> Hashtable metadata, boolean clientTransfer)
> throws VersioningException, RepositoryManagerException,
> DataTransferException, CatalogException {
>
> Product p = XmlRpcStructFactory.getProductFromXmlRpc(productHash);
>
> try {
> // first, create the product
> p.setTransferStatus(Product.STATUS_TRANSFER);
> catalogProduct(p);
>
> // now add the metadata
> Metadata m = new Metadata();
> m.addMetadata((Hashtable)metadata);
>
> *addMetadata(p, m);*
>
> if (!clientTransfer) {
> LOG.log(Level.FINEST,
> "File Manager: ingest: no client transfer enabled, "
> + "server transfering product: [" + p.getProductName() +
> "]");
>
> // version the product
> Versioner versioner = null;
> try {
> versioner = GenericFileManagerObjectFactory
> .getVersionerFromClassName(p.getProductType().getVersioner());
> versioner.createDataStoreReferences(p, m);
> } catch (Exception e) {
> LOG.log(Level.SEVERE,
> "ingestProduct: VersioningException when versioning Product: "
> + p.getProductName() + " with Versioner "
> + p.getProductType().getVersioner() + ": Message: "
> + e.getMessage());
> throw new VersioningException(e);
> }
>
> // add the newly versioned references to the data store
> addProductReferences(p);
>
> // now transfer the product
> try {
> dataTransfer.transferProduct(p);
> // now update the product's transfer status in the data store
> p.setTransferStatus(Product.STATUS_RECEIVED);
>
> try {
> catalog.setProductTransferStatus(p);
> } catch (CatalogException e) {
> LOG.log(Level.SEVERE, "ingestProduct: CatalogException "
> + "when updating product transfer status for Product: "
> + p.getProductName() + " Message: " + e.getMessage());
> throw e;
> }
> } catch (Exception e) {
> LOG.log(Level.SEVERE,
> "ingestProduct: DataTransferException when transfering
> Product: "
> + p.getProductName() + ": Message: " + e.getMessage());
> throw new DataTransferException(e);
> }
> }
>
> // that's it!
> return p.getProductId();
> } catch (Exception e) {
> e.printStackTrace();
> throw new CatalogException("Error ingesting product [" + p + "] : "
> + e.getMessage());
> }
>
>
> Am I missing something?
>
> Thanks,
> Niloofar

Re: unable to build OODT

[Cameron Goodale]

Andy, thanks for the thorough root cause analysis. One word of caution re:
maven on the mac, the latest round of updates will update the default maven
that comes installed to version 3.

It seems you are out of the woods now but something to be aware of when
using maven on the mac.

Best regards,

Cameron
On Mar 16, 2011 10:18 PM, "Andy Bauch"  wrote:
> Cameron:
>
> Andys-Mac-Book-Pro:Desktop andybauch$ md5 apache-oodt-0.2-src.tar.gz
> MD5 (apache-oodt-0.2-src.tar.gz) = a90f7b457f3b1dc9eb460343ee280481
>
> Chris:
>
> After trying to build in Windows with Maven 2.2.1 I ran into the XMLTest
> failure that others were encountering. Unfortunately I can't recall the
> subsequent error I was looking into when I decided to switch gears to my
> Mac.
>
> The steps I can take to reproduce the issue (which in reality appears to
be
> an issue with my Unarchiver application) are:
>
> 1. wget "
http://www.fightrice.com/mirrors/apache//oodt/apache-oodt-0.2-src.tar.gz";
> 2. In Finder double-click on apache-oodt-0.2-src.tar.gz to run Unarchiver
> 3. cd apache-oodt-0.2-src
> 4. find ./ -type f | grep 644
>
> When I use the tar command as you did everything seems to extract
properly.

Re: unable to build OODT

[Cameron Goodale]

Andy, did you run a checksum on the source tar ball?

Cameron
On Mar 16, 2011 9:09 PM, "Andy Bauch"  wrote:
> I tried building with Maven 2.2.1 per your suggestion but was running into
> different issues entirely.
>
> I decided to switch gears and start building on my Mac (which apparently
had
> Maven 2.2.1 installed by default). After a couple hours of build errors I
> tracked the issue down to about 20 source files whose names ended with
> "644" for some odd reason. I tried downloading the OODT source again
> from another computer and confirmed the oddly-named files.
>
> I identified all of the weird files with "find ./ -type f | grep 644" from
> the root OODT directory and renamed them all, removing the "644".
After
> doing so my build and install were successful.
>
> Thanks.

File Manager Policy Wiki is UP!

[Cameron Goodale]

Everyone,

After several discussions surrounding 'How should I define my FileManager
Policy?' I decided (with the help of John Tran and Chris Mattmann) to start
a wiki page on the topic of policy for File Manager.

You can find it here:
https://cwiki.apache.org/confluence/display/OODT/Everything+you+want+to+know+about+File+Manger+Policy

Please give it a look and leave comments or questions, or even add in your
own experiences.  File Manager Policy is really whatever you make of it, so
the wiki page is meant to collect some 'Best Practice' solutions to help
folks setup their policy correctly the first time.

Cheers!

Cameron

Re: Observation from the Peanut Gallery

[Cameron Goodale]

Adam,

My efforts have been focused largely on getting some of the cas components
documented thoroughly.  My current component of interest is CAS Workflow,
but I have also used the CAS FileManager and CAS Crawler on projects in the
past with great success.

My thought is that the 3 components I have listed do not depend on or use
Tomcat or Jetty in their standard config, so at  best I will increase the
documentation surrounding Setup and config.

I hope that when I have the docs published, you are willing to go through
them and provide feedback.

Thanks,

-Cameron

Sent from a Tin Can attached to a String

On Sun, Feb 13, 2011 at 6:59 PM, Adam Estrada  wrote:

> All,
>
> Would it be possible to distribute a preconfigured instance of Jetty or
> Tomcat with OODT to help eliminate the confusing configuration process? I
> think that a preconfigured setup just to get you up and running would be
> *very* helpful. I have tried several times to get everything going and I am
> nearly there but it's been a rough ride to say the least. Other projects do
> this in order to help streamline the initial setup and evaluation of the
> technology...
>
> Any thoughts on this?
>
> Adam
>
>

Re: [ANNOUNCE] Release of Apache OODT 0.2

[Cameron Goodale]

Guess it's about time to give the website an update in the NEWS section and
update all the download links to point at the new 0.2 release.  JIRA issue
and website updates are coming, so stay tuned.

-Cameron

On Mon, Feb 7, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Paul: Awesome job, Mr. RM!
>
> Long live Apache OODT 0.2!
>
> Cheers,
> Chris
>
> On Feb 4, 2011, at 5:27 PM, Ramirez, Paul M (388J) wrote:
>
> > The Apache OODT community is proud to announce our
> > 0.2 release, which is our first release as an Apache TLP.
> >
> > Please visit our project website to learn more about Apache
> > OODT, a data grid framework for transparent discovery of
> > disparate science resources.
> >
> > *http://oodt.apache.org/*
> >
> > Thanks,
> > Apache OODT PPMC
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>


-- 

Sent from a Tin Can attached to a String

Re: Workflow monitor webapp

[Cameron Goodale]

Rishi and Chris,

Sorry I am late to the Workflow WebApp party, but I noticed all your JIRA
issues reference using Tomcat 6.0.18, but the web-app-tomcat xml config file
has 5.5.9 in the filename.

Could this issue be caused by running a tomcat 5.X webapp in Tomcat 6?


-Cameron


On Tue, Feb 8, 2011 at 6:43 PM, Verma, Rishi (317I) <
rishi.ve...@jpl.nasa.gov> wrote:

> Hi Chris,
>
> I've detailed the specifics within the following JIRA issues:
> https://issues.apache.org/jira/browse/OODT-131
> https://issues.apache.org/jira/browse/OODT-132<
> https://issues.apache.org/jira/browse/OODT-13>
> https://issues.apache.org/jira/browse/OODT-133
>
> Essentially the problem is the hyperlinks within the workflow webapp return
> Java/JSP exceptions. These occur from a fresh install of the workflow
> manager webapp (ie. no editing of config files or modifications performed).
> I'm not sure what configuration is necessary, it seems that editing the
> "META-INF/workflow-webapp-tomcat-5.5.9.xml" may be necessary?
>
> Thanks for your help!
> Rishi
>
> On Feb 7, 2011, at 3:55 PM, Mattmann, Chris A (388J) wrote:
>
> Hey Rishi,
>
> Great questions. Let's start with some specifics -- which links in
> particular are failing? Can you show some screenshots (in particular, filing
> a JIRA issue, or set of them would be great).
>
> I'd be happy to help out here!
>
> Cheers,
> Chris
>
> On Feb 7, 2011, at 1:30 PM, Verma, Rishi (317I) wrote:
>
> Hello -
>
> I was just wondering, is there any documentation (xdoc, slides, notes)
> available for how to configure and run an instance of the OODT Workflow
> Monitor webapp with an active workflow manager?
>
> I've successfully built and deployed the workflow monitor webapp to tomcat;
> however, many of the workflow status hyperlinks yield JSP exceptions even
> though I have a running instance of workflow manager at the required URL.
> This is undoubtedly due to a lack of configuration of the webapp on my part.
> Problem is I'm just not sure which files need to be configured with what
> parameters.
>
> Thanks!
> Rishi
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>


-- 

Sent from a Tin Can attached to a String

Re: [VOTE] Apache OODT 0.2 Release

[Cameron Goodale]

+1 from me

Build took 4 re-tries due to maven hanging...but it did finish and I noticed
453 javadoc warnings.  Annoying yes, blockers No.  I will open issues in
JIRA to resolve these two minor irritations.

Formal testing:  I installed the Workflow component and successfully ran
through the test workflow from the user guide.  Worked Great!

Thanks for the bump Paul, it was just what I needed.

-Cameron




On Mon, Jan 31, 2011 at 5:54 PM, Mattmann, Chris A (388J) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Paul,
>
> +1 from me. I downloaded this, verified the signatures, and checksums:
>
> [chipotle:oodt-dev/0.2-release/rc2] mattmann% gpg --verify *.asc
> gpg: Signature made Tue Jan 25 22:23:47 2011 PST using RSA key ID 2C47D568
> gpg: Good signature from "Paul Michael Ramirez (CODE SIGNING KEY) <
> prami...@apache.org>"
> gpg: WARNING: This key is not certified with a trusted signature!
> gpg:  There is no indication that the signature belongs to the
> owner.
> Primary key fingerprint: FA78 6E77 5FF1 0023 D990  4D13 A5DB C018 2C47 D568
> [chipotle:oodt-dev/0.2-release/rc2] mattmann% md5
> apache-oodt-0.2-src.tar.gz
> MD5 (apache-oodt-0.2-src.tar.gz) = a90f7b457f3b1dc9eb460343ee280481
> [chipotle:oodt-dev/0.2-release/rc2] mattmann% cat *md5
> MD5 (apache-oodt-0.2-src.tar.gz) = a90f7b457f3b1dc9eb460343ee280481
> [chipotle:oodt-dev/0.2-release/rc2] mattmann% openssl sha1
> apache-oodt-0.2-src.tar.gz
> SHA1(apache-oodt-0.2-src.tar.gz)= 954c74f1fc8255729669dea318732db7b759ca54
> [chipotle:oodt-dev/0.2-release/rc2] mattmann% cat *.sha1
> SHA1(apache-oodt-0.2-src.tar.gz)= 954c74f1fc8255729669dea318732db7b759ca54
> [chipotle:oodt-dev/0.2-release/rc2] mattmann%
>
> All unit tests pass.  And most importantly I ran an end-to-end system test
> by upgrading the EDRN environment now depend on Apache OODT 0.2 with the
> following components:
>
> filemgr
> metadata
> curator
> crawler
>
> Great job on the release!
>
> Cheers,
> Chris
>
> On Jan 25, 2011, at 10:24 PM, Ramirez, Paul M (388J) wrote:
>
> > Hi Folks,
> >
> > I have posted a 2nd release candidate for the Apache OODT 0.2 release.
> The
> > source code is at:
> >
> > http://people.apache.org/~pramirez/apache-oodt-0.2/rc2/
> >
> > For more detailed information, see the included CHANGES.txt file for
> details on
> > release contents and latest changes. The release was made using the OODT
> > release process, documented on the Wiki here:
> >
> > https://cwiki.apache.org/confluence/display/OODT/Release+Process
> >
> > The release was made from the OODT 0.2 branch (r1063465) at:
> >
> > http://svn.apache.org/repos/asf/oodt/branches/0.2/
> >
> > Please vote on releasing these packages as Apache OODT 0.2. The vote is
> > open for the next 72 hours.
> >
> > Only votes from OODT PMC are binding, but folks are welcome to check the
> > release candidate and voice their approval or disapproval. The vote
> passes
> > if at least three binding +1 votes are cast.
> >
> > [ ] +1 Release the packages as Apache OODT 
> >
> > [ ] -1 Do not release the packages because...
> >
> > Thanks!
> >
> > Paul Ramirez
> >
> > P.S. Here is my +1.
> >
>
>
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>


-- 

Sent from a Tin Can attached to a String

Re: Branched components refer to nonexistent incubator version

[Cameron Goodale]

Paul,

I have only seen SK give it a +1 so far.  I wanted to run through the
FileManager and Workflow components before I give my +1...it built ok
(ignoring the maven stalls getting external jars) I just wanted to check a
little deeper on the two components I am familiar with.  Look for mine
tomorrow morning.

-Cameron

On Wed, Jan 19, 2011 at 2:24 PM, Ramirez, Paul M (388J) <
paul.m.rami...@jpl.nasa.gov> wrote:

> Hey All,
>
> It was brought to my attention that I forgot to update the pom in some of
> the branched components when I went around changing "0.2-incubating" to
> "0.2". If I update this should I roll another RC? Anyhow, let me know. Also
> JPL email is being weird and I haven't seen anyone else vote yet except SK.
>
> Thanks,
> Paul




-- 

Sent from a Tin Can attached to a String