Re: [ANNOUNCE] Welcome Nga Chung as an OODT PMC member and committer!

[Mattmann, Chris A (388J)]

Welcome Nga!!

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Sean Kelly 
Reply-To: "dev@oodt.apache.org" 
Date: Tuesday, April 2, 2013 5:43 AM
To: "dev@oodt.apache.org" 
Cc: "nch...@apache.org" 
Subject: [ANNOUNCE] Welcome Nga Chung as an OODT PMC member and committer!

>Folks:
>
>I'm happy to report that Nga Chung has accepted our invitation to become
>a committer on Apache OODT and to join the OODT Project Management
>Committee. WOOT!
>
>Welcome, Nga. Please feel free to introduce yourself.
>
>--k
>

Re: Google Summer of Code 2013 Proposal - CAS-Product JAX-RS using Apache CXF

[Mattmann, Chris A (388J)]

Hey Ross,

This sounds like an excellent project. I'd be happy to mentor you again,
or else, if some other Apache OODT PMC member would like to step up and
do it, you're in for a treat.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Ross Laidlaw 
Reply-To: "dev@oodt.apache.org" 
Date: Sunday, March 31, 2013 4:23 PM
To: "dev@oodt.apache.org" 
Subject: Google Summer of Code 2013 Proposal - CAS-Product JAX-RS using
Apache CXF

>Hi All,
>
>I'd like to propose a project for GSoC 2013 and request to be considered
>as
>a candidate to work on the project.
>
>The goal would be to implement a JAX-RS option for the CAS-Product
>(webapp/fmprod) module of OODT.  This would initially sit alongside the
>existing data, RDF and RSS servlets and would offer a more powerful means
>for the CAS-Product module to interact with the File Manager.  The
>implementation would use Apache CXF, to complement existing JAX-RS
>implementations in OODT such as those in PCS and CAS-Curator.
>
>I worked on improvements to the CAS-Product RSS servlets last year for
>GSoC
>2012 (e.g. see refs [1] to [8] below).  As a PMC member and committer for
>OODT, I also have experience of submitting patches via JIRA, using Review
>Board and committing updates to the repository.  I therefore feel that I
>would be a good candidate to work on this project for GSoC 2013.
>
>
>Ross
>
>
>[1] https://issues.apache.org/jira/browse/OODT-402
>[2] https://issues.apache.org/jira/browse/OODT-449
>[3] https://issues.apache.org/jira/browse/OODT-452
>[4] https://issues.apache.org/jira/browse/OODT-453
>[5] https://issues.apache.org/jira/browse/OODT-458
>[6] https://issues.apache.org/jira/browse/OODT-469
>[7] https://issues.apache.org/jira/browse/OODT-471
>[8] https://issues.apache.org/jira/browse/OODT-483

ApacheCon NA 2013 slide request

[Mattmann, Chris A (388J)]

Hey Guys,

A small request, see:

https://cwiki.apache.org/confluence/display/OODT/OODT+ApacheCon+2013+Wiki+P
age


For those of you that talked at ApacheCon NA 2013, and haven't uploaded
your
slides to the above page, I would appreciate you doing so.

slideshare.net is a great way to share/link (I believe all the other links
are
using slideshare)

Thanks, all!

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Re: Interested in participating GSoC 2013 with OODT

[Mattmann, Chris A (388J)]

Awesome thanks!

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Rajith Siriwardana 
Reply-To: "dev@oodt.apache.org" 
Date: Thursday, March 28, 2013 3:41 AM
To: dev 
Subject: Re: Interested in participating GSoC 2013 with OODT

>Thanks guys, me too looking forward to work with you guys.
>
>
>Thank you,
>
>T.W.R.O. Siriwardana
>StMIEEE
>Undergraduate
>Department of Computer Science & Engineering
>Faculty of Engineering
>University of Moratuwa
>Sri Lanka.
>
>
>
>On Thu, Mar 28, 2013 at 9:14 AM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hi Rajith!
>>
>> Thank you for your interest in OODT-219! Would love to have you help out
>> on the project.
>>
>> The mailing list doesn't accept attachments, so can you head over the
>> OODT-219 and attach
>> the diagram there?
>>
>
>I have attached the diagram to OODT-219.
>
>
>
>> Thank you and looking forward to working with you on this!
>>
>> Cheers,
>> Chris
>>
>> From: Rajith Siriwardana > rajithsiriward...@ieee.org>>
>> Reply-To: "dev@oodt.apache.org<mailto:dev@oodt.apache.org>" <
>> dev@oodt.apache.org<mailto:dev@oodt.apache.org>>
>> Date: Wednesday, March 27, 2013 9:20 AM
>> To: dev mailto:dev@oodt.apache.org>>
>> Subject: Re: Interested in participating GSoC 2013 with OODT
>>
>> hi
>> small mistake, forgot to attach the high level deployment architecture
>> diagram which i was talking about. note that this architecture diagram
>> changed to use mk_livestatus replacing Nagios XML engine to query
>>stats. I
>> can provide more details upon request  about the the project I carried
>>out.
>>
>> Thank you,
>>
>> T.W.R.O. Siriwardana
>> StMIEEE
>> Undergraduate
>> Department of Computer Science & Engineering
>> Faculty of Engineering
>> University of Moratuwa
>> Sri Lanka.
>>
>>
>> On Wed, Mar 27, 2013 at 9:38 PM, Rajith Siriwardana <
>> rajithsiriward...@ieee.org<mailto:rajithsiriward...@ieee.org>> wrote:
>> hi,
>> I'm a 4th year (final year) undergraduate of Faculty of Engineering
>> University of Moratuwa majoring in Computer Science and Engineering and
>>I
>> would like contribute to OODT.
>> I did my internship last year which spanned for 9 months in WSO2[1].
>>which
>> is a middle-ware company and I worked with their WSO2 Stratos
>>product[2].
>> Basically I was assigned for developing WSO2 Stratos system level health
>> monitoring component and WSO2 Stratos heartbeat monitoring package.  For
>> WSO2 Stratos health monitoring component[3] before finalizing Nagios[4]
>>for
>> monitoring with mk_livestatus[5] replacing NagiosXML engine for query
>> stats, which is shown in the architecture image, I did some work with
>> Ganglia (how to use it for WSO2 Stratos system level health monitoring).
>> Furthermore I took part in test automation hackathon carried out for
>>WSO2
>> ESB[6] around 2 months.
>> So I believe I have enough knowledge to complete the GSoC 2013 project
>>  "Monitor that plugs into ganglia" OODT-219<
>> https://issues.apache.org/jira/browse/OODT-219> [7] with some guidance.
>> Please give me any advice, guidance you'd like to extend.
>> You can access my Linkedin profile via [8]
>>
>> [1]http://wso2.com/
>> [2]http://wso2.com/cloud/stratos/
>> [3]http://mail.wso2.org/mailarchive/architecture/2012-June/008827.html
>> [4]http://www.nagios.org/
>> [5]http://mathias-kettner.de/checkmk_livestatus.html
>> [6]http://wso2.com/products/enterprise-service-bus/
>> [7]https://issues.apache.org/jira/browse/OODT-219
>> [8]http://lk.linkedin.com/in/rajithsiriwardana/
>>
>> Thank you,
>>
>> T.W.R.O. Siriwardana
>> StMIEEE
>> Undergraduate
>> Department of Computer Science & Engineering
>> Faculty of Engineering
>> University of Moratuwa
>> Sri Lanka.
>>
>>
>>

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hi Chintu,

-Original Message-

From: , "Chintu [SGT INC]  (GSFC-586.0)" 
Date: Thursday, March 28, 2013 9:02 AM
To: jpluser , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M (388J)" 
Subject: Re: pushPull component problem

>Chris,
>
>I figured out the problem. Don't know the solution.
>
>So the problem is the symbolic links on FTP server.
>
>/pub/OI-daily-v2/NetCDF/2013/AVHRR  is actually
>/san1/oisst/NetCDF/2013/AVHRR
>Where pub -> san1
>  And OI-daily-v2 -> oisst
>
>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it works.
>Otherwise it does not work.
>
>Another example is : ftp://aftp.cmdl.noaa.gov
>
>/data/radiation/surfrad/dra/2013
>->/data/radiation/surfrad/Desert_Rock_NV/2013
>
>
>
>I don't know how to fix this. Well, not exactly sure where to look. It
>appears that the filename filters are being applied based on the path
>specified in DirStructParserFiles instead of just the filename. Because I
>can see the correct path when I print RemoteFile (path =
>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' )
>on STDOUT.

Interesting. I know that different implementations of the Protocols handle
symlinks in various ways.

Brian Foster is usually excellent at tracking these types of issues down
(CC'ing him directly). Brian any thoughts?

Also see other replies from Rishi and I.

Cheers,
Chris

>
>
>Hopefully you guys can fix this fast or let me know where to look.
>
>Regards
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hi Chintu,

Looking at jVFTP doesn't look like it barfed, looks like it ran,
but just didn't match any files. How about a regex that matches all files,
or something else similar?

The other option I would do is try and run one of the sample configs
(e.g., for MODIS.xml)
and see if those work?

Also try Rishi's suggestions too and let me know what you think.

Thanks.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: , "Chintu [SGT INC]  (GSFC-586.0)" 
Date: Thursday, March 28, 2013 9:28 AM
To: jpluser , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M (388J)" 
Subject: Re: pushPull component problem

>Sorry about not explaining properly. When I said "failed to run", I meant
>class not found. Did not really look deep enough what the problem is.
>Apparently I did not include all jar files properly.
>
>Anyway, I managed to run both ftp client (of course I did change
>ProtocolFactor.xml file). But still no luck. Attaching files with output.
>
>Ftp4CheFtp => Completely barfed
>jvFTP => runs fine but same output as the other plugins.
>
>
>
>Regards
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>On 3/28/13 12:13 PM, "Mattmann, Chris A" 
>wrote:
>
>>Hey Chintu,
>>
>>Thanks -- sorry to be asking all these questions, but what does "failed
>>to
>>run" mean?
>>
>>Did you update the protocol plugins config to ref the new plugins and
>>their FQCN (fully
>>qualified class names?) Do you have a log file for that?
>>
>>Cheers,
>>Chris
>>
>>++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattm...@nasa.gov
>>WWW:  http://sunset.usc.edu/~mattmann/
>>++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++
>>
>>
>>
>>
>>-Original Message-
>>From: , "Chintu [SGT INC]  (GSFC-586.0)" 
>>Date: Thursday, March 28, 2013 9:12 AM
>>To: jpluser , "dev@oodt.apache.org"
>>
>>Cc: "Ramirez, Paul M (388J)" 
>>Subject: Re: pushPull component problem
>>
>>>Since we are not using OODT 0.6, it did not compile straight away.
>>>However, I did manage to compile it with "0.5" version of
>>>cas-protocol-api. But when I was running it, It failed to run.
>>>
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>
>>>On 3/28/13 12:04 PM, "Mattmann, Chris A" 
>>>wrote:
>>>
Hey Chintu,

Thanks for identifying this!

Have you tried the alternative plugins? They use a diff set of FTP
protocol
plugins (that unfortunately are not ALv2 licensed or Category A
approved,
so we can't ship with Apache OODT).

They may do better at handling that symlink than the default version of
commons-ftp
we're using.

Let me know. We'll investigate.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++




-Original Message-
From: , "Chintu [SGT INC]  (GSFC-586.0)"

Date: Thursday, March 28, 2013 9:02 AM
To: jpluser , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M (388J)" 
Subject: Re: pushPull component problem

>Chris,
>
>I figured out the problem. Don't know the solution.
>
>So the problem is the symbolic links on FTP server.
>
>/pub/OI-daily-v2/NetCDF/2013/AVHRR  is actually
>/san1/oisst/NetCDF/2013/AVHRR
>Where pub -> san1
>  And OI-daily-v2 -> oisst
>
>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>works.
>Otherwise it does not work.
>
>Another exam

Re: Interested in participating GSoC 2013 with OODT

[Mattmann, Chris A (388J)]

Hi Rajith,

We're looking forward to you participating.

I just outlined the process to someone else who is trying to participate
in Apache Mesos through GSOC 2013, so I'll repaste here:


  1.  ASF is accepted/declined as GSoC 2013 org (let's hope it's accepted
:) )
  2.  Rajith talks with OODT mentor and OODT mentor agrees to mentor
  3. Rajith comes up with proposal, referencing the GSoC project and JIRA
issue, inputs into
Google Melange and inputs into Google Melange for GSOC 2013
http://www.google-melange.com/gsoc/homepage/google/gsoc2013
  4.  OODT mentor signs up for Melange
http://www.google-melange.com/gsoc/homepage/google/gsoc2013
and lists ASF as organization
  5. Rajith requests OODT mentor as mentor (ASF GSoC 2013 admins
approve/decline proposed project)
  6.  Rajith and OODT mentor find out if the project was approved
  7.  Project happens
  8. OODT mentor and Rajith must coordinate, and then work together
through the process.


HTH!

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Rajith Siriwardana 
Reply-To: "dev@oodt.apache.org" 
Date: Wednesday, March 27, 2013 9:08 AM
To: "dev@oodt.apache.org" 
Subject: Interested in participating GSoC 2013 with OODT

>hi,
>I'm a 4th year (final year) undergraduate of Faculty of Engineering
>University of Moratuwa majoring in Computer Science and Engineering and I
>would like contribute to OODT.
>I did my internship last year which spanned for 9 months in WSO2[1]. which
>is a middle-ware company and I worked with their WSO2 Stratos product[2].
>Basically I was assigned for developing WSO2 Stratos system level health
>monitoring component and WSO2 Stratos heartbeat monitoring package.  For
>WSO2 Stratos health monitoring component[3] before finalizing Nagios[4]
>for
>monitoring with mk_livestatus[5] replacing NagiosXML engine for query
>stats, which is shown in the architecture image, I did some work with
>Ganglia (how to use it for WSO2 Stratos system level health monitoring).
>Furthermore I took part in test automation hackathon carried out for WSO2
>ESB[6] around 2 months.
>So I believe I have enough knowledge to complete the *GSoC 2013 project
> "Monitor that plugs into ganglia"
>OODT-219
> [7]* with some guidance.
>Please give me any advice, guidance you'd like to extend.
>You can access my Linkedin profile via [8]
>
>[1]http://wso2.com/
>[2]http://wso2.com/cloud/stratos/
>[3]http://mail.wso2.org/mailarchive/architecture/2012-June/008827.html
>[4]http://www.nagios.org/
>[5]http://mathias-kettner.de/checkmk_livestatus.html
>[6]http://wso2.com/products/enterprise-service-bus/
>[7]https://issues.apache.org/jira/browse/OODT-219
>[8]http://lk.linkedin.com/in/rajithsiriwardana/
>
>Thank you,
>
>T.W.R.O. Siriwardana
>StMIEEE
>Undergraduate
>Department of Computer Science & Engineering
>Faculty of Engineering
>University of Moratuwa
>Sri Lanka.

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hey Chintu,

Thanks -- sorry to be asking all these questions, but what does "failed to
run" mean?

Did you update the protocol plugins config to ref the new plugins and
their FQCN (fully
qualified class names?) Do you have a log file for that?

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++




-Original Message-
From: , "Chintu [SGT INC]  (GSFC-586.0)" 
Date: Thursday, March 28, 2013 9:12 AM
To: jpluser , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M (388J)" 
Subject: Re: pushPull component problem

>Since we are not using OODT 0.6, it did not compile straight away.
>However, I did manage to compile it with "0.5" version of
>cas-protocol-api. But when I was running it, It failed to run.
>
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>On 3/28/13 12:04 PM, "Mattmann, Chris A" 
>wrote:
>
>>Hey Chintu,
>>
>>Thanks for identifying this!
>>
>>Have you tried the alternative plugins? They use a diff set of FTP
>>protocol
>>plugins (that unfortunately are not ALv2 licensed or Category A approved,
>>so we can't ship with Apache OODT).
>>
>>They may do better at handling that symlink than the default version of
>>commons-ftp
>>we're using.
>>
>>Let me know. We'll investigate.
>>
>>Cheers,
>>Chris
>>
>>++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattm...@nasa.gov
>>WWW:  http://sunset.usc.edu/~mattmann/
>>++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++
>>
>>
>>
>>
>>-Original Message-
>>From: , "Chintu [SGT INC]  (GSFC-586.0)" 
>>Date: Thursday, March 28, 2013 9:02 AM
>>To: jpluser , "dev@oodt.apache.org"
>>
>>Cc: "Ramirez, Paul M (388J)" 
>>Subject: Re: pushPull component problem
>>
>>>Chris,
>>>
>>>I figured out the problem. Don't know the solution.
>>>
>>>So the problem is the symbolic links on FTP server.
>>>
>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR  is actually
>>>/san1/oisst/NetCDF/2013/AVHRR
>>>Where pub -> san1
>>>  And OI-daily-v2 -> oisst
>>>
>>>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it
>>>works.
>>>Otherwise it does not work.
>>>
>>>Another example is : ftp://aftp.cmdl.noaa.gov
>>>
>>>/data/radiation/surfrad/dra/2013
>>>->/data/radiation/surfrad/Desert_Rock_NV/2013
>>>
>>>
>>>
>>>I don't know how to fix this. Well, not exactly sure where to look. It
>>>appears that the filename filters are being applied based on the path
>>>specified in DirStructParserFiles instead of just the filename. Because
>>>I
>>>can see the correct path when I print RemoteFile (path =
>>>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz'
>>>)
>>>on STDOUT.
>>>
>>>
>>>Hopefully you guys can fix this fast or let me know where to look.
>>>
>>>Regards
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>
>>>On 3/28/13 10:15 AM, "Mattmann, Chris A" 
>>>wrote:
>>>
Hey Chintu,

This totally helped, thanks.

Can you take a look at the plugins here?

https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html


Try installing those, and let me know if either of those FTP
plugins works better and actually downloads the filesŠ

In the meanwhile I'll look at the logs and see what I can come up
with.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++




-Original Message-
From: , "Chintu [SGT INC]  (GSFC-586.0)"

Date: Thursday, March 28, 2013 7:12 AM
To: jpluser , "dev@oodt.apache.org"

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hey Chintu,

Thanks for identifying this!

Have you tried the alternative plugins? They use a diff set of FTP protocol
plugins (that unfortunately are not ALv2 licensed or Category A approved,
so we can't ship with Apache OODT).

They may do better at handling that symlink than the default version of
commons-ftp 
we're using.

Let me know. We'll investigate.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++




-Original Message-
From: , "Chintu [SGT INC]  (GSFC-586.0)" 
Date: Thursday, March 28, 2013 9:02 AM
To: jpluser , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M (388J)" 
Subject: Re: pushPull component problem

>Chris,
>
>I figured out the problem. Don't know the solution.
>
>So the problem is the symbolic links on FTP server.
>
>/pub/OI-daily-v2/NetCDF/2013/AVHRR  is actually
>/san1/oisst/NetCDF/2013/AVHRR
>Where pub -> san1
>  And OI-daily-v2 -> oisst
>
>So if you use "starting_path=/san1/oisst/NetCDF/2013/AVHRR" then it works.
>Otherwise it does not work.
>
>Another example is : ftp://aftp.cmdl.noaa.gov
>
>/data/radiation/surfrad/dra/2013
>->/data/radiation/surfrad/Desert_Rock_NV/2013
>
>
>
>I don't know how to fix this. Well, not exactly sure where to look. It
>appears that the filename filters are being applied based on the path
>specified in DirStructParserFiles instead of just the filename. Because I
>can see the correct path when I print RemoteFile (path =
>'/san1/oisst/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz' )
>on STDOUT.
>
>
>Hopefully you guys can fix this fast or let me know where to look.
>
>Regards
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>On 3/28/13 10:15 AM, "Mattmann, Chris A" 
>wrote:
>
>>Hey Chintu,
>>
>>This totally helped, thanks.
>>
>>Can you take a look at the plugins here?
>>
>>https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html
>>
>>
>>Try installing those, and let me know if either of those FTP
>>plugins works better and actually downloads the filesŠ
>>
>>In the meanwhile I'll look at the logs and see what I can come up
>>with.
>>
>>Cheers,
>>Chris
>>
>>++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattm...@nasa.gov
>>WWW:  http://sunset.usc.edu/~mattmann/
>>++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++
>>
>>
>>
>>
>>-Original Message-
>>From: , "Chintu [SGT INC]  (GSFC-586.0)" 
>>Date: Thursday, March 28, 2013 7:12 AM
>>To: jpluser , "dev@oodt.apache.org"
>>
>>Cc: "Ramirez, Paul M (388J)" 
>>Subject: Re: pushPull component problem
>>
>>>I have attached verbose log. I have added extra print statements and
>>>compiled to see what is returned from remote site. Hope this helps.
>>>
>>>
>>>--
>>>Chintu Mistry
>>>NASA Goddard Space Flight Center
>>>Bldg L40B, Room S776
>>>Office: 240 684 0477
>>>Mobile: 770 310 1047
>>>
>>>
>>>
>>>
>>>
>>>
>>>From: , Chris A 
>>>Date: Thursday, March 28, 2013 10:05 AM
>>>To: Chintu Mistry , "dev@oodt.apache.org"
>>>
>>>Cc: "Ramirez, Paul M" 
>>>Subject: Re: pushPull component problem
>>>
>>>
>>>
>>>Hey Chintu,
>>>
>>>
>>>Thanks for sending this ‹ really helped verify it's something else.
>>>
>>>
>>>Can you capture the log of push pull, with verbose logging turned on,
>>>and
>>>maybe do a pastebin?
>>>
>>>
>>>Cheers,
>>>Chris
>>>
>>>
>>>
>>>
>>>From: , "Chintu [SGT INC] (GSFC-586.0)" 
>>>Date: Thursday, March 28, 2013 7:03 AM
>>>To: jpluser , "dev@oodt.apache.org"
>>>
>>>Cc: "Ramirez, Paul M (388J)" 
>>>Subject: Re: pushPull component problem
>>>
>>>
>>>




Something else is wrong. I know that the regex is fine. I don't even
have regex, it is straight up file name. It should match.


Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047





From: , Chris A 
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M" 
Subject: Re: pushPull component problem



Hey Chintu,


Sorry that 

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hey Chintu,

This totally helped, thanks.

Can you take a look at the plugins here?

https://cwiki.apache.org/OODT/oodt-push-pull-plugins.html


Try installing those, and let me know if either of those FTP
plugins works better and actually downloads the filesŠ

In the meanwhile I'll look at the logs and see what I can come up
with.

Cheers,
Chris

++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++




-Original Message-
From: , "Chintu [SGT INC]  (GSFC-586.0)" 
Date: Thursday, March 28, 2013 7:12 AM
To: jpluser , "dev@oodt.apache.org"

Cc: "Ramirez, Paul M (388J)" 
Subject: Re: pushPull component problem

>I have attached verbose log. I have added extra print statements and
>compiled to see what is returned from remote site. Hope this helps.
>
>
>-- 
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>
>
>
>
>
>From: , Chris A 
>Date: Thursday, March 28, 2013 10:05 AM
>To: Chintu Mistry , "dev@oodt.apache.org"
>
>Cc: "Ramirez, Paul M" 
>Subject: Re: pushPull component problem
>
>
>
>Hey Chintu,
>
>
>Thanks for sending this ‹ really helped verify it's something else.
>
>
>Can you capture the log of push pull, with verbose logging turned on, and
>maybe do a pastebin?
>
>
>Cheers,
>Chris
>
>
>
>
>From: , "Chintu [SGT INC] (GSFC-586.0)" 
>Date: Thursday, March 28, 2013 7:03 AM
>To: jpluser , "dev@oodt.apache.org"
>
>Cc: "Ramirez, Paul M (388J)" 
>Subject: Re: pushPull component problem
>
>
>
>>
>>
>>
>>
>>Something else is wrong. I know that the regex is fine. I don't even
>>have regex, it is straight up file name. It should match.
>>
>>
>>Thanks for looking into this.
>>-- 
>>Chintu Mistry
>>NASA Goddard Space Flight Center
>>Bldg L40B, Room S776
>>Office: 240 684 0477
>>Mobile: 770 310 1047
>>
>>
>>
>>
>>
>>From: , Chris A 
>>Date: Thursday, March 28, 2013 9:01 AM
>>To: Chintu Mistry , "dev@oodt.apache.org"
>>
>>Cc: "Ramirez, Paul M" 
>>Subject: Re: pushPull component problem
>>
>>
>>
>>Hey Chintu,
>>
>>
>>Sorry that you have been having trouble and thanks for pasting the
>>relevant portions
>>of push pull config to diagnose this.
>>
>>
>>I think you have an issue in your RemoteFileSpecs file
>>(TEST_AVHRR_ONLY.xml) --
>>one quick way to debug this is to use the tool that Brian Foster wrote:
>>
>>
>>(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
>>$ cd bin
>>$ java ­Djava.ext.dirs=../lib
>>org.apache.oodt.cas.pushpull.util.ExpressionValidator
>>
>>
>>It's a simple GUI that will allow you to put in regular expressions from
>>your push pull
>>RmoteFileSpecs files, and then put in file names from the server to see
>>if they match.
>>
>>
>>Can you run the above and let me know if it's matching? If not, we've
>>found your issue.
>>If it is matching, we'll move on to the next set of things to check.
>>
>>
>>Thanks!
>>
>>
>>Cheers,
>>Chris
>>
>>
>>From: , "Chintu [SGT INC] (GSFC-586.0)" 
>>Date: Thursday, March 28, 2013 5:47 AM
>>To: "dev@oodt.apache.org" 
>>Cc: jpluser , "Ramirez, Paul M (388J)"
>>
>>Subject: pushPull component problem
>>
>>
>>
>>>Hi,
>>>
>>>
>>>I have been playing around with push pull for last two days and trying
>>>to download some data from remote location without success.
>>>I would appreciate if you guys can shed some light on what is going on
>>>and why isn't push pull find files to download even though its clearly
>>>there at remote location.
>>>
>>>
>>>Here is my configuration. I am trying to download
>>>/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc
>>>.gz file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and
>>>it lists all the files as well. However for some reason it
>>> does not match to the rule in TEST_AVHRR_ONLY.xml. Not sure what I am
>>>doing wrong. I have tried almost all combinations and still could not
>>>get it to work.
>>>
>>>
>>>In my /etc/pushpull.properties file, I have types=false so that it
>>>downloads the file even if its not specified in mime-types.xml file.
>>>org.apache.oodt.cas.pushpull.allow.only.defined.types=false
>>>
>>>
>>>ExternalSources.xml
>>>
>>>
>>>
>>>anonymous
>>>u...@host.com
>>>
>>>
>>>
>>>
>>>
>>>
>>>RemoteSpecs.xml
>>>
>>>>>period="1m" runOnReboot="yes"/>
>>>>>dir="[CAS_PP_RESOURCES]/examples/DirStructXmlParserFiles">
>>>>>parser="org.apache.oodt.cas.pushpull.filerestrictions.parsers.DirStructX
>>>mlParser"/>
>>>
>>>>>queryElement="Filename"/>

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hey Chintu,

Thanks for sending this — really helped verify it's something else.

Can you capture the log of push pull, with verbose logging turned on, and maybe 
do a pastebin?

Cheers,
Chris


From: , "Chintu [SGT INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Thursday, March 28, 2013 7:03 AM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Cc: "Ramirez, Paul M (388J)" 
mailto:paul.m.rami...@jpl.nasa.gov>>
Subject: Re: pushPull component problem

[cid:63493DAB-916B-45FE-B016-391572B374E2]
[cid:0CE0F486-1CE2-490B-B015-2666D41AF8EC]

Something else is wrong. I know that the regex is fine. I don't even have 
regex, it is straight up file name. It should match.

Thanks for looking into this.
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

From: , Chris A 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Date: Thursday, March 28, 2013 9:01 AM
To: Chintu Mistry mailto:chintu.mis...@nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Cc: "Ramirez, Paul M" 
mailto:paul.m.rami...@jpl.nasa.gov>>
Subject: Re: pushPull component problem

Hey Chintu,

Sorry that you have been having trouble and thanks for pasting the relevant 
portions
of push pull config to diagnose this.

I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:

(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib 
org.apache.oodt.cas.pushpull.util.ExpressionValidator

It's a simple GUI that will allow you to put in regular expressions from your 
push pull
RmoteFileSpecs files, and then put in file names from the server to see if they 
match.

Can you run the above and let me know if it's matching? If not, we've found 
your issue.
If it is matching, we'll move on to the next set of things to check.

Thanks!

Cheers,
Chris

From: , "Chintu [SGT INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Cc: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"Ramirez, Paul M (388J)" 
mailto:paul.m.rami...@jpl.nasa.gov>>
Subject: pushPull component problem

Hi,

I have been playing around with push pull for last two days and trying to 
download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why 
isn't push pull find files to download even though its clearly there at remote 
location.

Here is my configuration. I am trying to download  
/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz 
file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all 
the files as well. However for some reason it does not match to the rule in 
TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all 
combinations and still could not get it to work.

In my /etc/pushpull.properties file, I have types=false so that it downloads 
the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false

ExternalSources.xml



anonymous
u...@host.com




RemoteSpecs.xml








TEST_AVHRR_ONLY.xml









Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

Re: pushPull component problem

[Mattmann, Chris A (388J)]

Hey Chintu,

Sorry that you have been having trouble and thanks for pasting the relevant 
portions
of push pull config to diagnose this.

I think you have an issue in your RemoteFileSpecs file (TEST_AVHRR_ONLY.xml) --
one quick way to debug this is to use the tool that Brian Foster wrote:

(in a push pull deployment dir, e.g., $PUSHPULL_HOME)
$ cd bin
$ java –Djava.ext.dirs=../lib 
org.apache.oodt.cas.pushpull.util.ExpressionValidator

It's a simple GUI that will allow you to put in regular expressions from your 
push pull
RmoteFileSpecs files, and then put in file names from the server to see if they 
match.

Can you run the above and let me know if it's matching? If not, we've found 
your issue.
If it is matching, we'll move on to the next set of things to check.

Thanks!

Cheers,
Chris

From: , "Chintu [SGT INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Thursday, March 28, 2013 5:47 AM
To: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Cc: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"Ramirez, Paul M (388J)" 
mailto:paul.m.rami...@jpl.nasa.gov>>
Subject: pushPull component problem

Hi,

I have been playing around with push pull for last two days and trying to 
download some data from remote location without success.
I would appreciate if you guys can shed some light on what is going on and why 
isn't push pull find files to download even though its clearly there at remote 
location.

Here is my configuration. I am trying to download  
/pub/OI-daily-v2/NetCDF/2013/AVHRR/avhrr-only-v2.20130326_preliminary.nc.gz 
file from eclipse.ncdc.noaa.gov. It logs-in to the server fine and it lists all 
the files as well. However for some reason it does not match to the rule in 
TEST_AVHRR_ONLY.xml. Not sure what I am doing wrong. I have tried almost all 
combinations and still could not get it to work.

In my /etc/pushpull.properties file, I have types=false so that it downloads 
the file even if its not specified in mime-types.xml file.
org.apache.oodt.cas.pushpull.allow.only.defined.types=false

ExternalSources.xml



anonymous
u...@host.com




RemoteSpecs.xml








TEST_AVHRR_ONLY.xml









Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

Re: Interested in participating GSoC 2013 with OODT

[Mattmann, Chris A (388J)]

Hi Rajith!

Thank you for your interest in OODT-219! Would love to have you help out on the 
project.

The mailing list doesn't accept attachments, so can you head over the OODT-219 
and attach
the diagram there?

Thank you and looking forward to working with you on this!

Cheers,
Chris

From: Rajith Siriwardana 
mailto:rajithsiriward...@ieee.org>>
Reply-To: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Date: Wednesday, March 27, 2013 9:20 AM
To: dev mailto:dev@oodt.apache.org>>
Subject: Re: Interested in participating GSoC 2013 with OODT

hi
small mistake, forgot to attach the high level deployment architecture diagram 
which i was talking about. note that this architecture diagram changed to use 
mk_livestatus replacing Nagios XML engine to query stats. I can provide more 
details upon request  about the the project I carried out.

Thank you,

T.W.R.O. Siriwardana
StMIEEE
Undergraduate
Department of Computer Science & Engineering
Faculty of Engineering
University of Moratuwa
Sri Lanka.


On Wed, Mar 27, 2013 at 9:38 PM, Rajith Siriwardana 
mailto:rajithsiriward...@ieee.org>> wrote:
hi,
I'm a 4th year (final year) undergraduate of Faculty of Engineering University 
of Moratuwa majoring in Computer Science and Engineering and I would like 
contribute to OODT.
I did my internship last year which spanned for 9 months in WSO2[1]. which is a 
middle-ware company and I worked with their WSO2 Stratos product[2]. Basically 
I was assigned for developing WSO2 Stratos system level health monitoring 
component and WSO2 Stratos heartbeat monitoring package.  For WSO2 Stratos 
health monitoring component[3] before finalizing Nagios[4] for monitoring with 
mk_livestatus[5] replacing NagiosXML engine for query stats, which is shown in 
the architecture image, I did some work with Ganglia (how to use it for WSO2 
Stratos system level health monitoring).
Furthermore I took part in test automation hackathon carried out for WSO2 
ESB[6] around 2 months.
So I believe I have enough knowledge to complete the GSoC 2013 project  
"Monitor that plugs into ganglia" 
OODT-219 [7] with some guidance.
Please give me any advice, guidance you'd like to extend.
You can access my Linkedin profile via [8]

[1]http://wso2.com/
[2]http://wso2.com/cloud/stratos/
[3]http://mail.wso2.org/mailarchive/architecture/2012-June/008827.html
[4]http://www.nagios.org/
[5]http://mathias-kettner.de/checkmk_livestatus.html
[6]http://wso2.com/products/enterprise-service-bus/
[7]https://issues.apache.org/jira/browse/OODT-219
[8]http://lk.linkedin.com/in/rajithsiriwardana/

Thank you,

T.W.R.O. Siriwardana
StMIEEE
Undergraduate
Department of Computer Science & Engineering
Faculty of Engineering
University of Moratuwa
Sri Lanka.

Re: Integrate Apache OODT file manager as an extension to GFac

[Mattmann, Chris A (388J)]

Great to hear, Sanjaya!

Cheers,
Chris


On 3/25/13 8:15 PM, "Sanjaya Medonsa"  wrote:

>Thanks Chris! That works. I could able to successfully ingest a file.
>
>Best Regards,
>Sanjaya
>
>On Tue, Mar 26, 2013 at 4:29 AM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> Thanks. Did you create a catalog directory? If so, delete it and restart
>> your File Manager.
>> It will create this directory for you.
>>
>> Let me know and thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On 3/25/13 1:11 PM, "Sanjaya Medonsa"  wrote:
>>
>> >Thanks Chris!
>> >
>> >I have followed the OODT File Manager user guide to start file manager
>> >server to ingest file. I couldn't able to ingest a sample file due to
>> >following exception.
>> >
>> >I couldn't able to figure out the issue by googling on the issue. I
>>have
>> >also noticed that file named 'segments' should be available in catalog
>> >directory. Any idea on possible root cause for the issue?
>> >
>> >INFO: Running Met Extractor:
>> 
>>>[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocat
>>>io
>> >nExtractor]
>> >for product type: [GenericFile]
>> >Mar 26, 2013 12:40:19 AM
>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>> >addCompleteProductToIndex
>> >WARNING: Unable to index product: [blah.txt]: Message: read past EOF
>> >java.io.IOException: read past EOF
>> >at
>> 
>>>org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.jav
>>>a:
>> >60)
>> >at
>> 
>>>org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.j
>>>av
>> >a:33)
>> >at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
>> >at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
>> >at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
>> >at org.apache.lucene.store.Lock$With.run(Lock.java:99)
>> >at org.apache.lucene.index.IndexWriter.(IndexWriter.java:259)
>> >at org.apache.lucene.index.IndexWriter.(IndexWriter.java:204)
>> >at
>> 
>>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToIn
>>>de
>> >x(LuceneCatalog.java:993)
>> >at
>> 
>>>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(L
>>>uc
>> >eneCatalog.java:363)
>> >at
>> 
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReference
>>>s(
>> >XmlRpcFileManager.java:1122)
>> >at
>> 
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRp
>>>cF
>> >ileManager.java:740)
>> >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >at
>> 
>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
>>>a:
>> >39)
>> >at
>> 
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
>>>Im
>> >pl.java:25)
>> >at java.lang.reflect.Method.invoke(Method.java:597)
>> >at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>> >at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>> >at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>> >at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>> >at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>> >at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>> >at java.lang.Thread.run(Thread.java:662)
>> >
>> >
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >
>> >
>> >On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
>> >chris.a.mattm...@jpl.nasa.gov> wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> Great work!
>> >>
>> >> I would also have a look at:
>> >>
>> >> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>> >>
>> >>
>> >> That should help learn how to use CAS-PGE (and file manager).
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On 3/24/13 6:43 PM, "Sanjaya Medonsa"  wrote:
>> >>
>> >> >Hi Dev,
>> >> > I have implemented a new module named as oodt-integration in
>> >>Airavata
>> >> >which currently contains implementation which extends Apache OODT
>>PGE
>> >>task
>> >> >instance to stage files into a local directory. Still I am figuring
>>out
>> >> >best way to integrate configurations. My main concern is to how to
>>set
>> >>up
>> >> >a
>> >> >test environment to test the implementation. My plan is to follow
>>the
>> >> >following approach.
>> >> >1. Set up CAS File Manager using Airavata with the
>> >>help of
>> >> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>> >> >2. On Airavata is there any sample workflow that
>>uses
>> >>file
>> >> >as an input to GFac. Please let me know what is the best sample
>> >>workflow
>> >> >that can be used to test my implementation.
>> >> >
>> >> >Best Regards,
>> >> >Sanjaya
>> >>
>> >>
>>
>>

Re: Integrate Apache OODT file manager as an extension to GFac

[Mattmann, Chris A (388J)]

Hi Sanjaya,

Thanks. Did you create a catalog directory? If so, delete it and restart
your File Manager.
It will create this directory for you.

Let me know and thanks!

Cheers,
Chris


On 3/25/13 1:11 PM, "Sanjaya Medonsa"  wrote:

>Thanks Chris!
>
>I have followed the OODT File Manager user guide to start file manager
>server to ingest file. I couldn't able to ingest a sample file due to
>following exception.
>
>I couldn't able to figure out the issue by googling on the issue. I have
>also noticed that file named 'segments' should be available in catalog
>directory. Any idea on possible root cause for the issue?
>
>INFO: Running Met Extractor:
>[org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocatio
>nExtractor]
>for product type: [GenericFile]
>Mar 26, 2013 12:40:19 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
>addCompleteProductToIndex
>WARNING: Unable to index product: [blah.txt]: Message: read past EOF
>java.io.IOException: read past EOF
>at
>org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:
>60)
>at
>org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav
>a:33)
>at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:46)
>at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:47)
>at org.apache.lucene.index.IndexWriter$1.doBody(IndexWriter.java:264)
>at org.apache.lucene.store.Lock$With.run(Lock.java:99)
>at org.apache.lucene.index.IndexWriter.(IndexWriter.java:259)
>at org.apache.lucene.index.IndexWriter.(IndexWriter.java:204)
>at
>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addCompleteProductToInde
>x(LuceneCatalog.java:993)
>at
>org.apache.oodt.cas.filemgr.catalog.LuceneCatalog.addProductReferences(Luc
>eneCatalog.java:363)
>at
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addProductReferences(
>XmlRpcFileManager.java:1122)
>at
>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcF
>ileManager.java:740)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>39)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:25)
>at java.lang.reflect.Method.invoke(Method.java:597)
>at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
>at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
>at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
>at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
>at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
>at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
>at java.lang.Thread.run(Thread.java:662)
>
>
>
>Best Regards,
>Sanjaya
>
>
>
>On Mon, Mar 25, 2013 at 8:38 AM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> Great work!
>>
>> I would also have a look at:
>>
>> https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html
>>
>> https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide
>>
>>
>> That should help learn how to use CAS-PGE (and file manager).
>>
>> Cheers,
>> Chris
>>
>> On 3/24/13 6:43 PM, "Sanjaya Medonsa"  wrote:
>>
>> >Hi Dev,
>> > I have implemented a new module named as oodt-integration in
>>Airavata
>> >which currently contains implementation which extends Apache OODT PGE
>>task
>> >instance to stage files into a local directory. Still I am figuring out
>> >best way to integrate configurations. My main concern is to how to set
>>up
>> >a
>> >test environment to test the implementation. My plan is to follow the
>> >following approach.
>> >1. Set up CAS File Manager using Airavata with the
>>help of
>> >https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>> >2. On Airavata is there any sample workflow that uses
>>file
>> >as an input to GFac. Please let me know what is the best sample
>>workflow
>> >that can be used to test my implementation.
>> >
>> >Best Regards,
>> >Sanjaya
>>
>>

Re: Integrate Apache OODT file manager as an extension to GFac

[Mattmann, Chris A (388J)]

Hi Sanjaya,

Great work!

I would also have a look at:

https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html

https://cwiki.apache.org/confluence/display/OODT/OODT+Filemgr+User+Guide


That should help learn how to use CAS-PGE (and file manager).

Cheers,
Chris

On 3/24/13 6:43 PM, "Sanjaya Medonsa"  wrote:

>Hi Dev,
> I have implemented a new module named as oodt-integration in Airavata
>which currently contains implementation which extends Apache OODT PGE task
>instance to stage files into a local directory. Still I am figuring out
>best way to integrate configurations. My main concern is to how to set up
>a
>test environment to test the implementation. My plan is to follow the
>following approach.
>1. Set up CAS File Manager using Airavata with the help of
>https://cwiki.apache.org/OODT/oodt-filemgr-user-guide.html
>2. On Airavata is there any sample workflow that uses file
>as an input to GFac. Please let me know what is the best sample workflow
>that can be used to test my implementation.
>
>Best Regards,
>Sanjaya

Re: Flexible, use case specific metadata cataloging for CAS

[Mattmann, Chris A (388J)]

Lewis,

Since I still have not officially received this email from dev@oodt,
(grr) and since I saw it on the mail archives, I'm going to copy your
email below, with the same subject, and hope it gets threaded right :)

Comments below:


On 3/20/13 5:55 PM, Lewis John Mcgibbney (lewi...@gmail.com) wrote:

>
>All,
>
>I picked up OODT today and immediately thought about an implementation of
>Apache Gora [0] for abstracting persistence within the CAS metadata
>catalogue.

+1, I've wanted this for a while. A GoraCatalog implementation of the FM
Catalog
Interface.

>Right now, for me, the persistence of my metadata catalogue to Lucene or
>MySQL is sufficient and I have no immediate justification for using some
>alternative storage mechanism however I noticed that there are a few areas
>where OODT could generally benefit from the Gora implementation.
>It is natural that product discovery via daemon driven CAS crawler (for
>example) will fire product streams of varying nature towards the catalogue
>storage mechanism. Lucene or MySQL my not be best best option to store
>such
>streams of data and/or the best way to later retrieve that data. Gora
>would
>enable a much more comprehensive variety of data stores to be available
>for
>persistence of catalogue metadata and would also provide a much more
>flexible model specifically geared towards better solutions for metadata
>cataloguing. Currently we support Amazon DynamoDB, Accumulo, Cassandra,
>HBase, HDFS, HSQLDB and MySQL. We have patches for Solr, MongoDB and
>various file based stores. There is also interest to implement an Oracle
>NoSQL DB don't ask.

Haha!

>I notice that the SolrIndexer tool implemented by Paul provides an
>expressive number of options for indexing to your Solr HTTP server. The
>gora-solr module would provide all these plus more.
>I suppose this entirely depends on the requirements for expanding metadata
>catalogues within the File Manager.
>Is it envisaged that such an implementation is required for some use cases
>or would be required?

Yes, please, help! :)

>As Gora builds on Hadoop principles, I suppose it would also enable folks
>use their metadata catalogues in different, possibly useful, use-case
>adaptable ways.
>Just an initial thought.

A great one at that, I would be super +1 for a GoraCatalog to help in these
situations and would be keen to work on it with you.

Cheers,
Chris

>Thanks
>Lewis
>
>
>[0] http://gora.apache.org 
>--
>*Lewis*
>
>

Re: Curator

[Mattmann, Chris A (388J)]

Hi Guys,

I'm also OK with this, but it's best to let things sit for 48+ hours to
allow all others
in OODT in different time zones to also express their opinions.

Jordan, if you don't hear any objections here in OODT ville from anyone
else, then I think
you are OK to proceed. When you guys become famous, throw some Apache
Curator love at Apache
OODT CAS Curator ok? ^_^

Cheers,
Chris


On 3/24/13 8:24 AM, "Jordan Zimmerman"  wrote:

>Thank you!
>
>
>Jordan Zimmerman
>
>On Mar 24, 2013, at 8:03 AM, Sean Kelly  wrote:
>
>> I don't foresee any issues. Welcome to the incubator!
>>
>> Best
>> --k
>>
>> --
>> Sean Kelly
>> Apache Software Foundation
>>
>> On 2013.Mar.23, at 11.22p, Jordan Zimmerman  wrote:
>>
>>> Hello - I'm the lead on Curator - a ZooKeeper client that is now in
>>>the Apache Incubator. I'm doing the trademark search and came across
>>>OODT CAS-Curator. I don't think there's any overlap and, thus, there
>>>shouldn't be a problem with Apache Curator. Thoughts?
>>>
>>> -Jordan
>>
>

Re: Justification behind hardcoded paths within configuration

[Mattmann, Chris A (388J)]

Hey Lewis,

Yeah that was my bad I assume. No one really uses that code anymore
(at one point internally we used it at JPL when Sean Kelly was trying to
wrap File Manager in Jython so that we could have a Python FM server).
It actually really belongs in Agility (that's a separate issue), but
happy for you to leave it or fix it along the way.

Cheers,
Chris


On 3/23/13 2:14 PM, "Lewis John Mcgibbney" 
wrote:

>Hi All,
>I recently came across [0], which as you can see (amongst others) has a
>hardcoded path to the luce-core dependency.
>As I am upgrading this library to 4.2.0, this is not ideal.
>I can manually change it and include it within my patch, but this seems
>more of a workaround than solution.
>Can someone please explain why these particular paths are hard coded?
>Thank you, have a great weekend
>Lewis
>
>[0]
>http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/python/setclas
>spath.sh
>
>
>-- 
>*Lewis*

Google Summer of Code 2013: Plug the Resource Manager into Ganglia

[Mattmann, Chris A (388J)]

Hey Guys,

I posted a GSoC project:

https://issues.apache.org/jira/browse/OODT-219


This would involve plugging the Resource Manager into Ganglia by writing
a GangliaAssignmentMonitor.

If anyone is interested, let me know, here on list.

Cheers,
Chris

Re: OpsUI Functionality

[Mattmann, Chris A (388J)]

Hi Gabe,

Best bets:


  1.  Look at https://issues.apache.org/jira/browse/OODT-155
  2.  Look at https://issues.apache.org/jira/browse/OODT-156
  3.  Look at https://issues.apache.org/jira/browse/OODT-157

Beyond that:

Try them out. Deploy them, use them, look at the code:

http://svn.apache.org/repos/asf/oodt/trunk/webapp/components/ - core Wicket 
modules
http://svn.apache.org/repos/asf/oodt/trunk/webapp/wmonitor/ - Workflow Monitor 
app
http://svn.apache.org/repos/asf/oodt/trunk/webapp/fmbrowser/ - File Manager 
Browser app
http://svn.apache.org/repos/asf/oodt/trunk/pcs/opsui/ - OPSUI

Enjoy!

Cheers,
Chris


From: , "Gabriel M (388J)" 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 10:03 AM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Cc: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OpsUI Functionality

How would I find out the capabilities of the various apps that make up the 
OpsUI?

Gabe =)

From: , "Chris A (388J)" 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:51 AM
To: resneck 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Cc: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OpsUI Functionality

Hi Gabe,

The information that's in there is:

File Manager Browser App and Pages
Workflow Monitor App and Pages
PCS Stat
PCS Trace (integrated with File Manager Browser)
CAS product web service API (integrated with data download)

FM Browser and Workflow Monitor are usable independently of one another, and 
combined together
as sub widgets using Apache Wicket (http://wicket.apache.org/) an awesome web 
framework that is
simply Java (no XML) + XHTML.

Java and XHTML for the win.

Cheers,
Chris


From: , "Gabriel M (388J)" 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:48 AM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: Re: OpsUI Functionality

Yeah!  I'll forward all of this along to the dev list.
I already checked out the wiki page, but I was looking for a bit more detail.  
Specifically, what information from the File, Workflow, and Resource Managers 
is displayed and how does the user interact with them?  I skipped through the 
video, but didn't see the OpsUI in there and assumed it was included there just 
as a guide for how to deploy OODT.  Did I miss it?
I'll check out the ApacheCon presentations.
Thanks!

Gabe =)


From: , "Chris A (388J)" 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:40 AM
To: resneck 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Subject: Re: OpsUI Functionality

Hey Gabe,

Sure, happy to help.

Check this out:

https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OPSUI
http://archive.apachecon.com/na2011/presentations/09-Wednesday/A-Apache_In_Space/

I can show you one for our Snow Science Computing Facility at some point next 
week.

Also can we copy dev@oodt.apache.org on this?

Cheers,
Chris


From: , "Gabriel M (388J)" 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:34 AM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: OpsUI Functionality

Hi, Chris!
I've been tasked with assessing the various operator tools that we might use in 
SMAP, and I'm trying to find more information on the OpsUI tool.  Is there 
somewhere I can find a list of its functionality and perhaps some juicy 
screenshots?
Thanks!

Gabe =)

Re: RADiX: OPSUI

[Mattmann, Chris A (388J)]

Heya Arni,

[dropping xdata-general@ since it's a JPL private list and this is Apache
public]

On 3/21/13 9:19 PM, "Arni Sumarlidason" 
wrote:

>Hello all,
>
>I looked a little bit into the Resource manager page not loading in the
>RADiX archetype... Does the resource manager page(s) need to be
>developed, or are they developed and not tied in properly?

The resource manager pages in OPSUI don't exist yet (need Wicket pages for
them). In lieu of that, we simply
allow the definition of a GANGLIA_URL environment variable, and then allow
the page to link to that.

>Can you provide any direction on how to best approach completing/fixing
>the OPSUI webapp?

FYI above :)

Also see:

https://issues.apache.org/jira/browse/OODT-157

For the OPSUI wicket JIRA issue. 156, and 155 are also worth looking at.

Happy for contributions (maybe start out with Balsamiq with the Resource
Manager suggest pages)?

Steps to accomplish that:

1. Think of something to do :)
2. Design wire frames
3. Create JIRA issue in https://issues.apache.org/jira/browse/OODT for
your improvements
4. Attach patches, work with PMC members, etc.
5. Have fun!

Cheers,
Chris

>
>Thank you for your time,
>
>Arni Sumarlidason | Software Engineer, Information Technology
>MDA | 820 West Diamond Ave | Gaithersburg, MD | USA
>O: 240-833-8200 D: 240-833-8318 M: 256-393-2803
>arni.sumarlida...@mdaus.com|
>http://www.mdaus.com
>

Re: Environment variables in config files

[Mattmann, Chris A (388J)]

Hey Guys,

Maybe I can help here and it would be great to document on the wiki
(please) :)

One of the "load bearing walls" of OODT, is cas-metadta, and a nifty class
inside
called PathUtils that I created a long time ago.

http://oodt.apache.org/components/maven/apidocs/org/apache/oodt/cas/metadat
a/util/PathUtils.html


My original vision for PathUtils was for it to be a class that would use
OODT metadata
(from somewhere, the workflow manager, the file manager, met extractors
[client or server
side, etc.] to do environment variable replacement in paths). Logical or
physical paths, etc.

This originally helped in the Versioning interface within File Manager:

http://oodt.apache.org/components/maven/filemgr/development/developer.html


(see architecture diagram)

http://oodt.apache.org/components/maven/apidocs/org/apache/oodt/cas/filemgr
/versioning/package-summary.html

http://oodt.apache.org/components/maven/apidocs/org/apache/oodt/cas/filemgr
/versioning/Versioner.html


Eventually I found PathUtils could be used for all sorts of other things.
Mixing metadata based
replacement with environment variable replacement. Mixing that with full
paths in config files. Recursively doing
this, etc.

At that point we built a bunch of core stuff on it, and the way I wanted
to do environment variables was
using the '[' and ']' separators e.g., [ENV VAR NAME] expands to
$ENV_VAR_NAME.

Why didn't I use the dollar? I didn't want it to get confused with Maven
paths, and other paths (e.g., $ replacement
in Java regexs, etc.) This also existed way before Spring got popular and
started to deal with some of these
facilities. So general rule of thumb in OODT: Environment variable
replacement happens a lot of places. Most of the
properties in config files are env var replaceable by default (and if not
by default by adding envReplace="true" as an
attribute on the tag, etc.)

If there are specific questions or places you find it's not, suggest that
it should be and file a JIRA
issue and we can discuss.

Cheers!

Chris


On 3/21/13 5:06 PM, "Lewis John Mcgibbney" 
wrote:

>Hi Ryan,
>So can you please confirm if the environment variables are to be placed
>into the env.sh script?
>If so I will duly update the documentation on the wiki.
>Thank you very much
>Lewis
>
>On Thu, Mar 21, 2013 at 5:00 PM, Gerard, Ryan S. (GSFC-586.0)[SGT INC] <
>ryan.s.ger...@nasa.gov> wrote:
>
>> Thanks Lewis,
>>
>> Yes, it appears OODT does support environment variables if put within
>> brackets.
>>
>> For example, [ENV_VARIABLE_NAME] can be inserted into any xml config
>>files
>> and it will be deferenced.
>>
>> Thanks for your reply.
>> Ryan
>>
>> On 3/21/13 6:38 PM, "Lewis John Mcgibbney" 
>> wrote:
>>
>> >Hi Ryan,
>> >Please excuse if I am wrong, however I think (if you are doing as I am
>>and
>> >using the RADiX module and Maven archetype) then all environment
>>variables
>> >should be set within [0] prior to building your project.
>> >I also found myself confused yesterday and spent a bit of time looking
>>at
>> >this.
>> >We need to document this somewhere as the definition of environment
>> >variables is critical but not very clear within the OODT documentation.
>> >If one of the OODT team can confirm or correct me then we can get such
>> >information at least logged.
>> >Thanks
>> >Lewis
>> >
>> >[0] *http://s.apache.org/pg*
>> >
>> >On Thu, Mar 21, 2013 at 3:13 PM, Gerard, Ryan S. (GSFC-586.0)[SGT INC]
>><
>> >ryan.s.ger...@nasa.gov> wrote:
>> >
>> >> Is there a special OODT syntax to use environment variables in OODT?
>>I
>> >> thought I ran across something a while ago but perhaps its wasn't for
>> >> config files. I'm referring to files such as events.xml within
>> >>cas-workflow
>> >> and tasks.xml.
>> >>
>> >> Thanks,
>> >> Ryan
>> >>
>> >
>> >
>> >
>> >--
>> >*Lewis*
>>
>>
>
>
>-- 
>*Lewis*

Re: OpsUI Functionality

[Mattmann, Chris A (388J)]

Hi Gabe,

The information that's in there is:

File Manager Browser App and Pages
Workflow Monitor App and Pages
PCS Stat
PCS Trace (integrated with File Manager Browser)
CAS product web service API (integrated with data download)

FM Browser and Workflow Monitor are usable independently of one another, and 
combined together
as sub widgets using Apache Wicket (http://wicket.apache.org/) an awesome web 
framework that is
simply Java (no XML) + XHTML.

Java and XHTML for the win.

Cheers,
Chris


From: , "Gabriel M (388J)" 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:48 AM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: Re: OpsUI Functionality

Yeah!  I'll forward all of this along to the dev list.
I already checked out the wiki page, but I was looking for a bit more detail.  
Specifically, what information from the File, Workflow, and Resource Managers 
is displayed and how does the user interact with them?  I skipped through the 
video, but didn't see the OpsUI in there and assumed it was included there just 
as a guide for how to deploy OODT.  Did I miss it?
I'll check out the ApacheCon presentations.
Thanks!

Gabe =)


From: , "Chris A (388J)" 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:40 AM
To: resneck 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Subject: Re: OpsUI Functionality

Hey Gabe,

Sure, happy to help.

Check this out:

https://cwiki.apache.org/confluence/display/OODT/Quick+Start+for+PCS+OPSUI
http://archive.apachecon.com/na2011/presentations/09-Wednesday/A-Apache_In_Space/

I can show you one for our Snow Science Computing Facility at some point next 
week.

Also can we copy dev@oodt.apache.org on this?

Cheers,
Chris


From: , "Gabriel M (388J)" 
mailto:gabriel.m.resn...@jpl.nasa.gov>>
Date: Thursday, March 21, 2013 9:34 AM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: OpsUI Functionality

Hi, Chris!
I've been tasked with assessing the various operator tools that we might use in 
SMAP, and I'm trying to find more information on the OpsUI tool.  Is there 
somewhere I can find a list of its functionality and perhaps some juicy 
screenshots?
Thanks!

Gabe =)

Re: My Hadoop Summit Talk: NASA+BigData

[Mattmann, Chris A (388J)]

Hey Bruce,

On 3/20/13 7:56 AM, "Bruce Barkstrom"  wrote:

>I'll subside after one minor note on the "sky is the archive."

Don't ever subside! I appreciate your feedback and commentary and
wholly look up to you for advice and help.

Your cynicism at the conference is totally understood amidst as you
mention your ability to download the conference (or something similar ^_^)
off of your Gmail web page :)

>
>I once had a course from W. W. Morgan, the U. Chicago prof who
>developed the atlas of stellar types (A, O, B, etc.).  He had
>the spectrum of a "standard type R".  As I recall, two weeks
>after he published his atlas with the spectra, the star defining
>the type became a variable.

Precisely.

>
>Also, I note that on this very Google Mail page, I can get
>a "Free Guide to Big Data", as well as the "IBM Big Data
>Free eBook".  I suppose I don't need to go to a conference
>to become informed.

Nah, but it would be less fun without you there! Who else will represent
the society of troublemakers, and scientific reality, that is,
the people actually doing the work?!!

Take care my friend.

Cheers,
Chris


>
>Bruce B.
>
>On Wed, Mar 20, 2013 at 10:21 AM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hey Bruce,
>>
>> A couple points:
>>
>> On 3/20/13 5:46 AM, "Bruce Barkstrom"  wrote:
>>
>> >That may be a bit better.
>> >
>> >However, it still isn't clear to me how the physics of the instruments
>> >and of the data processing gets into what users understand they
>> >can do with the data.
>>
>> Yeah agreed. At the same time, this is kind of difficult to throw into
>> a 45 min with 15 mins "techie talk" that I haven't even prepared yet,
>> and even harder to throw in to a 100 word (what you see on the website)
>> and 200 word (longer, what I sent you) abstract that they requested.
>>
>> >
>> >As I understand Big Data and analytics, it usually appears to using
>> >a lot of statistics to find unexpected correlations in the data, but
>> >the techniques aren't looking for causation.  If you're dealing with
>> >scientific data, you're usually trying to get to physical causation.
>> >That means, I think, that users need to understand how the
>> >physics and math constrain what they can do.
>>
>> ++50 agreed.
>>
>> >
>> >Let me see if I can identify a more concrete example of a
>> >concern.  Usually, when we want to deal with physically
>> >connected phenomena, we want disparate data to be
>> >observing the same chunk of space at the same time.
>> >If the Big Data user picks up one piece of data from region
>> >X_1 and t_1 and then develops a correlation with observations
>> >with data from X_2 and t_2, where X_1 /= X_2 and t_1 /= t_2,
>> >it isn't clear why that correlation has anything to do with
>> >physical causation.  Of, to put it another way, Big Data
>> >may just give more examples of the "cherry picking"
>> >climate deniers do when they select data without
>> >paying attention to the statistical and physical significance
>> >of their "results".
>>
>> Totally agree. This is the big difference between card
>> carrying statisticians a lot of time and *computer science*
>> oriented *machine learning* people.
>>
>> >
>> >So, even though the data rates are large by today's
>> >standards, I'm not sure that, by itself, is impressive.
>>
>> Well I have to say it is impressive. Can you show me a disk
>> that can today write 700 TB/data per second? Or the filesystem
>> drivers and parallel I/O necessary to software them? Imagine in
>> astronomy, where they are moving into the time domain, and
>> away from the "sky is the archive" "so just reobserve next
>> time" mentality, and thus triage, which is super important,
>> isn't the main driver and archival is now becoming important,
>> and necessary in these eventually 700TB/sec producing systems.
>>
>> There are all sorts of IO, hardware, computer science, and
>> other advances that we don't have that are needed, and that
>> these types of examples like the SKA will drive.
>>
>> OTOH, the sheer infrastructure, domestic and international policy,
>> investment, and excitement and sense of nationality that many of
>> these new Big Data systems (especially the SKA) are creating in
>> their respective countries (e.g., in South Africa), is 

Re: My Hadoop Summit Talk: NASA+BigData

[Mattmann, Chris A (388J)]

Hey Bruce,

A couple points:

On 3/20/13 5:46 AM, "Bruce Barkstrom"  wrote:

>That may be a bit better.
>
>However, it still isn't clear to me how the physics of the instruments
>and of the data processing gets into what users understand they
>can do with the data.

Yeah agreed. At the same time, this is kind of difficult to throw into
a 45 min with 15 mins "techie talk" that I haven't even prepared yet,
and even harder to throw in to a 100 word (what you see on the website)
and 200 word (longer, what I sent you) abstract that they requested.

>
>As I understand Big Data and analytics, it usually appears to using
>a lot of statistics to find unexpected correlations in the data, but
>the techniques aren't looking for causation.  If you're dealing with
>scientific data, you're usually trying to get to physical causation.
>That means, I think, that users need to understand how the
>physics and math constrain what they can do.

++50 agreed.

>
>Let me see if I can identify a more concrete example of a
>concern.  Usually, when we want to deal with physically
>connected phenomena, we want disparate data to be
>observing the same chunk of space at the same time.
>If the Big Data user picks up one piece of data from region
>X_1 and t_1 and then develops a correlation with observations
>with data from X_2 and t_2, where X_1 /= X_2 and t_1 /= t_2,
>it isn't clear why that correlation has anything to do with
>physical causation.  Of, to put it another way, Big Data
>may just give more examples of the "cherry picking"
>climate deniers do when they select data without
>paying attention to the statistical and physical significance
>of their "results".

Totally agree. This is the big difference between card
carrying statisticians a lot of time and *computer science*
oriented *machine learning* people.

>
>So, even though the data rates are large by today's
>standards, I'm not sure that, by itself, is impressive.

Well I have to say it is impressive. Can you show me a disk
that can today write 700 TB/data per second? Or the filesystem
drivers and parallel I/O necessary to software them? Imagine in
astronomy, where they are moving into the time domain, and
away from the "sky is the archive" "so just reobserve next
time" mentality, and thus triage, which is super important,
isn't the main driver and archival is now becoming important,
and necessary in these eventually 700TB/sec producing systems.

There are all sorts of IO, hardware, computer science, and
other advances that we don't have that are needed, and that
these types of examples like the SKA will drive.

OTOH, the sheer infrastructure, domestic and international policy,
investment, and excitement and sense of nationality that many of
these new Big Data systems (especially the SKA) are creating in
their respective countries (e.g., in South Africa), is enough
to at least suggest to my evidence based mind that there is
something impressive here.

>Maybe the relevant example would be all those statistics
>on dams built or tons of steel produced by the Soviet
>Union.  The hype would be more interesting if it could
>talk about what new phenomena or understanding
>these techniques will produce - not just the data rate
>or the total amount of data being produced.

Agreed, lots of data has been generated for a while. However,
the volume (total and discrete); velocity, and variety (in
data types, metadata, etc.) are certainly such that they are
worthy of current study, at least in the area of data management.

>
>Maybe it's just a glorified popularity contest; if so,
>it would seem to be at about the level of interest
>of the new season of "Dancing with the Stars".

Perhaps, but I know you guys are interested in that show :)
Who's not?

>I suppose the hype is necessary to generate the
>funding (which has its uses), but I'm not sure it
>will do as much as a few million sent to appropriate
>super PACs to move the politics of climate change
>along.

Think of this as an IT super PAC for next generation data management
techniques and systems to deal with data volumes and varieties that
we don't have hardware or CS tools to manage yet. I'm not talking
about writing to tape and letting it die the morgue. I'm talking about
even simple things like making it available after you write it to spinning
disk.

Cheers,
Chris

>
>Bruce B.
>
>On Wed, Mar 20, 2013 at 1:16 AM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hey Bruce,
>>
>> Hah!
>>
>> Unfortunately all you get is the short summary through
>> the website which does make it scientifically hard to
>> judge, however, then again this isn't science, it's

Re: My Hadoop Summit Talk: NASA+BigData

[Mattmann, Chris A (388J)]

Hey Bruce,

Hah!

Unfortunately all you get is the short summary through
the website which does make it scientifically hard to
judge, however, then again this isn't science, it's a
glorified popularity contest.

I have a little bit more detailed abstract that I wrote up,
pasted below (of course the part that they don't use to solicit votes):

---longer abstract
The NASA Jet Propulsion Laboratory, California Institute of
Technology contributes to many Big Data projects for Earth science such as
the
U.S. National Climate Assessment (NCA) and for astronomy such as next
generation astronomical instruments like the Square Kilometre Array (SKA)
that
will generate unprecedented volumes of data (700TB/sec!).
 
Through these projects, we are addressing four key
challenges critical for the Hadoop community and broader open source Big
Data
community to consider: (1) unobtrusively integrating science algorithms
into
large scale processing systems; (2) selecting and deploying high powered
data
movement technologies for data staging and remote data acquisition;
processing,
and delivery to our customers and users; (3) better leveraging of cloud
computing (storage and processing) technologies in NASA missions; and (4)
technologies for automatically and rapidly extracting text and metadata
from
the file formats, by some estimates ranging from a few thousand to over
fifty
thousand in total.
 
This talk will focus on those Big Data challenges, how NASA
JPL is addressing them both technologically (Hadoop, OODT, Tika, Nutch,
Solr)
and from a community standpoint (Apache, interacting with open source,
etc.).
I¹ll also discuss the future of Big Data at JPL and NASA and how others
can get
Involved.
-

You can think of that as the longer version of what I submitted. *grin*

Cheers,
Chris



On 3/19/13 7:20 PM, "Bruce Barkstrom"  wrote:

>OK, so you've got a three-word summary of some
>hyperbole with Dumbo, the Flying Elephant.
>How are you going to deal with the real
>scientific constraints on the physics of combining real
>measurement technologies and "mashing stuff together"?
>
>You need to remember that imaging instruments integrate
>radiances with spectral responses and Point Spread Function
>weighted averages over the FOV of whatever the instrument
>was looking at - and that's just the instantaneous (L1 measurement).
>If you do orthorectification, you've got variations in the uncertainties
>across the image where the parts of the image where you've
>increased the resolving power (by putting interpolated points
>closer together) and have also increased the noise from the
>orthorectification process that acts as a noise multiplier.
>
>Next, you've got stuff like cloud identification (and rejection or
>acceptance) - which depends on spectral response, solar illumination
>(during the day) and temperature and cloud property stuff during
>the night - and finally, you've got temporal interpolation (not just
>creating an average through emission driven by solar illumination
>during the day and IR cooling at night.  Where (the hel)l is
>the physics that deals with this stuff?  If you do get some
>statistical stuff, why should anyone believe it contributes to
>our understanding of climate change?
>
>I won't vote, but you can think of this as my input to your
>scientific conscience.
>
>Bruce B.
>
>On Tue, Mar 19, 2013 at 7:51 PM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hey Guys,
>>
>> I proposed a talk for NASA and Big Data at the Hadoop Summit:
>>
>> 
>>http://hadoopsummit2013.uservoice.com/forums/196822-future-of-apache-hado
>>op
>> /suggestions/3733470-nasa-science-and-technology-for-big-data-junkies-
>>
>>
>> If you still have votes, and would like to support my talk, I'd
>>certainly
>> appreciate it!
>>
>> Thank you for considering.
>>
>> Cheers,
>> Chris Mattmann
>> Vote Herder
>>
>>



default.xml
Description: default.xml

My Hadoop Summit Talk: NASA+BigData

[Mattmann, Chris A (388J)]

Hey Guys,

I proposed a talk for NASA and Big Data at the Hadoop Summit:

http://hadoopsummit2013.uservoice.com/forums/196822-future-of-apache-hadoop
/suggestions/3733470-nasa-science-and-technology-for-big-data-junkies-


If you still have votes, and would like to support my talk, I'd certainly
appreciate it!

Thank you for considering.

Cheers,
Chris Mattmann 
Vote Herder

FW: GSoC 2013

[Mattmann, Chris A (388J)]

[Apologies for cross post]

Guys, to play in the GSoC 2013 spec, we just need to tag issues in JIRA
with the gsoc2013 tag.

I'll try and come up with  few projects soon :)

Cheers,
Chris


On 3/15/13 11:15 AM, "Luciano Resende"  wrote:

>On Fri, Mar 15, 2013 at 11:01 AM, Manish Agrawal 
>wrote:
>> Hi
>>
>> I am Manish Agrawal, a 3rd year student of Mathematics and computing
>> department from IIT Delhi.
>>
>> I want to participate in GSoC 2013 through one of the ASF projects. I
>>would
>> be really thankful if you could please suggest me how should I proceed
>>for
>> the same.
>>
>> Hoping for a reply.
>>
>> Thanks
>> Manish Agrawal
>
>Google is sponsoring GSoC 2013, and Apache Software Foundation is
>planing to participate again.
>More information about Apache Participation in GSoC is available at :
>http://community.apache.org/gsoc.html.
>
>The proper way to find a project idea would be to identify an Apache
>Project in the area of your interest and start discussions with them
>via the project mailing list.
>
>The projects are starting to create their project ideas, and you can
>start browsing them at
>https://issues.apache.org/jira/secure/IssueNavigator!executeAdvanced.jspa?
>jqlQuery=labels+=+gsoc2013&runQuery=true&clear=true
>
>
>-- 
>Luciano Resende
>http://people.apache.org/~lresende
>http://twitter.com/lresende1975
>http://lresende.blogspot.com/

Re: Min Pool Size in WorkflowManager Thread Pool (0.3)

[Mattmann, Chris A (388J)]

Hi Gabe,

The best docs I can point you at are here:

http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/package-s
ummary.html#package_description



I would recommend also reading:

http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/Pooled
Executor.html


That really explains what the purpose of those diff properties are.

Cheers,
Chris


On 3/14/13 1:57 PM, "Resneck, Gabriel M (388J)"
 wrote:

>Hi, guys!
>I've run into a problem with the Workflow Manager in release 0.3.  The
>number of active threads allowed by the pool seems to be dictated by the
>minimum thread count when the queue supplied to the pool object is
>unlimited.  The default number (6) is a bit low for our purposes, so I
>was wondering how you guys have dealt with this issue in the past.  Did
>you simply increase the minimum pool size or implement another solution?
>If you increased the minimum pool size, what was the highest that you
>have used and did you see any issues as a result?
>Thanks!
>
>Gabe =)
>

Re: Research project on integrating geoservices with Apache Airavata

[Mattmann, Chris A (388J)]

Hi Amila,

Thanks!

Regarding the National Climate Assessment, the place to get more info on
our Snow project is:

http://snow.jpl.nasa.gov/

If you have any questions let me know.

Cheers,
Chris


On 3/10/13 10:41 AM, "AMILA RANATUNGA"  wrote:

>Hi,
>
>Thank you very much for co-operation and guidance towards this project and
>your willingness to in cooperate with us. Since we are doing our
>background
>research we are looking into how these projects works and what we can make
>out of those ( Airavata and SIS integration).
>
>The flow you have given get us some understanding about the  Snow project
>for the U.S. National Climate Assessment. We would like to have more
>details about this project if you have so,  to get some overall
>understanding.
>
>Thank You !
>
>
>
>
>
>On Fri, Mar 8, 2013 at 11:18 PM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hey Guys,
>>
>> One other thing I might point you at is the work going on in Apache SIS
>> [1]:
>>
>> http://incubator.apache.org/sis/
>>
>> I recently made a presentation on SIS to the NOAA FOSS meet up here:
>>
>> 
>>http://svn.apache.org/repos/asf/sis/presentations/NOAA-Meetup-Feb21-2013/
>>Ap
>> acheSIS-NOAAMeetup-Final.fodp
>>
>> Right now SIS has support for a Quad Tree, and there is information on
>>how
>> to connect it with Apache OODT:
>>
>> https://cwiki.apache.org/confluence/display/SIS/SIS+Wiki
>>
>> 
>>https://cwiki.apache.org/confluence/display/SIS/OODT+File+Manager+to+SIS+
>>Co
>> nnection+Demo
>>
>>
>> SIS is currently undergoing a humongous change bringing over GeoTK,
>>which
>> is essentially a fully supported
>> Java spatial library originated by Martin Desruisseaux.
>>
>> You may consider doing some Airavata and SIS integration, and
>>potentially
>> looking at some of the OODT integration with geospatial as well.
>>
>> See:
>>
>> http://snow.jpl.nasa.gov/
>>
>> That is an Apache OODT data system integrated with GDAL workflows, and
>> pushing data to GeoServer. Would be great to bring all the projects
>> together here.
>>
>> I'm copying dev@oodt and dev@sis for their feedback. Maybe we could do a
>> few geospatial projects during GSoC between the communities this summer.
>> We did a Geospatial project with Ross Laidlaw as my GSoC student (now on
>> the SIS and OODT PMCs) last summer.
>>
>> Cheers,
>> Chris
>>
>>
>> On 3/7/13 10:20 PM, "Sameera Jayaratna" 
>> wrote:
>>
>> >Hi all,
>> >
>> >We are a group of final year students from the Department of Computer
>> >Science and Engineering, University of Moratuwa, Sri Lanka. We are
>>doing a
>> >research project on Integration of Open Geo-Spatial Consortium¹s WPS
>>[1]
>> >with Apache Airavata under the supervision of Dr. Shahani Markus
>> >Weerawarana.
>> >
>> >The outcome of this project would be a geoscience gateway leveraging
>> >Apache
>> >Airavata and OGC¹s standards-based geo-services. As the initial step we
>> >are
>> >doing a background study on Apache Airavata, scientific workflows,
>> >scientific gateways, geoscience workflows and geo-services. We would
>>like
>> >to explore some solid examples of scientific workflows and resources
>>used
>> >to integrate them apart from what is published on Apache Airavata web
>> >site.
>> >
>> >We would like to receive any thoughts, comments and any other useful
>> >resources.
>> >
>> >[1] http://www.opengeospatial.org/standards/wps
>> >
>> >Thank you.
>> >Sameera.
>> >
>> >--
>> >*Sameera Jayaratna*
>> >*Undergraduate*
>> >*Department of Computer Science And Engineering*
>> >*University of Moratuwa*
>> >*Sri Lanka*
>>
>>

FW: [OPENING] Google Summer of Code Applications

[Mattmann, Chris A (388J)]

FYI

On 3/10/13 5:10 PM, "Lewis John Mcgibbney" 
wrote:

>I just told a huge lie.
>I got my dates mixed up...
>Students have from between April 22nd and May 3rd to get proposals in.
>Sorry about the mix up.
>
>Lewis
>
>On Sun, Mar 10, 2013 at 5:09 PM, Lewis John Mcgibbney <
>lewis.mcgibb...@gmail.com> wrote:
>
>> Hi All,
>>
>> We have from the 18th until the 29th to submit this years GSoC
>> proposals[0].
>>
>> Just a gentle reminder for any potential guys wanting to formally
>>apply...
>>
>> The idea would be to sort out any discrepancies just now and to develop
>> your proposal to a comprehensive standard.
>>
>> I am interested in mentoring another project this year, so can work with
>> folks who wish to progress with proposals.
>>
>> Thanks
>>
>> Lewis
>>
>> [0] http://www.google-melange.com/gsoc/events/google/gsoc2013
>>
>> --
>> *Lewis*
>>
>
>
>
>-- 
>*Lewis*

Re: Research project on integrating geoservices with Apache Airavata

[Mattmann, Chris A (388J)]

Hey Guys,

One other thing I might point you at is the work going on in Apache SIS
[1]:

http://incubator.apache.org/sis/

I recently made a presentation on SIS to the NOAA FOSS meet up here:

http://svn.apache.org/repos/asf/sis/presentations/NOAA-Meetup-Feb21-2013/Ap
acheSIS-NOAAMeetup-Final.fodp

Right now SIS has support for a Quad Tree, and there is information on how
to connect it with Apache OODT:

https://cwiki.apache.org/confluence/display/SIS/SIS+Wiki

https://cwiki.apache.org/confluence/display/SIS/OODT+File+Manager+to+SIS+Co
nnection+Demo


SIS is currently undergoing a humongous change bringing over GeoTK, which
is essentially a fully supported
Java spatial library originated by Martin Desruisseaux.

You may consider doing some Airavata and SIS integration, and potentially
looking at some of the OODT integration with geospatial as well.

See:

http://snow.jpl.nasa.gov/

That is an Apache OODT data system integrated with GDAL workflows, and
pushing data to GeoServer. Would be great to bring all the projects
together here.

I'm copying dev@oodt and dev@sis for their feedback. Maybe we could do a
few geospatial projects during GSoC between the communities this summer.
We did a Geospatial project with Ross Laidlaw as my GSoC student (now on
the SIS and OODT PMCs) last summer.

Cheers,
Chris


On 3/7/13 10:20 PM, "Sameera Jayaratna"  wrote:

>Hi all,
>
>We are a group of final year students from the Department of Computer
>Science and Engineering, University of Moratuwa, Sri Lanka. We are doing a
>research project on Integration of Open Geo-Spatial Consortium¹s WPS [1]
>with Apache Airavata under the supervision of Dr. Shahani Markus
>Weerawarana.
>
>The outcome of this project would be a geoscience gateway leveraging
>Apache
>Airavata and OGC¹s standards-based geo-services. As the initial step we
>are
>doing a background study on Apache Airavata, scientific workflows,
>scientific gateways, geoscience workflows and geo-services. We would like
>to explore some solid examples of scientific workflows and resources used
>to integrate them apart from what is published on Apache Airavata web
>site.
>
>We would like to receive any thoughts, comments and any other useful
>resources.
>
>[1] http://www.opengeospatial.org/standards/wps
>
>Thank you.
>Sameera.
>
>-- 
>*Sameera Jayaratna*
>*Undergraduate*
>*Department of Computer Science And Engineering*
>*University of Moratuwa*
>*Sri Lanka*

Re: QuakeSim workflow with Airavata

[Mattmann, Chris A (388J)]

Hey Guys,

We have a workflow in OODT on our Snow project for the U.S. National
Climate Assessment:

http://snow.jpl.nasa.gov/

That does just that (plumbs through GDAL; outputs to GeoTIFF; pushes
GeoTIFF both in Near Real Time and Historically to a map run by Leaflet
and backed by GeoServer).

There might be some room for synergy here. I'm copying the dev@oodt lists.

Also once you guys get this working you should present it in the ESIP
Geospatial Cluster and we'd love to have you present in the NASA ESDSWG
Geospatial group (copy to both those lists too).

Thanks and happy to chat more.

Cheers,
Chris


On 3/8/13 6:45 AM, "Jun Wang"  wrote:

>Hi, All
>
>We work on the QuakeSim project (http://quakesim.org) and are also
>interested in integrate GeoServer related stuff and for RDAHMM GPS
>processing with Airavata.
>
>Here is the slides to explain the basic idea.
>https://docs.google.com/file/d/0B5ipnY2ZK0lxRmI2aHBjOHl0NU0/edit?usp=shari
>ng
>
>Any suggestions and feedback are welcome.
>
>Thanks.
>
>Jun Wang

Re: Problem arising from XSD validation while configuring Crawler

[Mattmann, Chris A (388J)]

s/Gave/Gabe/

Sent from my iPhone

On Mar 5, 2013, at 4:29 PM, "Mattmann, Chris A (388J)" 
 wrote:

> Hey Gave,
> 
> Okey dokey great to hear!
> 
> Cheers,
> Chris
> 
> 
> On 3/5/13 9:10 AM, "Resneck, Gabriel M (388J)"
>  wrote:
> 
>> We ended up downloading the XSD file and changing the schemaLocation field
>> value in each of the XML files in the crawler policy directory to the
>> absolute path of that XSD file.  It worked a treat!
>> 
>> Gabe =)
>> 
>> 
>> On 3/5/13 12:29 AM, "Mattmann, Chris A (388J)"
>>  wrote:
>> 
>>> Hi Gabe,
>>> 
>>> This should be fixed in current OODT 0.5 and OODT 0.6-SNAPSHOT trunk. Can
>>> you try them?
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> On 2/28/13 3:22 PM, "Resneck, Gabriel M (388J)"
>>>  wrote:
>>> 
>>>> We in SMAP Land have been running into an issue while distributing jobs
>>>> to nodes with no internet access.
>>>> 
>>>> Here's the stack trace:
>>>> 
>>>> org.apache.oodt.cas.resource.structs.exceptions.JobInputException:
>>>> Failed
>>>> to run task
>>>> at 
>>>> org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
>>>> at 
>>>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.r
>>>> u
>>>> n
>>>> (XmlRpcBatchStub.java:229)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: 
>>>> org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExce
>>>> p
>>>> t
>>>> ion: PGETask failed : Line 12 in XML document from URL
>>>> [file:/project/spdm/subsys/deploy/spdm-crawler/policy/crawler-config.xml
>>>> ]
>>>> is invalid; nested exception is org.xml.sax.SAXParseException:
>>>> cvc-elt.1:
>>>> Cannot find the declaration of element 'beans'.
>>>> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:456)
>>>> at 
>>>> org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>>> ... 2 more
>>>> Caused by: 
>>>> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException:
>>>> Line 12 in XML document from URL
>>>> [file:/project/spdm/subsys/deploy/spdm-crawler/policy/crawler-config.xml
>>>> ]
>>>> is invalid; nested exception is org.xml.sax.SAXParseException:
>>>> cvc-elt.1:
>>>> Cannot find the declaration of element 'beans'.
>>>> at 
>>>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBean
>>>> D
>>>> e
>>>> finitions(XmlBeanDefinitionReader.java:369)
>>>> at 
>>>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDe
>>>> f
>>>> i
>>>> nitions(XmlBeanDefinitionReader.java:313)
>>>> at 
>>>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDe
>>>> f
>>>> i
>>>> nitions(XmlBeanDefinitionReader.java:290)
>>>> at 
>>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.l
>>>> o
>>>> a
>>>> dBeanDefinitions(AbstractBeanDefinitionReader.java:131)
>>>> at 
>>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.l
>>>> o
>>>> a
>>>> dBeanDefinitions(AbstractBeanDefinitionReader.java:147)
>>>> at 
>>>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.l
>>>> o
>>>> a
>>>> dBeanDefinitions(AbstractBeanDefinitionReader.java:173)
>>>> at 
>>>> org.springframework.context.support.AbstractXmlApplicationContext.loadBe
>>>> a
>>>> n
>>>> Definitions(AbstractXmlApplicationContext.java:112)
>>>> at 
>>>> org.springframework.context.support.AbstractXmlApplicationContext.loadBe
>>>> a
>>>> n
>>>> Definitions(AbstractXmlApplicationContext.java:79)
>>>> at 
>>>> org.springframework.context.support.AbstractRefreshableApplicationContex
>>>> t
>>>> .
>>>> refreshBeanFactory(AbstractRefreshableApplicationContext.java:101)
>>>> at 
>>>> org.springframework.context.support.AbstractApplicationContext

Re: Problem arising from XSD validation while configuring Crawler

[Mattmann, Chris A (388J)]

Hey Gave,

Okey dokey great to hear!

Cheers,
Chris


On 3/5/13 9:10 AM, "Resneck, Gabriel M (388J)"
 wrote:

>We ended up downloading the XSD file and changing the schemaLocation field
>value in each of the XML files in the crawler policy directory to the
>absolute path of that XSD file.  It worked a treat!
>
>Gabe =)
>
>
>On 3/5/13 12:29 AM, "Mattmann, Chris A (388J)"
> wrote:
>
>>Hi Gabe,
>>
>>This should be fixed in current OODT 0.5 and OODT 0.6-SNAPSHOT trunk. Can
>>you try them?
>>
>>Cheers,
>>Chris
>>
>>On 2/28/13 3:22 PM, "Resneck, Gabriel M (388J)"
>> wrote:
>>
>>>We in SMAP Land have been running into an issue while distributing jobs
>>>to nodes with no internet access.
>>>
>>>Here's the stack trace:
>>>
>>>org.apache.oodt.cas.resource.structs.exceptions.JobInputException:
>>>Failed
>>>to run task
>>>at 
>>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
>>>at 
>>>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.r
>>>u
>>>n
>>>(XmlRpcBatchStub.java:229)
>>>at java.lang.Thread.run(Thread.java:619)
>>>Caused by: 
>>>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExce
>>>p
>>>t
>>>ion: PGETask failed : Line 12 in XML document from URL
>>>[file:/project/spdm/subsys/deploy/spdm-crawler/policy/crawler-config.xml
>>>]
>>>is invalid; nested exception is org.xml.sax.SAXParseException:
>>>cvc-elt.1:
>>>Cannot find the declaration of element 'beans'.
>>>at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:456)
>>>at 
>>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>>... 2 more
>>>Caused by: 
>>>org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException:
>>>Line 12 in XML document from URL
>>>[file:/project/spdm/subsys/deploy/spdm-crawler/policy/crawler-config.xml
>>>]
>>>is invalid; nested exception is org.xml.sax.SAXParseException:
>>>cvc-elt.1:
>>>Cannot find the declaration of element 'beans'.
>>>at 
>>>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBean
>>>D
>>>e
>>>finitions(XmlBeanDefinitionReader.java:369)
>>>at 
>>>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDe
>>>f
>>>i
>>>nitions(XmlBeanDefinitionReader.java:313)
>>>at 
>>>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDe
>>>f
>>>i
>>>nitions(XmlBeanDefinitionReader.java:290)
>>>at 
>>>org.springframework.beans.factory.support.AbstractBeanDefinitionReader.l
>>>o
>>>a
>>>dBeanDefinitions(AbstractBeanDefinitionReader.java:131)
>>>at 
>>>org.springframework.beans.factory.support.AbstractBeanDefinitionReader.l
>>>o
>>>a
>>>dBeanDefinitions(AbstractBeanDefinitionReader.java:147)
>>>at 
>>>org.springframework.beans.factory.support.AbstractBeanDefinitionReader.l
>>>o
>>>a
>>>dBeanDefinitions(AbstractBeanDefinitionReader.java:173)
>>>at 
>>>org.springframework.context.support.AbstractXmlApplicationContext.loadBe
>>>a
>>>n
>>>Definitions(AbstractXmlApplicationContext.java:112)
>>>at 
>>>org.springframework.context.support.AbstractXmlApplicationContext.loadBe
>>>a
>>>n
>>>Definitions(AbstractXmlApplicationContext.java:79)
>>>at 
>>>org.springframework.context.support.AbstractRefreshableApplicationContex
>>>t
>>>.
>>>refreshBeanFactory(AbstractRefreshableApplicationContext.java:101)
>>>at 
>>>org.springframework.context.support.AbstractApplicationContext.obtainFre
>>>s
>>>h
>>>BeanFactory(AbstractApplicationContext.java:394)
>>>at 
>>>org.springframework.context.support.AbstractApplicationContext.refresh(A
>>>b
>>>s
>>>tractApplicationContext.java:324)
>>>at 
>>>org.springframework.context.support.FileSystemXmlApplicationContext.>>t
>>>>
>>>(FileSystemXmlApplicationContext.java:124)
>>>at 
>>>org.springframework.context.support.FileSystemXmlApplicationContext.>>t
>>>>
>>>(FileSystemXmlApplicationContext.java:68)
>>>at 
>>>org.apache.oodt.cas.pge.PGETaskInstance.setCrawlerConfigurations(PG

Re: Including Workflow Manager in Resource Manager Package using Maven Assembly Plug-In

[Mattmann, Chris A (388J)]

>   
>   
>   
>
>
>
>Here is the assembly spec for the Workflow Manager:
>
>
>   dist
>   
>   tar.gz
>   zip
>   
>   true
>   ${project.artifactId}-${project.version}
>   false
>   
>   
>   ${basedir}/src/main/bin
>   bin
>   
>   755
>   
>   
>   ${basedir}/src/main/resources/etc
>   etc
>   
>   
>   
>   
> ${basedir}/src/main/resources/logs
>   logs
>   
>   
>   
> ${basedir}/src/main/resources/policy
>   policy
>   
>   
>   
>   ${basedir}/src/main/ojdbc-dist
>   lib
>   
>   
>   
>   
>   
>   lib
>   false
>   true
>   
> true
>   
>   
>   
>
>
>
>Here is the POM for the Resource Manager:
>
>
>xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
>   xsi:schemaLocation='http://maven.apache.org/POM/4.0.0
>http://maven.apache.org/maven-v4_0_0.xsd'>
>   4.0.0
>   
>   gov.nasa.smap
>   spdm
>   0.4.0-dev
>   
>
>   gov.nasa.smap.spdm
>   spdm-resource
>   ${spdm-version}
>   jar
>
>   SPDM Resource Manager
>   
>   The SPDM Resource Manager is an extension of PCS Resource Manager to
>manage the job resources.
>   
>
>   
>   
>   org.apache.oodt
>   cas-resource
>   ${oodt-version}
>   
>   
>   gov.nasa.smap.spdm
>   spdm-filemgr
>   ${spdm-version}
>   
>   
>   gov.nasa.smap.spdm
>   spdm-crawler
>   runtime
>   ${spdm-version}
>   
>   
>   net.sf.saxon
>   saxon9he
>   
>   
>   
>   
>   org.apache.oodt
>   cas-pge
>   ${oodt-version}
>   
>   
>   net.sf.saxon
>   saxon-dom
>   
>   
>   
>   
>   net.sf.saxon
>   Saxon-HE
>   9.4
>   
>   
>
>   
>   
>   
>   org.apache.maven.plugins
>   maven-assembly-plugin
>   
>   
>   
> src/main/assembly/assembly.xml
>   
>   
>   
>   
>   package
>   
>   single
>   
>   
>   
>       
>   
>   
>
>
>
>
>Aand, here is the assembly spec for the Resource Manager.  I suspect
>that this is what I'll have to change, I'm just unsure of what changes I
>need to make to move the Workflow Manager JAR into the lib directory:
>
>
>   dist
>   
>   tar.gz
>   zip
>   
>   true
>   ${project.artifactId}-${project.version}
>   false
>   
>   
>   ${basedir}/src/main/bin
>   bin
>   
>   755
>   
>   
>   ${basedir}/src/main/resources/etc
>   etc

Re: Problem arising from XSD validation while configuring Crawler

[Mattmann, Chris A (388J)]

Hi Gabe,

This should be fixed in current OODT 0.5 and OODT 0.6-SNAPSHOT trunk. Can
you try them?

Cheers,
Chris

On 2/28/13 3:22 PM, "Resneck, Gabriel M (388J)"
 wrote:

>We in SMAP Land have been running into an issue while distributing jobs
>to nodes with no internet access.
>
>Here's the stack trace:
>
>org.apache.oodt.cas.resource.structs.exceptions.JobInputException: Failed
>to run task
>at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
>at 
>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run
>(XmlRpcBatchStub.java:229)
>at java.lang.Thread.run(Thread.java:619)
>Caused by: 
>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcept
>ion: PGETask failed : Line 12 in XML document from URL
>[file:/project/spdm/subsys/deploy/spdm-crawler/policy/crawler-config.xml]
>is invalid; nested exception is org.xml.sax.SAXParseException: cvc-elt.1:
>Cannot find the declaration of element 'beans'.
>at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:456)
>at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>... 2 more
>Caused by: 
>org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException:
>Line 12 in XML document from URL
>[file:/project/spdm/subsys/deploy/spdm-crawler/policy/crawler-config.xml]
>is invalid; nested exception is org.xml.sax.SAXParseException: cvc-elt.1:
>Cannot find the declaration of element 'beans'.
>at 
>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDe
>finitions(XmlBeanDefinitionReader.java:369)
>at 
>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefi
>nitions(XmlBeanDefinitionReader.java:313)
>at 
>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefi
>nitions(XmlBeanDefinitionReader.java:290)
>at 
>org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loa
>dBeanDefinitions(AbstractBeanDefinitionReader.java:131)
>at 
>org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loa
>dBeanDefinitions(AbstractBeanDefinitionReader.java:147)
>at 
>org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loa
>dBeanDefinitions(AbstractBeanDefinitionReader.java:173)
>at 
>org.springframework.context.support.AbstractXmlApplicationContext.loadBean
>Definitions(AbstractXmlApplicationContext.java:112)
>at 
>org.springframework.context.support.AbstractXmlApplicationContext.loadBean
>Definitions(AbstractXmlApplicationContext.java:79)
>at 
>org.springframework.context.support.AbstractRefreshableApplicationContext.
>refreshBeanFactory(AbstractRefreshableApplicationContext.java:101)
>at 
>org.springframework.context.support.AbstractApplicationContext.obtainFresh
>BeanFactory(AbstractApplicationContext.java:394)
>at 
>org.springframework.context.support.AbstractApplicationContext.refresh(Abs
>tractApplicationContext.java:324)
>at 
>org.springframework.context.support.FileSystemXmlApplicationContext.
>(FileSystemXmlApplicationContext.java:124)
>at 
>org.springframework.context.support.FileSystemXmlApplicationContext.
>(FileSystemXmlApplicationContext.java:68)
>at 
>org.apache.oodt.cas.pge.PGETaskInstance.setCrawlerConfigurations(PGETaskIn
>stance.java:371)
>at 
>org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.jav
>a:348)
>at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>... 3 more
>Caused by: org.xml.sax.SAXParseException: cvc-elt.1: Cannot find the
>declaration of element 'beans'.
>at 
>org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
> Source)
>at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
>at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
>at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
>at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
>at 
>org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown
>Source)
>at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown
>Source)
>at 
>org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown
>Source)
>at 
>org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRo
>otElementHook(Unknown Source)
>at 
>org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispa
>tcher.dispatch(Unknown Source)
>at 
>org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
>at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
>at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
>at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
>at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
>at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
>at 
>org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(D
>efaultDocumentLoader.java:73)
>at 
>org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDe
>finitions(XmlB

Re: svn commit: r1450390 - in /oodt/trunk: mvn/archetypes/opsui/pom.xml mvn/archetypes/radix/pom.xml pom.xml

[Mattmann, Chris A (388J)]

Hey Paul,

Can you file some JIRA issues for these? Looks like a significant amount
of work went into RADIX updates over ApacheCon NA 2013 and it would be
great to have JIRA tickets for these in the change log.

Cheers,
Chris


On 2/26/13 12:22 PM, "prami...@apache.org"  wrote:

>Author: pramirez
>Date: Tue Feb 26 20:22:06 2013
>New Revision: 1450390
>
>URL: http://svn.apache.org/r1450390
>Log:
>Tie archetypes into the  OODT build so that they get picked up and
>published to Maven Central.
>
>Modified:
>oodt/trunk/mvn/archetypes/opsui/pom.xml
>oodt/trunk/mvn/archetypes/radix/pom.xml
>oodt/trunk/pom.xml
>
>Modified: oodt/trunk/mvn/archetypes/opsui/pom.xml
>URL: 
>http://svn.apache.org/viewvc/oodt/trunk/mvn/archetypes/opsui/pom.xml?rev=1
>450390&r1=1450389&r2=1450390&view=diff
>==
>
>--- oodt/trunk/mvn/archetypes/opsui/pom.xml (original)
>+++ oodt/trunk/mvn/archetypes/opsui/pom.xml Tue Feb 26 20:22:06 2013
>@@ -7,7 +7,7 @@
>   0.4-SNAPSHOT
>   maven-archetype
> 
>-  opsui-archetype
>+  OODT :: Archetypes :: OpsUI
> 
>   
> 
>
>Modified: oodt/trunk/mvn/archetypes/radix/pom.xml
>URL: 
>http://svn.apache.org/viewvc/oodt/trunk/mvn/archetypes/radix/pom.xml?rev=1
>450390&r1=1450389&r2=1450390&view=diff
>==
>
>--- oodt/trunk/mvn/archetypes/radix/pom.xml (original)
>+++ oodt/trunk/mvn/archetypes/radix/pom.xml Tue Feb 26 20:22:06 2013
>@@ -7,7 +7,7 @@
>   0.6-SNAPSHOT
>   maven-archetype
> 
>-  RADiX Distribution of OODT
>+  OODT :: Archetypes :: RADiX
> 
>   
> 
>
>Modified: oodt/trunk/pom.xml
>URL: 
>http://svn.apache.org/viewvc/oodt/trunk/pom.xml?rev=1450390&r1=1450389&r2=
>1450390&view=diff
>==
>
>--- oodt/trunk/pom.xml (original)
>+++ oodt/trunk/pom.xml Tue Feb 26 20:22:06 2013
>@@ -59,6 +59,7 @@ the License.
> curator
> pge
> mvn/plugins/cas-install
>+mvn/archetypes
> pushpull
> product
> profile
>
>

ApacheCon NA 2013 Post Mortem: Apache in Science!

[Mattmann, Chris A (388J)]

Hey Guys,

For all those at ApacheCon NA 2013, just wanted to thank you for being
there and for making the Apache in Science Track a success!

For all those that weren't there, we missed you!

I've created a page on the wiki:

https://cwiki.apache.org/confluence/display/OODT/OODT+ApacheCon+2013+Wiki+P
age


Please feel free to link your slides. All of the abstracts are linked so
far, and Rishi and my presentation were the only ones I could find online.

I think we should definitely have another Apache in Science track at the
next ApacheCon.

Thanks all!

Cheers,
Chris

Re: Including Workflow Manager in Resource Manager Package using Maven Assembly Plug-In

[Mattmann, Chris A (388J)]

Hi Gabe,

Can you post your assembly specs, and your pom.xmls?

WM should depend on RM mainly for the Job interface and JobInput
interface, so that sounds right, so not sure what you are seeingŠ

Cheers,
Chris


On 2/23/13 11:33 AM, "Resneck, Gabriel M (388J)"
 wrote:

>Hi, all!
>I know that the OODT community is home to some Maven gurus, and I'm
>hoping to tap into that expertise.
>I have recently changed the SMAP code in such a way that our
>project-specific extensions to the Workflow Manager depend upon our
>extensions to the Resource Manager.  This is actually similar to the
>setup in OODT 0.5, however, these changes have upset how the Maven
>assembly plug-in builds our project.  Before, the dependency went the
>other way, which meant that JARs important to the Workflow Manager would
>make their way into the lib directory of Resource Manager which needed
>those JARs.
>Our assembly specification is basically the same as the one used in the
>core.  Does anyone happen to know how I could change it to include the
>Workflow Manager JAR that we create in the lib directory of the Resource
>Manager.
>Thanks heaps!
>
>Gabe =)
>

Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens

[Mattmann, Chris A (388J)]

Hi Sanjaya,

Great. You may simply want to extend (sub-class) PgeTaskInstance, and
create one for Airavata?

Cheers,
Chris

On 2/20/13 10:29 AM, "Sanjaya Medonsa"  wrote:

>Thanks Chris! As you suggested previously, I am looking into CAS-PGE and
>plan is to reuse the same code. I have looked at PGETaskInstance where
>most
>of these pre/post task execution implementation resides. I believe
>FileManageFileStager class should be able to reuse  easily. First I'll
>focus on the input fileStaging and then I am planning to focus on
>ingesting
>products and updating metadata as post execution task.
>
>Best Regards,
>Sanjaya
>
>On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hi Sanjaya,
>>
>> I would seriously recommend looking at OODT CAS-PGE, which already does
>> file staging, and connects to the file manager using queries. You could
>> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot
>>of
>> the existing FM to WM to RM and crawl infrastructure from OODT.
>>
>> Cheers,
>> Chris
>>
>> On 2/19/13 10:52 AM, "Sanjaya Medonsa"  wrote:
>>
>> >Thanks Lahiru! I have gone through the test classes and the classes in
>> >package org.apache.airavata.gfac. It was really helpful to understand
>>the
>> >new architecture. I have listed down my approach based on new
>>architecture
>> >to use Apache OODT products as an input to Airavata.
>> > 1. Introduce new Data Type to represent Apache OODT Product
>>as an
>> >DataType. Basically new DataType is added into the
>>GFacParameterTypes.xsd.
>> > 2. With new Architecture In Handlers and Out Handlers replaces
>> >the
>> >Pre/Post execution chains in old architecture. For the moment I am
>> >focusing
>> >on using Apache OODT product ID or file path as an input and stage the
>> >file
>> >(product) into host where actual execution happens. File staging
>>requires
>> >to retrieve product from a File Manager server to the host where
>>execution
>> >occurs. File staging can be implemented as an* In Handler* and needs
>>to be
>> >configured as a new item in the list of configured In Handlers.
>> > 3. Handler should first verify the input parameter types
>>listed
>> >in
>> >Service Description of the Application context of the
>> >*JobExecutionContext*.
>> >If input type matches the new parameter type, in handler stage file
>>into
>> >host machine using Apache OODT file manager component. Corresponding
>>input
>> >value can be retrieved from *In MessageContex*t. If a parameter type in
>> >MessageContext matches the new input type, then corresponding value is
>>the
>> >id or file path to product managed by Apache OODT File Manager server.
>> >
>> >Best Regards,
>> >Sanjaya
>> >
>> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
>> >wrote:
>> >
>> >> Hi Sanjaya,
>> >>
>> >> If you want to understand the new architecture by looking in to the
>> >>code,
>> >> please just refer the package org.apache.airavata.gfac, do not refer
>> >>any of
>> >> the classes in org.apache.airavata.core.gfac.
>> >>
>> >> Best place to start with is referring is from test classes
>> >> (LocalProviderTest,GramProviderTest), from tehre you can start
>>looking
>> >>in
>> >> to GFacAPI class and see how to execution is flawing.
>> >>
>> >> if you have further questions please post on the list and more than
>> >>happy
>> >> to help. I will be doing some documentation about the architecture,
>> >>once I
>> >> am done, will post in to the list. And we will be having an
>>architecture
>> >> review this week, so please watch the mailing list, if possible
>>please
>> >>try
>> >> to join us.
>> >>
>> >> Regards
>> >> Lahiru
>> >>
>> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa
>>> >> >wrote:
>> >>
>> >> > Thanks Suresh and Chris! It seems I am moving on the correct path.
>>I
>> >>have
>> >> > followed the email thread on improved GFac architecture. Though I
>>am
>> >>not
>> >> > entirely clear on the improved GFac architecture, proposed
>>integration
>> >> with
&g

[NOTICE] Congrats to Sean Kelly, the new Apache OODT PMC chair!

[Mattmann, Chris A (388J)]

Hey Everyone,

Today at the board meeting, the board VOTEd and approved to make Sean
Kelly our new Apache OODT PMC chair.

Great job, Sean!

Cheers,
Chris

Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens

[Mattmann, Chris A (388J)]

Hi Sanjaya,

I would seriously recommend looking at OODT CAS-PGE, which already does
file staging, and connects to the file manager using queries. You could
wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of
the existing FM to WM to RM and crawl infrastructure from OODT.

Cheers,
Chris

On 2/19/13 10:52 AM, "Sanjaya Medonsa"  wrote:

>Thanks Lahiru! I have gone through the test classes and the classes in
>package org.apache.airavata.gfac. It was really helpful to understand the
>new architecture. I have listed down my approach based on new architecture
>to use Apache OODT products as an input to Airavata.
> 1. Introduce new Data Type to represent Apache OODT Product as an
>DataType. Basically new DataType is added into the GFacParameterTypes.xsd.
> 2. With new Architecture In Handlers and Out Handlers replaces
>the
>Pre/Post execution chains in old architecture. For the moment I am
>focusing
>on using Apache OODT product ID or file path as an input and stage the
>file
>(product) into host where actual execution happens. File staging requires
>to retrieve product from a File Manager server to the host where execution
>occurs. File staging can be implemented as an* In Handler* and needs to be
>configured as a new item in the list of configured In Handlers.
> 3. Handler should first verify the input parameter types listed
>in
>Service Description of the Application context of the
>*JobExecutionContext*.
>If input type matches the new parameter type, in handler stage file into
>host machine using Apache OODT file manager component. Corresponding input
>value can be retrieved from *In MessageContex*t. If a parameter type in
>MessageContext matches the new input type, then corresponding value is the
>id or file path to product managed by Apache OODT File Manager server.
>
>Best Regards,
>Sanjaya
>
>On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake
>wrote:
>
>> Hi Sanjaya,
>>
>> If you want to understand the new architecture by looking in to the
>>code,
>> please just refer the package org.apache.airavata.gfac, do not refer
>>any of
>> the classes in org.apache.airavata.core.gfac.
>>
>> Best place to start with is referring is from test classes
>> (LocalProviderTest,GramProviderTest), from tehre you can start looking
>>in
>> to GFacAPI class and see how to execution is flawing.
>>
>> if you have further questions please post on the list and more than
>>happy
>> to help. I will be doing some documentation about the architecture,
>>once I
>> am done, will post in to the list. And we will be having an architecture
>> review this week, so please watch the mailing list, if possible please
>>try
>> to join us.
>>
>> Regards
>> Lahiru
>>
>> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa > >wrote:
>>
>> > Thanks Suresh and Chris! It seems I am moving on the correct path. I
>>have
>> > followed the email thread on improved GFac architecture. Though I am
>>not
>> > entirely clear on the improved GFac architecture, proposed integration
>> with
>> > OODT is primarily based on the GFac extension, PreExecustionChain,
>>which
>> > has not been modified with the Architecture improvements (As per one
>>of
>> the
>> > replies from Lahiru, output extension is supported with new
>> Architecture. I
>> > assume input extension is also supported).
>> >
>> > I have looked into provenance manager and related implementation.
>>Still I
>> > am unclear how Airavata support provenance aware work flow processing.
>> >
>> > Best Regards,
>> > Sanjaya
>> >
>> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru 
>>wrote:
>> >
>> > > Hi Sanjaya,
>> > >
>> > > This sounds very exciting. Both Airavata and OODT projects have good
>> > > synergies and have been long looking for volunteers who can bridge
>>them
>> > > both. Please do not hesitate to ask any questions to either or both
>>the
>> > dev
>> > > lists. The more engaged you are, you will find use cases and
>>feedback
>> > which
>> > > should help your MSc project.
>> > >
>> > > Your plan sounds good. If you are following dev list, you may have
>> > > noticed, the GFac architecture has been improved to properly support
>> this
>> > > kind of handler architecture.
>> > >
>> > > You may also want to look at Airavata Registry API which has
>> organically
>> > > emerg

Re: Build failed in Jenkins: oodt-trunk #840

[Mattmann, Chris A (388J)]

Hey Luca, any idea why this broke? Looks like a broken channel and some
Jenkins-side error and not our own.

If it persists, we'll contact builds@

Cheers,
Chris


On 2/18/13 10:39 PM, "Apache Jenkins Server" 
wrote:

>See 
>
>Changes:
>
>[luca] Updated CHANGES.txt for OODT-558
>
>[luca] Allowing web-grid servlet to be configured via external XML file
>(OODT-558)
>
>--
>[...truncated 3812 lines...]
>845b downloaded  (wagon-http-shared-1.0-beta-2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/jtidy/jtidy/4aug2000r7-dev/jtidy-4aug2000r7-
>dev.pom
>1K downloaded  (jtidy-4aug2000r7-dev.pom)
>Downloading: 
>http://repo1.maven.org/maven2/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.pom
>2K downloaded  (xml-apis-1.0.b2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/reporting/maven-reporting-a
>pi/2.0.6/maven-reporting-api-2.0.6.pom
>1K downloaded  (maven-reporting-api-2.0.6.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/reporting/maven-reporting/2
>.0.6/maven-reporting-2.0.6.pom
>1K downloaded  (maven-reporting-2.0.6.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.0-al
>pha-7/doxia-sink-api-1.0-alpha-7.pom
>424b downloaded  (doxia-sink-api-1.0-alpha-7.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/doxia/doxia/1.0-alpha-7/dox
>ia-1.0-alpha-7.pom
>3K downloaded  (doxia-1.0-alpha-7.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/maven-error-diagnostics/2.0
>.6/maven-error-diagnostics-2.0.6.pom
>1K downloaded  (maven-error-diagnostics-2.0.6.pom)
>Downloading: 
>http://repo1.maven.org/maven2/commons-cli/commons-cli/1.0/commons-cli-1.0.
>pom
>2K downloaded  (commons-cli-1.0.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-api/2.0.6/mave
>n-plugin-api-2.0.6.pom
>1K downloaded  (maven-plugin-api-2.0.6.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh-external/1.
>0-beta-2/wagon-ssh-external-1.0-beta-2.pom
>1K downloaded  (wagon-ssh-external-1.0-beta-2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh-common/1.0-
>beta-2/wagon-ssh-common-1.0-beta-2.pom
>851b downloaded  (wagon-ssh-common-1.0-beta-2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interactivity-api
>/1.0-alpha-4/plexus-interactivity-api-1.0-alpha-4.pom
>6K downloaded  (plexus-interactivity-api-1.0-alpha-4.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-container-default
>/1.0-alpha-7/plexus-container-default-1.0-alpha-7.pom
>1K downloaded  (plexus-container-default-1.0-alpha-7.pom)
>Downloading: 
>http://repo1.maven.org/maven2/plexus/plexus-containers/1.0.2/plexus-contai
>ners-1.0.2.pom
>471b downloaded  (plexus-containers-1.0.2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/plexus/plexus-root/1.0.3/plexus-root-1.0.3.p
>om
>5K downloaded  (plexus-root-1.0.3.pom)
>Downloading: 
>http://repo1.maven.org/maven2/plexus/plexus-utils/1.0.2/plexus-utils-1.0.2
>.pom
>740b downloaded  (plexus-utils-1.0.2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/maven-plugin-descriptor/2.0
>.6/maven-plugin-descriptor-2.0.6.pom
>1K downloaded  (maven-plugin-descriptor-2.0.6.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/maven-monitor/2.0.6/maven-m
>onitor-2.0.6.pom
>1K downloaded  (maven-monitor-2.0.6.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh/1.0-beta-2/
>wagon-ssh-1.0-beta-2.pom
>1K downloaded  (wagon-ssh-1.0-beta-2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/com/jcraft/jsch/0.1.27/jsch-0.1.27.pom
>965b downloaded  (jsch-0.1.27.pom)
>Downloading: 
>http://repo1.maven.org/maven2/classworlds/classworlds/1.1/classworlds-1.1.
>pom
>3K downloaded  (classworlds-1.1.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-utils/1.5.15/plex
>us-utils-1.5.15.pom
>6K downloaded  (plexus-utils-1.5.15.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/codehaus/plexus/plexus/2.0.2/plexus-2.0.
>2.pom
>11K downloaded  (plexus-2.0.2.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.1
>2/plexus-interpolation-1.12.pom
>889b downloaded  (plexus-interpolation-1.12.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/codehaus/plexus/plexus-components/1.1.14
>/plexus-components-1.1.14.pom
>5K downloaded  (plexus-components-1.1.14.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/sonatype/plexus/plexus-build-api/0.0.4/p
>lexus-build-api-0.0.4.pom
>2K downloaded  (plexus-build-api-0.0.4.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/sonatype/spice/spice-parent/10/spice-par
>ent-10.pom
>2K downloaded  (spice-parent-10.pom)
>Downloading: 
>http://repo1.maven.org/maven2/org/sonatype/forge/forge-parent/3/forge-pare
>nt-3.pom
>4K downloaded  (forge-parent-3.p

Re: Proposal To Use Apache OODT products as input to Apache Airavata Workflows and staging product files into node where execution happens

[Mattmann, Chris A (388J)]

+1, sounds like a great idea Sanjaya!

I'm copying dev@oodt so they can be in the conversation too.

Cheers,
Chris

On 2/17/13 10:22 AM, "Sanjaya Medonsa"  wrote:

>Hi Dev Team,
>As I have posted previously, I am working on a Apache Airavata + Apache
>OODT integration as my MSc project. Following is one of the possible
>integration to leverage Apache OODT file management capability into Apache
>Airavata. Please review the proposal and let me know your feedback.
>
>Proposal To Use Apache OODT products as input to Apache Airavata Workflows
>and staging product files into node where execution happens
>==
>==
>1. Introduce "Apache OODT Product" as a new GFacParameterType. New "Apache
>OODT Product" input type can sepcify "Product ID" or "File Path to
>Product"
>as an input to Apache Airavata workflows.
>2. Introduce new PreExecuteChain to retrieve Apache OODT Products from
>File
>Manager Server managed using Apache OODT.
>1. Using Apache OODT File Manager componenet transfer Product from Server
>to input directory of the application as configured using XBaya-GUI under
>advanced configuration. (Here the assumption is that Products are
>accesible
>through Apache OODT File Manager server)
>2. Finally reset the input value to local file path. I think we can remove
>the OODT Product parameter from invocation context and add new file
>parameter with value set to 'local path of the transferred product'. I am
>not quite sure what are the implications of changing input parameter type
>during the execution.
>
>Similar approach has been implemented for GridFTP and HTTP.
>
>Best Regards,
>Sanjaya

Re: [GSoC Mentors] Google Summer of Code 2013

[Mattmann, Chris A (388J)]

You are the man. :)

Thanks my friend.

Cheers,
Chris

On 2/15/13 2:30 PM, "Ross Laidlaw"  wrote:

>Hi Chris,
>
>Many thanks for your kind words of encouragement, much appreciated!
>I'm looking forward to making more contributions in the near future.
>For example, I've been working on a solution for OODT-470 and hope to
>have a proposal ready soon for discussion/review.  I'll try to use the
>usual channels and post to JIRA and the review board when it's ready
>:)
>
>Ross
>
>
>
>On 15 February 2013 02:32, Mattmann, Chris A (388J)
> wrote:
>> Heya Ross,
>>
>> You were an amazing GSoC student last year, and a great member of the
>>OODT
>> PMC and SIS PMC.
>>
>> Please continue to contribute and we would be happy to have you as a
>>GSoC
>> again this summer!
>>
>> Cheers,
>> Chris
>>
>> On 2/13/13 11:23 PM, "Ross Laidlaw"  wrote:
>>
>>>Hi Chris & Cam,
>>>
>>>I'm listening in on the OODT and SIS lists and I've been following
>>>along with great interest.  My GSoC experience last summer was very
>>>positive.  I found OODT to be highly accessible because of its
>>>component-based structure.  This allowed me to focus on one area (File
>>>Manager) and start working on my project without having to learn
>>>everything upfront about each part of OODT.  The approach for my
>>>project was to break it down into smaller deliverable steps.  I liked
>>>this idea as it helped me to gain momentum.  I found the OODT and SIS
>>>communities to be very knowledgable, welcoming and encouraging.  All
>>>of these aspects gave me a lot of enthusiasm to contribute and to
>>>complete GSoC.
>>>
>>>I've tried to spread the word about GSoC to my classmates and
>>>colleagues.  Every year there are tons of applicants and there should
>>>be lots of healthy competition again this year for places.  I'm still
>>>a student and I'm very keen to apply this year if the OODT/SIS
>>>communities would be happy for me to contribute again via GSoC.
>>>
>>>Ross
>>>
>>>
>>>
>>>On 13 February 2013 22:09, Cameron Goodale  wrote:
>>>> Chris,
>>>>
>>>> I think it would be a good idea to check in with Ross who was our GSoC
>>>> student last summer, and ask him for feedback on his experience with
>>>>OODT.
>>>>  I hope he is listening to the dev list still, but if not we should
>>>> check-in and see what he is doing now.
>>>>
>>>> My hope is that if Ross gives us feedback (good or bad or both) we can
>>>>work
>>>> to improve this summer for our next GSoC student.  My apologies in
>>>>advance
>>>> if you already did this when the GSoC last year was closed out.
>>>>
>>>> Just a few questions to get the feedback pump primed would be:
>>>>
>>>> Was the mini-project challenging enough?
>>>> Did the OODT Devs provide enough information and support?
>>>> Did you encounter anything that was frustrating or unnecessarily
>>>>difficult?
>>>>
>>>> Any thoughts?
>>>>
>>>>
>>>> -Cam chee
>>>>
>>>>
>>>>
>>>> On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
>>>> chris.a.mattm...@jpl.nasa.gov> wrote:
>>>>
>>>>> [Sorry for cross posting]
>>>>>
>>>>> Guys,
>>>>>
>>>>> FYI please note that you can participate as a mentor from a PMC via
>>>>>Apache
>>>>> as they are a GSoC org. ComDev will coordinate our participation but
>>>>>start
>>>>> thinking about what projects we may want to do.
>>>>>
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>> From: Carol Smith mailto:car...@google.com>>
>>>>> Date: Monday, February 11, 2013 11:02 AM
>>>>> To: Google Summer of Code Mentors List <
>>>>> google-summer-of-code-mentors-l...@googlegroups.com>>>> google-summer-of-code-mentors-l...@googlegroups.com>>
>>>>> Subject: [GSoC Mentors] Google Summer of Code 2013
>>>>>
>>>>> Hi GSoC mentors and org admins,
>>>>>
>>>>> We've announced that we're doing Google Summer of Code 2013 [1]. Yay!
>>>>>
>>>>> If you would 

Re: [GSoC Mentors] Google Summer of Code 2013

[Mattmann, Chris A (388J)]

Cam, 

Great approach to reaching out to Ross -- would you like to be a GSoC
mentor this summer?
*nudge*

Cheers,
Chris

On 2/14/13 12:51 PM, "Cameron Goodale"  wrote:

>Ross,
>
>Thanks for the feedback and I am glad to hear that your GSoC experience
>was
>good enough that you are considering doing it again. Is there anything we
>can do differently to improve this summer?  (don't worry you won't hurt my
>feelings)
>
>Cheers,
>
>
>Cameron
>
>
>On Wed, Feb 13, 2013 at 11:23 PM, Ross Laidlaw
>wrote:
>
>> Hi Chris & Cam,
>>
>> I'm listening in on the OODT and SIS lists and I've been following
>> along with great interest.  My GSoC experience last summer was very
>> positive.  I found OODT to be highly accessible because of its
>> component-based structure.  This allowed me to focus on one area (File
>> Manager) and start working on my project without having to learn
>> everything upfront about each part of OODT.  The approach for my
>> project was to break it down into smaller deliverable steps.  I liked
>> this idea as it helped me to gain momentum.  I found the OODT and SIS
>> communities to be very knowledgable, welcoming and encouraging.  All
>> of these aspects gave me a lot of enthusiasm to contribute and to
>> complete GSoC.
>>
>> I've tried to spread the word about GSoC to my classmates and
>> colleagues.  Every year there are tons of applicants and there should
>> be lots of healthy competition again this year for places.  I'm still
>> a student and I'm very keen to apply this year if the OODT/SIS
>> communities would be happy for me to contribute again via GSoC.
>>
>> Ross
>>
>>
>>
>> On 13 February 2013 22:09, Cameron Goodale  wrote:
>> > Chris,
>> >
>> > I think it would be a good idea to check in with Ross who was our GSoC
>> > student last summer, and ask him for feedback on his experience with
>> OODT.
>> >  I hope he is listening to the dev list still, but if not we should
>> > check-in and see what he is doing now.
>> >
>> > My hope is that if Ross gives us feedback (good or bad or both) we can
>> work
>> > to improve this summer for our next GSoC student.  My apologies in
>> advance
>> > if you already did this when the GSoC last year was closed out.
>> >
>> > Just a few questions to get the feedback pump primed would be:
>> >
>> > Was the mini-project challenging enough?
>> > Did the OODT Devs provide enough information and support?
>> > Did you encounter anything that was frustrating or unnecessarily
>> difficult?
>> >
>> > Any thoughts?
>> >
>> >
>> > -Cam chee
>> >
>> >
>> >
>> > On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
>> > chris.a.mattm...@jpl.nasa.gov> wrote:
>> >
>> >> [Sorry for cross posting]
>> >>
>> >> Guys,
>> >>
>> >> FYI please note that you can participate as a mentor from a PMC via
>> Apache
>> >> as they are a GSoC org. ComDev will coordinate our participation but
>> start
>> >> thinking about what projects we may want to do.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> From: Carol Smith mailto:car...@google.com>>
>> >> Date: Monday, February 11, 2013 11:02 AM
>> >> To: Google Summer of Code Mentors List <
>> >> google-summer-of-code-mentors-l...@googlegroups.com> >> google-summer-of-code-mentors-l...@googlegroups.com>>
>> >> Subject: [GSoC Mentors] Google Summer of Code 2013
>> >>
>> >> Hi GSoC mentors and org admins,
>> >>
>> >> We've announced that we're doing Google Summer of Code 2013 [1]. Yay!
>> >>
>> >> If you would like to help spread the word about GSoC, we have
>> >> presentations [2], logos [3], and flyers [4] for you to use. Please
>>host
>> >> meetups, tell your friends and colleagues about the program, go to
>> >> conferences, talk to people about the program, and just generally do
>>all
>> >> the awesome word-of-mouth stuff you do every year to promote the
>> program.
>> >>
>> >> The GSoC calendar, FAQ, and events timeline have all been updated
>>with
>> >> this year's important dates, so please refer to those for the
>>milestones
>> >> for this year's program. NB: the normal timelin

Apache OODT mentioned in my SDTimes article on Apache in Science

[Mattmann, Chris A (388J)]

Hey Guys,

Mentioned Apache OODT (along with a number of other ones) in an article I
wrote for the SDTimes on Apache in Science:

http://www.sdtimes.com/content/article.aspx?ArticleID=39397&page=2

Just wanted to let you guys know

Cheers,
Chris

Re: [GSoC Mentors] Google Summer of Code 2013

[Mattmann, Chris A (388J)]

Heya Ross,

You were an amazing GSoC student last year, and a great member of the OODT
PMC and SIS PMC.

Please continue to contribute and we would be happy to have you as a GSoC
again this summer!

Cheers,
Chris

On 2/13/13 11:23 PM, "Ross Laidlaw"  wrote:

>Hi Chris & Cam,
>
>I'm listening in on the OODT and SIS lists and I've been following
>along with great interest.  My GSoC experience last summer was very
>positive.  I found OODT to be highly accessible because of its
>component-based structure.  This allowed me to focus on one area (File
>Manager) and start working on my project without having to learn
>everything upfront about each part of OODT.  The approach for my
>project was to break it down into smaller deliverable steps.  I liked
>this idea as it helped me to gain momentum.  I found the OODT and SIS
>communities to be very knowledgable, welcoming and encouraging.  All
>of these aspects gave me a lot of enthusiasm to contribute and to
>complete GSoC.
>
>I've tried to spread the word about GSoC to my classmates and
>colleagues.  Every year there are tons of applicants and there should
>be lots of healthy competition again this year for places.  I'm still
>a student and I'm very keen to apply this year if the OODT/SIS
>communities would be happy for me to contribute again via GSoC.
>
>Ross
>
>
>
>On 13 February 2013 22:09, Cameron Goodale  wrote:
>> Chris,
>>
>> I think it would be a good idea to check in with Ross who was our GSoC
>> student last summer, and ask him for feedback on his experience with
>>OODT.
>>  I hope he is listening to the dev list still, but if not we should
>> check-in and see what he is doing now.
>>
>> My hope is that if Ross gives us feedback (good or bad or both) we can
>>work
>> to improve this summer for our next GSoC student.  My apologies in
>>advance
>> if you already did this when the GSoC last year was closed out.
>>
>> Just a few questions to get the feedback pump primed would be:
>>
>> Was the mini-project challenging enough?
>> Did the OODT Devs provide enough information and support?
>> Did you encounter anything that was frustrating or unnecessarily
>>difficult?
>>
>> Any thoughts?
>>
>>
>> -Cam chee
>>
>>
>>
>> On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
>> chris.a.mattm...@jpl.nasa.gov> wrote:
>>
>>> [Sorry for cross posting]
>>>
>>> Guys,
>>>
>>> FYI please note that you can participate as a mentor from a PMC via
>>>Apache
>>> as they are a GSoC org. ComDev will coordinate our participation but
>>>start
>>> thinking about what projects we may want to do.
>>>
>>> Cheers,
>>> Chris
>>>
>>> From: Carol Smith mailto:car...@google.com>>
>>> Date: Monday, February 11, 2013 11:02 AM
>>> To: Google Summer of Code Mentors List <
>>> google-summer-of-code-mentors-l...@googlegroups.com>> google-summer-of-code-mentors-l...@googlegroups.com>>
>>> Subject: [GSoC Mentors] Google Summer of Code 2013
>>>
>>> Hi GSoC mentors and org admins,
>>>
>>> We've announced that we're doing Google Summer of Code 2013 [1]. Yay!
>>>
>>> If you would like to help spread the word about GSoC, we have
>>> presentations [2], logos [3], and flyers [4] for you to use. Please
>>>host
>>> meetups, tell your friends and colleagues about the program, go to
>>> conferences, talk to people about the program, and just generally do
>>>all
>>> the awesome word-of-mouth stuff you do every year to promote the
>>>program.
>>>
>>> The GSoC calendar, FAQ, and events timeline have all been updated with
>>> this year's important dates, so please refer to those for the
>>>milestones
>>> for this year's program. NB: the normal timeline for the program has
>>>been
>>> modified for this year. You'll probably want to examine the dates
>>>closely
>>> to make sure you know when important things are happening.
>>>
>>> Please consider translating the presentations and/or flyers into your
>>> native language and submitting them directly to me to post on the wiki.
>>> Localization for our material is integral to reaching the widest
>>>possible
>>> audience around the world. If you decide to translate a flyer, please
>>>fill
>>> out our form to request a thank you gift for your effort. [5]
>>>
>>> If you decide to

Re: [GSoC Mentors] Google Summer of Code 2013

[Mattmann, Chris A (388J)]

Hi Cam,

Thanks for pushing this forward. I think Ross replied later, so I'll go
look at his replies to this and reply there.

Thanks!

Cheers,
Chris

On 2/13/13 2:09 PM, "Cameron Goodale"  wrote:

>Chris,
>
>I think it would be a good idea to check in with Ross who was our GSoC
>student last summer, and ask him for feedback on his experience with OODT.
> I hope he is listening to the dev list still, but if not we should
>check-in and see what he is doing now.
>
>My hope is that if Ross gives us feedback (good or bad or both) we can
>work
>to improve this summer for our next GSoC student.  My apologies in advance
>if you already did this when the GSoC last year was closed out.
>
>Just a few questions to get the feedback pump primed would be:
>
>Was the mini-project challenging enough?
>Did the OODT Devs provide enough information and support?
>Did you encounter anything that was frustrating or unnecessarily
>difficult?
>
>Any thoughts?
>
>
>-Cam chee
>
>
>
>On Mon, Feb 11, 2013 at 12:24 PM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> [Sorry for cross posting]
>>
>> Guys,
>>
>> FYI please note that you can participate as a mentor from a PMC via
>>Apache
>> as they are a GSoC org. ComDev will coordinate our participation but
>>start
>> thinking about what projects we may want to do.
>>
>> Cheers,
>> Chris
>>
>> From: Carol Smith mailto:car...@google.com>>
>> Date: Monday, February 11, 2013 11:02 AM
>> To: Google Summer of Code Mentors List <
>> google-summer-of-code-mentors-l...@googlegroups.com> google-summer-of-code-mentors-l...@googlegroups.com>>
>> Subject: [GSoC Mentors] Google Summer of Code 2013
>>
>> Hi GSoC mentors and org admins,
>>
>> We've announced that we're doing Google Summer of Code 2013 [1]. Yay!
>>
>> If you would like to help spread the word about GSoC, we have
>> presentations [2], logos [3], and flyers [4] for you to use. Please host
>> meetups, tell your friends and colleagues about the program, go to
>> conferences, talk to people about the program, and just generally do all
>> the awesome word-of-mouth stuff you do every year to promote the
>>program.
>>
>> The GSoC calendar, FAQ, and events timeline have all been updated with
>> this year's important dates, so please refer to those for the milestones
>> for this year's program. NB: the normal timeline for the program has
>>been
>> modified for this year. You'll probably want to examine the dates
>>closely
>> to make sure you know when important things are happening.
>>
>> Please consider translating the presentations and/or flyers into your
>> native language and submitting them directly to me to post on the wiki.
>> Localization for our material is integral to reaching the widest
>>possible
>> audience around the world. If you decide to translate a flyer, please
>>fill
>> out our form to request a thank you gift for your effort. [5]
>>
>> If you decide to host a meetup, please email me to let me know the date,
>> time, and location so I can put it on the GSoC calendar. Also, remember
>>to
>> take pictures at your meetup and write up a blog post for our blog using
>> our provided template for formatting [6]. If you need promotional items
>>for
>> your attendees, please fill out our form [7] to request some; we're
>>happy
>> to send some along. We can provide up to about 25 pens, notebooks, or
>> stickers and/or a few t-shirts. Please keep in mind, though, that
>>shipping
>> restrictions and timeline vary country-to-country; request items early
>>to
>> make sure they get there on time! If you have questions about hosting
>> meetups, please see the section in our FAQ [8].
>>
>> Please consider applying to participate as an organization again this
>>year
>> or maybe joining as a mentor for your favorite organization if they are
>> selected this year.
>>
>> We rely on you for your help for the success of this program, so thank
>>you
>> in advance for all the work you do!
>>
>> [1] -
>> 
>>http://google-opensource.blogspot.com/2013/02/flip-bits-not-burgers-googl
>>e-summer-of.html
>> [2] -
>> http://code.google.com/p/google-summer-of-code/wiki/ProgramPresentations
>> [3] - http://code.google.com/p/google-summer-of-code/wiki/GsocLogos
>> [4] - http://code.google.com/p/google-summer-of-code/wiki/GsocFlyers
>> [5] - http://goo.gl/gEHDO
>> [6] - http://goo.gl/wbZrt
>> [7] - htt

Wiki page added summarizing a project to integrate Apache OODT and the Wings workflow system

[Mattmann, Chris A (388J)]

Hi Guys,

There has been some work going on that should be in the process for being
contributed to the ASF by Varun Ratnakar (CC'ed on this email) that
integrates that the Wings workflow system (http://wings-workflows.org/)
with Apache OODT. You can find the wiki page here:

https://cwiki.apache.org/confluence/display/OODT/Integrating+OODT+with+Wing
s


Welcome Varun, and looking forward to your contributions!

Cheers,
Chris

FW: [GSoC Mentors] Google Summer of Code 2013

[Mattmann, Chris A (388J)]

[Sorry for cross posting]

Guys,

FYI please note that you can participate as a mentor from a PMC via Apache as 
they are a GSoC org. ComDev will coordinate our participation but start 
thinking about what projects we may want to do.

Cheers,
Chris

From: Carol Smith mailto:car...@google.com>>
Date: Monday, February 11, 2013 11:02 AM
To: Google Summer of Code Mentors List 
mailto:google-summer-of-code-mentors-l...@googlegroups.com>>
Subject: [GSoC Mentors] Google Summer of Code 2013

Hi GSoC mentors and org admins,

We've announced that we're doing Google Summer of Code 2013 [1]. Yay!

If you would like to help spread the word about GSoC, we have presentations 
[2], logos [3], and flyers [4] for you to use. Please host meetups, tell your 
friends and colleagues about the program, go to conferences, talk to people 
about the program, and just generally do all the awesome word-of-mouth stuff 
you do every year to promote the program.

The GSoC calendar, FAQ, and events timeline have all been updated with this 
year's important dates, so please refer to those for the milestones for this 
year's program. NB: the normal timeline for the program has been modified for 
this year. You'll probably want to examine the dates closely to make sure you 
know when important things are happening.

Please consider translating the presentations and/or flyers into your native 
language and submitting them directly to me to post on the wiki. Localization 
for our material is integral to reaching the widest possible audience around 
the world. If you decide to translate a flyer, please fill out our form to 
request a thank you gift for your effort. [5]

If you decide to host a meetup, please email me to let me know the date, time, 
and location so I can put it on the GSoC calendar. Also, remember to take 
pictures at your meetup and write up a blog post for our blog using our 
provided template for formatting [6]. If you need promotional items for your 
attendees, please fill out our form [7] to request some; we're happy to send 
some along. We can provide up to about 25 pens, notebooks, or stickers and/or a 
few t-shirts. Please keep in mind, though, that shipping restrictions and 
timeline vary country-to-country; request items early to make sure they get 
there on time! If you have questions about hosting meetups, please see the 
section in our FAQ [8].

Please consider applying to participate as an organization again this year or 
maybe joining as a mentor for your favorite organization if they are selected 
this year.

We rely on you for your help for the success of this program, so thank you in 
advance for all the work you do!

[1] - 
http://google-opensource.blogspot.com/2013/02/flip-bits-not-burgers-google-summer-of.html
[2] - http://code.google.com/p/google-summer-of-code/wiki/ProgramPresentations
[3] - http://code.google.com/p/google-summer-of-code/wiki/GsocLogos
[4] - http://code.google.com/p/google-summer-of-code/wiki/GsocFlyers
[5] - http://goo.gl/gEHDO
[6] - http://goo.gl/wbZrt
[7] - http://goo.gl/0BsR8
[8] - http://goo.gl/2NGfp

Cheers,
Carol

--
You received this message because you are subscribed to the Google Groups 
"Google Summer of Code Mentors List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
google-summer-of-code-mentors-list+unsubscr...@googlegroups.com.
To post to this group, send email to 
google-summer-of-code-mentors-l...@googlegroups.com.
Visit this group at 
http://groups.google.com/group/google-summer-of-code-mentors-list?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [HEADS UP] Apache OODT now builds with Maven3 too

[Mattmann, Chris A (388J)]

Thanks guys ;)

On 2/7/13 11:47 AM, "Michael Joyce"  wrote:

>Woot! Nice. Now I can't give Cam more Stack Overflow points though =(.
>
>Mike
>
>
>Mike
>
>
>On Wed, Feb 6, 2013 at 3:54 PM, Cameron Goodale 
>wrote:
>
>> Chris,
>>
>> This is a huge win!  Now I can let Apple install Maven 3 with my next
>>Java
>> update and I don't have to revert.
>>
>> -Cam
>>
>>
>> On Sun, Feb 3, 2013 at 3:41 PM, Mattmann, Chris A (388J) <
>> chris.a.mattm...@jpl.nasa.gov> wrote:
>>
>> > Hey Guys,
>> >
>> > With r1441987, OODT 0.6 trunk now builds with Maven3. Yay!
>> >
>> > Trunk still builds with Maven2 as well.
>> >
>> > [INFO] CAS Product Server  SUCCESS
>> > [2.791s]
>> > [INFO] CAS Workflow Manager Monitor Web App .. SUCCESS
>> > [2.341s]
>> > [INFO] Catalog and Archive File Management Browser ... SUCCESS
>> > [1.561s]
>> > [INFO] Catalog and Archive Workflow Management GUI Editor  SUCCESS
>> > [9.351s]
>> > [INFO] Process Control System Operator Interface Webapp .. SUCCESS
>> > [4.225s]
>> > [INFO] OODT Process Control System JAX-RS service layer .. SUCCESS
>> > [4.131s]
>> > [INFO] Apache OODT ... SUCCESS
>> > [37.794s]
>> > [INFO]
>> > 
>>
>> > [INFO]
>> > 
>>
>> > [INFO] BUILD SUCCESSFUL
>> > [INFO]
>> > 
>>
>> > [INFO] Total time: 7 minutes 6 seconds
>> > [INFO] Finished at: Sun Feb 03 15:40:18 PST 2013
>> > [INFO] Final Memory: 100M/191M
>> > [INFO]
>> > 
>>
>> > [chipotle:~/tmp/oodt0.6] mattmann%
>> >
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>>

[HEADS UP] New Release Process on Wiki

[Mattmann, Chris A (388J)]

Hey Guys,

Heads up: per https://issues.apache.org/jira/browse/INFRA-5784

We have a new release process for Apache OODT mirroring per the use of
svnpubsub now:

http://www.apache.org/dev/release.html#upload-ci


Cheers,
Chris

[HEADS UP] Apache OODT now builds with Maven3 too

[Mattmann, Chris A (388J)]

Hey Guys,

With r1441987, OODT 0.6 trunk now builds with Maven3. Yay!

Trunk still builds with Maven2 as well.

[INFO] CAS Product Server  SUCCESS
[2.791s]
[INFO] CAS Workflow Manager Monitor Web App .. SUCCESS
[2.341s]
[INFO] Catalog and Archive File Management Browser ... SUCCESS
[1.561s]
[INFO] Catalog and Archive Workflow Management GUI Editor  SUCCESS
[9.351s]
[INFO] Process Control System Operator Interface Webapp .. SUCCESS
[4.225s]
[INFO] OODT Process Control System JAX-RS service layer .. SUCCESS
[4.131s]
[INFO] Apache OODT ... SUCCESS
[37.794s]
[INFO] 

[INFO] 

[INFO] BUILD SUCCESSFUL
[INFO] 

[INFO] Total time: 7 minutes 6 seconds
[INFO] Finished at: Sun Feb 03 15:40:18 PST 2013
[INFO] Final Memory: 100M/191M
[INFO] 

[chipotle:~/tmp/oodt0.6] mattmann%


Cheers,
Chris

Re: Patch For Env Vars

[Mattmann, Chris A (388J)]

Thanks Mike, thanks for creating the Review Board too!

I went ahead and commented back check it out.

Thanks again.

Cheers,
Chris

From: , "Michael D (388L)" 
mailto:michael.d.sta...@jpl.nasa.gov>>
Date: Tuesday, January 29, 2013 12:13 PM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Cc: "dev@oodt.apache.org<mailto:dev@oodt.apache.org>" 
mailto:dev@oodt.apache.org>>
Subject: Re: Patch For Env Vars

Cris et. all,

I have attached a patch for OODT-553 to fix the EnvUtilities to call 
System.getEnvironment instead of run env.  I have some suspicions that the 
latter (exec env) is causing problems on my project, and this patch updates it 
to use the native java mechanisms which work as of java 1.5, or 1.6.  This 
should be more stable and portable.

The test case needs to run on UNIX to get access to USER and HOME environment 
variables to work, so I make that check first.

See below for previous discussion.

-Michael Starch (starch)


On Jan 17, 2013, at 9:37 PM, Mattmann, Chris A (388J) wrote:

Hi Mike,

From: , "Michael D (388L)" 
mailto:michael.d.sta...@jpl.nasa.gov>>
Date: Thursday, January 17, 2013 12:05 PM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: Patch For Env Vars

Chris,

Here is the patch for the Env Vars (svn diff from trunk level).  Let me know if 
it is accepted by OODT, and if I need to send it somewhere else (Not sure where 
to send it for review).

Thank you! You should:

  1.  File a JIRA issue in Apache OODT — 
https://issues.apache.org/jira/browse/OODT
  2.  Attach this patch to the issue
  3.  Request someone on the dev@oodt list (by replying to the thread) to 
review the patch and commit it

I added some unit tests, the fix to use System.getenv and noticed that the 
EnvUtilities.java is in win/dos format not unix.  I did not change the format, 
just a heads up.

Awesome if you have time file a patch for fixing the line format too (and a 
separate issue as you pointed out).

The tests make the assumption that that USER and HOME variables are defined.  
If this is a bad assumption, a new avenue of testing would need to be 
investigated.

How about if those env vars aren't defined based on OS, then don't run them 
(e.g., you could do a test for OS and if it's e.g., windoze, then don't run 
them or something?)

BTW this and the above can and should be discussed on list. This is for 2 
benefits:

  1.  Others can benefit from our convo in the Apache community
  2.  You earn the merit you deserve by having others see what great and cool 
things you are doing via your discussion

Thanks Mike and see you on list! Appreciate your contribution.

Cheers,
Chris


-Michael

Re: [jira] [Updated] (OODT-553) Update org.apache.oodt.commons.exec.EnvUtilities to Use System.getEnvironment

[Mattmann, Chris A (388J)]

Thanks Mike feel free to use http://reviews.apache.org and the Review Board 
instance too if you like 

Sent from my iPhone

On Jan 29, 2013, at 3:09 PM, "Michael Starch (JIRA)"  wrote:

> 
> [ 
> https://issues.apache.org/jira/browse/OODT-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>  ]
> 
> Michael Starch updated OODT-553:
> 
> 
>Attachment: OODT-553.patch.2013-01-29.txt
> 
> This is a proposed patch.  
> 
> It uses System.getEnvironment() (instead of exec "env").  The test cases 
> (additional) will only run on UNIX systems because they require USER and HOME 
> environment variables to be set.
> 
> Could another user review it?  I am new.
> 
> Enjoy!
> 
>> Update org.apache.oodt.commons.exec.EnvUtilities to Use System.getEnvironment
>> -
>> 
>>Key: OODT-553
>>URL: https://issues.apache.org/jira/browse/OODT-553
>>Project: OODT
>> Issue Type: Improvement
>> Components: commons
>>   Reporter: Michael Starch
>>   Priority: Minor
>>Fix For: 0.6
>> 
>>Attachments: OODT-553.patch.2013-01-29.txt
>> 
>> 
>> The environment variables utilities use a command line invocation of 'env' 
>> to get the environment variables due to the fact that java 1.4 did not have 
>> a System.getEnvironment call.
>> This is no longer the best solution (and is suspected of causing some 
>> errors).
> 
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Requested Read-only Git Mirror for Apache OODT

[Mattmann, Chris A (388J)]

Dear all,

This is now complete:

https://github.com/apache/oodt


Apache OODT is available on Github. If you make a pull request over there
too I *think* that we'll get an email here on the dev@oodt list to
integrate it. Yay!

Cheers,
Chris

On 1/12/13 10:04 AM, "Mattmann, Chris A (388J)"
 wrote:

>Hey Guys,
>
>FYI https://issues.apache.org/jira/browse/INFRA-5759
>
>I requested a GitHub read-only Git mirror for Apache OODT. Yay!
>
>I'm not proposing we move to Git for our CM system :) I just wanted folks
>on Github to be able to send us pull requests with improvements if they
>have any.
>
>Cheers,
>Chris
>

Filed issue to move to svnpubsub based releasing

[Mattmann, Chris A (388J)]

Hey Guys,

Per: http://www.apache.org/dev/release-publishing.html#distribution_dist

We need to migrate to svnpubsub based release mirroring.

I've filed an issue here for it:

https://issues.apache.org/jira/browse/INFRA-5784


Cheers,
Chris

Re: [jira] [Commented] (OODT-551) DataSourceCatalog implementation does not preserve order of metadata values

[Mattmann, Chris A (388J)]

Bruce,

Great point -- related to that, at some level ordering does have some
level of semantics.

The proposal is to enable those semantics through a flag, and then let
unstructured types still get unstructured behavior by default for back
compat.

Cheers,
Chris

On 1/17/13 6:55 AM, "Bruce Barkstrom"  wrote:

>Sounds like that could get interesting if the metadata had to deal
>with values that are part of a time series - such as versions of
>files created on different dates or identifying files that are similar
>except for the time interval of the data they contain.
>
>Bruce B.
>
>On Thu, Jan 17, 2013 at 9:52 AM, Luca Cinquini (JIRA)
>wrote:
>
>>
>> [
>> 
>>https://issues.apache.org/jira/browse/OODT-551?page=com.atlassian.jira.pl
>>ugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556230#com
>>ment-13556230]
>>
>> Luca Cinquini commented on OODT-551:
>> 
>>
>> Having a flag sounds good to me. I can have that done by the end of the
>> week, unless you have it already.
>> And I will fix the unit test, sorry about that.
>>
>> > DataSourceCatalog implementation does not preserve order of metadata
>> values
>> >
>> 
>>-
>>--
>> >
>> > Key: OODT-551
>> > URL: https://issues.apache.org/jira/browse/OODT-551
>> > Project: OODT
>> >  Issue Type: Bug
>> >  Components: file manager
>> >Affects Versions: 0.5
>> >Reporter: Luca Cinquini
>> >Assignee: Luca Cinquini
>> > Fix For: 0.6
>> >
>> > Attachments: OODT-551.luca.patch.txt
>> >
>> >
>> > The table that stores the metadata (key, value) pairs for the File
>> Manager database-based implementation has no primary key - as a
>> consequence, values are not guaranteed to be returned in any order,
>>which
>> is a problem for applications that rely on the order of the values (for
>> example, among different metadata keys).
>>
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA
>> administrators
>> For more information on JIRA, see:
>>http://www.atlassian.com/software/jira
>>

Re: Environment Variables in OODT

[Mattmann, Chris A (388J)]

Thanks Mike!

Cheers,
Chris

On 1/17/13 8:24 AM, "Starch, Michael D (388L)"
 wrote:

>Sounds good.   I tested the code and it does return environment variables
>as expected.  Looks like this is something I can improve for our next
>software delivery on PEATE (v9.0.1).  If so, a patch for apache should be
>easy.
>
>-Michael
>
>
>On Jan 15, 2013, at 11:05 PM, Mattmann, Chris A (388J) wrote:
>
>> +1, BFost :)
>> 
>> Mike?
>> 
>> Cheers,
>> Chris
>> 
>> On 1/15/13 10:24 PM, "Brian Foster"  wrote:
>> 
>>> 
>>> hey michael,
>>> 
>>> System.getenv() didn't exist in jdk 1.4 when the code was initially
>>> written and System.getenv(String) was deprecated in jdk 1.4... i
>>>believe
>>> it was rewritten and undeprecated in jdk 1.5 and System.getenv() was
>>> added... so ya it probably should be using System.getenv()... a patch
>>> with several unit-tests would be welcomed as far as i'm concerned...
>>> chris?
>>> 
>>> -brian
>>> 
>>> On Jan 15, 2013, at 10:58 AM, Starch, Michael D (388L) wrote:
>>> 
>>>> Hi All,
>>>> 
>>>> Why does the OODT Commons code for reading environment variables run
>>>> the command line utility "env" instead of calling
>>>> System.getEnvironment()?
>>>> 
>>>> 
>>>> 
>>>>http://svn.apache.org/repos/asf/oodt/trunk/commons/src/main/java/org/ap
>>>>ac
>>>> he/oodt/commons/exec/EnvUtilities.java
>>>> 
>>>> We are running into transient problems in our version where
>>>>environment
>>>> variables are sometimes being set to null.  I am wondering if this is
>>>>a
>>>> problem with the added complexity of running env over and over again
>>>> getting env variables, instead of using the built in java mechanisms
>>>>for
>>>> getting this info.
>>>> 
>>>> -Michael
>>> 
>> 
>

Re: Trouble with cas-curator

[Mattmann, Chris A (388J)]

Hi Marlon,

Sorry it took a sec to get back to you on this. Here's some help inline
below:

On 1/16/13 11:40 AM, "Marlon Pierce"  wrote:

>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA1
>
>Hi OODT devs--
>
>I'm evaluating OODT's CAS Curation and following the steps at
>http://oodt.apache.org/components/maven/curator/user/basic.html, but
>I'm having problems.  I've attached a screenshot.
>
>* I checked out the code from SVN, grabbing the 0.5 release tag:  svn
>checkout https://svn.apache.org/repos/asf/oodt/tags/0.5/curator
>cas-curator
>
>* I've downloaded Tika 1.2 and modified the sample python script to
>use my local paths and version of Tika. Note you have a typo in line
>24 after \n.  This all works fine.


Thanks for finding this I went ahead and filed a JIRA issue to fix the
docs:

https://issues.apache.org/jira/browse/OODT-552


I'll fix it shortly.

>
>* I've created the directory structures for staging and extractors and
>populated with appropriate files.
>156-56-179-234:cas-curator marpierc$ ls -tlr extractors/mp3extractor/
>total 51144
>- -rw-r--r--@ 1 marpierc  marpierc  26173420 Jan 16 10:27 tika-app-1.2.jar
>- -rwxr-xr-x  1 marpierc  marpierc   888 Jan 16 10:34
>mp3PythonExtractor.py
>- -rw-r--r--  1 marpierc  marpierc   348 Jan 16 11:13
>mp3PythonExtractor.config
>- -rw-r--r--  1 marpierc  marpierc   188 Jan 16 14:20
>config.properties

Can you please cat config.properties and paste to this list?

>
>
>* As shown in the screenshot, I still get "No Metadata Detected"

It seems to have been filtered? Can you use min.us to share to list or
something like that?


>
>* I see the following errors in catalina.out, which don't seem to be
>relevant (looking for value for
>org.apache.oodt.cas.curator.dataDefinition.uploadPath):
>
>
>Caused by: java.lang.NullPointerException
>   at
>org.apache.oodt.cas.metadata.util.PathUtils.replaceEnvVariables(PathUtils.
>java:66)
>   at
>org.apache.oodt.cas.metadata.util.PathUtils.replaceEnvVariables(PathUtils.
>java:59)
>   at
>org.apache.oodt.cas.metadata.util.PathUtils.replaceEnvVariables(PathUtils.
>java:55)
>   at
>org.apache.oodt.cas.curation.service.CurationServiceConfig.evaluateParamet
>er(CurationServiceConfig.java:204)
>   at
>org.apache.oodt.cas.curation.service.CurationServiceConfig.getPolicyUpload
>Path(CurationServiceConfig.java:176)
>   at
>org.apache.oodt.cas.curation.service.PolicyResource.getPolicies(PolicyReso
>urce.java:176)
>   at
>org.apache.oodt.cas.curation.service.PolicyResource.browseCatalog(PolicyRe
>source.java:115)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Ah yes, those are relevant (I think ;) ).

That path should be set to [FILEMGR_HOME]/policy. Can you refresh that in
the config and try again?

>
>
>Any help is appreciated--

Let me know how it goes. Will be around tonight and tomorrow and happy to
help debug.

Cheers,
Chris

>
>
>Marlon
>
>
>-BEGIN PGP SIGNATURE-
>Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
>Comment: GPGTools - http://gpgtools.org
>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
>iQEcBAEBAgAGBQJQ9wJBAAoJEOEgD2XReDo5pKsIAI0pzzaoRYevbuB25zxhK9vz
>kS51CYOjetkoHM1tkhs+XAf07wYNoPHXhDIvt+o2PfWP9QqjIEYF/D8okubY6NMQ
>Df9qhQL74+X5EGvHehLwJXDtW2TYzzuOyvJG5kpFUazfSHitClAJrnIRmWaO2Y11
>5VtYmNx1cZnXi2rw92AI6M9cmDeCHxtwQg5oQ9IJBpmBrSXBUyPcelUeNphYw/NO
>/I863rufsVaW9ZCVoaYaq8gEUwyVwObEm6HK0bK90ptHQ2MylJqwQ68TcduYw0tY
>vu3ndDv/GeBIVLJzQ/lGA3nDC6EZw3vMkqyZnWTYGC+R9q1kIePYSvBtog8YbV8=
>=8US7
>-END PGP SIGNATURE-

FW: Trouble with cas-curator

[Mattmann, Chris A (388J)]

Request for help from Marlon, CC'ing him.


On 1/16/13 11:40 AM, "Marlon Pierce"  wrote:

>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA1
>
>Hi OODT devs--
>
>I'm evaluating OODT's CAS Curation and following the steps at
>http://oodt.apache.org/components/maven/curator/user/basic.html, but
>I'm having problems.  I've attached a screenshot.
>
>* I checked out the code from SVN, grabbing the 0.5 release tag:  svn
>checkout https://svn.apache.org/repos/asf/oodt/tags/0.5/curator
>cas-curator
>
>* I've downloaded Tika 1.2 and modified the sample python script to
>use my local paths and version of Tika. Note you have a typo in line
>24 after \n.  This all works fine.
>
>* I've created the directory structures for staging and extractors and
>populated with appropriate files.
>156-56-179-234:cas-curator marpierc$ ls -tlr extractors/mp3extractor/
>total 51144
>- -rw-r--r--@ 1 marpierc  marpierc  26173420 Jan 16 10:27 tika-app-1.2.jar
>- -rwxr-xr-x  1 marpierc  marpierc   888 Jan 16 10:34
>mp3PythonExtractor.py
>- -rw-r--r--  1 marpierc  marpierc   348 Jan 16 11:13
>mp3PythonExtractor.config
>- -rw-r--r--  1 marpierc  marpierc   188 Jan 16 14:20
>config.properties
>
>
>* As shown in the screenshot, I still get "No Metadata Detected"
>
>* I see the following errors in catalina.out, which don't seem to be
>relevant (looking for value for
>org.apache.oodt.cas.curator.dataDefinition.uploadPath):
>
>
>Caused by: java.lang.NullPointerException
>   at
>org.apache.oodt.cas.metadata.util.PathUtils.replaceEnvVariables(PathUtils.
>java:66)
>   at
>org.apache.oodt.cas.metadata.util.PathUtils.replaceEnvVariables(PathUtils.
>java:59)
>   at
>org.apache.oodt.cas.metadata.util.PathUtils.replaceEnvVariables(PathUtils.
>java:55)
>   at
>org.apache.oodt.cas.curation.service.CurationServiceConfig.evaluateParamet
>er(CurationServiceConfig.java:204)
>   at
>org.apache.oodt.cas.curation.service.CurationServiceConfig.getPolicyUpload
>Path(CurationServiceConfig.java:176)
>   at
>org.apache.oodt.cas.curation.service.PolicyResource.getPolicies(PolicyReso
>urce.java:176)
>   at
>org.apache.oodt.cas.curation.service.PolicyResource.browseCatalog(PolicyRe
>source.java:115)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>
>Any help is appreciated--
>
>
>Marlon
>
>
>-BEGIN PGP SIGNATURE-
>Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
>Comment: GPGTools - http://gpgtools.org
>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
>iQEcBAEBAgAGBQJQ9wJBAAoJEOEgD2XReDo5pKsIAI0pzzaoRYevbuB25zxhK9vz
>kS51CYOjetkoHM1tkhs+XAf07wYNoPHXhDIvt+o2PfWP9QqjIEYF/D8okubY6NMQ
>Df9qhQL74+X5EGvHehLwJXDtW2TYzzuOyvJG5kpFUazfSHitClAJrnIRmWaO2Y11
>5VtYmNx1cZnXi2rw92AI6M9cmDeCHxtwQg5oQ9IJBpmBrSXBUyPcelUeNphYw/NO
>/I863rufsVaW9ZCVoaYaq8gEUwyVwObEm6HK0bK90ptHQ2MylJqwQ68TcduYw0tY
>vu3ndDv/GeBIVLJzQ/lGA3nDC6EZw3vMkqyZnWTYGC+R9q1kIePYSvBtog8YbV8=
>=8US7
>-END PGP SIGNATURE-

Related research funding page for OODT on wiki

[Mattmann, Chris A (388J)]

Hey Guys,

I added a page on the wiki for related research funding for Apache OODT.
If you've been funded by the NIH, NASA, DARPA, NSF, etc., to work on
something related to OODT, please feel free to add it to the wiki and to
use my template!

https://cwiki.apache.org/confluence/display/OODT/Related+research+funding+f
or+OODT


Thanks!

Cheers,
Chris

Re: Environment Variables in OODT

[Mattmann, Chris A (388J)]

+1, BFost :)

Mike?

Cheers,
Chris

On 1/15/13 10:24 PM, "Brian Foster"  wrote:

>
>hey michael,
>
>System.getenv() didn't exist in jdk 1.4 when the code was initially
>written and System.getenv(String) was deprecated in jdk 1.4... i believe
>it was rewritten and undeprecated in jdk 1.5 and System.getenv() was
>added... so ya it probably should be using System.getenv()... a patch
>with several unit-tests would be welcomed as far as i'm concerned...
>chris?
>
>-brian
>
>On Jan 15, 2013, at 10:58 AM, Starch, Michael D (388L) wrote:
>
>> Hi All,
>> 
>> Why does the OODT Commons code for reading environment variables run
>>the command line utility "env" instead of calling
>>System.getEnvironment()?
>> 
>> 
>>http://svn.apache.org/repos/asf/oodt/trunk/commons/src/main/java/org/apac
>>he/oodt/commons/exec/EnvUtilities.java
>> 
>> We are running into transient problems in our version where environment
>>variables are sometimes being set to null.  I am wondering if this is a
>>problem with the added complexity of running env over and over again
>>getting env variables, instead of using the built in java mechanisms for
>>getting this info.
>> 
>> -Michael
>

Re: Environment Variables in OODT

[Mattmann, Chris A (388J)]

Hi Mike,

Basically this is a legacy decision as when the system was written we were
in the midst of JDK4, which didn't have Environment variable support.
Nowadays, we could probably update this. Would you like to file a JIRA
issue and provide a patch?

Cheers,
Chris

On 1/15/13 10:58 AM, "Starch, Michael D (388L)"
 wrote:

>Hi All,
>
>Why does the OODT Commons code for reading environment variables run the
>command line utility "env" instead of calling System.getEnvironment()?
>
>http://svn.apache.org/repos/asf/oodt/trunk/commons/src/main/java/org/apach
>e/oodt/commons/exec/EnvUtilities.java
>
>We are running into transient problems in our version where environment
>variables are sometimes being set to null.  I am wondering if this is a
>problem with the added complexity of running env over and over again
>getting env variables, instead of using the built in java mechanisms for
>getting this info.
>
>-Michael

Might want to stay away from JDK7

[Mattmann, Chris A (388J)]

Hey Guys,

Bruce Barkstrom and several others off list have pointed out the following
articles:

Garber, L., 2012: Have Java's Security Issues Gotten out of Hand?,
Computer, Dec. 2012, pp. 18-21.


http://www.usatoday.com/story/tech/2013/01/11/homeland-security-disable-jav
a-security-vulnerability/1828011/


My suggestion would be to try and stick with earlier versions as it seems
not to affect them though it may be a fundamental flaw in the actual
language design, not sure yet.

Just felt that it was worth noting.

Thanks for listening.

Cheers,
Chris

Requested Read-only Git Mirror for Apache OODT

[Mattmann, Chris A (388J)]

Hey Guys,

FYI https://issues.apache.org/jira/browse/INFRA-5759

I requested a GitHub read-only Git mirror for Apache OODT. Yay!

I'm not proposing we move to Git for our CM system :) I just wanted folks
on Github to be able to send us pull requests with improvements if they
have any.

Cheers,
Chris

[NOTICE] Added a ton of new material on the Apache OODT Web Page

[Mattmann, Chris A (388J)]

Guys FYI I added a ton of new material on the Home Page, as you can
probably tell. Including:

1. Links to Jenkins in the Bottom Footer
2. Links to PEAR channel in Bottom Footer
3. Links to PyPI in Bottom Footer
4. Links to Gplus sites
5. News and announcements for 0.5 release and for ApacheCon NA 2011 and
2013
6. Link to Jenkins on testing page
7. Various link fixes and typo fixes throughout

Yay!

Cheers,
Chris

[NOTICE] Created Google Plus Apache OODT community

[Mattmann, Chris A (388J)]

Hey Guys,

I created a Google Plus Apache OODT community page here:

http://s.apache.org/8vM

Feel free to join up!

Cheers,
Chris

P.S. Note the Gplus community is NOT a replacement for the mailing lists,
etc. All project discussions *and more importantly* decisions happen here.
I simply created the community at Gplus hoping to cast the wider net and
get more people interested in Apache OODT and to bring them back here. If
you see conversations happening over there that should be happening here
on the mailing list, please instruct people to come here.

[RESULT] [VOTE] Apache OODT 0.5 RC #1

[Mattmann, Chris A (388J)]

Merry Christmas Apache OODT users and devs!

The VOTE to release 0.5 has passed with the following tallies:

+1 PMC

Chris Mattmann
Sean Kelly
Luca Cinquini
Mike Joyce

I'll push the release out to the mirrors and send an ANNOUNCE very soon.
I'll also update the website shortly.

Merry Christmas!

Cheers,
Chris

[NOTICE] Added XMLPS Quick Start Guide to the Wiki

[Mattmann, Chris A (388J)]

ŠMerry Christmas everyone :)

It's been a long time coming, and Mike Joyce's recent run-in with some of
its lack of documentation gave me the impetus to finally write this guide:

https://cwiki.apache.org/confluence/display/OODT/XMLPS+Quick+Start+Guide


I linked it from the front page of the wiki.

Thanks all!

Cheers,
Chris

Re: svn commit: r1424428 - /oodt/trunk/CHANGES.txt

[Mattmann, Chris A (388J)]

Hey Luca yep I took care of it, Happy Holidays!

On 12/22/12 12:36 PM, "Cinquini, Luca (3880)" 
wrote:

>Hi Chris,
>   you have done this already, correct ?
>Happy holidays,
>thanks, Luca
>
>On Dec 20, 2012, at 12:58 PM, Mattmann, Chris A (388J) wrote:
>
>> Hey Luca,
>> 
>> Can you update these changes to be in 0.6?
>> 
>> Cheers,
>> Chris
>> 
>> On 12/20/12 4:13 AM, "l...@apache.org"  wrote:
>> 
>>> Author: luca
>>> Date: Thu Dec 20 12:13:21 2012
>>> New Revision: 1424428
>>> 
>>> URL: http://svn.apache.org/viewvc?rev=1424428&view=rev
>>> Log:
>>> Updated CHANGES.txt
>>> 
>>> Modified:
>>>   oodt/trunk/CHANGES.txt
>>> 
>>> Modified: oodt/trunk/CHANGES.txt
>>> URL: 
>>> 
>>>http://svn.apache.org/viewvc/oodt/trunk/CHANGES.txt?rev=1424428&r1=14244
>>>27
>>> &r2=1424428&view=diff
>>> 
>>>
>>>==
>>> 
>>> --- oodt/trunk/CHANGES.txt (original)
>>> +++ oodt/trunk/CHANGES.txt Thu Dec 20 12:13:21 2012
>>> @@ -5,6 +5,11 @@ Release 0.5
>>> 
>>> Release Date: December 18, 2012
>>> 
>>> +* OODT-547 Switch newest Curator upudate metadata method to use
>>>XML-RPC
>>> FileManager client.
>>> +   Switch oler Curator update metadata method to use shared
>>> instance of Catalog Factory
>>> +   to minimize use of resources.
>>> +   (luca)
>>> +
>>> * OODT-541 Implement a LenientDataSourceCatalog that allows for
>>> dynamically named
>>>  metadata fields, effectively bypassing the validation layer.
>>>  (luca)
>>> 
>>> 
>> 
>

Re: [VOTE] Apache OODT 0.5 RC #1

[Mattmann, Chris A (388J)]

WOOT thanks dude!

On 12/20/12 8:43 PM, "Sean Kelly"  wrote:

>
>> Will do thanks dude. Are you +1 on the release?
>
>See: http://s.apache.org/WQC
>
>--k

Re: [VOTE] Apache OODT 0.5 RC #1

[Mattmann, Chris A (388J)]

Will do thanks dude. Are you +1 on the release?

On 12/20/12 10:13 AM, "Sean Kelly"  wrote:

>OK, zip's fine. We should update the release procedure wiki to match.
>
>Thanks!
>
>--k
>
>On 2012.Dec.20, at 12.11p, "Mattmann, Chris A (388J)"
> wrote:
>
>> Hey Sean,
>> 
>> Yeah I used the Maven release plugin to generate that source zip.
>> 
>> In Apache Tika, we decided since 0.4 to provide Tika as a zip based on
>> Jukka's rationale here [1].
>> 
>> What do you think? I can generate the .tar.gz but it seems like the .zip
>> is a sufficient superset of the bits that doesn't corrupt long filenames
>> on 
>> other platforms.
>> 
>> Thoughts? That OK?
>> 
>> Cheers,
>> Chris
>> 
>> [1] http://s.apache.org/SQ9
>> 
>> On 12/20/12 7:48 AM, "Sean Kelly"  wrote:
>> 
>>>> I have posted a 1st release candidate for the Apache OODT 0.5 release.
>>>> The
>>>> source code is at:
>>>> 
>>>> http://people.apache.org/~mattmann/apache-oodt-0.5/rc1/
>>> 
>>> Only .zip? No .tar.gz?
>>> 
>>> --k
>>> 
>> 
>

Re: svn commit: r1424428 - /oodt/trunk/CHANGES.txt

[Mattmann, Chris A (388J)]

Hey Luca,

Can you update these changes to be in 0.6?

Cheers,
Chris

On 12/20/12 4:13 AM, "l...@apache.org"  wrote:

>Author: luca
>Date: Thu Dec 20 12:13:21 2012
>New Revision: 1424428
>
>URL: http://svn.apache.org/viewvc?rev=1424428&view=rev
>Log:
>Updated CHANGES.txt
>
>Modified:
>oodt/trunk/CHANGES.txt
>
>Modified: oodt/trunk/CHANGES.txt
>URL: 
>http://svn.apache.org/viewvc/oodt/trunk/CHANGES.txt?rev=1424428&r1=1424427
>&r2=1424428&view=diff
>==
>
>--- oodt/trunk/CHANGES.txt (original)
>+++ oodt/trunk/CHANGES.txt Thu Dec 20 12:13:21 2012
>@@ -5,6 +5,11 @@ Release 0.5
> 
> Release Date: December 18, 2012
> 
>+* OODT-547 Switch newest Curator upudate metadata method to use XML-RPC
>FileManager client.
>+   Switch oler Curator update metadata method to use shared
>instance of Catalog Factory
>+   to minimize use of resources.
>+   (luca)
>+
> * OODT-541 Implement a LenientDataSourceCatalog that allows for
>dynamically named
>   metadata fields, effectively bypassing the validation layer.
>   (luca)
>
>

Re: [VOTE] Apache OODT 0.5 RC #1

[Mattmann, Chris A (388J)]

Hey Sean,

Yeah I used the Maven release plugin to generate that source zip.

In Apache Tika, we decided since 0.4 to provide Tika as a zip based on
Jukka's rationale here [1].

What do you think? I can generate the .tar.gz but it seems like the .zip
is a sufficient superset of the bits that doesn't corrupt long filenames
on 
other platforms.

Thoughts? That OK?

Cheers,
Chris

[1] http://s.apache.org/SQ9

On 12/20/12 7:48 AM, "Sean Kelly"  wrote:

>> I have posted a 1st release candidate for the Apache OODT 0.5 release.
>>The
>> source code is at:
>> 
>> http://people.apache.org/~mattmann/apache-oodt-0.5/rc1/
>
>Only .zip? No .tar.gz?
>
>--k
>

[VOTE] Apache OODT 0.5 RC #1

[Mattmann, Chris A (388J)]

Hi Folks,

I have posted a 1st release candidate for the Apache OODT 0.5 release. The
source code is at:

http://people.apache.org/~mattmann/apache-oodt-0.5/rc1/

For more detailed information, see the included CHANGES.txt file for
details on
release contents and latest changes. The release was made using the OODT
release process, documented on the Wiki here:

https://cwiki.apache.org/confluence/display/OODT/Release+Process

The release was made from the OODT 0.5 tag (r1423763) at:

http://svn.apache.org/repos/asf/oodt/tags/0.5/

A staged Maven repository is available at:

https://repository.apache.org/content/repositories/orgapacheoodt-050/


Please vote on releasing these packages as Apache OODT 0.5. The vote is
open for the next 72 hours.

Only votes from OODT PMC are binding, but folks are welcome to check the
release candidate and voice their approval or disapproval. The vote passes
if at least three binding +1 votes are cast.

[ ] +1 Release the packages as Apache OODT 0.5

[ ] -1 Do not release the packages because...

Thanks!

Chris Mattmann
(on behalf of the Apache OODT PMC)

P.S. Here is my +1.

Re: passing objects between tasks / workflows

[Mattmann, Chris A (388J)]

Hey Lindsey,

Thanks for your email! Some comments below:

On 12/14/12 7:26 AM, "Lindsey Davis"  wrote:

>Hello,
>
>I  work on the NRAO ALMA pipeline software. I have recently begun
>developing
>some prototype OODT workflows for use in the production pipeline.

Great to hear!

>
>I would like to be able to pass an object between workflows and tasks. The
>object is modest in size, serializable in the Java sense, and can be
>encoded
>to and decoded from a string. XML is not a useful option here.

There are a few different ways to do this, listed below in lowest->highest
level of complexity:

1. All Tasks within the same Workflow have the capability to pass
information to one another via a Shared workflow context, provided to the
Task during execution by this interface:

http://oodt.apache.org/components/maven/xref/org/apache/oodt/cas/workflow/s
tructs/WorkflowTaskInstance.html#44


That provided Metadata is readable/writeable by all of the WorkflowTasks
in a particular WorkflowInstance. So, concretely, you could feasiable set
a object that you want the downstream tasks to have access to by doing
(within your WorkflowTaskInstance):

/* first workflow task */
String serializedObj = yourSerializationFunc(obj);
metadata.replaceMetadata(/* key name */ "objectKey", /* value */
serializedObj);

Then:

/* downstream workflow task */

String serializedObj = metadata.getMetadata(/* key name */ "objectKey");
// unserialize it, do something with it, etc.

2. If your serialized object makes more sense as a file, then you could
simply ingest that object separately (as part of an upstream workflow
task, or via some external ingestion process) into the File Manager:

http://oodt.apache.org/components/maven/filemgr/user/basic.html


Once that file is ingested, it can be referenced automatically using
CAS-PGE:

https://cwiki.apache.org/OODT/cas-pge-learn-by-example.html


There are other ways to do this, but try starting out with #1 or #2.


>
>Does OODT provide any builtin support for this? If not are there any OODT
>imposed limits on the size of strings that can be passed around, or
>any other related issues I should be aware of.

Yep see above. In terms of imposed limits on the size of strings to pass
around that is dictated by the underlying JVMs that the OODT daemons are
run inside of. If you are running Workflow Manager (without Resource
Manager) then you are size limited by the total # of workflow instances
running * workflow instance_i's memory footprint which has to be less than
the values passed to -Xms and -Xmx args sent to the JVM. If you are
running in Resource Manager mode, then you are limited by not only the
above values for WM, but also by the same JVM args sent to RM, and
ultimately the JVM args passed to each BatchStub which is executing the
underlying job on a compute node.

Hope that helps!

If you have more questions, keep em' coming!

Cheers,
Chris

Re: OODT 0.3 branch

[Mattmann, Chris A (388J)]

Hey Chintu,

Thanks much. One thing you could try to use to speed up as well would be
to:

1. Subclass, or roll your own LocalDataTransfer class -- call it
MoveBasedLocalDataTransfer
2. Replace the class in that class from using FileUtils.copyFile or
FileUtils.moveFile to calls to ExecHelper.execute("cp ...") and ("mv ...")
3. In your calls to the crawler, pass --dataTransferFactory for your new
MoveBased...one

See if that improves it at all. If you want, file a JIRA issue too and I
could try and wire up such a transferer for you.

Thanks!

Cheers,
Chris

On 12/14/12 5:23 AM, "Mistry, Chintu (GSFC-586.0)[COLUMBUS TECHNOLOGIES
AND SERVICES INC]"  wrote:

>Thank you for example.
>
>In our case, the file size can vary from 10K to 200MB. About 12000 files
>make up of 262GB data.
>
>We are using IBM GPFS for our storage, which is suppose to be faster for
>this kind of activity. The parallel performance that we are seeing in our
>test case is far from isolated filesystem test (with simple copy and move
>without OODT).
>
>So far the best combination I could find is to use "move" and keep almost
>1:1 ratio of FM and CR. Will still dig more into it.
>
>Thanks
>--
>Chintu Mistry
>NASA Goddard Space Flight Center
>Bldg L40B, Room S776
>Office: 240 684 0477
>Mobile: 770 310 1047
>
>From: Cameron Goodale mailto:good...@apache.org>>
>Date: Friday, December 14, 2012 12:03 AM
>To: "dev@oodt.apache.org<mailto:dev@oodt.apache.org>"
>mailto:dev@oodt.apache.org>>
>Cc: "Mistry, Chintu (GSFC-586.0)[COLUMBUS TECHNOLOGIES AND SERVICES INC]"
>mailto:chintu.mis...@nasa.gov>>
>Subject: Re: OODT 0.3 branch
>
>Chintu,
>
>I see that your test data volume is 262GB, but I am curious about the
>make up of the data.  On average what is your file size and how many
>files?
>
>The reason I ask is because the process of extraction and ingestion can
>vary wildly.  On the LMMP project I was ingesting 12GB DEMs over NFS and
>it was a slow process.  It was basically serial with 1CR+1FM, but we
>didn't have a requirement to push large volumes of data.
>
>On our recent Snow Data System I am processing 160 workflow jobs in
>parallel and OODT could handle the load, it turned out the filesystem was
>our major bottleneck.  We used a SAN initially when doing development,
>but when we increased the number of jobs in parallel the I/O became so
>bad we moved to GlusterFS. GlusterFS had speed improvements over the SAN,
>but we had to be careful about heavy writing, moving, deleting since the
>clustering would try to replicate the data.  Turns out Gluster is great
>for heavy writting OR heavy reading, but not both at the same time.
>Finally we are using NAS and it works great.
>
>My point is the file system plays a major role in performance when
>ingesting data.  The ultimate speed test would be if you could actually
>write the data into the final archive directory and basically do an
>ingestion in place (skip data transfer entirely), but I know that is
>rarely possible.
>
>This is an interesting challenge to see what configuration will yield the
>best through put/performance.  I look forward to hearing more about your
>progress on this.
>
>
>Best Regards,
>
>
>
>Cameron
>
>
>On Wed, Dec 12, 2012 at 7:28 PM, Mattmann, Chris A (388J)
>mailto:chris.a.mattm...@jpl.nasa.gov>>
>wrote:
>Hi Chintu,
>
>From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC]
>(GSFC-586.0)" 
>mailto:chintu.mis...@nasa.gov><mailto:chintu.mistr
>y...@nasa.gov<mailto:chintu.mis...@nasa.gov>>>
>Date: Wednesday, December 12, 2012 12:02 PM
>To: jpluser 
>mailto:chris.a.mattm...@jpl.nasa.gov>o:chris.a.mattm...@jpl.nasa.gov<mailto:chris.a.mattm...@jpl.nasa.gov>>>,
>"dev@oodt.apache.org<mailto:dev@oodt.apache.org><mailto:d...@oodt.apache.or
>g<mailto:dev@oodt.apache.org>>"
>mailto:dev@oodt.apache.org><mailto:d...@oodt.apache.or
>g<mailto:dev@oodt.apache.org>>>
>Subject: Re: OODT 0.3 branch
>
>If you are saying that FM can handle multiple connections at one time,
>
>Yep I'm saying that it can.
>
>then multiple crawlers pointing to same FM should increase performance
>significantly.
>
>Well that really depends to be honest. It sounds like you guys are
>hitting an IO bottleneck potentially in data transfer? What file sizes
>are you transferring? If you are IO bound on the data transfer part, the
>product isn't fully ingested until:
>
>
>  1.  it's entry is added to the catalog
>  2.  The data transfer finishes
>
>Are you checking the FM for status along the way? Also realize that the
>FM will

Re: OODT 0.3 branch

[Mattmann, Chris A (388J)]

Thanks Cam, for the use cases, and insight.

Cheers,
Chris

On 12/13/12 9:03 PM, "Cameron Goodale"  wrote:

>Chintu,
>
>I see that your test data volume is 262GB, but I am curious about the make
>up of the data.  On average what is your file size and how many files?
>
>The reason I ask is because the process of extraction and ingestion can
>vary wildly.  On the LMMP project I was ingesting 12GB DEMs over NFS and
>it
>was a slow process.  It was basically serial with 1CR+1FM, but we didn't
>have a requirement to push large volumes of data.
>
>On our recent Snow Data System I am processing 160 workflow jobs in
>parallel and OODT could handle the load, it turned out the filesystem was
>our major bottleneck.  We used a SAN initially when doing development, but
>when we increased the number of jobs in parallel the I/O became so bad we
>moved to GlusterFS. GlusterFS had speed improvements over the SAN, but we
>had to be careful about heavy writing, moving, deleting since the
>clustering would try to replicate the data.  Turns out Gluster is great
>for
>heavy writting OR heavy reading, but not both at the same time.  Finally
>we
>are using NAS and it works great.
>
>My point is the file system plays a major role in performance when
>ingesting data.  The ultimate speed test would be if you could actually
>write the data into the final archive directory and basically do an
>ingestion in place (skip data transfer entirely), but I know that is
>rarely
>possible.
>
>This is an interesting challenge to see what configuration will yield the
>best through put/performance.  I look forward to hearing more about your
>progress on this.
>
>
>Best Regards,
>
>
>
>Cameron
>
>
>On Wed, Dec 12, 2012 at 7:28 PM, Mattmann, Chris A (388J) <
>chris.a.mattm...@jpl.nasa.gov> wrote:
>
>> Hi Chintu,
>>
>> From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC]
>> (GSFC-586.0)" mailto:chintu.mis...@nasa.gov>>
>> Date: Wednesday, December 12, 2012 12:02 PM
>> To: jpluser > chris.a.mattm...@jpl.nasa.gov>>, "dev@oodt.apache.org> dev@oodt.apache.org>" mailto:dev@oodt.apache.org>>
>> Subject: Re: OODT 0.3 branch
>>
>> If you are saying that FM can handle multiple connections at one time,
>>
>> Yep I'm saying that it can.
>>
>> then multiple crawlers pointing to same FM should increase performance
>> significantly.
>>
>> Well that really depends to be honest. It sounds like you guys are
>>hitting
>> an IO bottleneck potentially in data transfer? What file sizes are you
>> transferring? If you are IO bound on the data transfer part, the product
>> isn't fully ingested until:
>>
>>
>>   1.  it's entry is added to the catalog
>>   2.  The data transfer finishes
>>
>> Are you checking the FM for status along the way? Also realize that the
>>FM
>> will never be faster than the file system, so if it takes the file
>>system X
>> minutes to transfer a file F1, Y to transfer F2, and Z to transfer F3,
>>then
>> you still have to wait at least the max(X,Y,Z) time, regardless for the
>>3
>> ingestions to complete.
>>
>> But that¹s not what we saw in our tests.
>>
>> For example,
>> I saw barely 2 minutes performance difference between 2FM-6CR and
>>3FM-6CR.
>>
>> 1) 2 hour  6 minutes to process 262G   (1FM 3CR - 3CR to 1FM)
>> 2) 1 hour 58 minutes to process 262G   (1FM 6CR - 6CR to 1FM)
>> 3) 1 hour 39 minutes to process 262G   (2FM 6CR - 3CR to 1FM)
>> 4) 1 hour 39 minutes to process 262G   (2FM 9CR - 4+CR to 1FM)
>> 5) 1 hour 37 minutes to process 262G   (3FM 9CR - 3CR to 1FM)
>> 6) 2 hourto process 262G   (3FM 20CR - 6+CR to 1FM)
>> 7) 28 minutesto process 262G   (6FM 9CR - 1+CR to 1FM)   => This is
>>my
>> latest test and this is good number.
>>
>> What would be interesting is simply looking at the speed for how long it
>> takes to cp the files (which I bet is what's happening) versus mv'ing
>>the
>> files by hand. If mv is faster, I'd:
>>
>>
>>   1.  Implement a Data Transfer implementation that simply replaces the
>> calls to FileUtils.copyFile or .moveFile with systemCalls (see
>>ExecHelper
>> from oodt-commons) to UNIX equivalents.
>>   2.  Plug that data transfer in to your crawler invocations via the cmd
>> line.
>>
>> HTH!
>>
>> Cheers,
>> Chris
>>
>>
>> From: , Chris A > chris.a.mattm...@jpl.nasa.gov>>
>> Date: Wednesday, December 12, 2012 2:51 PM
>> T

FW: Resource Manager issue

[Mattmann, Chris A (388J)]

FYI I thought this would be of interest to the community...

On 12/13/12 5:30 PM, "Jain, Peyush (GSFC-5860)" 
wrote:

>>On 12/12/12 6:28 PM, "Mattmann, Chris A (388J)" wrote:
>>
>>>Hey Peyush,
>>>
>>>
>>>On 12/12/12 9:32 AM, "Jain, Peyush (GSFC-5860)"
>>>wrote:
>>>
>>>>Hi Chris,
>>>>
>>>>
>>>>Can you tell us how Job Scheduler talks to Resource and Job Monitors?
>>>>Is
>>>>Job Scheduler giving or getting information (regarding node
>>>>availability)
>>>>to Resource Monitor?
>>>
>>>Sure, specifically:
>>>
>>>The LRUScheduler keeps an instance to an instance of a Monitor,
>>>specifically for use in obtaining load (per node), nodes (by id and
>>>URL),
>>>incrementing and reducing load and so forth. So the scheduler uses the
>>>Monitor to profile the nodes that it assigns load onto and that it
>>>removes
>>>load from.
>>>
>>>>
>>>>If Job Scheduler is giving information to Resource Monitor and
>>>>executing
>>>>the jobs when node is available then why do we need Resource Monitor?
>>>
>>>The monitor is meant to be an interface for managing load with respect
>>>to
>>>nodes. Ultimately right now that information is managed partially in XML
>>>files, and partially in memory during the resource manager running, but
>>>ideally we have always wanted to have a generic GangliaMonitor to get
>>>information from a system like Ganglia, and then plug it into the
>>>scheduler. Right now the monitoring is "virtual" by profiling what we've
>>>sent to a node, and its capacity, etc. But we'd like it to be more real
>>>time (and have hacked together solutions for this that haven't made
>>>their
>>>way back into the OODT Apache trunk).
>>>
>>>
>>>>I
>>>>can see a need for Resource Monitor if it is monitoring the nodes and
>>>>sending information (node availability) to Job Scheduler so that Job
>>>>Scheduler can execute a task (from Job Queue).
>>>
>>>Yep you got it -- that's basically what the current one is doing.
>>>
>>>Cheers,
>>>Chris
>>>
>>>>
>>>>Thanks,
>>>>Peyush
>>>>
>>>>
>>>>On 12/12/12 9:12 AM, "Iwunze, Michael C  (GSFC-4700)[NOAA-JPSS]" wrote:
>>>>
>>>>>
>>>>>-- Forwarded Message
>>>>>From: "Mattmann, Chris A"
>>>>>Date: Tue, 11 Dec 2012 15:39:25 -0600
>>>>>To: Michael Iwunze
>>>>>Subject: Re: Resource Manager issue
>>>>>
>>>>>Hey Mike,
>>>>>
>>>>>
>>>>>On 12/11/12 11:29 AM, "Iwunze, Michael C  (GSFC-4700)[NOAA-JPSS]"
>>>>>wrote:
>>>>>
>>>>>>Hi Chris,
>>>>>>
>>>>>> How are you doing?  Thanks for all your help. Things seem to be
>>>>>>working
>>>>>>fine.
>>>>>
>>>>>Doing great and getting ready for the holidays. Going to head to El
>>>>>Paso,
>>>>>TX to watch USC play G. Tech in the Sun Bowl! Even though the Trojans
>>>>>didn't have a great season I am still going to head out and to support
>>>>>them. My wife and kid are coming too so it'll be a family vacation
>>>>>around
>>>>>New Years :)
>>>>>
>>>>>As for the integration work into JPSS/GRAVITE and that working fine,
>>>>>that
>>>>>is totally awesome and great to hear! :)
>>>>>
>>>>>>I am trying to get an overview of how all the extensions points
>>>>>>interact with each other based on the Resource Manager document
>>>>>>online.
>>>>>>Extension points meaning the resource manager client/server, batch
>>>>>>manager,
>>>>>>Job scheduler, Job monitor and Resource Monitor. Without delving deep
>>>>>>into
>>>>>>the code, my understanding is that once jobs are submitted via the
>>>>>>Workflow
>>>>>>to the Resource manager the job scheduler queues up jobs using the
>>>>>>job
>>>>>>queue. During this process the resource monitor checks for available
>>>>>>nodes
>>>>>>and the job monitor checks if a job is done executing on the nodes.
>>>>>>If
>>>>>>a
>>>>>>node is available the job is sent to the batch manager via the
>>>>>>scheduler
>>>>>>for
>>>>>>execution if not it sits in the queue. Is my explanation close to
>>>>>>accurate?
>>>>>
>>>>>+1 totally accurate and pretty much what happens.
>>>>>
>>>>>>In addition does the Job scheduler directly communicate with both
>>>>>>monitors
>>>>>>for node availability?
>>>>>
>>>>>Yep it sure does -- it has a reference to them in the code I believe.
>>>>>
>>>>>Cheers,
>>>>>Chris
>>>>>
>>>>>>
>>>>>>Thanks
>>>>>>mike
>>>>>>
>>>>>-- End of Forwarded Message
>>>>>
>>>>
>

[DISCUSS] Release 0.5?

[Mattmann, Chris A (388J)]

Hey Guys,

We've got 57 issues fixed for 0.5, a ton of improvements over 0.4,
critical bug fixes, a semi-fully functioning wengine (and fully
functioning workflow manager, and cas-pge).

It's time for a release.

If no one beats me to it, watch for an RC by tomorrow evening.

Cheers,
Chris

Re: OODT 0.3 branch

[Mattmann, Chris A (388J)]

Hi Chintu,

From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Wednesday, December 12, 2012 12:02 PM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OODT 0.3 branch

If you are saying that FM can handle multiple connections at one time,

Yep I'm saying that it can.

then multiple crawlers pointing to same FM should increase performance 
significantly.

Well that really depends to be honest. It sounds like you guys are hitting an 
IO bottleneck potentially in data transfer? What file sizes are you 
transferring? If you are IO bound on the data transfer part, the product isn't 
fully ingested until:


  1.  it's entry is added to the catalog
  2.  The data transfer finishes

Are you checking the FM for status along the way? Also realize that the FM will 
never be faster than the file system, so if it takes the file system X minutes 
to transfer a file F1, Y to transfer F2, and Z to transfer F3, then you still 
have to wait at least the max(X,Y,Z) time, regardless for the 3 ingestions to 
complete.

But that’s not what we saw in our tests.

For example,
I saw barely 2 minutes performance difference between 2FM-6CR and 3FM-6CR.

1) 2 hour  6 minutes to process 262G   (1FM 3CR - 3CR to 1FM)
2) 1 hour 58 minutes to process 262G   (1FM 6CR - 6CR to 1FM)
3) 1 hour 39 minutes to process 262G   (2FM 6CR - 3CR to 1FM)
4) 1 hour 39 minutes to process 262G   (2FM 9CR - 4+CR to 1FM)
5) 1 hour 37 minutes to process 262G   (3FM 9CR - 3CR to 1FM)
6) 2 hourto process 262G   (3FM 20CR - 6+CR to 1FM)
7) 28 minutesto process 262G   (6FM 9CR - 1+CR to 1FM)   => This is my 
latest test and this is good number.

What would be interesting is simply looking at the speed for how long it takes 
to cp the files (which I bet is what's happening) versus mv'ing the files by 
hand. If mv is faster, I'd:


  1.  Implement a Data Transfer implementation that simply replaces the calls 
to FileUtils.copyFile or .moveFile with systemCalls (see ExecHelper from 
oodt-commons) to UNIX equivalents.
  2.  Plug that data transfer in to your crawler invocations via the cmd line.

HTH!

Cheers,
Chris


From: , Chris A 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Date: Wednesday, December 12, 2012 2:51 PM
To: "Mistry, Chintu (GSFC-586.0)[COLUMBUS TECHNOLOGIES AND SERVICES INC]" 
mailto:chintu.mis...@nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OODT 0.3 branch

Hey Chintu,

From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Tuesday, December 11, 2012 2:41 PM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OODT 0.3 branch

Answers inline below.

---snip

Gotcha, so you are using different product types. So, each crawler is crawling 
various product types in each one of the staging area dirs, that looks like 
e.g.,

/STAGING_AREA_BASE
  /dir1 – 1st crawler
   - file1 of product type 1
   - file2 of product type 3

 /dir2 – 2nd crawler
   - file3 of product type 3

 /dir3 – 3rd crawler
   - file4 of product type 2

Is that what the staging area looks like? - YES

And then your FM is ingesting all 3 product types (I just picked 3 arbitrarily 
could have been N) into:

ARCHIVE_BASE/{ProductTypeName}/{MMDD}

Correct?  - YES

If so, I would imagine if FM1 and FM2 and FM3 would actually speed up the 
ingestion process compared to just using 1 FM with 1, or 2 or 3 crawlers all 
talking to it.

Let me ask a few more questions:

Do you see e.g., in the above example that file4 is ingested before file2? What 
about file3 before file2? If not, there is something wiggy going on.
   - I have not checked that. I guess I can check that. Can FM handle 
multiple connections at the same time ?


Yep FM can handle multiple connections at one time up to a limit (I think hard 
defaulted to ~100-200 by the underlying XMLRPC 2.1 library). We're using an old 
library currently but have a goal to upgrade to the latest version where I 
think this # is configurable.

Cheers,
Chris

Re: OODT 0.3 branch

[Mattmann, Chris A (388J)]

Hey Chintu,

From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Tuesday, December 11, 2012 2:41 PM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OODT 0.3 branch

Answers inline below.

---snip

Gotcha, so you are using different product types. So, each crawler is crawling 
various product types in each one of the staging area dirs, that looks like 
e.g.,

/STAGING_AREA_BASE
  /dir1 – 1st crawler
   - file1 of product type 1
   - file2 of product type 3

 /dir2 – 2nd crawler
   - file3 of product type 3

 /dir3 – 3rd crawler
   - file4 of product type 2

Is that what the staging area looks like? - YES

And then your FM is ingesting all 3 product types (I just picked 3 arbitrarily 
could have been N) into:

ARCHIVE_BASE/{ProductTypeName}/{MMDD}

Correct?  - YES

If so, I would imagine if FM1 and FM2 and FM3 would actually speed up the 
ingestion process compared to just using 1 FM with 1, or 2 or 3 crawlers all 
talking to it.

Let me ask a few more questions:

Do you see e.g., in the above example that file4 is ingested before file2? What 
about file3 before file2? If not, there is something wiggy going on.
   - I have not checked that. I guess I can check that. Can FM handle 
multiple connections at the same time ?


Yep FM can handle multiple connections at one time up to a limit (I think hard 
defaulted to ~100-200 by the underlying XMLRPC 2.1 library). We're using an old 
library currently but have a goal to upgrade to the latest version where I 
think this # is configurable.

Cheers,
Chris

Re: OODT 0.3 branch

[Mattmann, Chris A (388J)]

Hey Chintu,


From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Tuesday, December 11, 2012 2:41 PM
To: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OODT 0.3 branch

Answers inline below.

We will share information on apache.org at one point, but we are not there yet.

Thanks, OK,  please see inline below:

--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

From: , Chris A 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Date: Tuesday, December 11, 2012 5:23 PM
To: "Mistry, Chintu (GSFC-586.0)[COLUMBUS TECHNOLOGIES AND SERVICES INC]" 
mailto:chintu.mis...@nasa.gov>>, 
"dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Subject: Re: OODT 0.3 branch

Hey Chintu,

Thanks for reaching out! Replies inline below:

From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Tuesday, December 11, 2012 1:50 PM
To: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Cc: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: OODT 0.3 branch

Hi Chris,

We are trying measure a performance of how fast filemanager+crawler is 
performing.

Here is what we are trying to do:

  *   Total data to process : 262GB
  *   3 file managers and 9 crawlers
  *where 3 crawlers are sending file location to  file manager to process 
the file
  *   We have our own schema running on postgresql database
  *   Custom H5 Extactor using h5dump utility

Cool this sounds like an awesome test. Would you be willing to share some of 
the info on the OODT wiki?

https://cwiki.apache.org/confluence/display/OODT/Home

Questions:
1) I have tried using FileUtils.copyFile vs FileUtils.moveFile, but I don't see 
any difference in processing time. Both my LandingZone and Archive Area are 
located on same Filesystem(GPFS). It is roughly taking 100 minutes to process 
262G data. Can you shed any light on why don't we see any performance change ?

This may have to do with the way that the JDK (what version are you using?) 
implements the actual arraycopy methods, and how the apache commons-io library 
wraps those methods. Let me know what JDK version you're using and we can 
investigate it.

- java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.5) (rhel-1.50.1.11.5.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

OK thanks. I found this article:

http://stackoverflow.com/questions/300559/move-copy-file-operations-in-java

It doesn't really go into too much detail but the nice thing is that if you 
need a different, or faster DataTransfer, you can always sub-class or implement 
your own that makes a call to e.g., "mv" or "cp" at the UNIX level if you think 
it'll speed it up.

Looking at: 
http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html

http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html#copyFile(java.io.File,
 
java.io.File)
http://commons.apache.org/io/api-release/org/apache/commons/io/FileUtils.html#moveFile(java.io.File,
 
java.io.File)

Note for moveFile:

"When the destination file is on another file system, do a "copy and delete".

I wonder how it detects that? I wonder if it always thinks it's on another 
filesystem using JDK and GFS? If so that might explain what you are seeing in 
that there is no difference between copyFile and moveFile?

2) The other thing also is that I don't see any performance gain between 
running 2 FM or 3FM. I thought that I would see some performance gain due to 
concurrency. Same goes for multiple crawlers. I was hoping to see pretty 
obvious performance change if I increase number of crawlers. What are thoughts 
on running things in parallel to increase performance.

How are you situating the additional file managers? Are you having 1 crawler 
ingest to 3? Or is there a 1:1 correspondence between each crawler and FM? And, 
what do you mean by no performance gain? Do you mean that you don't see 3x 
speed in terms of e.g. Product ingestion of met into the catalog? Of file 
transfer speed?

- All 3 FM are running on one machine. Each crawler instance is crawling 
different directory. And 3 Crawlers are connected to 1st FM. Other 3 are 
connected to second FM and last 3 crawlers are connected to third FM. When I 
say performance difference between 2 and 3FM, I meant they take identically 
same amount of time to process same amount of data concurrently.

So, I think the big thing here is to understand how the crawlers work and how 
they march through files. Basically the crawl method re

Re: OODT 0.3 branch

[Mattmann, Chris A (388J)]

Hey Chintu,

Thanks for reaching out! Replies inline below:

From: , "Chintu [COLUMBUS TECHNOLOGIES AND SERVICES INC] (GSFC-586.0)" 
mailto:chintu.mis...@nasa.gov>>
Date: Tuesday, December 11, 2012 1:50 PM
To: "dev@oodt.apache.org" 
mailto:dev@oodt.apache.org>>
Cc: jpluser 
mailto:chris.a.mattm...@jpl.nasa.gov>>
Subject: OODT 0.3 branch

Hi Chris,

We are trying measure a performance of how fast filemanager+crawler is 
performing.

Here is what we are trying to do:

  *   Total data to process : 262GB
  *   3 file managers and 9 crawlers
  *where 3 crawlers are sending file location to  file manager to process 
the file
  *   We have our own schema running on postgresql database
  *   Custom H5 Extactor using h5dump utility

Cool this sounds like an awesome test. Would you be willing to share some of 
the info on the OODT wiki?

https://cwiki.apache.org/confluence/display/OODT/Home

Questions:
1) I have tried using FileUtils.copyFile vs FileUtils.moveFile, but I don't see 
any difference in processing time. Both my LandingZone and Archive Area are 
located on same Filesystem(GPFS). It is roughly taking 100 minutes to process 
262G data. Can you shed any light on why don't we see any performance change ?

This may have to do with the way that the JDK (what version are you using?) 
implements the actual arraycopy methods, and how the apache commons-io library 
wraps those methods. Let me know what JDK version you're using and we can 
investigate it.

2) The other thing also is that I don't see any performance gain between 
running 2 FM or 3FM. I thought that I would see some performance gain due to 
concurrency. Same goes for multiple crawlers. I was hoping to see pretty 
obvious performance change if I increase number of crawlers. What are thoughts 
on running things in parallel to increase performance.

How are you situating the additional file managers? Are you having 1 crawler 
ingest to 3? Or is there a 1:1 correspondence between each crawler and FM? And, 
what do you mean by no performance gain? Do you mean that you don't see 3x 
speed in terms of e.g. Product ingestion of met into the catalog? Of file 
transfer speed?

Are the 3 crawlers crawling the same staging area concurrently? Or are they 
separated out by buckets? And, which crawler are you using? The 
MetExtractorProductCrawler or the AutoDetectCrawler? Also, what is the 
versioning policy for the FM on a per product basis? Are all products being 
ingested of the same ProductType and ultimately of the same versioner and 
ultimate disk location?

3) Like I said earlier, we are running crawler to push data to file manager. If 
I run it that way, then "data transfer(copy or move)" is happing on the crawler 
side. I can not find any way to let file manager handle "data transfer" using 
on of your runtime options. Please let me know if you guys know how to do that ?

If you want the FM to handle the transfer you have to use the low level File 
Manager Client and omit the clientTransfer option:

[chipotle:local/filemgr/bin] mattmann% ./filemgr-client
filemgr-client --url  --operation [ [params]]
operations:
--addProductType --typeName  --typeDesc  --repository  
--versionClass 
--ingestProduct --productName  --productStructure  
--productTypeName  --metadataFile  
[--clientTransfer --dataTransfer ] 
--refs ...
--hasProduct --productName 
--getProductTypeByName --productTypeName 
--getNumProducts --productTypeName 
--getFirstPage --productTypeName 
--getNextPage --productTypeName  --currentPageNum 
--getPrevPage --productTypeName  --currentPageNum 
--getLastPage --productTypeName 
--getCurrentTransfer
--getCurrentTransfers
--getProductPctTransferred --productId  --productTypeName 
--getFilePctTransferred --origRef 

[chipotle:local/filemgr/bin] mattmann%

That is just a CMD line exposure of the underlying FM client Java API which 
lets you do server side transfers on ingest by passing clientTransfer == false 
to this method:

http://oodt.apache.org/components/maven/xref/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.html#1168

We have enough processing power to run multiple FM and Crawlers for 
scalability. But for some reason crawler is not scaling enough.


We'll get it scaling out for ya. Can you please provide answers to the above 
questions and we'll go from there? Thanks!

Thanks!

Cheers,
Chris




Regards
--
Chintu Mistry
NASA Goddard Space Flight Center
Bldg L40B, Room S776
Office: 240 684 0477
Mobile: 770 310 1047

Re: [jira] [Created] (OODT-539) New maven archetype for generating a template MetExtractor project

[Mattmann, Chris A (388J)]

Hey Rishi,

Check out the Maven archetype effort called OODT RADIX that you've seen up
on the Wiki and in other places. It lives inside of cas-mvn-plugins
archetypes folder:

http://svn.apache.org/repos/asf/oodt/trunk/mvn/archetypes/


You may want to put this archetype in there. I know that Paul and Cameron
did some work on a met extractor type as part of the OPSUI so it would be
great to integrate. Even though it is currently cas-mvn-plugins, it's so
close to cas-mvn, I would question the rename -- probably good enough as
it is.

However, for your idea of archetypes for everything, that.sounds.awesome.
Goal for ApacheCon NA 2013? Fast Feather talk showing all of our progress?
YES! :)

Cheers,
Chris

On 12/5/12 3:54 PM, "Verma, Rishi (388J)"  wrote:

>Hi All -
>
>Small point here, but the issue below is currently tagged under the
>'cas-mvn-plugins' component; however, the proposed work is more of maven
>archetype related work.
>
>I'd like to propose we rename the component 'cas-mvn-plugins' to be
>'cas-mvn' so that it can include code for cas-related maven archetypes.
>
>Any thoughts?
>
>Rishi 
>
>On Dec 5, 2012, at 3:46 PM, Rishi Verma (JIRA) wrote:
>
>> Rishi Verma created OODT-539:
>> 
>> 
>> Summary: New maven archetype for generating a template
>>MetExtractor project
>> Key: OODT-539
>> URL: https://issues.apache.org/jira/browse/OODT-539
>> Project: OODT
>>  Issue Type: New Feature
>>  Components: cas-mvn-plugins
>>Affects Versions: 0.4
>>Reporter: Rishi Verma
>>Assignee: Rishi Verma
>>Priority: Minor
>> Fix For: 0.5
>> 
>> 
>> Create a new maven archetype for allowing developers to quickly
>>generate MetExtractor projects on their own.
>> 
>> The rationale for creating this archetype is to make it simpler and
>>quicker to write new met extractors. Writing new met extractors is
>>pretty easy, but establishing the correct maven directory structure, pom
>>file(s), and build cycle can be tricky. I myself often just replicate
>>the directory structure and pom files from other projects instead of
>>creating/writing new met extractor projects from scratch. This can
>>however lead to human errors and also unwanted code and config. Having
>>an archetype which generates a preferred met extractor-specific maven
>>directory structure, pom files, and template met extractor code would
>>make the job of creating new custom met extractors more consistent and
>>automatic.
>> 
>> The proposed capability is the following:
>> 1. New archetype to generate a full maven project for writing custom
>>met extractor(s)
>> 2. Pre-populate generated project with template MetExtractor(s) which
>>work out-of-the box and make it obvious WHERE developer should add their
>>custom extraction code
>> 3. Ensure generated project pom allows a build of met extractor project
>>out of the box
>> 
>> To start out, I would recommend we include an extended
>>CmdLineMetExtractor class which works out-of-the box. We can include
>>more custom met extractors and template code later.
>> 
>> 
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA
>>administrators
>> For more information on JIRA, see:
>>http://www.atlassian.com/software/jira
>

[ANNOUNCE] Welcome Mike Joyce as an Apache OODT PMC member and Committer

[Mattmann, Chris A (388J)]

Hi Folks,

Mike Joyce was recently elected to the Apache OODT PMC and as a committer on 
the project.
Welcome Mike! Feel free to say a bit about yourself.

Cheers,
Chris

Re: svn commit: r1412053 - /oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java

[Mattmann, Chris A (388J)]

Hey Luca,

Appreciate it, saw that and no worries and feel free to catch the same
thing for me :) Appreciate the peer review!

Also feel free to use: http://reviews.apache.org/

Many of use on the Apache OODT PMC use it and like
using it to review patches and changes. You can request
an account with infra@ if you don't have one yet, and 
you can tie back to JIRA, etc.

Happy Thanksgiving!

Cheers,
Chris

On Nov 22, 2012, at 3:47 AM, Cinquini, Luca (3880) wrote:

> Thanks Chris, comment added. You are right that I should have mentioned the 
> issue number in the commit.
> thanks for the careful review :)
> Luca
> 
> On Nov 21, 2012, at 5:09 PM, Mattmann, Chris A (388J) wrote:
> 
>> Hey Luca,
>> 
>> Ah, perfect. If it was for OODT-522, then if you ref that in your commit 
>> message that helps to 
>> tie to the issue that you are fixing. And it helps people like me who peer 
>> review the commits
>> to know that it's for something that you already filed one for :)
>> 
>> +1 to adding a comment on OODT-522 about what you updated.
>> 
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>> On Nov 21, 2012, at 3:57 PM, Cinquini, Luca (3880) wrote:
>> 
>>> Hi Chris,
>>> sure, but I have a question first: this is really a fix on the 
>>> following issue, which got filed and solved a few weeks ago:
>>> 
>>> https://issues.apache.org/jira/browse/OODT-522
>>> 
>>> Would it be more appropriate to insert a comment in this Jira issue, 
>>> bringing its development to a close ?
>>> 
>>> thanks, Luca
>>> 
>>> On Nov 21, 2012, at 10:19 AM, Mattmann, Chris A (388J) wrote:
>>> 
>>>> Hi Luca,
>>>> 
>>>> Can you please file a JIRA issue for this? This was a change I believe
>>>> that took some time to track down so having it in a JIRA issue will help
>>>> others track and understand how it was fixed. It should also appear in
>>>> our change log.
>>>> 
>>>> Thanks!
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Nov 21, 2012, at 2:50 AM, 
>>>>  wrote:
>>>> 
>>>>> Author: luca
>>>>> Date: Wed Nov 21 10:50:30 2012
>>>>> New Revision: 1412053
>>>>> 
>>>>> URL: http://svn.apache.org/viewvc?rev=1412053&view=rev
>>>>> Log:
>>>>> Removing product references upon metadata update, as they will be added 
>>>>> again later on.
>>>>> 
>>>>> Modified:
>>>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>>>> 
>>>>> Modified: 
>>>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>>>> URL: 
>>>>> http://svn.apache.org/viewvc/oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java?rev=1412053&r1=1412052&r2=1412053&view=diff
>>>>> ==
>>>>> --- 
>>>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>>>>  (original)
>>>>> +++ 
>>>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>>>>  Wed Nov 21 10:50:30 2012
>>>>> @@ -523,6 +523,12 @@ public class MetadataResource extends Cu
>>>>>   // retrieve existing metadata
>>>>>   metadata = catalog.getMetadata(product);
>>>>> 
>>>>> +  // remove product references (as they will be added later)
>>>>> +  metadata.removeMetadata("reference_orig");
>>>>> +  metadata.removeMetadata("reference_data_store");
>>>>> +  metadata.removeMetadata("reference_fileSize");
>>>>> +  metadata.removeMetadata("reference_mimeType");
>>>>> +  
>>>>>   // merge new and existing metadata
>>>>>   metadata.addMetadata(newMetadata);
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

Re: svn commit: r1412053 - /oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java

[Mattmann, Chris A (388J)]

Hey Luca,

Ah, perfect. If it was for OODT-522, then if you ref that in your commit 
message that helps to 
tie to the issue that you are fixing. And it helps people like me who peer 
review the commits
to know that it's for something that you already filed one for :)

+1 to adding a comment on OODT-522 about what you updated.

Thanks!

Cheers,
Chris

On Nov 21, 2012, at 3:57 PM, Cinquini, Luca (3880) wrote:

> Hi Chris,
>   sure, but I have a question first: this is really a fix on the 
> following issue, which got filed and solved a few weeks ago:
> 
> https://issues.apache.org/jira/browse/OODT-522
> 
> Would it be more appropriate to insert a comment in this Jira issue, bringing 
> its development to a close ?
> 
> thanks, Luca
> 
> On Nov 21, 2012, at 10:19 AM, Mattmann, Chris A (388J) wrote:
> 
>> Hi Luca,
>> 
>> Can you please file a JIRA issue for this? This was a change I believe
>> that took some time to track down so having it in a JIRA issue will help
>> others track and understand how it was fixed. It should also appear in
>> our change log.
>> 
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>> On Nov 21, 2012, at 2:50 AM, 
>>  wrote:
>> 
>>> Author: luca
>>> Date: Wed Nov 21 10:50:30 2012
>>> New Revision: 1412053
>>> 
>>> URL: http://svn.apache.org/viewvc?rev=1412053&view=rev
>>> Log:
>>> Removing product references upon metadata update, as they will be added 
>>> again later on.
>>> 
>>> Modified:
>>>  
>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>> 
>>> Modified: 
>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>> URL: 
>>> http://svn.apache.org/viewvc/oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java?rev=1412053&r1=1412052&r2=1412053&view=diff
>>> ==
>>> --- 
>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>>  (original)
>>> +++ 
>>> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>>>  Wed Nov 21 10:50:30 2012
>>> @@ -523,6 +523,12 @@ public class MetadataResource extends Cu
>>> // retrieve existing metadata
>>> metadata = catalog.getMetadata(product);
>>> 
>>> +  // remove product references (as they will be added later)
>>> +  metadata.removeMetadata("reference_orig");
>>> +  metadata.removeMetadata("reference_data_store");
>>> +  metadata.removeMetadata("reference_fileSize");
>>> +  metadata.removeMetadata("reference_mimeType");
>>> +  
>>> // merge new and existing metadata
>>> metadata.addMetadata(newMetadata);
>>> 
>>> 
>>> 
>> 
> 

Re: svn commit: r1412053 - /oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java

[Mattmann, Chris A (388J)]

Hi Luca,

Can you please file a JIRA issue for this? This was a change I believe
that took some time to track down so having it in a JIRA issue will help
others track and understand how it was fixed. It should also appear in
our change log.

Thanks!

Cheers,
Chris

On Nov 21, 2012, at 2:50 AM, 
  wrote:

> Author: luca
> Date: Wed Nov 21 10:50:30 2012
> New Revision: 1412053
> 
> URL: http://svn.apache.org/viewvc?rev=1412053&view=rev
> Log:
> Removing product references upon metadata update, as they will be added again 
> later on.
> 
> Modified:
>
> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
> 
> Modified: 
> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
> URL: 
> http://svn.apache.org/viewvc/oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java?rev=1412053&r1=1412052&r2=1412053&view=diff
> ==
> --- 
> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>  (original)
> +++ 
> oodt/trunk/curator/src/main/java/org/apache/oodt/cas/curation/service/MetadataResource.java
>  Wed Nov 21 10:50:30 2012
> @@ -523,6 +523,12 @@ public class MetadataResource extends Cu
>   // retrieve existing metadata
>   metadata = catalog.getMetadata(product);
> 
> +  // remove product references (as they will be added later)
> +  metadata.removeMetadata("reference_orig");
> +  metadata.removeMetadata("reference_data_store");
> +  metadata.removeMetadata("reference_fileSize");
> +  metadata.removeMetadata("reference_mimeType");
> +  
>   // merge new and existing metadata
>   metadata.addMetadata(newMetadata);
> 
> 
> 

Re: [oodt-dev] Re: ApacheCon NA 2013

[Mattmann, Chris A (388J)]

Thanks Luca!

Cheers,
Chris

On Nov 11, 2012, at 5:19 PM, Cinquini, Luca (3880) wrote:

> Hi all,
>   I also submitted two proposed talks:
> 
> 1) Leveraging OODT tools and components within Climate Science and the Earth 
> System Grid Federation
> 
> 2) Detecting radio-astronomical "Fast Radio Transient Events" via an 
> OODT-based metadata processing pipeline
> 
> thanks, Luca
> 
> On Nov 11, 2012, at 4:51 PM, Andrew Hart wrote:
> 
>> Hey all,
>> 
>> Great to see all the activity for ApacheCon NA! I've just made a couple of 
>> submissions to this as well:
>> 
>> "Apache OODT and Radioastronomy: Preparing for the Data Deluge" is designed 
>> to serve as an introduction to OODT as a tool in the 'big data' arsenal by 
>> highlighting several large-scale radioastronomy experiments that currently 
>> leverage OODT in their data management systems.
>> 
>> "Having your Cake and Eating it Too with Apache OODT and Apache Solr" covers 
>> the way that the flexible search capabilities of Apache Solr can be 
>> leveraged as a compliment to the robust data and metadata management 
>> capabilities of Apache OODT, as well as existing tool support in OODT for 
>> making that easier.
>> 
>> "Effective Data Curation with Apache OODT" highlights the Apache OODT File 
>> Manager Curation tool as a powerful way to interact with the OODT File 
>> Manager either via its web interface or JAX-RS web services layer that 
>> allows third party tools to manipulate the File Manager via a standard 
>> interface.
>> 
>> Fingers crossed that together we get enough submissions accepted that we can 
>> get our own track again!
>> 
>> Best,
>> 
>> Andrew.
>> 
>> On 11/11/2012 12:21 PM, Verma, Rishi (388J) wrote:
>>> Hey Chris and all,
>>> 
>>> I just submitted a talk proposal myself, titled: "Searching for cancer 
>>> biomarkers with the help of Apache OODT". This talk will explore the 
>>> application of OODT data processing technologies like Workflow, PGEs, and 
>>> Resource Manager, and how they are furthering cancer biomarker discovery 
>>> goals for the Early Detection Research Network (EDRN). Another goal of the 
>>> talk is also to show how new projects can easily get up to speed and start 
>>> using OODT tools, to meet their own data processing needs.
>>> 
>>> Thanks,
>>> Rishi
>>> 
>>> On Nov 9, 2012, at 1:42 PM, Mattmann, Chris A (388J) wrote:
>>> 
>>>> Hi Folks,
>>>> 
>>>> I went ahead and submitted a talk titled:
>>>> 
>>>> "Wengines, Workflows, and 2 years of advanced data processing in Apache 
>>>> OODT."
>>>> 
>>>> Here's the link: http://na.apachecon.com/proposals/120/
>>>> 
>>>> Keep em' coming there is still time left! :) The more quality
>>>> Apache OODT related talks, the more chance we have to
>>>> get a full on session/track at the meeting.
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Nov 8, 2012, at 6:05 AM, Crichton, Daniel J (4231) wrote:
>>>> 
>>>>> Hi Chris,
>>>>> 
>>>>> I submitted a talk on "Apache OODT: From the beginning to big data
>>>>> management" to discuss the origin, architectural principles, and path for
>>>>> going from a research project to a scalable, open source framework.
>>>>> 
>>>>> It would be great to get some talks and discuss how Apache OODT can
>>>>> continue to evolve to fit the needs of the big data management community.
>>>>> 
>>>>> Best,
>>>>> 
>>>>> Dan
>>>>> -- 
>>>>> 
>>>>> Dan Crichton
>>>>> Program Manager, Planetary Data System Engineering, Solar System
>>>>> Exploration
>>>>> Program Manager, Data Systems and Technology, Earth Science and Technology
>>>>> Principal Computer Scientist
>>>>> Jet Propulsion Laboratory
>>>>> (818) 354-9155
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 11/6/12 12:48 PM, "Mattmann, Chris A (388J

Re: [oodt-dev] Re: ApacheCon NA 2013

[Mattmann, Chris A (388J)]

Great work Andrew!

Cheers,
Chris

On Nov 11, 2012, at 3:51 PM, Andrew Hart wrote:

> Hey all,
> 
> Great to see all the activity for ApacheCon NA! I've just made a couple of 
> submissions to this as well:
> 
> "Apache OODT and Radioastronomy: Preparing for the Data Deluge" is designed 
> to serve as an introduction to OODT as a tool in the 'big data' arsenal by 
> highlighting several large-scale radioastronomy experiments that currently 
> leverage OODT in their data management systems.
> 
> "Having your Cake and Eating it Too with Apache OODT and Apache Solr" covers 
> the way that the flexible search capabilities of Apache Solr can be leveraged 
> as a compliment to the robust data and metadata management capabilities of 
> Apache OODT, as well as existing tool support in OODT for making that easier.
> 
> "Effective Data Curation with Apache OODT" highlights the Apache OODT File 
> Manager Curation tool as a powerful way to interact with the OODT File 
> Manager either via its web interface or JAX-RS web services layer that allows 
> third party tools to manipulate the File Manager via a standard interface.
> 
> Fingers crossed that together we get enough submissions accepted that we can 
> get our own track again!
> 
> Best,
> 
> Andrew.
> 
> On 11/11/2012 12:21 PM, Verma, Rishi (388J) wrote:
>> Hey Chris and all,
>> 
>> I just submitted a talk proposal myself, titled: "Searching for cancer 
>> biomarkers with the help of Apache OODT". This talk will explore the 
>> application of OODT data processing technologies like Workflow, PGEs, and 
>> Resource Manager, and how they are furthering cancer biomarker discovery 
>> goals for the Early Detection Research Network (EDRN). Another goal of the 
>> talk is also to show how new projects can easily get up to speed and start 
>> using OODT tools, to meet their own data processing needs.
>> 
>> Thanks,
>> Rishi
>> 
>> On Nov 9, 2012, at 1:42 PM, Mattmann, Chris A (388J) wrote:
>> 
>>> Hi Folks,
>>> 
>>> I went ahead and submitted a talk titled:
>>> 
>>> "Wengines, Workflows, and 2 years of advanced data processing in Apache 
>>> OODT."
>>> 
>>> Here's the link: http://na.apachecon.com/proposals/120/
>>> 
>>> Keep em' coming there is still time left! :) The more quality
>>> Apache OODT related talks, the more chance we have to
>>> get a full on session/track at the meeting.
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> On Nov 8, 2012, at 6:05 AM, Crichton, Daniel J (4231) wrote:
>>> 
>>>> Hi Chris,
>>>> 
>>>> I submitted a talk on "Apache OODT: From the beginning to big data
>>>> management" to discuss the origin, architectural principles, and path for
>>>> going from a research project to a scalable, open source framework.
>>>> 
>>>> It would be great to get some talks and discuss how Apache OODT can
>>>> continue to evolve to fit the needs of the big data management community.
>>>> 
>>>> Best,
>>>> 
>>>> Dan
>>>> -- 
>>>> 
>>>> Dan Crichton
>>>> Program Manager, Planetary Data System Engineering, Solar System
>>>> Exploration
>>>> Program Manager, Data Systems and Technology, Earth Science and Technology
>>>> Principal Computer Scientist
>>>> Jet Propulsion Laboratory
>>>> (818) 354-9155
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 11/6/12 12:48 PM, "Mattmann, Chris A (388J)"
>>>>   wrote:
>>>> 
>>>>> Hey Folks,
>>>>> 
>>>>> Please consider submitting an OODT related talk to ApacheCon NA 2013!
>>>>> 
>>>>> http://na.apachecon.com/
>>>>> 
>>>>> If you are using OODT, one of its developers; think it's cool; heard
>>>>> about it somewhere
>>>>> but don't know what it is, whatever! Please consider submitting a talk.
>>>>> Many of the
>>>>> Apache OODT PMC will be there and we'll be meeting up, happy to answer
>>>>> questions
>>>>> and talk about the project.
>>>>> 
>>>>> Thanks!
>>>>> 
>>>>> Cheers,
>>>>> Chris
>>>>> 
>>

Re: Workflow Task/Condition Instances...

[Mattmann, Chris A (388J)]

Hi Bruce,

Thanks for the insights. It makes me believe that what we're doing is sound
since we've got your first 2 points covered on a per-workflow, and a per-task
basis, down to the hierarchies and appropriate time tracking and scheduling.

I'll think about the appropriate algorithms for duration and scheduling based
on that and I look forward to reading more as I go along implementing.

Thanks and take care!

Cheers,
Chris

On Nov 10, 2012, at 6:40 AM, Bruce Barkstrom wrote:

> Thanks.  It may take a week or two for the WMSCI folks
> to request a review.  My submission deadline is Nov. 16,
> which should be no problem since I already have a first
> draft and it's only about 5 pages long.
> 
> There are a few subtleties about the scheduling
> problem:
> 
> 1.  To do scheduling, you'll need to include an estimate
> of the (wall clock) duration of each job.  When you do the
> computation, you'll have an earliest start, latest start,
> earliest finish, and latest finish for each job.  You'll usually
> want to compute slack times and find the critical path.
> This kind of work usually appears as a Gantt chart, which
> may mean you'll have to find software to draw this kind
> of chart.  MS Project can show you what that looks like,
> but I haven't found a short way to get enough access
> to the innards of that software.
> 
> 2.   I suspect you're going to have a lot of jobs to track.
> You may want to figure out how to create hierarchies
> and to have different time horizons (more precision in
> times close to now; less precision in times far from now).
> 
> 3.  This stuff can be coupled to cost estimation as well.
> 
> 4.  You may need to consider reprocessing of product
> lines.  That gets into versioning.
> 
> 5.  Schedules are really stochastic.
> 
> As a strong suggestion, do a simple, deterministic
> duration schedule algorithm first.
> 
> Bruce B.
> 
> On Fri, Nov 9, 2012 at 10:29 PM, Mattmann, Chris A (388J) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
> 
>> Hey Bruce,
>> 
>> Yes please provide my name as a reviewer and I have scoped out your ESI
>> paper I believe when you brought it up before. We definitely need to
>> incorporate
>> some of the logic you mention below especially when we are dealing with
>> the types
>> of operational scenarios you mention in ES data production.
>> 
>> Cheers,
>> Chris
>> 
>> On Nov 10, 2012, at 2:30 AM, Bruce Barkstrom wrote:
>> 
>>> I'd strongly suggest looking at my paper in ESI.  It's got fairly
>> detailed
>>> data structures - in Ada, as you might expect from me.  While that
>>> isn't necessarily critical - I'd recommend doing the calculations in
>>> RAM because the topological sort (which is a breadth first search
>>> - if I recall correctly, although you'd be wise to check) is easiest to
>>> implement as a recursive traversal of the production graph when that's
>>> represented as an adjacency list, rather than an adjacency matrix.
>>> 
>>> There are some subtleties that are likely to occur when you want to
>>> do a multi-time scale approach.  In other words, you might want
>>> a detailed schedule for the next day, but not for the next year.
>>> Those different time scales need to produce different schedules
>>> with different granularities.  It will complicate the data structures.
>>> 
>>> There's also an issue for design when the production is done with
>>> product lines.  I don't expect these to show up for many production
>>> problems.  They do show up in operational production and in
>>> climate data record production.
>>> 
>>> I've worked on these problems - but it's not what I'm doing right now.
>>> 
>>> I'm expecting to submit a paper to an odd conference called WMSCI
>>> 2013.  The paper is on using Formal Concept Analysis that is related
>>> to some of the set theory work that folks doing ontologies use, but
>>> can also help in classifying things.  WMSCI is a bit of an oddball
>>> conference - all kinds of system work, including semiotics, information
>>> theory.  It's held in Orlando in July - and has a rather large contingent
>>> of participants from Latin America.  I think they complicate the
>> refereeing
>>> process by using double blind and other methods of reviewing submissions.
>>> I need to provide names of prospective reviewers to get the paper through
>>> the submission.  I'd appreciate it if I could put your name in as a
>>

Re: Workflow Task/Condition Instances...

[Mattmann, Chris A (388J)]

Hey Bruce,

Yes please provide my name as a reviewer and I have scoped out your ESI
paper I believe when you brought it up before. We definitely need to incorporate
some of the logic you mention below especially when we are dealing with the 
types
of operational scenarios you mention in ES data production.

Cheers,
Chris

On Nov 10, 2012, at 2:30 AM, Bruce Barkstrom wrote:

> I'd strongly suggest looking at my paper in ESI.  It's got fairly detailed
> data structures - in Ada, as you might expect from me.  While that
> isn't necessarily critical - I'd recommend doing the calculations in
> RAM because the topological sort (which is a breadth first search
> - if I recall correctly, although you'd be wise to check) is easiest to
> implement as a recursive traversal of the production graph when that's
> represented as an adjacency list, rather than an adjacency matrix.
> 
> There are some subtleties that are likely to occur when you want to
> do a multi-time scale approach.  In other words, you might want
> a detailed schedule for the next day, but not for the next year.
> Those different time scales need to produce different schedules
> with different granularities.  It will complicate the data structures.
> 
> There's also an issue for design when the production is done with
> product lines.  I don't expect these to show up for many production
> problems.  They do show up in operational production and in
> climate data record production.
> 
> I've worked on these problems - but it's not what I'm doing right now.
> 
> I'm expecting to submit a paper to an odd conference called WMSCI
> 2013.  The paper is on using Formal Concept Analysis that is related
> to some of the set theory work that folks doing ontologies use, but
> can also help in classifying things.  WMSCI is a bit of an oddball
> conference - all kinds of system work, including semiotics, information
> theory.  It's held in Orlando in July - and has a rather large contingent
> of participants from Latin America.  I think they complicate the refereeing
> process by using double blind and other methods of reviewing submissions.
> I need to provide names of prospective reviewers to get the paper through
> the submission.  I'd appreciate it if I could put your name in as a
> reviewer.
> 
> Bruce B.
> 
> On Thu, Nov 8, 2012 at 11:27 PM, Mattmann, Chris A (388J) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
> 
>> Thanks Bruce -- yep we carry it forward as a model called "Workflow" which
>> has a corresponding Java class hierarchy behind it, and a serialized
>> representation
>> in XML, and in a DB format.
>> 
>> Cheers,
>> Chris
>> 
>> On Nov 9, 2012, at 7:53 AM, Bruce Barkstrom wrote:
>> 
>>> You might consider building the graph in advance and use
>>> it for planning purposes.
>>> 
>>> Bruce B.
>>> 
>>> On Thu, Nov 8, 2012 at 4:01 AM, Chris A Mattmann
>>> wrote:
>>> 
>>>> Hey Brian,
>>>> 
>>>> That sounds Perfect! I like the idea of simply creating a
>> RunnableInstance
>>>> and think it fits the architecture well!
>>>> +50 to proceed down that path.
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Nov 8, 2012, at 3:16 PM, Brian Foster wrote:
>>>> 
>>>>> 
>>>>> hey chris,
>>>>> 
>>>>> I think i work this out... i'm gonna create a RunnableInstance class...
>>>> this will then hold the state, start/end times, type (TASK or
>> CONDITION),
>>>> and the class to run (WorkflowTaskInstance or
>> WorkflowConditionInstance).
>>>>> 
>>>>> -brian
>>>>> 
>>>>> On Nov 7, 2012, at 3:05 PM, Brian Foster wrote:
>>>>> 
>>>>>> hey chris,
>>>>>> 
>>>>>> so i'm going through this WorkflowProcessor stuff and finishing it
>>>> up... I'm trying to make it so that WorkflowProcessor is used by all
>>>> Workflow Engines... WorkflowInstance holds the state, times, etc.. for a
>>>> Workflow, and then a WorkflowProcessor is response for analyzing that
>>>> WorkflowInstance and determining what to run next in the Workflow... in
>>>> this process i've notices that we really don't have any
>>>> "WorkflowInstance-like" class for Tasks and Conditions... A
>>>> WorkflowTaskInstance is actually the runnable task/condition job... I
>> know
>>>> this would be a

Re: ApacheCon NA 2013

[Mattmann, Chris A (388J)]

Hi Folks,

I went ahead and submitted a talk titled:

"Wengines, Workflows, and 2 years of advanced data processing in Apache OODT."

Here's the link: http://na.apachecon.com/proposals/120/

Keep em' coming there is still time left! :) The more quality
Apache OODT related talks, the more chance we have to
get a full on session/track at the meeting.

Cheers,
Chris

On Nov 8, 2012, at 6:05 AM, Crichton, Daniel J (4231) wrote:

> Hi Chris,
> 
> I submitted a talk on "Apache OODT: From the beginning to big data
> management" to discuss the origin, architectural principles, and path for
> going from a research project to a scalable, open source framework.
> 
> It would be great to get some talks and discuss how Apache OODT can
> continue to evolve to fit the needs of the big data management community.
> 
> Best,
> 
> Dan
> -- 
> 
> Dan Crichton
> Program Manager, Planetary Data System Engineering, Solar System
> Exploration
> Program Manager, Data Systems and Technology, Earth Science and Technology
> Principal Computer Scientist
> Jet Propulsion Laboratory
> (818) 354-9155
> 
> 
> 
> 
> 
> 
> 
> On 11/6/12 12:48 PM, "Mattmann, Chris A (388J)"
>  wrote:
> 
>> Hey Folks,
>> 
>> Please consider submitting an OODT related talk to ApacheCon NA 2013!
>> 
>> http://na.apachecon.com/
>> 
>> If you are using OODT, one of its developers; think it's cool; heard
>> about it somewhere
>> but don't know what it is, whatever! Please consider submitting a talk.
>> Many of the
>> Apache OODT PMC will be there and we'll be meeting up, happy to answer
>> questions
>> and talk about the project.
>> 
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>> ++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattm...@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++
>> 
> 


++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++

Re: Workflow Task/Condition Instances...

[Mattmann, Chris A (388J)]

Thanks Bruce -- yep we carry it forward as a model called "Workflow" which
has a corresponding Java class hierarchy behind it, and a serialized 
representation
in XML, and in a DB format.

Cheers,
Chris

On Nov 9, 2012, at 7:53 AM, Bruce Barkstrom wrote:

> You might consider building the graph in advance and use
> it for planning purposes.
> 
> Bruce B.
> 
> On Thu, Nov 8, 2012 at 4:01 AM, Chris A Mattmann
> wrote:
> 
>> Hey Brian,
>> 
>> That sounds Perfect! I like the idea of simply creating a RunnableInstance
>> and think it fits the architecture well!
>> +50 to proceed down that path.
>> 
>> Cheers,
>> Chris
>> 
>> On Nov 8, 2012, at 3:16 PM, Brian Foster wrote:
>> 
>>> 
>>> hey chris,
>>> 
>>> I think i work this out... i'm gonna create a RunnableInstance class...
>> this will then hold the state, start/end times, type (TASK or CONDITION),
>> and the class to run (WorkflowTaskInstance or WorkflowConditionInstance).
>>> 
>>> -brian
>>> 
>>> On Nov 7, 2012, at 3:05 PM, Brian Foster wrote:
>>> 
 hey chris,
 
 so i'm going through this WorkflowProcessor stuff and finishing it
>> up... I'm trying to make it so that WorkflowProcessor is used by all
>> Workflow Engines... WorkflowInstance holds the state, times, etc.. for a
>> Workflow, and then a WorkflowProcessor is response for analyzing that
>> WorkflowInstance and determining what to run next in the Workflow... in
>> this process i've notices that we really don't have any
>> "WorkflowInstance-like" class for Tasks and Conditions... A
>> WorkflowTaskInstance is actually the runnable task/condition job... I know
>> this would be a big structural change, but i think WorkflowTaskInstance and
>> WorkflowConditionInstance should be similar to WorkflowInstance (it should
>> hold the state, times, etc... for a task/condition)... then possibly
>> introduce a WorkflowTaskExecutable and WorkflowConditionExecutable
>> interface, or something like that, which CAS-PGE and all other runnable
>> tasks would implement instead... then i've already created a version of
>> WorkflowInstance which would hold the WorkflowTaskInstances and
>> WorkflowConditionInstance for it's task, preConds, and postConds... this
>> way when one loads a WorkflowInstance they would have access to all the
>> state, times, etc.. of everything in that workflow... this makes is very
>> easy to create a WorkflowProcessor which then is stateless and when given a
>> WorkflowInstance can easy determine what is the next thing to run or what
>> state the workflow is in... this new design i working toward also ditches
>> the whole WorkflowProcessorListener interface stuff, which i never liked in
>> the first place, but just never came up with something better.
 
 -brian
>>> 
>> 
>> 

Re: [oodt-dev] Re: ApacheCon NA 2013

[Mattmann, Chris A (388J)]

Very cool, Dan! Keep em' coming people! ;)

Cheers,
Chris

On Nov 8, 2012, at 6:06 AM, Crichton, Daniel J (4231) wrote:

> Hi Chris,
> 
> I submitted a talk on "Apache OODT: From the beginning to big data
> management" to discuss the origin, architectural principles, and path for
> going from a research project to a scalable, open source framework.
> 
> It would be great to get some talks and discuss how Apache OODT can
> continue to evolve to fit the needs of the big data management community.
> 
> Best,
> 
> Dan
> -- 
> 
> Dan Crichton
> Program Manager, Planetary Data System Engineering, Solar System
> Exploration
> Program Manager, Data Systems and Technology, Earth Science and Technology
> Principal Computer Scientist
> Jet Propulsion Laboratory
> (818) 354-9155
> 
> 
> 
> 
> 
> 
> 
> On 11/6/12 12:48 PM, "Mattmann, Chris A (388J)"
>  wrote:
> 
>> Hey Folks,
>> 
>> Please consider submitting an OODT related talk to ApacheCon NA 2013!
>> 
>> http://na.apachecon.com/
>> 
>> If you are using OODT, one of its developers; think it's cool; heard
>> about it somewhere
>> but don't know what it is, whatever! Please consider submitting a talk.
>> Many of the
>> Apache OODT PMC will be there and we'll be meeting up, happy to answer
>> questions
>> and talk about the project.
>> 
>> Thanks!
>> 
>> Cheers,
>> Chris
>> 
>> ++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattm...@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++
>> 
> 
> 


++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattm...@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
Phone: +1 (818) 354-8810
++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++