Thanks Chris! As you suggested previously, I am looking into CAS-PGE and plan is to reuse the same code. I have looked at PGETaskInstance where most of these pre/post task execution implementation resides. I believe FileManageFileStager class should be able to reuse easily. First I'll focus on the input fileStaging and then I am planning to focus on ingesting products and updating metadata as post execution task.
Best Regards, Sanjaya On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) < chris.a.mattm...@jpl.nasa.gov> wrote: > Hi Sanjaya, > > I would seriously recommend looking at OODT CAS-PGE, which already does > file staging, and connects to the file manager using queries. You could > wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot of > the existing FM to WM to RM and crawl infrastructure from OODT. > > Cheers, > Chris > > On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sanjaya...@gmail.com> wrote: > > >Thanks Lahiru! I have gone through the test classes and the classes in > >package org.apache.airavata.gfac. It was really helpful to understand the > >new architecture. I have listed down my approach based on new architecture > >to use Apache OODT products as an input to Airavata. > > 1. Introduce new Data Type to represent Apache OODT Product as an > >DataType. Basically new DataType is added into the GFacParameterTypes.xsd. > > 2. With new Architecture In Handlers and Out Handlers replaces > >the > >Pre/Post execution chains in old architecture. For the moment I am > >focusing > >on using Apache OODT product ID or file path as an input and stage the > >file > >(product) into host where actual execution happens. File staging requires > >to retrieve product from a File Manager server to the host where execution > >occurs. File staging can be implemented as an* In Handler* and needs to be > >configured as a new item in the list of configured In Handlers. > > 3. Handler should first verify the input parameter types listed > >in > >Service Description of the Application context of the > >*JobExecutionContext*. > >If input type matches the new parameter type, in handler stage file into > >host machine using Apache OODT file manager component. Corresponding input > >value can be retrieved from *In MessageContex*t. If a parameter type in > >MessageContext matches the new input type, then corresponding value is the > >id or file path to product managed by Apache OODT File Manager server. > > > >Best Regards, > >Sanjaya > > > >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake > ><glah...@gmail.com>wrote: > > > >> Hi Sanjaya, > >> > >> If you want to understand the new architecture by looking in to the > >>code, > >> please just refer the package org.apache.airavata.gfac, do not refer > >>any of > >> the classes in org.apache.airavata.core.gfac. > >> > >> Best place to start with is referring is from test classes > >> (LocalProviderTest,GramProviderTest), from tehre you can start looking > >>in > >> to GFacAPI class and see how to execution is flawing. > >> > >> if you have further questions please post on the list and more than > >>happy > >> to help. I will be doing some documentation about the architecture, > >>once I > >> am done, will post in to the list. And we will be having an architecture > >> review this week, so please watch the mailing list, if possible please > >>try > >> to join us. > >> > >> Regards > >> Lahiru > >> > >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa <sanjaya...@gmail.com > >> >wrote: > >> > >> > Thanks Suresh and Chris! It seems I am moving on the correct path. I > >>have > >> > followed the email thread on improved GFac architecture. Though I am > >>not > >> > entirely clear on the improved GFac architecture, proposed integration > >> with > >> > OODT is primarily based on the GFac extension, PreExecustionChain, > >>which > >> > has not been modified with the Architecture improvements (As per one > >>of > >> the > >> > replies from Lahiru, output extension is supported with new > >> Architecture. I > >> > assume input extension is also supported). > >> > > >> > I have looked into provenance manager and related implementation. > >>Still I > >> > am unclear how Airavata support provenance aware work flow processing. > >> > > >> > Best Regards, > >> > Sanjaya > >> > > >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sma...@apache.org> > >>wrote: > >> > > >> > > Hi Sanjaya, > >> > > > >> > > This sounds very exciting. Both Airavata and OODT projects have good > >> > > synergies and have been long looking for volunteers who can bridge > >>them > >> > > both. Please do not hesitate to ask any questions to either or both > >>the > >> > dev > >> > > lists. The more engaged you are, you will find use cases and > >>feedback > >> > which > >> > > should help your MSc project. > >> > > > >> > > Your plan sounds good. If you are following dev list, you may have > >> > > noticed, the GFac architecture has been improved to properly support > >> this > >> > > kind of handler architecture. > >> > > > >> > > You may also want to look at Airavata Registry API which has > >> organically > >> > > emerged with the need, but can greatly benefit for OODT's provenance > >> > > capabilities. > >> > > > >> > > Suresh > >> > > > >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" < > >> > > chris.a.mattm...@jpl.nasa.gov> wrote: > >> > > > >> > > > +1, sounds like a great idea Sanjaya! > >> > > > > >> > > > I'm copying dev@oodt so they can be in the conversation too. > >> > > > > >> > > > Cheers, > >> > > > Chris > >> > > > > >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjaya...@gmail.com> > >>wrote: > >> > > > > >> > > >> Hi Dev Team, > >> > > >> As I have posted previously, I am working on a Apache Airavata + > >> > Apache > >> > > >> OODT integration as my MSc project. Following is one of the > >>possible > >> > > >> integration to leverage Apache OODT file management capability > >>into > >> > > Apache > >> > > >> Airavata. Please review the proposal and let me know your > >>feedback. > >> > > >> > >> > > >> Proposal To Use Apache OODT products as input to Apache Airavata > >> > > Workflows > >> > > >> and staging product files into node where execution happens > >> > > >> > >> > > > >> > > >> > >>========================================================================= > >>= > >> > > >> ============================== > >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. > >>New > >> > > "Apache > >> > > >> OODT Product" input type can sepcify "Product ID" or "File Path > >>to > >> > > >> Product" > >> > > >> as an input to Apache Airavata workflows. > >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT Products > >> from > >> > > >> File > >> > > >> Manager Server managed using Apache OODT. > >> > > >> 1. Using Apache OODT File Manager componenet transfer Product > >>from > >> > > Server > >> > > >> to input directory of the application as configured using > >>XBaya-GUI > >> > > under > >> > > >> advanced configuration. (Here the assumption is that Products are > >> > > >> accesible > >> > > >> through Apache OODT File Manager server) > >> > > >> 2. Finally reset the input value to local file path. I think we > >>can > >> > > remove > >> > > >> the OODT Product parameter from invocation context and add new > >>file > >> > > >> parameter with value set to 'local path of the transferred > >> product'. I > >> > > am > >> > > >> not quite sure what are the implications of changing input > >>parameter > >> > > type > >> > > >> during the execution. > >> > > >> > >> > > >> Similar approach has been implemented for GridFTP and HTTP. > >> > > >> > >> > > >> Best Regards, > >> > > >> Sanjaya > >> > > > > >> > > > >> > > > >> > > >> > >> > >> > >> -- > >> System Analyst Programmer > >> PTI Lab > >> Indiana University > >> > >
