Thanks Chris! As you suggested best way to reuse the PgeTaskInstance is to sub-class, as it contains several protected methods which are very useful for integrating Apache OODT with workflow execution.
Best Regards, Sanjaya On Fri, Feb 22, 2013 at 9:22 AM, Mattmann, Chris A (388J) < [email protected]> wrote: > Hi Sanjaya, > > Great. You may simply want to extend (sub-class) PgeTaskInstance, and > create one for Airavata? > > Cheers, > Chris > > On 2/20/13 10:29 AM, "Sanjaya Medonsa" <[email protected]> wrote: > > >Thanks Chris! As you suggested previously, I am looking into CAS-PGE and > >plan is to reuse the same code. I have looked at PGETaskInstance where > >most > >of these pre/post task execution implementation resides. I believe > >FileManageFileStager class should be able to reuse easily. First I'll > >focus on the input fileStaging and then I am planning to focus on > >ingesting > >products and updating metadata as post execution task. > > > >Best Regards, > >Sanjaya > > > >On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) < > >[email protected]> wrote: > > > >> Hi Sanjaya, > >> > >> I would seriously recommend looking at OODT CAS-PGE, which already does > >> file staging, and connects to the file manager using queries. You could > >> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot > >>of > >> the existing FM to WM to RM and crawl infrastructure from OODT. > >> > >> Cheers, > >> Chris > >> > >> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <[email protected]> wrote: > >> > >> >Thanks Lahiru! I have gone through the test classes and the classes in > >> >package org.apache.airavata.gfac. It was really helpful to understand > >>the > >> >new architecture. I have listed down my approach based on new > >>architecture > >> >to use Apache OODT products as an input to Airavata. > >> > 1. Introduce new Data Type to represent Apache OODT Product > >>as an > >> >DataType. Basically new DataType is added into the > >>GFacParameterTypes.xsd. > >> > 2. With new Architecture In Handlers and Out Handlers replaces > >> >the > >> >Pre/Post execution chains in old architecture. For the moment I am > >> >focusing > >> >on using Apache OODT product ID or file path as an input and stage the > >> >file > >> >(product) into host where actual execution happens. File staging > >>requires > >> >to retrieve product from a File Manager server to the host where > >>execution > >> >occurs. File staging can be implemented as an* In Handler* and needs > >>to be > >> >configured as a new item in the list of configured In Handlers. > >> > 3. Handler should first verify the input parameter types > >>listed > >> >in > >> >Service Description of the Application context of the > >> >*JobExecutionContext*. > >> >If input type matches the new parameter type, in handler stage file > >>into > >> >host machine using Apache OODT file manager component. Corresponding > >>input > >> >value can be retrieved from *In MessageContex*t. If a parameter type in > >> >MessageContext matches the new input type, then corresponding value is > >>the > >> >id or file path to product managed by Apache OODT File Manager server. > >> > > >> >Best Regards, > >> >Sanjaya > >> > > >> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake > >> ><[email protected]>wrote: > >> > > >> >> Hi Sanjaya, > >> >> > >> >> If you want to understand the new architecture by looking in to the > >> >>code, > >> >> please just refer the package org.apache.airavata.gfac, do not refer > >> >>any of > >> >> the classes in org.apache.airavata.core.gfac. > >> >> > >> >> Best place to start with is referring is from test classes > >> >> (LocalProviderTest,GramProviderTest), from tehre you can start > >>looking > >> >>in > >> >> to GFacAPI class and see how to execution is flawing. > >> >> > >> >> if you have further questions please post on the list and more than > >> >>happy > >> >> to help. I will be doing some documentation about the architecture, > >> >>once I > >> >> am done, will post in to the list. And we will be having an > >>architecture > >> >> review this week, so please watch the mailing list, if possible > >>please > >> >>try > >> >> to join us. > >> >> > >> >> Regards > >> >> Lahiru > >> >> > >> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa > >><[email protected] > >> >> >wrote: > >> >> > >> >> > Thanks Suresh and Chris! It seems I am moving on the correct path. > >>I > >> >>have > >> >> > followed the email thread on improved GFac architecture. Though I > >>am > >> >>not > >> >> > entirely clear on the improved GFac architecture, proposed > >>integration > >> >> with > >> >> > OODT is primarily based on the GFac extension, PreExecustionChain, > >> >>which > >> >> > has not been modified with the Architecture improvements (As per > >>one > >> >>of > >> >> the > >> >> > replies from Lahiru, output extension is supported with new > >> >> Architecture. I > >> >> > assume input extension is also supported). > >> >> > > >> >> > I have looked into provenance manager and related implementation. > >> >>Still I > >> >> > am unclear how Airavata support provenance aware work flow > >>processing. > >> >> > > >> >> > Best Regards, > >> >> > Sanjaya > >> >> > > >> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <[email protected]> > >> >>wrote: > >> >> > > >> >> > > Hi Sanjaya, > >> >> > > > >> >> > > This sounds very exciting. Both Airavata and OODT projects have > >>good > >> >> > > synergies and have been long looking for volunteers who can > >>bridge > >> >>them > >> >> > > both. Please do not hesitate to ask any questions to either or > >>both > >> >>the > >> >> > dev > >> >> > > lists. The more engaged you are, you will find use cases and > >> >>feedback > >> >> > which > >> >> > > should help your MSc project. > >> >> > > > >> >> > > Your plan sounds good. If you are following dev list, you may > >>have > >> >> > > noticed, the GFac architecture has been improved to properly > >>support > >> >> this > >> >> > > kind of handler architecture. > >> >> > > > >> >> > > You may also want to look at Airavata Registry API which has > >> >> organically > >> >> > > emerged with the need, but can greatly benefit for OODT's > >>provenance > >> >> > > capabilities. > >> >> > > > >> >> > > Suresh > >> >> > > > >> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" < > >> >> > > [email protected]> wrote: > >> >> > > > >> >> > > > +1, sounds like a great idea Sanjaya! > >> >> > > > > >> >> > > > I'm copying dev@oodt so they can be in the conversation too. > >> >> > > > > >> >> > > > Cheers, > >> >> > > > Chris > >> >> > > > > >> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <[email protected]> > >> >>wrote: > >> >> > > > > >> >> > > >> Hi Dev Team, > >> >> > > >> As I have posted previously, I am working on a Apache > >>Airavata + > >> >> > Apache > >> >> > > >> OODT integration as my MSc project. Following is one of the > >> >>possible > >> >> > > >> integration to leverage Apache OODT file management capability > >> >>into > >> >> > > Apache > >> >> > > >> Airavata. Please review the proposal and let me know your > >> >>feedback. > >> >> > > >> > >> >> > > >> Proposal To Use Apache OODT products as input to Apache > >>Airavata > >> >> > > Workflows > >> >> > > >> and staging product files into node where execution happens > >> >> > > >> > >> >> > > > >> >> > > >> >> > >> > >>>>======================================================================= > >>>>== > >> >>= > >> >> > > >> ============================== > >> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. > >> >>New > >> >> > > "Apache > >> >> > > >> OODT Product" input type can sepcify "Product ID" or "File > >>Path > >> >>to > >> >> > > >> Product" > >> >> > > >> as an input to Apache Airavata workflows. > >> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT > >>Products > >> >> from > >> >> > > >> File > >> >> > > >> Manager Server managed using Apache OODT. > >> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product > >> >>from > >> >> > > Server > >> >> > > >> to input directory of the application as configured using > >> >>XBaya-GUI > >> >> > > under > >> >> > > >> advanced configuration. (Here the assumption is that Products > >>are > >> >> > > >> accesible > >> >> > > >> through Apache OODT File Manager server) > >> >> > > >> 2. Finally reset the input value to local file path. I think > >>we > >> >>can > >> >> > > remove > >> >> > > >> the OODT Product parameter from invocation context and add new > >> >>file > >> >> > > >> parameter with value set to 'local path of the transferred > >> >> product'. I > >> >> > > am > >> >> > > >> not quite sure what are the implications of changing input > >> >>parameter > >> >> > > type > >> >> > > >> during the execution. > >> >> > > >> > >> >> > > >> Similar approach has been implemented for GridFTP and HTTP. > >> >> > > >> > >> >> > > >> Best Regards, > >> >> > > >> Sanjaya > >> >> > > > > >> >> > > > >> >> > > > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> System Analyst Programmer > >> >> PTI Lab > >> >> Indiana University > >> >> > >> > >> > >
