Hi Sanjaya, Great. You may simply want to extend (sub-class) PgeTaskInstance, and create one for Airavata?
Cheers, Chris On 2/20/13 10:29 AM, "Sanjaya Medonsa" <sanjaya...@gmail.com> wrote: >Thanks Chris! As you suggested previously, I am looking into CAS-PGE and >plan is to reuse the same code. I have looked at PGETaskInstance where >most >of these pre/post task execution implementation resides. I believe >FileManageFileStager class should be able to reuse easily. First I'll >focus on the input fileStaging and then I am planning to focus on >ingesting >products and updating metadata as post execution task. > >Best Regards, >Sanjaya > >On Wed, Feb 20, 2013 at 9:00 PM, Mattmann, Chris A (388J) < >chris.a.mattm...@jpl.nasa.gov> wrote: > >> Hi Sanjaya, >> >> I would seriously recommend looking at OODT CAS-PGE, which already does >> file staging, and connects to the file manager using queries. You could >> wrap or sub-class CAS-PGE in Airavata and I think avoid rewriting a lot >>of >> the existing FM to WM to RM and crawl infrastructure from OODT. >> >> Cheers, >> Chris >> >> On 2/19/13 10:52 AM, "Sanjaya Medonsa" <sanjaya...@gmail.com> wrote: >> >> >Thanks Lahiru! I have gone through the test classes and the classes in >> >package org.apache.airavata.gfac. It was really helpful to understand >>the >> >new architecture. I have listed down my approach based on new >>architecture >> >to use Apache OODT products as an input to Airavata. >> > 1. Introduce new Data Type to represent Apache OODT Product >>as an >> >DataType. Basically new DataType is added into the >>GFacParameterTypes.xsd. >> > 2. With new Architecture In Handlers and Out Handlers replaces >> >the >> >Pre/Post execution chains in old architecture. For the moment I am >> >focusing >> >on using Apache OODT product ID or file path as an input and stage the >> >file >> >(product) into host where actual execution happens. File staging >>requires >> >to retrieve product from a File Manager server to the host where >>execution >> >occurs. File staging can be implemented as an* In Handler* and needs >>to be >> >configured as a new item in the list of configured In Handlers. >> > 3. Handler should first verify the input parameter types >>listed >> >in >> >Service Description of the Application context of the >> >*JobExecutionContext*. >> >If input type matches the new parameter type, in handler stage file >>into >> >host machine using Apache OODT file manager component. Corresponding >>input >> >value can be retrieved from *In MessageContex*t. If a parameter type in >> >MessageContext matches the new input type, then corresponding value is >>the >> >id or file path to product managed by Apache OODT File Manager server. >> > >> >Best Regards, >> >Sanjaya >> > >> >On Tue, Feb 19, 2013 at 1:29 AM, Lahiru Gunathilake >> ><glah...@gmail.com>wrote: >> > >> >> Hi Sanjaya, >> >> >> >> If you want to understand the new architecture by looking in to the >> >>code, >> >> please just refer the package org.apache.airavata.gfac, do not refer >> >>any of >> >> the classes in org.apache.airavata.core.gfac. >> >> >> >> Best place to start with is referring is from test classes >> >> (LocalProviderTest,GramProviderTest), from tehre you can start >>looking >> >>in >> >> to GFacAPI class and see how to execution is flawing. >> >> >> >> if you have further questions please post on the list and more than >> >>happy >> >> to help. I will be doing some documentation about the architecture, >> >>once I >> >> am done, will post in to the list. And we will be having an >>architecture >> >> review this week, so please watch the mailing list, if possible >>please >> >>try >> >> to join us. >> >> >> >> Regards >> >> Lahiru >> >> >> >> On Mon, Feb 18, 2013 at 1:25 PM, Sanjaya Medonsa >><sanjaya...@gmail.com >> >> >wrote: >> >> >> >> > Thanks Suresh and Chris! It seems I am moving on the correct path. >>I >> >>have >> >> > followed the email thread on improved GFac architecture. Though I >>am >> >>not >> >> > entirely clear on the improved GFac architecture, proposed >>integration >> >> with >> >> > OODT is primarily based on the GFac extension, PreExecustionChain, >> >>which >> >> > has not been modified with the Architecture improvements (As per >>one >> >>of >> >> the >> >> > replies from Lahiru, output extension is supported with new >> >> Architecture. I >> >> > assume input extension is also supported). >> >> > >> >> > I have looked into provenance manager and related implementation. >> >>Still I >> >> > am unclear how Airavata support provenance aware work flow >>processing. >> >> > >> >> > Best Regards, >> >> > Sanjaya >> >> > >> >> > On Mon, Feb 18, 2013 at 6:35 AM, Suresh Marru <sma...@apache.org> >> >>wrote: >> >> > >> >> > > Hi Sanjaya, >> >> > > >> >> > > This sounds very exciting. Both Airavata and OODT projects have >>good >> >> > > synergies and have been long looking for volunteers who can >>bridge >> >>them >> >> > > both. Please do not hesitate to ask any questions to either or >>both >> >>the >> >> > dev >> >> > > lists. The more engaged you are, you will find use cases and >> >>feedback >> >> > which >> >> > > should help your MSc project. >> >> > > >> >> > > Your plan sounds good. If you are following dev list, you may >>have >> >> > > noticed, the GFac architecture has been improved to properly >>support >> >> this >> >> > > kind of handler architecture. >> >> > > >> >> > > You may also want to look at Airavata Registry API which has >> >> organically >> >> > > emerged with the need, but can greatly benefit for OODT's >>provenance >> >> > > capabilities. >> >> > > >> >> > > Suresh >> >> > > >> >> > > On Feb 17, 2013, at 1:25 PM, "Mattmann, Chris A (388J)" < >> >> > > chris.a.mattm...@jpl.nasa.gov> wrote: >> >> > > >> >> > > > +1, sounds like a great idea Sanjaya! >> >> > > > >> >> > > > I'm copying dev@oodt so they can be in the conversation too. >> >> > > > >> >> > > > Cheers, >> >> > > > Chris >> >> > > > >> >> > > > On 2/17/13 10:22 AM, "Sanjaya Medonsa" <sanjaya...@gmail.com> >> >>wrote: >> >> > > > >> >> > > >> Hi Dev Team, >> >> > > >> As I have posted previously, I am working on a Apache >>Airavata + >> >> > Apache >> >> > > >> OODT integration as my MSc project. Following is one of the >> >>possible >> >> > > >> integration to leverage Apache OODT file management capability >> >>into >> >> > > Apache >> >> > > >> Airavata. Please review the proposal and let me know your >> >>feedback. >> >> > > >> >> >> > > >> Proposal To Use Apache OODT products as input to Apache >>Airavata >> >> > > Workflows >> >> > > >> and staging product files into node where execution happens >> >> > > >> >> >> > > >> >> > >> >> >> >>>>======================================================================= >>>>== >> >>= >> >> > > >> ============================== >> >> > > >> 1. Introduce "Apache OODT Product" as a new GFacParameterType. >> >>New >> >> > > "Apache >> >> > > >> OODT Product" input type can sepcify "Product ID" or "File >>Path >> >>to >> >> > > >> Product" >> >> > > >> as an input to Apache Airavata workflows. >> >> > > >> 2. Introduce new PreExecuteChain to retrieve Apache OODT >>Products >> >> from >> >> > > >> File >> >> > > >> Manager Server managed using Apache OODT. >> >> > > >> 1. Using Apache OODT File Manager componenet transfer Product >> >>from >> >> > > Server >> >> > > >> to input directory of the application as configured using >> >>XBaya-GUI >> >> > > under >> >> > > >> advanced configuration. (Here the assumption is that Products >>are >> >> > > >> accesible >> >> > > >> through Apache OODT File Manager server) >> >> > > >> 2. Finally reset the input value to local file path. I think >>we >> >>can >> >> > > remove >> >> > > >> the OODT Product parameter from invocation context and add new >> >>file >> >> > > >> parameter with value set to 'local path of the transferred >> >> product'. I >> >> > > am >> >> > > >> not quite sure what are the implications of changing input >> >>parameter >> >> > > type >> >> > > >> during the execution. >> >> > > >> >> >> > > >> Similar approach has been implemented for GridFTP and HTTP. >> >> > > >> >> >> > > >> Best Regards, >> >> > > >> Sanjaya >> >> > > > >> >> > > >> >> > > >> >> > >> >> >> >> >> >> >> >> -- >> >> System Analyst Programmer >> >> PTI Lab >> >> Indiana University >> >> >> >>
