Diagram can be found here[1]. thanks Milinda
[1] https://issues.apache.org/jira/secure/attachment/12539096/GFac%20Improvements.png On Fri, Aug 3, 2012 at 4:47 PM, Milinda Pathirage < [email protected]> wrote: > Hi Suresh, > > Sorry I forgot that we can't send attachements to the list. Will attach > the diagram to issue [1]. > > Thanks > Milinda > > [1] https://issues.apache.org/jira/browse/AIRAVATA-477 > > > On Fri, Aug 3, 2012 at 4:43 PM, Suresh Marru <[email protected]> wrote: > >> Hi Milinda, >> >> Thanks for the detailed feedback. I did not read through in detail yet, >> but did you intend any visual diagram? If so, since you cannot attach >> files, can you please crate a jira on this improvement and attach to it? >> >> Thanks, >> Suresh >> On Aug 3, 2012, at 4:31 PM, Milinda Pathirage wrote: >> >> > Hi Devs, >> > >> > I found out several issues in internal architecture of current GFac >> > implementation while implementing cloud bursting support for GFac. As a >> > solution to these issues and to make it easy to extend GFac to support >> > different scenarios involve in cloud bursting, I did several >> improvements >> > to internal architecture. Below are the changes I did. Please feel free >> to >> > comment on new improvements. >> > >> > *Handler Based Architecture >> > *Execution of a GFac job includes step like input file staging, >> > initializing execution environment(cluster setup, etc.) and output file >> > staging. In the current trunk implementation some of these steps(input >> data >> > movement) happen inside provider initialization and it limits the >> > flexibility of having different execution sequences. For example in >> Hadoop >> > cloud bursting scenario, we may have to setup HDFS cluster first then >> move >> > the data to HDFS and execute the Hadoop job. So in this case we need to >> > have flexibility to setup HDFS cluster before doing any data movements. >> If >> > we follow the current implementation pattern, we had to implement >> support >> > for these different scenarios in the providers initialization method and >> > this will make provider logic complex. >> > >> > Also handler based architecture allow us to reuse already available >> > handlers in different execution flows. For example we can have GridFTP >> to >> > Amazon S3 data movement handler which can be used with different >> > providers(Hadoop, etc.). >> > >> > Another improvement we can do on top of handler architecture is >> dynamically >> > changing handler execution sequence based on application description. >> > Initial implementation will come with the static handler sequence. But >> I am >> > planning to scheduler which decides handler sequence based on >> application >> > description. >> > >> > Flow of information between handlers can be done via parameters and >> handler >> > implementor should document required parameters by the handler and what >> are >> > the parameters that handler will make available to other handlers. >> > >> > Following is a visual representation of new handler architecture. >> > >> > ** >> > >> > >> > *Changes to context hierarchy >> > *JobExecutionContext carries the data and configuration information >> related >> > to single job execution. JobExecutionContext will contain input and >> output >> > MessageContext instances which carry data related to input and output. >> > There will be a SecurityContext which can be used to set security >> related >> > properties to each message context. ApplicationContext contains >> information >> > related to application and there will be a ApplicationContext associated >> > with JobExecutionContext. >> > >> > >> > New GFac code can be found at [1]. I am done with the initial >> > implementation and we need to do following things before first release >> of >> > new GFac core. >> > >> > - Migrate existing code to new implementation >> > - Complete workflow notifier based on new EventBus API >> > - Migrate Hadoop provider and Whirr based cloud deployment code to new >> > architecture >> > >> > >> > Thanks >> > Milinda >> > >> > >> > [1] >> > >> https://svn.codespot.com/a/apache-extras.org/airavata-gsoc-sandbox/trunk/milinda/gfac-core >> > >> > -- >> > Milinda Pathirage >> > PhD Student Indiana University, Bloomington; >> > E-mail: [email protected] >> > Web: http://mpathirage.com >> > Blog: http://blog.mpathirage.com >> >> > > > -- > Milinda Pathirage > PhD Student Indiana University, Bloomington; > E-mail: [email protected] > Web: http://mpathirage.com > Blog: http://blog.mpathirage.com > -- Milinda Pathirage PhD Student Indiana University, Bloomington; E-mail: [email protected] Web: http://mpathirage.com Blog: http://blog.mpathirage.com
