Hi Dimuthu, Thank you very much for your help.
Regards, Lahiru Jayathilake > On May 9, 2018, at 7:56 PM, DImuthu Upeksha <[email protected]> > wrote: > > Hi Lahiru > > This is a good analysis. I can see that you have gone through the code > thoroughly. Please find my comments in line and let me know if anything is > not clear. > > On Sat, May 5, 2018 at 6:39 PM, Lahiru Jayathilake <[email protected] > <mailto:[email protected]>> wrote: > Hi Everyone, > > I have few things to be clarified regarding the architecture of the workflow > managers and this is related to Airavata-2718 [1]. > > A Data Parsing Workflow Manager should be implemented and it should be > separated from the PostWorkflowManager(by this method a user has the ability > to run the DataParsingWorkflowManager or not). > My first question is, should there be a separate class for Data Parsing? Why > I am asking this is according to this code block [2] PostWorkflowManager > checks the TaskType and then only attach the relevant task to the tasks list. > If there are tasks related to the data parsing they must come to the tasks > list. In that case, I believe a separate workflow manager is not required. > > My assumption is this processModel [3] has all the tasks related to any of > the Workflow Managers. > > In the process model object, you don't have API level tasks related to data > parsing. Data parsing is currently performed outside the normal workflow. So > you have to come up with your own set of Helix Tasks and statically compose > the DAG using the Data Parsing Workflow Manager. > > > > > [The rest of the email is written assuming there should be a separate Data > Parsing Workflow Manager] > > In PostWorkflowManager, always a Completing Task is added to the end of the > tasks list [4]. The problem is if there are other tasks(such as tasks related > to the data parsing) they must come before the CompletingTask in order for a > successful workflow. (I raised this question in a GSoC student meeting and > Dimuthu gave his support to figure this out) > > As the solution, there should be a separate workflow manager just to attach > the completing task to the tasks list (suppose the workflow manager to be > CompletingTaskWorkflowManager). > Consider the following diagrams. > > Agreed > > > The current way of adding the completing task > <Screen Shot 2018-05-06 at 12.29.16 AM.png> > <Screen Shot 2018-05-06 at 12.29.16 AM.png> > The suggested way of adding tasks and the completing task > <Screen Shot 2018-05-06 at 12.36.24 AM.png> > > > In order to get the work done, there should be some kind of a communication > mechanism between workflow managers to attach tasks to the tasks list. One > way to do is make a connection from one workflow manager to the next workflow > manager, similar to the AiravataTask's architecture. > > This communication framework should be generic and you should think of how > you are going to deploy workflow managers (In single JVM or multiple JVMs). > However I'm not sure that we can convert Workflow Managers into Helix Tasks > as Helix Tasks are supposed to be finished after doing some work and Workflow > Managers are supposed be running in a loop. Instead, you can directly use > zookeeper to coordinate with Workflow Managers. > > > While I was going through the PostWorkflowManager's code base I realized > following changes have to be done. > According to the suggested method, there can be workflow managers other than > PostWorkflowManager then "launchWorkflow(...)" [5] and > "registerWorkflow(...)" [6] methods cannot be called inside > PostWorkflowManager. Since CompletingTaskWorkflowManager is the last one to > attach a task, it would be the best place to call the aforementioned methods. > No. These methods invoke the Workflow that is related to Post Processing. So > you should have these methods. However you might have to remove registering > completing task into this workflow [4]. > The list "allTasks" [7] has to be a common list for all the workflow managers > or the list should be passed through the workflow managers until the > CompletingTaskWorkflowManager receives it so the CompletingTaskWorkflowManger > can launch the workflow. > No. You don't have to pass the tasks that are not relevant to other workflow > managers. Each Workflow Manager creates tasks that are only relevant to it. > For example, Pre workflow Manager creates tasks like Environment Setup, Input > Staging and Job Submission while Post Workflow Manager only creates Output > Staging Tasks assuming that you transfer Completing Task into a new Workflow > Manager. > Can you confirm whether the above points are correct or wrong? and your > comments and suggestions would be very helpful. > > > [1] https://issues.apache.org/jira/browse/AIRAVATA-2718 > <https://issues.apache.org/jira/browse/AIRAVATA-2718> > [2] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L175 > > <https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L175> > [3] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L164 > > <https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L164> > [4] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L217 > > <https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L217> > > [5] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L219 > > <https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L219> > [6] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L222 > > <https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L222> > [7] > https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L168 > > <https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PostWorkflowManager.java#L168> > > Thanks & Regards, > -- > Lahiru Jayathilake > Department of Computer Science and Engineering, > Faculty of Engineering, > University of Moratuwa > > <https://lk.linkedin.com/in/lahirujayathilake>
