Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Amila, Please see my comments inline. On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara thejaka.am...@gmail.com wrote: Hello, I am sorry, I am bit late on this thread. But when reading through this thread I simply got lost, what this thread is discussing. I have few questions. 1. @Shameera : Is XWF actually a language to define workflow ? To my understanding it was an intermediate representation to convert workflow defined in UI to java object model. Was XWF ever used by any airavata user to define a workflow graph ? Yes, XWF is the language defined and used by Airavata to explain the workflows but it is not well documented. Both server and client sides read this description language to generate runtime java representation. so when you used XBaya to create a workflow with multiple applications, under the hood XBaya generate xwf which describe that workflow to server. From initial description what I understood is we are looking for a improved intermediate representation, not a language which describes workflows. 2. So what is the exact purpose of this proposed language ? - Is it to hide parallelism in workflows ? - Is it to replace the XBAYA functionalities ? (i hope not) Actually initial idea was to identify good, well-defined and scientific domain friendly language. Whole purpose of this effort is reduce the entry barrier of the end user. But later it is understood that introducing a new language won't fix our issue. 3. What are we trying to achieve by this proposed language which we cannot achieved through workflow UI tool ? 4. Who is going to use this language ? As I explained, our direction has been changed.By introducing a new language we are get nothing but nice description file. No functional improvements etc ... The current language should use all airavata client(Currently we have only XBaya) side applications to explain the workflow to the server side. 5. Why would a user prefer (assuming intended audience of proposed language is end users) a language over a Work Flow UI tool such as XBAYA ? (In other words what are the things we can do with language which we cannot do with UI) Let's say you are going to write a new web base client for Airavata which generate workflows and launch it. What you need to do is do some magic with UI and finally come up with description language and send it to server side. Here you need to learn how to write a valid XWF file and write your own JS code to generate it. But if airavata has a JavaScript API which can be used to generate XWF for you by getting JSON objects as input then it would be great help for client side developers. Thank, Shameera. Sorry, if above questions are in-appropriate, just wanted to understand what exactly needed. Thanks -Thejaka Amila On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva supu...@gmail.com wrote: I think I'm not suggesting to create a Workflow interpreter using Python etc. What I'm suggesting is to remove the Worflow aspect from core Airavata and move it to a more higher level component. The more I think about it, the model I'm suggesting is similar to what Hadoop, Storm etc has done for distributed system computations. This model is proven to be successful over the years. Keeping what Airavata does at its core can help you to build a more robust system. If we look at Airavata as middleware to execute applications on computing resources we can simplify what Airavata does and focus on improving the core functionality. All the successful systems have thrived on defining what it does at its core and keeping it simple and being excellent at what it does. In that regard keeping workflow aspect out of Airavata can help you to focus on the core problem. That is to execute an application on a remote computing resource in a fault tolerant and scalable manner. What I'm suggesting is to give the Orchestrator the capability to execute a Driver program that is specified by the user. (This program can be written in Python, Java or any other language). This driver program is similar to what you define in a Hadoop or Storm configuration. The driver program specifies the flow of the computation. It specifies what are the applications needs to be executed, how to manipulate input output. The driver program is the workflow for the user. Because the user specifies the program he can program it to handle workflow steering etc. Every time the user wants to execute this program he can tell Airavata to execute the Driver program. I'm also not 100% sure about all the details. But this can be a new way to look at how systems like Airavata should behave. Your thoughts and suggestions are more than welcome. Thanks, Supun.. On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi Supun, I think we were in two context, because I as suggesting a way to serialize and deserialize
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Supun, I think we were in two context, because I as suggesting a way to serialize and deserialize the workflow description while you are suggesting to implement some kind of workflow interpreter using Python, where Python client can send thrift calls to Airavata server to run the application. I can see with your suggested approach we can control the workflow execution process from client side which make it easy to implement workflow debugger.As you mentioned this is a major change to Airavata. So we should neatly think as this will change our existing architecture. Still if someone need to use different language java, php, JS etc ... to run the same workflow which generated by Python, we need a language independent workflow description. My initial question was what is the best language for this?. But as I have explained in one of my previous reply, It is not matter what language we used Either we can use XML or JSON to write this description, what matters is how easy to generate workflow with the provided API. Hence it would be great to have set of neat APIs in different languages. Thanks, Shameera. On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva supu...@gmail.com wrote: Hi Shameera, Using python is a radical approach for workflow I think. But I believe this is a very beautiful and new approach.Here is a possible scenario for implementation and running using a Python based library. The Python library facilitates the creation of Applications and submitting them to Airavata for execution. The underlying functionality is exactly similar to what Java clients provides.The only difference is that, Python library should have a more fluent API than Java for easy creation of workflows. We can generate the Python clients that talk to Airavata server using Thrift. Here is an example off the top of my head to a Python script created by user for a Workflow. This is a very crude example and we need to come up with a much better API if we are going to go along this path. First we need to write a Python script that can execute a workflow using Airavata. import airavata host = Host(localhost, ) app1 = Application(host, ) app2 = Application(host, ) # we will connect these applications as a workflow using some topology builder or other constructs wb = WorkFlowBuilder() wb.setApp(name1, app1) # you can do a simple output transformation here etc # connects the input of app1 to app2 etc wb.setApp(name2, app2).connectInput(name1) wb.submit() Now we can load this Python script from XBaya. When XBaya loads this script the Python script can output an XML configuration of the topology, XBaya can render. There are other ways like directly executing the Python script from command line and connecting XBaya indirectly as well. Now you can run the workflow from XBaya. Running the Workflow means just executing the Python script. XBaya gets the notifications through messaging and update the UI accordingly. The users need to write the Python script by hand. XBaya cannot create the script. Because workflow language is an actual python program the benefits are immense. For example user can do workflow steering in the workflow itself by subscribing to messages from Airavata. Thanks, Supun.. On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi Supun, I meant to say JS is a well-known client side scripting language i have messed scripting part. Even we use Python, ultimately we should convert this to java model in server side, somehow we need to serialized python representation to the language which java can parse and generate that model. In this case we need to parse python script in java isn't it? I am not exactly clear how you suggesting to use python here. More details on how end system works if we use Python would be great help to clearly understand your points. Thanks, Shameera. On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann chris.mattm...@gmail.com wrote: Have you guys considered using JCC [1] as a means to expose the workflow API currently in Java as a Python API? We are exploring its use in OODT, and we have already created a Tika [2] JCC-based python API. Cheers, Chris [1] http://lucene.apache.org/pylucene/jcc/ [2] http://github.com/chrismattmann/tika-python/ Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Supun Kamburugamuva supu...@gmail.com Reply-To: dev@oodt.apache.org Date: Thursday, October 16, 2014 at 3:43 PM To: dev d...@airavata.apache.org Cc: Alek Jones (Indiana) alek...@gmail.com, Suresh Marru sma...@apache.org, architect...@airavata.apache.org architect...@airavata.apache.org, dev@oodt.apache.org dev@oodt.apache.org Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata. Once we had an offline
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Supun, I meant to say JS is a well-known client side scripting language i have messed scripting part. Even we use Python, ultimately we should convert this to java model in server side, somehow we need to serialized python representation to the language which java can parse and generate that model. In this case we need to parse python script in java isn't it? I am not exactly clear how you suggesting to use python here. More details on how end system works if we use Python would be great help to clearly understand your points. Thanks, Shameera. On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann chris.mattm...@gmail.com wrote: Have you guys considered using JCC [1] as a means to expose the workflow API currently in Java as a Python API? We are exploring its use in OODT, and we have already created a Tika [2] JCC-based python API. Cheers, Chris [1] http://lucene.apache.org/pylucene/jcc/ [2] http://github.com/chrismattmann/tika-python/ Chris Mattmann chris.mattm...@gmail.com -Original Message- From: Supun Kamburugamuva supu...@gmail.com Reply-To: dev@oodt.apache.org Date: Thursday, October 16, 2014 at 3:43 PM To: dev d...@airavata.apache.org Cc: Alek Jones (Indiana) alek...@gmail.com, Suresh Marru sma...@apache.org, architect...@airavata.apache.org architect...@airavata.apache.org, dev@oodt.apache.org dev@oodt.apache.org Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata. Once we had an offline discussion about the Airavata Workflow language (with Milinda, Saliya and Shameera). In that discussion one thing came out was why we need to invent a different language when a simple library like Python will full fill of Airavata requirements. There are many benefits in using a Python library as the API for controlling Airavata workflows. 1. It is a library, gives the ultimate control over the execution and it can be simpler than any domain specific language that we can come with like XML, JSON etc 2. Most people use python and can learn it easily than any Domain specific language 3. You can easily create a Python library for Airavata because all the APIs are thrift based. 4. If you design the constructs correctly you can plug an XBaya. Any thoughts? Thanks, Supun.. On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva supu...@gmail.com wrote: Hi Shameera, Why you prefer JavaScript over a language like Python? Thanks, Supun.. On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi, First of all thanks everyone for giving valuable inputs. After doing some background search and talking to different people in the University who has used different workflow languages, I myself convinced that introducing an another workflow language is not what actually they need. By changing exiting workflow language to another will not solve problems. What they asking is a easy way to construct the workflows. Indirectly what they asking for a sort of API which they can use to generate the workflows and run it. Correct me if i am wrong here. As most of above replies depict, if we can get a simple API, as an example, for a web based application, JavaScript API would be a good solution, and probably JSON would be a good candidate for language, instead of XML. Airavata community already have started to implement web base GUI. Hence introducing a JSON base JavaScript API would be great help. WDYT? Thanks, Shameera. On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) alek...@gmail.com wrote: Hi, it is not dataflow instead focused on orchestrating REST services but you may find it useful datapoint - we created worfklow service that uses natively JavaScript and JSON to describe what happens during workflow execution: https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002 HTH, Alek On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru sma...@apache.org wrote: Hi Chris, Great to hear OODT community will be interested in adopting a JSON based workflow language and potentially a web based composer as well. Airavata previously had BPEL support initially through a home grown implementation [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by Alek on this topic is an interesting read. I am of the same opinion that we should adopt something more modern as the challenges from scientific workflows seems to be converging with the data flow patterns in business workflows. It will be great if we can all compile a list of potential candidates and hack them through. Suresh [1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1 [2] - http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run time_for_eScience_workflows [3] - http://www.computer.org/csdl/proceedings
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi, First of all thanks everyone for giving valuable inputs. After doing some background search and talking to different people in the University who has used different workflow languages, I myself convinced that introducing an another workflow language is not what actually they need. By changing exiting workflow language to another will not solve problems. What they asking is a easy way to construct the workflows. Indirectly what they asking for a sort of API which they can use to generate the workflows and run it. Correct me if i am wrong here. As most of above replies depict, if we can get a simple API, as an example, for a web based application, JavaScript API would be a good solution, and probably JSON would be a good candidate for language, instead of XML. Airavata community already have started to implement web base GUI. Hence introducing a JSON base JavaScript API would be great help. WDYT? Thanks, Shameera. On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) alek...@gmail.com wrote: Hi, it is not dataflow instead focused on orchestrating REST services but you may find it useful datapoint - we created worfklow service that uses natively JavaScript and JSON to describe what happens during workflow execution: https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002 HTH, Alek On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru sma...@apache.org wrote: Hi Chris, Great to hear OODT community will be interested in adopting a JSON based workflow language and potentially a web based composer as well. Airavata previously had BPEL support initially through a home grown implementation [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by Alek on this topic is an interesting read. I am of the same opinion that we should adopt something more modern as the challenges from scientific workflows seems to be converging with the data flow patterns in business workflows. It will be great if we can all compile a list of potential candidates and hack them through. Suresh [1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1 [2] - http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows [3] - http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Hi Guys, I've been interested in this too - we don't per have a specific OODT workflow language, but we specific workflows using XML, and other configuration (we are also thinking of moving to JSON for this). In the past I've also looked at YAWL and BPEL - both seem complex to me. I wonder at the end of the day if we should adopt something more modern like PIG or some other data flow type of language (PIG is really neat). Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Shameera Rathnayaka shameerai...@gmail.com Reply-To: architect...@airavata.apache.org architect...@airavata.apache.org Date: Thursday, September 18, 2014 8:26 AM To: architect...@airavata.apache.org architect...@airavata.apache.org, dev d...@airavata.apache.org Subject: Evaluate Suitable Scientific Workflow Language for Airavata. Hi All, As we all know Airavata has its own workflow language call XWF. When XWF was introduced, main focus points are interoperability and convertibility. But with years of experience it is convinced that above requirements are not really useful when we come to real world use cases. And XWF is XML based bulky language where we attache WSDLs and Workflow image it self. But with the recent changes WSDL part is being removed from XWF. It is worth to evaluate handy Scientific workflow languages in industry and find out pros and cons, at the end of this evaluation we need to come up with idea how we should improve Airavata workflow language, either we can improve existing XWF language, totally change to a new language available in industry or write a new light weight language. Basic requirements that we expect from new improvement are, high usability, flexible, light weight and real time monitoring support. As you can see above requirements are not direct comes with workflow languages but we need workflow language which
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Supun, Considering we are going to provide web base GUI support and JavaScript is a well-known client side language, I would select JavaScript over Python. I am not much familiar with Python, so would like to know the advantages we get by selecting python here. Thanks, Shameera. On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva supu...@gmail.com wrote: Hi Shameera, Why you prefer JavaScript over a language like Python? Thanks, Supun.. On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi, First of all thanks everyone for giving valuable inputs. After doing some background search and talking to different people in the University who has used different workflow languages, I myself convinced that introducing an another workflow language is not what actually they need. By changing exiting workflow language to another will not solve problems. What they asking is a easy way to construct the workflows. Indirectly what they asking for a sort of API which they can use to generate the workflows and run it. Correct me if i am wrong here. As most of above replies depict, if we can get a simple API, as an example, for a web based application, JavaScript API would be a good solution, and probably JSON would be a good candidate for language, instead of XML. Airavata community already have started to implement web base GUI. Hence introducing a JSON base JavaScript API would be great help. WDYT? Thanks, Shameera. On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) alek...@gmail.com wrote: Hi, it is not dataflow instead focused on orchestrating REST services but you may find it useful datapoint - we created worfklow service that uses natively JavaScript and JSON to describe what happens during workflow execution: https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002 HTH, Alek On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru sma...@apache.org wrote: Hi Chris, Great to hear OODT community will be interested in adopting a JSON based workflow language and potentially a web based composer as well. Airavata previously had BPEL support initially through a home grown implementation [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by Alek on this topic is an interesting read. I am of the same opinion that we should adopt something more modern as the challenges from scientific workflows seems to be converging with the data flow patterns in business workflows. It will be great if we can all compile a list of potential candidates and hack them through. Suresh [1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1 [2] - http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows [3] - http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) chris.a.mattm...@jpl.nasa.gov wrote: Hi Guys, I've been interested in this too - we don't per have a specific OODT workflow language, but we specific workflows using XML, and other configuration (we are also thinking of moving to JSON for this). In the past I've also looked at YAWL and BPEL - both seem complex to me. I wonder at the end of the day if we should adopt something more modern like PIG or some other data flow type of language (PIG is really neat). Cheers, Chris ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Shameera Rathnayaka shameerai...@gmail.com Reply-To: architect...@airavata.apache.org architect...@airavata.apache.org Date: Thursday, September 18, 2014 8:26 AM To: architect...@airavata.apache.org architect...@airavata.apache.org, dev d...@airavata.apache.org Subject: Evaluate Suitable Scientific Workflow Language for Airavata. Hi All, As we all know Airavata has its own workflow language call XWF. When XWF was introduced, main focus points are interoperability and convertibility. But with years of experience it is convinced that above requirements are not really useful when we come to real world use cases. And XWF is XML based bulky language where we attache WSDLs and Workflow image it self. But with the recent changes WSDL part is being removed from XWF. It is worth to evaluate handy Scientific workflow
