Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Amila Jayasekara]

>>> > to write a Python script that can execute a workflow using Airavata.
> >>> >
> >>> > import airavata
> >>> >
> >>> > host = Host("localhost", )
> >>> > app1 = Application(host, )
> >>> > app2 = Application(host, )
> >>> >
> >>> > # we will connect these applications as a workflow using some
> topology
> >>> > builder or other constructs
> >>> >
> >>> > wb = WorkFlowBuilder()
> >>> > wb.setApp("name1", app1)
> >>> > # you can do a simple output transformation here etc
> >>> >
> >>> > # connects the input of app1 to app2 etc
> >>> > wb.setApp("name2", app2).connectInput("name1")
> >>> >
> >>> > wb.submit()
> >>> >
> >>> > Now we can load this Python script from XBaya. When XBaya loads this
> >>> script
> >>> > the Python script can output an XML configuration of the topology,
> >>> XBaya
> >>> > can render. There are other ways like directly executing the Python
> >>> script
> >>> > from command line and connecting XBaya indirectly as well. Now you
> can
> >>> run
> >>> > the workflow from XBaya. Running the Workflow means just executing
> the
> >>> > Python script.
> >>> >
> >>> > XBaya gets the notifications through messaging and update the UI
> >>> > accordingly.
> >>> >
> >>> > The users need to write the Python script by hand. XBaya cannot
> create
> >>> the
> >>> > script. Because workflow language is an actual python program the
> >>> benefits
> >>> > are immense. For example user can do workflow steering in the
> workflow
> >>> > itself by subscribing to messages from Airavata.
> >>> >
> >>> > Thanks,
> >>> > Supun..
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> >>> > shameerai...@gmail.com> wrote:
> >>> >
> >>> > > Hi Supun,
> >>> > >
> >>> > > I meant to say JS is a well-known client side scripting language i
> >>> have
> >>> > > messed scripting part. Even we use Python, ultimately we should
> >>> convert
> >>> > > this to java model in server side, somehow we need to serialized
> >>> python
> >>> > > representation to the language which java can parse and generate
> that
> >>> > > model. In this case we need to parse python script in java isn't
> it?
> >>> I am
> >>> > > not exactly clear how you suggesting to use python here. More
> >>> details on
> >>> > > how end system works if we use Python would be great help to
> clearly
> >>> > > understand your points.
> >>> > >
> >>> > > Thanks,
> >>> > > Shameera.
> >>> > >
> >>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> >>> > chris.mattm...@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Have you guys considered using JCC [1] as a means
> >>> > > > to expose the workflow API currently in Java as a
> >>> > > > Python API?
> >>> > > >
> >>> > > > We are exploring its use in OODT, and we have already
> >>> > > > created a Tika [2] JCC-based python API.
> >>> > > >
> >>> > > > Cheers,
> >>> > > > Chris
> >>> > > >
> >>> > > > [1] http://lucene.apache.org/pylucene/jcc/
> >>> > > > [2] http://github.com/chrismattmann/tika-python/
> >>> > > >
> >>> > > > 
> >>> > > > Chris Mattmann
> >>> > > > chris.mattm...@gmail.com
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > -Original Message-
> >>> > > > From: Supun Kamburu

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Lahiru Gunathilake]

;> server
>>>> > using Thrift.
>>>> >
>>>> > Here is an example off the top of my head to a Python script created
>>>> by
>>>> > user for a Workflow. This is a very crude example and we need to come
>>>> up
>>>> > with a much better API if we are going to go along this path. First
>>>> we need
>>>> > to write a Python script that can execute a workflow using Airavata.
>>>> >
>>>> > import airavata
>>>> >
>>>> > host = Host("localhost", )
>>>> > app1 = Application(host, )
>>>> > app2 = Application(host, )
>>>> >
>>>> > # we will connect these applications as a workflow using some topology
>>>> > builder or other constructs
>>>> >
>>>> > wb = WorkFlowBuilder()
>>>> > wb.setApp("name1", app1)
>>>> > # you can do a simple output transformation here etc
>>>> >
>>>> > # connects the input of app1 to app2 etc
>>>> > wb.setApp("name2", app2).connectInput("name1")
>>>> >
>>>> > wb.submit()
>>>> >
>>>> > Now we can load this Python script from XBaya. When XBaya loads this
>>>> script
>>>> > the Python script can output an XML configuration of the topology,
>>>> XBaya
>>>> > can render. There are other ways like directly executing the Python
>>>> script
>>>> > from command line and connecting XBaya indirectly as well. Now you
>>>> can run
>>>> > the workflow from XBaya. Running the Workflow means just executing the
>>>> > Python script.
>>>> >
>>>> > XBaya gets the notifications through messaging and update the UI
>>>> > accordingly.
>>>> >
>>>> > The users need to write the Python script by hand. XBaya cannot
>>>> create the
>>>> > script. Because workflow language is an actual python program the
>>>> benefits
>>>> > are immense. For example user can do workflow steering in the workflow
>>>> > itself by subscribing to messages from Airavata.
>>>> >
>>>> > Thanks,
>>>> > Supun..
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>>> > shameerai...@gmail.com> wrote:
>>>> >
>>>> > > Hi Supun,
>>>> > >
>>>> > > I meant to say JS is a well-known client side scripting language i
>>>> have
>>>> > > messed scripting part. Even we use Python, ultimately we should
>>>> convert
>>>> > > this to java model in server side, somehow we need to serialized
>>>> python
>>>> > > representation to the language which java can parse and generate
>>>> that
>>>> > > model. In this case we need to parse python script in java isn't
>>>> it? I am
>>>> > > not exactly clear how you suggesting to use python here. More
>>>> details on
>>>> > > how end system works if we use Python would be great help to clearly
>>>> > > understand your points.
>>>> > >
>>>> > > Thanks,
>>>> > > Shameera.
>>>> > >
>>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>>> > chris.mattm...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Have you guys considered using JCC [1] as a means
>>>> > > > to expose the workflow API currently in Java as a
>>>> > > > Python API?
>>>> > > >
>>>> > > > We are exploring its use in OODT, and we have already
>>>> > > > created a Tika [2] JCC-based python API.
>>>> > > >
>>>> > > > Cheers,
>>>> > > > Chris
>>>> > > >
>>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>>> > > >
>>>> > > > 
>>>> > > > Chris Mattmann
>>>>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[K Yoshimoto]

I prefer python myself, but if the API is clean, it should be
usable from a large number of languages on the client side.

On Thu, Oct 16, 2014 at 06:46:36PM -0400, Shameera Rathnayaka wrote:
> Hi Supun,
> 
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
> 
> 
> Thanks,
> Shameera.
> 
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
> wrote:
> 
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerai...@gmail.com> wrote:
> >
> >> ​Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing some
> >> background search and talking to different people in the University who has
> >> used different workflow languages, I myself convinced that introducing an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language, instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alek...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:
> >>>
>  Hi Chris,
> 
>  Great to hear OODT community will be interested in adopting a JSON
>  based workflow language and potentially a web based composer as well.
>  Airavata previously had BPEL support initially through a home grown
>  implementation [1] by Alek Slominski and later through Apache ODE [2]. 
>  Also
>  a white paper [3] by Alek on this topic is an interesting read.
> 
>  I am of the same opinion that we should adopt something more modern as
>  the challenges from scientific workflows seems to be converging with the
>  data flow patterns in business workflows.
> 
>  It will be great if we can all compile a list of potential candidates
>  and hack them through.
> 
>  Suresh
>  [1] -
>  http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>  [2] -
>  http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>  [3] -
>  http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> 
> 
>  On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>  chris.a.mattm...@jpl.nasa.gov> wrote:
> 
>  > Hi Guys,
>  >
>  > I've been interested in this too - we don't per have a specific
>  > OODT workflow language, but we specific workflows using XML, and
>  > other configuration (we are also thinking of moving to JSON for
>  > this).
>  >
>  > In the past I've also looked at YAWL and BPEL - both seem complex
>  > to me.
>  >
>  > I wonder at the end of the day if we should adopt something more
>  > modern like PIG or some other data flow type of language (PIG
>  > is really neat).
>  >
>  > Cheers,
>  > Chris
>  >
>  >
>  > ++
>  > Chris Mattmann, Ph.D.
>  > Chief Architect
>  > Instrument Software and Science Data Systems Section (398)
>  > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>  > Office: 168-519, Mailstop: 168-527
>  > Email: chris.a.mattm...@nasa.gov
>  > WWW:  http://sunset.usc.edu/~mattmann/
>  > ++
>  > Adjunct Associate Professor, Computer Science Department
>  > University of Southern California, Los Angeles, CA 90089 USA
>  > ++
>  >
>  >
>  >
>  >
>  >
>  >
>  > -Original Message-
>  > From: Shameera Rathnayaka 
>  > Reply

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Shameera Rathnayaka]

ipt by hand. XBaya cannot create
>>> the
>>> > script. Because workflow language is an actual python program the
>>> benefits
>>> > are immense. For example user can do workflow steering in the workflow
>>> > itself by subscribing to messages from Airavata.
>>> >
>>> > Thanks,
>>> > Supun..
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>>> > shameerai...@gmail.com> wrote:
>>> >
>>> > > Hi Supun,
>>> > >
>>> > > I meant to say JS is a well-known client side scripting language i
>>> have
>>> > > messed scripting part. Even we use Python, ultimately we should
>>> convert
>>> > > this to java model in server side, somehow we need to serialized
>>> python
>>> > > representation to the language which java can parse and generate that
>>> > > model. In this case we need to parse python script in java isn't it?
>>> I am
>>> > > not exactly clear how you suggesting to use python here. More
>>> details on
>>> > > how end system works if we use Python would be great help to clearly
>>> > > understand your points.
>>> > >
>>> > > Thanks,
>>> > > Shameera.
>>> > >
>>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>>> > chris.mattm...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Have you guys considered using JCC [1] as a means
>>> > > > to expose the workflow API currently in Java as a
>>> > > > Python API?
>>> > > >
>>> > > > We are exploring its use in OODT, and we have already
>>> > > > created a Tika [2] JCC-based python API.
>>> > > >
>>> > > > Cheers,
>>> > > > Chris
>>> > > >
>>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>>> > > > [2] http://github.com/chrismattmann/tika-python/
>>> > > >
>>> > > > 
>>> > > > Chris Mattmann
>>> > > > chris.mattm...@gmail.com
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > -Original Message-
>>> > > > From: Supun Kamburugamuva 
>>> > > > Reply-To: 
>>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>>> > > > To: dev 
>>> > > > Cc: "Alek Jones (Indiana)" , Suresh Marru
>>> > > > , "architect...@airavata.apache.org"
>>> > > > , "dev@oodt.apache.org"
>>> > > > 
>>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>>> > Airavata.
>>> > > >
>>> > > > >Once we had an offline discussion about the Airavata Workflow
>>> language
>>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>>> came
>>> > > out
>>> > > > >was why we need to invent a different language when a simple
>>> library
>>> > > like
>>> > > > >Python will full fill of Airavata requirements.
>>> > > > >
>>> > > > >There are many benefits in using a Python library as the API for
>>> > > > >controlling Airavata workflows.
>>> > > > >
>>> > > > >1. It is a library, gives the ultimate control over the execution
>>> and
>>> > it
>>> > > > >can be simpler than any domain specific language that we can come
>>> with
>>> > > > >like
>>> > > > >XML, JSON etc
>>> > > > >2. Most people use python and can learn it easily than any Domain
>>> > > specific
>>> > > > >language
>>> > > > >3. You can easily create a Python library for Airavata because
>>> all the
>>> > > > >APIs
>>> > > > >are thrift based.
>>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>>> > > > >
>>> > > > >Any thought

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Amila Jayasekara]

muva 
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", )
>> > app1 = Application(host, )
>> > app2 = Application(host, )
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerai...@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattm...@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > 
>> > > > Chris Mattmann
>> > > > chris.mattm...@gmail.com
>> &

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Supun Kamburugamuva]

> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerai...@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattm...@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > 
> > > > Chris Mattmann
> > > > chris.mattm...@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Supun Kamburugamuva 
> > > > Reply-To: 
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev 
> > > > Cc: "Alek Jones (Indiana)" , Suresh Marru
> > > > , "architect...@airavata.apache.org"
> > > > , "dev@oodt.apache.org"
> > > > 
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > s

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Shameera Rathnayaka]

Hi Supun,

I think we were in two context, because I as suggesting a way to serialize
and deserialize the workflow description while you are suggesting to
implement
some kind of workflow interpreter using Python, where Python client can
send
thrift calls to Airavata server to run the application. I can see with your
suggested
approach we can control the workflow execution process from client side
which
make it easy to implement workflow debugger.As you mentioned this is a
major change
to Airavata. So we should neatly think as this will change our existing
architecture.

Still if someone need to use different language java, php, JS etc ... to
run the same
workflow which generated by Python, we need a language independent workflow
description.
My initial question was what is the best language for this?. But as I have
explained in
one of my previous reply, It is not matter what language we used Either we
can use
XML or JSON to write this description, what matters is how easy to generate
workflow with the provided API. Hence it would be great to have set of neat
APIs in
different languages.

Thanks,
Shameera.


On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva 
wrote:

> Hi Shameera,
>
> Using python is a radical approach for workflow I think. But I believe this
> is a very beautiful and new approach.Here is a possible scenario for
> implementation and running using a Python based library.
>
> The Python library facilitates the creation of Applications and submitting
> them to Airavata for execution. The underlying functionality is exactly
> similar to what Java clients provides.The only difference is that, Python
> library should have a more fluent API than Java for easy creation of
> workflows. We can generate the Python clients that talk to Airavata server
> using Thrift.
>
> Here is an example off the top of my head to a Python script created by
> user for a Workflow. This is a very crude example and we need to come up
> with a much better API if we are going to go along this path. First we need
> to write a Python script that can execute a workflow using Airavata.
>
> import airavata
>
> host = Host("localhost", )
> app1 = Application(host, )
> app2 = Application(host, )
>
> # we will connect these applications as a workflow using some topology
> builder or other constructs
>
> wb = WorkFlowBuilder()
> wb.setApp("name1", app1)
> # you can do a simple output transformation here etc
>
> # connects the input of app1 to app2 etc
> wb.setApp("name2", app2).connectInput("name1")
>
> wb.submit()
>
> Now we can load this Python script from XBaya. When XBaya loads this script
> the Python script can output an XML configuration of the topology, XBaya
> can render. There are other ways like directly executing the Python script
> from command line and connecting XBaya indirectly as well. Now you can run
> the workflow from XBaya. Running the Workflow means just executing the
> Python script.
>
> XBaya gets the notifications through messaging and update the UI
> accordingly.
>
> The users need to write the Python script by hand. XBaya cannot create the
> script. Because workflow language is an actual python program the benefits
> are immense. For example user can do workflow steering in the workflow
> itself by subscribing to messages from Airavata.
>
> Thanks,
> Supun..
>
>
>
>
>
>
>
>
> On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
> > Hi Supun,
> >
> > I meant to say JS is a well-known client side scripting language i have
> > messed scripting part. Even we use Python, ultimately we should convert
> > this to java model in server side, somehow we need to serialized python
> > representation to the language which java can parse and generate that
> > model. In this case we need to parse python script in java isn't it? I am
> > not exactly clear how you suggesting to use python here. More details on
> > how end system works if we use Python would be great help to clearly
> > understand your points.
> >
> > Thanks,
> > Shameera.
> >
> > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> chris.mattm...@gmail.com>
> > wrote:
> >
> > > Have you guys considered using JCC [1] as a means
> > > to expose the workflow API currently in Java as a
> > > Python API?
> > >
> > > We are exploring its use in OODT, and we have already
> > > created a Tika [2] JCC-based python API.
> > >
> > > Cheers,
> > > Chris
> > >
> > > [1] http://lucene.apache.org/pylucene/jcc/
> > > [2] http://github.com/chrismattmann/tika-pyth

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Supun Kamburugamuva]

Hi Shameera,

Using python is a radical approach for workflow I think. But I believe this
is a very beautiful and new approach.Here is a possible scenario for
implementation and running using a Python based library.

The Python library facilitates the creation of Applications and submitting
them to Airavata for execution. The underlying functionality is exactly
similar to what Java clients provides.The only difference is that, Python
library should have a more fluent API than Java for easy creation of
workflows. We can generate the Python clients that talk to Airavata server
using Thrift.

Here is an example off the top of my head to a Python script created by
user for a Workflow. This is a very crude example and we need to come up
with a much better API if we are going to go along this path. First we need
to write a Python script that can execute a workflow using Airavata.

import airavata

host = Host("localhost", )
app1 = Application(host, )
app2 = Application(host, )

# we will connect these applications as a workflow using some topology
builder or other constructs

wb = WorkFlowBuilder()
wb.setApp("name1", app1)
# you can do a simple output transformation here etc

# connects the input of app1 to app2 etc
wb.setApp("name2", app2).connectInput("name1")

wb.submit()

Now we can load this Python script from XBaya. When XBaya loads this script
the Python script can output an XML configuration of the topology, XBaya
can render. There are other ways like directly executing the Python script
from command line and connecting XBaya indirectly as well. Now you can run
the workflow from XBaya. Running the Workflow means just executing the
Python script.

XBaya gets the notifications through messaging and update the UI
accordingly.

The users need to write the Python script by hand. XBaya cannot create the
script. Because workflow language is an actual python program the benefits
are immense. For example user can do workflow steering in the workflow
itself by subscribing to messages from Airavata.

Thanks,
Supun..








On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
shameerai...@gmail.com> wrote:

> Hi Supun,
>
> I meant to say JS is a well-known client side scripting language i have
> messed scripting part. Even we use Python, ultimately we should convert
> this to java model in server side, somehow we need to serialized python
> representation to the language which java can parse and generate that
> model. In this case we need to parse python script in java isn't it? I am
> not exactly clear how you suggesting to use python here. More details on
> how end system works if we use Python would be great help to clearly
> understand your points.
>
> Thanks,
> Shameera.
>
> On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann 
> wrote:
>
> > Have you guys considered using JCC [1] as a means
> > to expose the workflow API currently in Java as a
> > Python API?
> >
> > We are exploring its use in OODT, and we have already
> > created a Tika [2] JCC-based python API.
> >
> > Cheers,
> > Chris
> >
> > [1] http://lucene.apache.org/pylucene/jcc/
> > [2] http://github.com/chrismattmann/tika-python/
> >
> > 
> > Chris Mattmann
> > chris.mattm...@gmail.com
> >
> >
> >
> >
> > -Original Message-
> > From: Supun Kamburugamuva 
> > Reply-To: 
> > Date: Thursday, October 16, 2014 at 3:43 PM
> > To: dev 
> > Cc: "Alek Jones (Indiana)" , Suresh Marru
> > , "architect...@airavata.apache.org"
> > , "dev@oodt.apache.org"
> > 
> > Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Once we had an offline discussion about the Airavata Workflow language
> > >(with Milinda, Saliya and Shameera). In that discussion one thing came
> out
> > >was why we need to invent a different language when a simple library
> like
> > >Python will full fill of Airavata requirements.
> > >
> > >There are many benefits in using a Python library as the API for
> > >controlling Airavata workflows.
> > >
> > >1. It is a library, gives the ultimate control over the execution and it
> > >can be simpler than any domain specific language that we can come with
> > >like
> > >XML, JSON etc
> > >2. Most people use python and can learn it easily than any Domain
> specific
> > >language
> > >3. You can easily create a Python library for Airavata because all the
> > >APIs
> > >are thrift based.
> > >4. If you design the constructs correctly you can plug an XBaya.
> > >
> > >Any thoughts?
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Shameera Rathnayaka]

Hi Supun,

I meant to say JS is a well-known client side scripting language i have
messed scripting part. Even we use Python, ultimately we should convert
this to java model in server side, somehow we need to serialized python
representation to the language which java can parse and generate that
model. In this case we need to parse python script in java isn't it? I am
not exactly clear how you suggesting to use python here. More details on
how end system works if we use Python would be great help to clearly
understand your points.

Thanks,
Shameera.

On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann 
wrote:

> Have you guys considered using JCC [1] as a means
> to expose the workflow API currently in Java as a
> Python API?
>
> We are exploring its use in OODT, and we have already
> created a Tika [2] JCC-based python API.
>
> Cheers,
> Chris
>
> [1] http://lucene.apache.org/pylucene/jcc/
> [2] http://github.com/chrismattmann/tika-python/
>
> 
> Chris Mattmann
> chris.mattm...@gmail.com
>
>
>
>
> -Original Message-
> From: Supun Kamburugamuva 
> Reply-To: 
> Date: Thursday, October 16, 2014 at 3:43 PM
> To: dev 
> Cc: "Alek Jones (Indiana)" , Suresh Marru
> , "architect...@airavata.apache.org"
> , "dev@oodt.apache.org"
> 
> Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Once we had an offline discussion about the Airavata Workflow language
> >(with Milinda, Saliya and Shameera). In that discussion one thing came out
> >was why we need to invent a different language when a simple library like
> >Python will full fill of Airavata requirements.
> >
> >There are many benefits in using a Python library as the API for
> >controlling Airavata workflows.
> >
> >1. It is a library, gives the ultimate control over the execution and it
> >can be simpler than any domain specific language that we can come with
> >like
> >XML, JSON etc
> >2. Most people use python and can learn it easily than any Domain specific
> >language
> >3. You can easily create a Python library for Airavata because all the
> >APIs
> >are thrift based.
> >4. If you design the constructs correctly you can plug an XBaya.
> >
> >Any thoughts?
> >
> >Thanks,
> >Supun..
> >
> >
> >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
> >wrote:
> >
> >> Hi Shameera,
> >>
> >> Why you prefer JavaScript over a language like Python?
> >>
> >> Thanks,
> >> Supun..
> >>
> >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> >> shameerai...@gmail.com> wrote:
> >>
> >>> ​Hi,
> >>>
> >>> First of all thanks everyone for giving valuable inputs. After doing
> >>>some
> >>> background search and talking to different people in the University
> >>>who has
> >>> used different workflow languages, I myself convinced that introducing
> >>>an
> >>> another workflow language is not what actually they need. By changing
> >>> exiting workflow language to another will not solve problems. What they
> >>> asking is a easy way to construct the workflows. Indirectly what they
> >>> asking for a sort of API which they can use to generate the workflows
> >>>and
> >>> run it. Correct me if i am wrong here.
> >>>
> >>> As most of above replies depict,  if we can get a simple API, as an
> >>> example, for a web based application, JavaScript API would be a good
> >>> solution, and probably JSON would be a good candidate for language,
> >>>instead
> >>> of XML.
> >>>
> >>> Airavata community already have started to implement web base GUI.
> >>>Hence
> >>> introducing a JSON base JavaScript API would be great help. WDYT?
> >>>
> >>> Thanks,
> >>> Shameera.
> >>>
> >>>
> >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >>> alek...@gmail.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> it is not dataflow instead focused on orchestrating REST services but
> >>>> you may find it useful datapoint - we created worfklow service that
> >>>>uses
> >>>> natively JavaScript and JSON to describe what happens during workflow
> >>>> execution:
> >>>>
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Chris Mattmann]

Have you guys considered using JCC [1] as a means
to expose the workflow API currently in Java as a
Python API?

We are exploring its use in OODT, and we have already
created a Tika [2] JCC-based python API.

Cheers,
Chris

[1] http://lucene.apache.org/pylucene/jcc/
[2] http://github.com/chrismattmann/tika-python/


Chris Mattmann
chris.mattm...@gmail.com




-Original Message-
From: Supun Kamburugamuva 
Reply-To: 
Date: Thursday, October 16, 2014 at 3:43 PM
To: dev 
Cc: "Alek Jones (Indiana)" , Suresh Marru
, "architect...@airavata.apache.org"
, "dev@oodt.apache.org"

Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.

>Once we had an offline discussion about the Airavata Workflow language
>(with Milinda, Saliya and Shameera). In that discussion one thing came out
>was why we need to invent a different language when a simple library like
>Python will full fill of Airavata requirements.
>
>There are many benefits in using a Python library as the API for
>controlling Airavata workflows.
>
>1. It is a library, gives the ultimate control over the execution and it
>can be simpler than any domain specific language that we can come with
>like
>XML, JSON etc
>2. Most people use python and can learn it easily than any Domain specific
>language
>3. You can easily create a Python library for Airavata because all the
>APIs
>are thrift based.
>4. If you design the constructs correctly you can plug an XBaya.
>
>Any thoughts?
>
>Thanks,
>Supun..
>
>
>On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
>wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerai...@gmail.com> wrote:
>>
>>> ​Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>>some
>>> background search and talking to different people in the University
>>>who has
>>> used different workflow languages, I myself convinced that introducing
>>>an
>>> another workflow language is not what actually they need. By changing
>>> exiting workflow language to another will not solve problems. What they
>>> asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows
>>>and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language,
>>>instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI.
>>>Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alek...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> it is not dataflow instead focused on orchestrating REST services but
>>>> you may find it useful datapoint - we created worfklow service that
>>>>uses
>>>> natively JavaScript and JSON to describe what happens during workflow
>>>> execution:
>>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>>
>>>> HTH,
>>>>
>>>> Alek
>>>>
>>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru 
>>>>wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Great to hear OODT community will be interested in adopting a JSON
>>>>> based workflow language and potentially a web based composer as well.
>>>>> Airavata previously had BPEL support initially through a home grown
>>>>> implementation [1] by Alek Slominski and later through Apache ODE
>>>>>[2]. Also
>>>>> a white paper [3] by Alek on this topic is an interesting read.
>>>>>
>>>>> I am of the same opinion that we should adopt something more modern
>>>>>as
>>>>> the challenges from scientific workflows seems to be converging with
>>>>>the
>>>>> data flow patterns in business workflows.
>>>>>
>>>>> It will be great if we can all compile a list of potential candidates
>>>>> and

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Supun Kamburugamuva]

Hi Shameera,

To create a web based UI I don't see an immediate advantage of using
JavaScript over a Python API. Because Airavata's API is Thrift (not any
language based API) you can generate code for any language (including
JavaScript). Anyway most probably you will not use direct Airavata API
calls for creating a web UI. I'm also not sure what you exactly mean by
JavaScript language as well. If you explain in more detail that would be
really helpful.

If you just give a Python API to create workflows, you really don't even
need XBaya to create/run workflows. You can create them as scripts and run
them from command line. You can debug your workflows written in Python
using normal python debuggers. You can generate the XBaya workflow by using
the structure you create in Python as well. Then you can use XBaya to
monitor the workflow. These things are hard to do with an JavaScript API.
As a language also I would pick Python over JavaScript.

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:46 PM, Shameera Rathnayaka  wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript is
> a well-known client side language, I would select JavaScript over Python. I
> am not much familiar with Python, so would like to know the advantages we
> get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
> wrote:
>
> > Hi Shameera,
> >
> > Why you prefer JavaScript over a language like Python?
> >
> > Thanks,
> > Supun..
> >
> > On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > shameerai...@gmail.com> wrote:
> >
> >> ​Hi,
> >>
> >> First of all thanks everyone for giving valuable inputs. After doing
> some
> >> background search and talking to different people in the University who
> has
> >> used different workflow languages, I myself convinced that introducing
> an
> >> another workflow language is not what actually they need. By changing
> >> exiting workflow language to another will not solve problems. What they
> >> asking is a easy way to construct the workflows. Indirectly what they
> >> asking for a sort of API which they can use to generate the workflows
> and
> >> run it. Correct me if i am wrong here.
> >>
> >> As most of above replies depict,  if we can get a simple API, as an
> >> example, for a web based application, JavaScript API would be a good
> >> solution, and probably JSON would be a good candidate for language,
> instead
> >> of XML.
> >>
> >> Airavata community already have started to implement web base GUI. Hence
> >> introducing a JSON base JavaScript API would be great help. WDYT?
> >>
> >> Thanks,
> >> Shameera.
> >>
> >>
> >> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> >> alek...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> it is not dataflow instead focused on orchestrating REST services but
> >>> you may find it useful datapoint - we created worfklow service that
> uses
> >>> natively JavaScript and JSON to describe what happens during workflow
> >>> execution:
> >>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> >>>
> >>> HTH,
> >>>
> >>> Alek
> >>>
> >>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru 
> wrote:
> >>>
>  Hi Chris,
> 
>  Great to hear OODT community will be interested in adopting a JSON
>  based workflow language and potentially a web based composer as well.
>  Airavata previously had BPEL support initially through a home grown
>  implementation [1] by Alek Slominski and later through Apache ODE
> [2]. Also
>  a white paper [3] by Alek on this topic is an interesting read.
> 
>  I am of the same opinion that we should adopt something more modern as
>  the challenges from scientific workflows seems to be converging with
> the
>  data flow patterns in business workflows.
> 
>  It will be great if we can all compile a list of potential candidates
>  and hack them through.
> 
>  Suresh
>  [1] -
> 
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>  [2] -
> 
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>  [3] -
> 
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> 
> 
>  On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>  chris.a.mattm...@jpl.nasa.gov> wrote:
> 
>  > Hi Guys,
>  >
>  > I've been interested in this too - we don't per have a specific
>  > OODT workflow language, but we specific workflows using XML, and
>  > other configuration (we are also thinking of moving to JSON for
>  > this).
>  >
>  > In the past I've also looked at YAWL and BPEL - both seem complex
>  > to me.
>  >
>  > I wonder at the end of the day if we should adopt something more
>  > modern like PIG or some other data flow type of language (PIG
>  > is really neat).
>  >
>  > Cheers,
> 

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Nipurn Doshi]

Hi,

How about I suggest a third option where I have the current code setup
which has access to thrift classes, airavata interface classes and a
complete framework ( Laravel ) that supports rest apis to generate json
data that we are looking at?

I am suggesting this only because I have done this in previous projects I
have worked on using Laravel.

Although, it's in PHP.
 On 16-Oct-2014 6:47 pm, "Shameera Rathnayaka" 
wrote:

> Hi Supun,
>
> Considering we are going to provide web base GUI support and  JavaScript
> is a well-known client side language, I would select JavaScript over
> Python. I am not much familiar with Python, so would like to know the
> advantages we get by selecting python here.
>
>
> Thanks,
> Shameera.
>
> On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
> wrote:
>
>> Hi Shameera,
>>
>> Why you prefer JavaScript over a language like Python?
>>
>> Thanks,
>> Supun..
>>
>> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> shameerai...@gmail.com> wrote:
>>
>>> ​Hi,
>>>
>>> First of all thanks everyone for giving valuable inputs. After doing
>>> some background search and talking to different people in the University
>>> who has used different workflow languages, I myself convinced that
>>> introducing an another workflow language is not what actually they need. By
>>> changing exiting workflow language to another will not solve problems. What
>>> they asking is a easy way to construct the workflows. Indirectly what they
>>> asking for a sort of API which they can use to generate the workflows and
>>> run it. Correct me if i am wrong here.
>>>
>>> As most of above replies depict,  if we can get a simple API, as an
>>> example, for a web based application, JavaScript API would be a good
>>> solution, and probably JSON would be a good candidate for language, instead
>>> of XML.
>>>
>>> Airavata community already have started to implement web base GUI. Hence
>>> introducing a JSON base JavaScript API would be great help. WDYT?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>>> alek...@gmail.com> wrote:
>>>
 Hi,

 it is not dataflow instead focused on orchestrating REST services but
 you may find it useful datapoint - we created worfklow service that uses
 natively JavaScript and JSON to describe what happens during workflow
 execution:
 https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002

 HTH,

 Alek

 On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru 
 wrote:

> Hi Chris,
>
> Great to hear OODT community will be interested in adopting a JSON
> based workflow language and potentially a web based composer as well.
> Airavata previously had BPEL support initially through a home grown
> implementation [1] by Alek Slominski and later through Apache ODE [2]. 
> Also
> a white paper [3] by Alek on this topic is an interesting read.
>
> I am of the same opinion that we should adopt something more modern as
> the challenges from scientific workflows seems to be converging with the
> data flow patterns in business workflows.
>
> It will be great if we can all compile a list of potential candidates
> and hack them through.
>
> Suresh
> [1] -
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> [2] -
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> [3] -
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>
>
> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattm...@nasa.gov
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
>>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Shameera Rathnayaka]

Hi Supun,

Considering we are going to provide web base GUI support and  JavaScript is
a well-known client side language, I would select JavaScript over Python. I
am not much familiar with Python, so would like to know the advantages we
get by selecting python here.


Thanks,
Shameera.

On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>> ​Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alek...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:
>>>
 Hi Chris,

 Great to hear OODT community will be interested in adopting a JSON
 based workflow language and potentially a web based composer as well.
 Airavata previously had BPEL support initially through a home grown
 implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
 a white paper [3] by Alek on this topic is an interesting read.

 I am of the same opinion that we should adopt something more modern as
 the challenges from scientific workflows seems to be converging with the
 data flow patterns in business workflows.

 It will be great if we can all compile a list of potential candidates
 and hack them through.

 Suresh
 [1] -
 http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
 [2] -
 http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
 [3] -
 http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf


 On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
 chris.a.mattm...@jpl.nasa.gov> wrote:

 > Hi Guys,
 >
 > I've been interested in this too - we don't per have a specific
 > OODT workflow language, but we specific workflows using XML, and
 > other configuration (we are also thinking of moving to JSON for
 > this).
 >
 > In the past I've also looked at YAWL and BPEL - both seem complex
 > to me.
 >
 > I wonder at the end of the day if we should adopt something more
 > modern like PIG or some other data flow type of language (PIG
 > is really neat).
 >
 > Cheers,
 > Chris
 >
 >
 > ++
 > Chris Mattmann, Ph.D.
 > Chief Architect
 > Instrument Software and Science Data Systems Section (398)
 > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
 > Office: 168-519, Mailstop: 168-527
 > Email: chris.a.mattm...@nasa.gov
 > WWW:  http://sunset.usc.edu/~mattmann/
 > ++
 > Adjunct Associate Professor, Computer Science Department
 > University of Southern California, Los Angeles, CA 90089 USA
 > ++
 >
 >
 >
 >
 >
 >
 > -Original Message-
 > From: Shameera Rathnayaka 
 > Reply-To: "architect...@airavata.apache.org"
 > 
 > Date: Thursday, September 18, 2014 8:26 AM
 > To: "architect...@airavata.apache.org" <
 architect...@airavata.apache.org>,
 > dev 
 > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
 >
 >> Hi All,
 >>
 >> As we all know Airavata has its own workflow language call XWF. When
 XWF
 >> was introduced, main focus points are interoperability and
>>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Supun Kamburugamuva]

Once we had an offline discussion about the Airavata Workflow language
(with Milinda, Saliya and Shameera). In that discussion one thing came out
was why we need to invent a different language when a simple library like
Python will full fill of Airavata requirements.

There are many benefits in using a Python library as the API for
controlling Airavata workflows.

1. It is a library, gives the ultimate control over the execution and it
can be simpler than any domain specific language that we can come with like
XML, JSON etc
2. Most people use python and can learn it easily than any Domain specific
language
3. You can easily create a Python library for Airavata because all the APIs
are thrift based.
4. If you design the constructs correctly you can plug an XBaya.

Any thoughts?

Thanks,
Supun..


On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva 
wrote:

> Hi Shameera,
>
> Why you prefer JavaScript over a language like Python?
>
> Thanks,
> Supun..
>
> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>> ​Hi,
>>
>> First of all thanks everyone for giving valuable inputs. After doing some
>> background search and talking to different people in the University who has
>> used different workflow languages, I myself convinced that introducing an
>> another workflow language is not what actually they need. By changing
>> exiting workflow language to another will not solve problems. What they
>> asking is a easy way to construct the workflows. Indirectly what they
>> asking for a sort of API which they can use to generate the workflows and
>> run it. Correct me if i am wrong here.
>>
>> As most of above replies depict,  if we can get a simple API, as an
>> example, for a web based application, JavaScript API would be a good
>> solution, and probably JSON would be a good candidate for language, instead
>> of XML.
>>
>> Airavata community already have started to implement web base GUI. Hence
>> introducing a JSON base JavaScript API would be great help. WDYT?
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> alek...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> it is not dataflow instead focused on orchestrating REST services but
>>> you may find it useful datapoint - we created worfklow service that uses
>>> natively JavaScript and JSON to describe what happens during workflow
>>> execution:
>>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>>
>>> HTH,
>>>
>>> Alek
>>>
>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:
>>>
 Hi Chris,

 Great to hear OODT community will be interested in adopting a JSON
 based workflow language and potentially a web based composer as well.
 Airavata previously had BPEL support initially through a home grown
 implementation [1] by Alek Slominski and later through Apache ODE [2]. Also
 a white paper [3] by Alek on this topic is an interesting read.

 I am of the same opinion that we should adopt something more modern as
 the challenges from scientific workflows seems to be converging with the
 data flow patterns in business workflows.

 It will be great if we can all compile a list of potential candidates
 and hack them through.

 Suresh
 [1] -
 http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
 [2] -
 http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
 [3] -
 http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf


 On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
 chris.a.mattm...@jpl.nasa.gov> wrote:

 > Hi Guys,
 >
 > I've been interested in this too - we don't per have a specific
 > OODT workflow language, but we specific workflows using XML, and
 > other configuration (we are also thinking of moving to JSON for
 > this).
 >
 > In the past I've also looked at YAWL and BPEL - both seem complex
 > to me.
 >
 > I wonder at the end of the day if we should adopt something more
 > modern like PIG or some other data flow type of language (PIG
 > is really neat).
 >
 > Cheers,
 > Chris
 >
 >
 > ++
 > Chris Mattmann, Ph.D.
 > Chief Architect
 > Instrument Software and Science Data Systems Section (398)
 > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
 > Office: 168-519, Mailstop: 168-527
 > Email: chris.a.mattm...@nasa.gov
 > WWW:  http://sunset.usc.edu/~mattmann/
 > ++
 > Adjunct Associate Professor, Computer Science Department
 > University of Southern California, Los Angeles, CA 90089 USA
 > ++
 >
 >
 >
 >
 >
 >
 > -Original M

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Supun Kamburugamuva]

Hi Shameera,

Why you prefer JavaScript over a language like Python?

Thanks,
Supun..

On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka  wrote:

> ​Hi,
>
> First of all thanks everyone for giving valuable inputs. After doing some
> background search and talking to different people in the University who has
> used different workflow languages, I myself convinced that introducing an
> another workflow language is not what actually they need. By changing
> exiting workflow language to another will not solve problems. What they
> asking is a easy way to construct the workflows. Indirectly what they
> asking for a sort of API which they can use to generate the workflows and
> run it. Correct me if i am wrong here.
>
> As most of above replies depict,  if we can get a simple API, as an
> example, for a web based application, JavaScript API would be a good
> solution, and probably JSON would be a good candidate for language, instead
> of XML.
>
> Airavata community already have started to implement web base GUI. Hence
> introducing a JSON base JavaScript API would be great help. WDYT?
>
> Thanks,
> Shameera.
>
>
> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> alek...@gmail.com> wrote:
>
>> Hi,
>>
>> it is not dataflow instead focused on orchestrating REST services but you
>> may find it useful datapoint - we created worfklow service that uses
>> natively JavaScript and JSON to describe what happens during workflow
>> execution:
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>>
>> HTH,
>>
>> Alek
>>
>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:
>>
>>> Hi Chris,
>>>
>>> Great to hear OODT community will be interested in adopting a JSON based
>>> workflow language and potentially a web based composer as well. Airavata
>>> previously had BPEL support initially through a home grown implementation
>>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>>> [3] by Alek on this topic is an interesting read.
>>>
>>> I am of the same opinion that we should adopt something more modern as
>>> the challenges from scientific workflows seems to be converging with the
>>> data flow patterns in business workflows.
>>>
>>> It will be great if we can all compile a list of potential candidates
>>> and hack them through.
>>>
>>> Suresh
>>> [1] -
>>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>>> [2] -
>>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>>> [3] -
>>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>>
>>>
>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>>> chris.a.mattm...@jpl.nasa.gov> wrote:
>>>
>>> > Hi Guys,
>>> >
>>> > I've been interested in this too - we don't per have a specific
>>> > OODT workflow language, but we specific workflows using XML, and
>>> > other configuration (we are also thinking of moving to JSON for
>>> > this).
>>> >
>>> > In the past I've also looked at YAWL and BPEL - both seem complex
>>> > to me.
>>> >
>>> > I wonder at the end of the day if we should adopt something more
>>> > modern like PIG or some other data flow type of language (PIG
>>> > is really neat).
>>> >
>>> > Cheers,
>>> > Chris
>>> >
>>> >
>>> > ++
>>> > Chris Mattmann, Ph.D.
>>> > Chief Architect
>>> > Instrument Software and Science Data Systems Section (398)
>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> > Office: 168-519, Mailstop: 168-527
>>> > Email: chris.a.mattm...@nasa.gov
>>> > WWW:  http://sunset.usc.edu/~mattmann/
>>> > ++
>>> > Adjunct Associate Professor, Computer Science Department
>>> > University of Southern California, Los Angeles, CA 90089 USA
>>> > ++
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -Original Message-
>>> > From: Shameera Rathnayaka 
>>> > Reply-To: "architect...@airavata.apache.org"
>>> > 
>>> > Date: Thursday, September 18, 2014 8:26 AM
>>> > To: "architect...@airavata.apache.org" <
>>> architect...@airavata.apache.org>,
>>> > dev 
>>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>>> >
>>> >> Hi All,
>>> >>
>>> >> As we all know Airavata has its own workflow language call XWF. When
>>> XWF
>>> >> was introduced, main focus points are interoperability and
>>> convertibility.
>>> >> But with years of experience it is convinced that above requirements
>>> are
>>> >> not really useful when we come to real world use cases. And XWF is XML
>>> >> based bulky language where we attache WSDLs and Workflow image it
>>> self.
>>> >> But
>>> >> with the recent changes WSDL part is being removed from XWF.
>>> >>
>>> >> It is worth to evaluate handy Scientific workflow languages in
>>> industry
>>> >> and
>>> >> find out pros and cons, at the end of this evaluation we ne

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Shameera Rathnayaka]

​Hi,

First of all thanks everyone for giving valuable inputs. After doing some
background search and talking to different people in the University who has
used different workflow languages, I myself convinced that introducing an
another workflow language is not what actually they need. By changing
exiting workflow language to another will not solve problems. What they
asking is a easy way to construct the workflows. Indirectly what they
asking for a sort of API which they can use to generate the workflows and
run it. Correct me if i am wrong here.

As most of above replies depict,  if we can get a simple API, as an
example, for a web based application, JavaScript API would be a good
solution, and probably JSON would be a good candidate for language, instead
of XML.

Airavata community already have started to implement web base GUI. Hence
introducing a JSON base JavaScript API would be great help. WDYT?

Thanks,
Shameera.


On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alek...@gmail.com> wrote:

> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:
>
>> Hi Chris,
>>
>> Great to hear OODT community will be interested in adopting a JSON based
>> workflow language and potentially a web based composer as well. Airavata
>> previously had BPEL support initially through a home grown implementation
>> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
>> [3] by Alek on this topic is an interesting read.
>>
>> I am of the same opinion that we should adopt something more modern as
>> the challenges from scientific workflows seems to be converging with the
>> data flow patterns in business workflows.
>>
>> It will be great if we can all compile a list of potential candidates and
>> hack them through.
>>
>> Suresh
>> [1] -
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> [2] -
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
>> [3] -
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>>
>>
>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> chris.a.mattm...@jpl.nasa.gov> wrote:
>>
>> > Hi Guys,
>> >
>> > I've been interested in this too - we don't per have a specific
>> > OODT workflow language, but we specific workflows using XML, and
>> > other configuration (we are also thinking of moving to JSON for
>> > this).
>> >
>> > In the past I've also looked at YAWL and BPEL - both seem complex
>> > to me.
>> >
>> > I wonder at the end of the day if we should adopt something more
>> > modern like PIG or some other data flow type of language (PIG
>> > is really neat).
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > ++
>> > Chris Mattmann, Ph.D.
>> > Chief Architect
>> > Instrument Software and Science Data Systems Section (398)
>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > Office: 168-519, Mailstop: 168-527
>> > Email: chris.a.mattm...@nasa.gov
>> > WWW:  http://sunset.usc.edu/~mattmann/
>> > ++
>> > Adjunct Associate Professor, Computer Science Department
>> > University of Southern California, Los Angeles, CA 90089 USA
>> > ++
>> >
>> >
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Shameera Rathnayaka 
>> > Reply-To: "architect...@airavata.apache.org"
>> > 
>> > Date: Thursday, September 18, 2014 8:26 AM
>> > To: "architect...@airavata.apache.org" <
>> architect...@airavata.apache.org>,
>> > dev 
>> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>> >
>> >> Hi All,
>> >>
>> >> As we all know Airavata has its own workflow language call XWF. When
>> XWF
>> >> was introduced, main focus points are interoperability and
>> convertibility.
>> >> But with years of experience it is convinced that above requirements
>> are
>> >> not really useful when we come to real world use cases. And XWF is XML
>> >> based bulky language where we attache WSDLs and Workflow image it self.
>> >> But
>> >> with the recent changes WSDL part is being removed from XWF.
>> >>
>> >> It is worth to evaluate handy Scientific workflow languages in industry
>> >> and
>> >> find out pros and cons, at the end of this evaluation we need to come
>> up
>> >> with idea how we should improve Airavata workflow language, either we
>> can
>> >> improve existing XWF language, totally change to a new language
>> available
>> >> in industry or write a new light weight language. Basic requirements
>> that
>> >> we expect from new improvement ar

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Bruce Barkstrom]

Thanks very much for the documentation.

Just glancing through it, I'm not sure I'd describe production
by embedding a particular language in the basic description.
In the large-scale production systems I architected for NASA's
ERBE and CERES production, I started with a Data Flow Diagram.
Now, I'd probably move to UML before getting close to a language-
specific design.

A key part of our work was two years getting an agreement on
the naming convention of the files that tied in to the multiple
instruments, multiple products, and multiple versions of each
element in the provenance.  I did a formal description of those
structures in an ESI paper (I guess 2009) on configuration management
in large-scale satellite data production.  That paper contains
some example figures of the provenance graphs - which are
probably easiest to visualize if you get the pdf version and
view the graphs with a zoom of 2800% or so.

One point that was important is that our production flow was
not completely automated - and humans had to invoke the
proper connection between input files, various pieces of
production executables, and output files - including variations
from one month to the next a as well as match*ing *which satellite
was involved (plus some other configuration issues).  In the
long run, we also needed to build production schedules so
the archive could check quality control on the proper running
of the process.  That work required interaction between objects
and staff on the archive and the science teams, which makes
for the usual schedule complications that are an intrinsic
part of production planning on big projects.

Also, in some work I'm doing currently, I am quite concerned
about user search states (or production states) that come from
a Markov model that allows simulation of user loadings on
an archive.  I haven't gone into full detail on this yet - but it
may mean that the interaction between the user and the archive
needs to keep track of the state of both agents.  The basic
notion for this work is that the archive and the user are
playing a two agent cooperative game where the archive
knows what it contains and the user knows what his or her
target is.  I think that requires maintaining agent states across
multiple sessions.  I don't recall whether that violates the
original source of the RESTFUL design, so I'll need to go
back to the original to check.

I'll try to look at what you've sent in more detail and provide
some more comments.  Regardless, I appreciate the information.

Bruce B.


On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
alek...@gmail.com> wrote:

> Hi,
>
> it is not dataflow instead focused on orchestrating REST services but you
> may find it useful datapoint - we created worfklow service that uses
> natively JavaScript and JSON to describe what happens during workflow
> execution:
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>
> HTH,
>
> Alek
>
> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:
>
> > Hi Chris,
> >
> > Great to hear OODT community will be interested in adopting a JSON based
> > workflow language and potentially a web based composer as well. Airavata
> > previously had BPEL support initially through a home grown implementation
> > [1] by Alek Slominski and later through Apache ODE [2]. Also a white
> paper
> > [3] by Alek on this topic is an interesting read.
> >
> > I am of the same opinion that we should adopt something more modern as
> the
> > challenges from scientific workflows seems to be converging with the data
> > flow patterns in business workflows.
> >
> > It will be great if we can all compile a list of potential candidates and
> > hack them through.
> >
> > Suresh
> > [1] -
> > http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > [2] -
> >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> > [3] -
> >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
> >
> >
> > On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > chris.a.mattm...@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattm...@nas

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Aleksander Slominski (NY)]

Hi,

it is not dataflow instead focused on orchestrating REST services but you
may find it useful datapoint - we created worfklow service that uses
natively JavaScript and JSON to describe what happens during workflow
execution:
https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002

HTH,

Alek

On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru  wrote:

> Hi Chris,
>
> Great to hear OODT community will be interested in adopting a JSON based
> workflow language and potentially a web based composer as well. Airavata
> previously had BPEL support initially through a home grown implementation
> [1] by Alek Slominski and later through Apache ODE [2]. Also a white paper
> [3] by Alek on this topic is an interesting read.
>
> I am of the same opinion that we should adopt something more modern as the
> challenges from scientific workflows seems to be converging with the data
> flow patterns in business workflows.
>
> It will be great if we can all compile a list of potential candidates and
> hack them through.
>
> Suresh
> [1] -
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> [2] -
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
> [3] -
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf
>
>
> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattm...@nasa.gov
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
> >
> >
> >
> >
> >
> > -Original Message-
> > From: Shameera Rathnayaka 
> > Reply-To: "architect...@airavata.apache.org"
> > 
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architect...@airavata.apache.org"  >,
> > dev 
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> >> Hi All,
> >>
> >> As we all know Airavata has its own workflow language call XWF. When XWF
> >> was introduced, main focus points are interoperability and
> convertibility.
> >> But with years of experience it is convinced that above requirements are
> >> not really useful when we come to real world use cases. And XWF is XML
> >> based bulky language where we attache WSDLs and Workflow image it self.
> >> But
> >> with the recent changes WSDL part is being removed from XWF.
> >>
> >> It is worth to evaluate handy Scientific workflow languages in industry
> >> and
> >> find out pros and cons, at the end of this evaluation we need to come up
> >> with idea how we should improve Airavata workflow language, either we
> can
> >> improve existing XWF language, totally change to a new language
> available
> >> in industry or write a new light weight language. Basic requirements
> that
> >> we expect from new improvement are, high usability, flexible, light
> weight
> >> and real time monitoring support. As you can see above requirements are
> >> not
> >> direct comes with workflow languages but we need workflow language which
> >> help to support above requirements.
> >>
> >> After reading few papers and googling, initially i have come up with
> >> following three existing languages,
> >> 1. YAWL 
> >> 2. WS-BPEL
> >> ​3. SIDL
> >> 
> >>
> >> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >> also uses slightly modified version of SIDL. Other than above three
> >> languages we can come up with simple workflow language base on json(or
> >> yaml) which support all our requirements for some extends.
> >>
> >> It would be grate if I can get more input regarding the $Subject form
> the
> >> airavata community. You all are more than welcome to provide any type of
> >> suggestions.
> >>
> >> Thanks,
> >> Shameera.
> >>
> >> ​
> >>
> >> --
> >> Best Regards,
> >> Shameera Rathnayaka.
> >>
> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> >> Blog : http://sh

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Bruce Barkstrom]

Can you make a copy of your IEEE article available
without the paywall?

Thanks.

Bruce B.

On Thu, Sep 18, 2014 at 2:54 PM, Lavanya Ramakrishnan  wrote:

> Here is my 2c -
>
> I think it is important to try and understand what your users are going to
> do with workflow and what kind of language they are used to
> (domain-specific, functional, etc). They are processes called user-centered
> design processes you can use to do this or do at a minimum an informal
> study.
>
>  A couple of years ago, we did an introspection on why all the existing
> workflow tools didn't have the uptake we had assumed it would. I have been
> part of a half dozen different tools over my career. We have since launched
> a project called Tigres - http://tigres.lbl.gov/ where we have learned a
> lot due to using a user-centered design approach. We have an IEEE eScience
> paper on our initial work - which you might find interesting. I am also
> happy to share more details on Tigres and/or the process.
>
> Lavanya
>
>
>
>
>
> On Thu, Sep 18, 2014 at 10:53 AM, BW  wrote:
>
> > Is there a list of graphical BEL workflow tools?
> >
> > On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> > chris.a.mattm...@jpl.nasa.gov> wrote:
> >
> > > Hi Guys,
> > >
> > > I've been interested in this too - we don't per have a specific
> > > OODT workflow language, but we specific workflows using XML, and
> > > other configuration (we are also thinking of moving to JSON for
> > > this).
> > >
> > > In the past I've also looked at YAWL and BPEL - both seem complex
> > > to me.
> > >
> > > I wonder at the end of the day if we should adopt something more
> > > modern like PIG or some other data flow type of language (PIG
> > > is really neat).
> > >
> > > Cheers,
> > > Chris
> > >
> > >
> > > ++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattm...@nasa.gov 
> > > WWW:  http://sunset.usc.edu/~mattmann/
> > > ++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Shameera Rathnayaka >
> > > Reply-To: "architect...@airavata.apache.org "
> > > >
> > > Date: Thursday, September 18, 2014 8:26 AM
> > > To: "architect...@airavata.apache.org " <
> > > architect...@airavata.apache.org >,
> > > dev >
> > > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> > >
> > > >Hi All,
> > > >
> > > >As we all know Airavata has its own workflow language call XWF. When
> XWF
> > > >was introduced, main focus points are interoperability and
> > convertibility.
> > > >But with years of experience it is convinced that above requirements
> are
> > > >not really useful when we come to real world use cases. And XWF is XML
> > > >based bulky language where we attache WSDLs and Workflow image it
> self.
> > > >But
> > > >with the recent changes WSDL part is being removed from XWF.
> > > >
> > > >It is worth to evaluate handy Scientific workflow languages in
> industry
> > > >and
> > > >find out pros and cons, at the end of this evaluation we need to come
> up
> > > >with idea how we should improve Airavata workflow language, either we
> > can
> > > >improve existing XWF language, totally change to a new language
> > available
> > > >in industry or write a new light weight language. Basic requirements
> > that
> > > >we expect from new improvement are, high usability, flexible, light
> > weight
> > > >and real time monitoring support. As you can see above requirements
> are
> > > >not
> > > >direct comes with workflow languages but we need workflow language
> which
> > > >help to support above requirements.
> > > >
> > > >After reading few papers and googling, initially i have come up with
> > > >following three existing languages,
> > > >1. YAWL 
> > > >2. WS-BPEL
> > > >​3. SIDL
> > > >
> > > >
> > > >In my opinion SIDL is more familiar with scientific domain,
> Radical-SAGA
> > > >also uses slightly modified version of SIDL. Other than above three
> > > >languages we can come up with simple workflow language base on json(or
> > > >yaml) which support all our requirements for some extends.
> > > >
> > > >It would be grate if I can get more input regarding the $Subject form
> > the
> > > >airavata community. You all are more than welcome to provide any type
> of
> > > >suggestions.
> > > >
> > > >Thanks,
> > > >Shameera.
> > > >
> > > >​
> > > >
> > > >--
> > > >Best Regards,
> > > >Shameera Rathnayaka.
> > > >
> > > 

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Mattmann, Chris A (3980)]

Great Bruce - these are amazing insights and I hope this
in the Airavata and OODT community check this out.

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Bruce Barkstrom 
Reply-To: "architect...@airavata.apache.org"

Date: Friday, September 19, 2014 9:29 AM
To: "dev@oodt.apache.org" 
Cc: "architect...@airavata.apache.org" ,
dev 
Subject: Re: Evaluate Suitable Scientific Workflow Language for Airavata.

>One factor that should be included in the group's deliberations
>on adding a workflow language to the other things in OODT
>is the impact on long-term maintenance.  While there's a lot
>of enthusiasm in the developer community right now, we need
>to think about what happens when development turns into
>maintenance.  The account that follows is based on my experience
>with trying to resurrect a W3C-related project to visualize RDF
>graphs.
>
>The project is called IsaViz.  It's even got a W3 web site:
>http://www.w3.org/2001/11/IsaViz/
>IsaViz identifies itself as a visual authoring tool for RDF.
>Right up near the top are two dates that should serve as
>a cautionary note for people who want to pick up this tool:
>Current Stable Version: October 2007 and Current Development Version:
>May 2007.  It also looks like the site was maintained by a single
>developer who did the development as a postdoc project.
>The overview page was last modified on Oct. 21, 2007.
>The installation instructions were last modified on Oct. 18, 2004.
>
>IsaViz uses a number of tools.  The Installation Instructions
>identify the following:
>
>   - A Java Virtual Machine version 1.3.0 or later (1.4 strongly
>   recommended - see Known problems
>   <http://www.w3.org/2001/11/IsaViz/overview.html#bugs>)
>   - A distribution of IsaViz, which contains the following Java JAR
>files:
>  - IsaViz itself *(isaviz.jar)*
>  - Zoomable Visual Transformation Machine *(zvtm.jar)*
>  - Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
>  - Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
>  2, Xerces 1.4.4 for IsaViz 1.2
>   - GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
>   supported in IsaViz 2, and has actually only been tested with
>version 1.9). *Note:
>   some instances of version 1.10.0 had a bug that produced incomplete SVG
>   files, but it has been corrected in subsequent releases *(newer
>versions
>   can be obtained on the graphviz.org site
>   <http://www.graphviz.org/pub/graphviz/CURRENT>).
>
>So, what complications ensue:
>1.  Java has moved way beyond version 1.3 or 1.4.  Since Java can
>deprecate
>code and
>since there's Oracle and OpenJDK, there may be some unpleasantries that
>might need
>fixes.  I haven't seen comments from the community on whether or not these
>might be
>significant.  The IsaViz documentation refers to the ancient time when Sun
>controlled
>the language.  Apparently, the IsaViz code was only tested with Sun's
>j2se/1.3 or 1.4.
>2.  I suppose the jar files from IsaViz version 2 would be the place to
>start in reconstructing
>this piece of software.  However, one might be careful about this when you
>get into the
>installation scripts from the Installation Instructions.
>3.  The Zoomable Visual Transformation Machine project is on Sourceforge.
>It's apparently
>done in Java.  However, the IsaViz code used version 0.9.0, while the
>current Sourceforge
>project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
>stable version (Aug. 2013)
>with a more recent development version (0.11.2 - snapshot; June 2014).  No
>idea if
>there would be any serious ramifications from this change.
>4.  The Installation Instructions have a link to the HP Jena site.  If you
>link to it, the
>page says "Oops! ..."  Jena was moved from HP to apache.  So if you want
>to
>do
>Jena, you now need to consult <https://jena.apache.org/>.  I'm not sure
>exactly how
>the apache Jena source code or binary installations compare with what
>IsaViz is expecting.
>As a note, Jena is a BIG chunk of software.  I think the tutorials on RDF
>(including OWL
>and related reasoners) are 

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Bruce Barkstrom]

One factor that should be included in the group's deliberations
on adding a workflow language to the other things in OODT
is the impact on long-term maintenance.  While there's a lot
of enthusiasm in the developer community right now, we need
to think about what happens when development turns into
maintenance.  The account that follows is based on my experience
with trying to resurrect a W3C-related project to visualize RDF
graphs.

The project is called IsaViz.  It's even got a W3 web site:
http://www.w3.org/2001/11/IsaViz/
IsaViz identifies itself as a visual authoring tool for RDF.
Right up near the top are two dates that should serve as
a cautionary note for people who want to pick up this tool:
Current Stable Version: October 2007 and Current Development Version:
May 2007.  It also looks like the site was maintained by a single
developer who did the development as a postdoc project.
The overview page was last modified on Oct. 21, 2007.
The installation instructions were last modified on Oct. 18, 2004.

IsaViz uses a number of tools.  The Installation Instructions
identify the following:

   - A Java Virtual Machine version 1.3.0 or later (1.4 strongly
   recommended - see Known problems
   )
   - A distribution of IsaViz, which contains the following Java JAR files:
  - IsaViz itself *(isaviz.jar)*
  - Zoomable Visual Transformation Machine *(zvtm.jar)*
  - Jena 2.1 for IsaViz 2.1, Jena 1.6.1 for IsaViz 1.2
  - Xerces-J version 2 *(xercesImpl.jar,xmlParserAPIs.jar)* for IsaViz
  2, Xerces 1.4.4 for IsaViz 1.2
   - GraphViz from AT&T version 1.8.9 or later (version 1.7.x is no longer
   supported in IsaViz 2, and has actually only been tested with
version 1.9). *Note:
   some instances of version 1.10.0 had a bug that produced incomplete SVG
   files, but it has been corrected in subsequent releases *(newer versions
   can be obtained on the graphviz.org site
   ).

So, what complications ensue:
1.  Java has moved way beyond version 1.3 or 1.4.  Since Java can deprecate
code and
since there's Oracle and OpenJDK, there may be some unpleasantries that
might need
fixes.  I haven't seen comments from the community on whether or not these
might be
significant.  The IsaViz documentation refers to the ancient time when Sun
controlled
the language.  Apparently, the IsaViz code was only tested with Sun's
j2se/1.3 or 1.4.
2.  I suppose the jar files from IsaViz version 2 would be the place to
start in reconstructing
this piece of software.  However, one might be careful about this when you
get into the
installation scripts from the Installation Instructions.
3.  The Zoomable Visual Transformation Machine project is on Sourceforge.
It's apparently
done in Java.  However, the IsaViz code used version 0.9.0, while the
current Sourceforge
project (at http://zvtm.sourceforge.net/) is now up to 0.11.1 for the
stable version (Aug. 2013)
with a more recent development version (0.11.2 - snapshot; June 2014).  No
idea if
there would be any serious ramifications from this change.
4.  The Installation Instructions have a link to the HP Jena site.  If you
link to it, the
page says "Oops! ..."  Jena was moved from HP to apache.  So if you want to
do
Jena, you now need to consult .  I'm not sure
exactly how
the apache Jena source code or binary installations compare with what
IsaViz is expecting.
As a note, Jena is a BIG chunk of software.  I think the tutorials on RDF
(including OWL
and related reasoners) are going to take a novice user (including many
scientists) a month
or two of dedicated time to work through.  I don't know how easy IsaViz
would be to install
without at least a basic understanding of RDF and of the related triple
store database.
5.  Xerces-J is the XML Java parser (see ), which is now
up to
version 2.11.0.  Again, it isn't clear what kinds of difficulties one would
encounter to use
this library.
6.  GraphViz (at ) is now at version 2.38.0-1.
AT&T seems
to be maintaining a lot of installation options.  I was interested in
Ubuntu - and then
there are different versions of that.

As an additional note, Linux has developed a bunch of variants.  A
particularly active
area of development is the creation of automated package managers - often
with centralized
control over installation procedures and source code libraries.  The
packages have dependencies
on the libraries -- and there's no guarantee that an RPM package has the
same dependencies
as a Debian package.  This is a bit like the DOI guarantee of providing a
unique location to
obtain original items - although publishers have been known to substitute
new versions of
the unique object for the "true" original.

At the same time, software packages with complex networks of dependencies
are
not exactly easy to maintain with Linux (or Unix) scripts.  Exploring the
integrity of
the whole package r

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Lavanya Ramakrishnan]

Here is my 2c -

I think it is important to try and understand what your users are going to
do with workflow and what kind of language they are used to
(domain-specific, functional, etc). They are processes called user-centered
design processes you can use to do this or do at a minimum an informal
study.

 A couple of years ago, we did an introspection on why all the existing
workflow tools didn't have the uptake we had assumed it would. I have been
part of a half dozen different tools over my career. We have since launched
a project called Tigres - http://tigres.lbl.gov/ where we have learned a
lot due to using a user-centered design approach. We have an IEEE eScience
paper on our initial work - which you might find interesting. I am also
happy to share more details on Tigres and/or the process.

Lavanya





On Thu, Sep 18, 2014 at 10:53 AM, BW  wrote:

> Is there a list of graphical BEL workflow tools?
>
> On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > Hi Guys,
> >
> > I've been interested in this too - we don't per have a specific
> > OODT workflow language, but we specific workflows using XML, and
> > other configuration (we are also thinking of moving to JSON for
> > this).
> >
> > In the past I've also looked at YAWL and BPEL - both seem complex
> > to me.
> >
> > I wonder at the end of the day if we should adopt something more
> > modern like PIG or some other data flow type of language (PIG
> > is really neat).
> >
> > Cheers,
> > Chris
> >
> >
> > ++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattm...@nasa.gov 
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++
> >
> >
> >
> >
> >
> >
> > -Original Message-
> > From: Shameera Rathnayaka >
> > Reply-To: "architect...@airavata.apache.org "
> > >
> > Date: Thursday, September 18, 2014 8:26 AM
> > To: "architect...@airavata.apache.org " <
> > architect...@airavata.apache.org >,
> > dev >
> > Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> >
> > >Hi All,
> > >
> > >As we all know Airavata has its own workflow language call XWF. When XWF
> > >was introduced, main focus points are interoperability and
> convertibility.
> > >But with years of experience it is convinced that above requirements are
> > >not really useful when we come to real world use cases. And XWF is XML
> > >based bulky language where we attache WSDLs and Workflow image it self.
> > >But
> > >with the recent changes WSDL part is being removed from XWF.
> > >
> > >It is worth to evaluate handy Scientific workflow languages in industry
> > >and
> > >find out pros and cons, at the end of this evaluation we need to come up
> > >with idea how we should improve Airavata workflow language, either we
> can
> > >improve existing XWF language, totally change to a new language
> available
> > >in industry or write a new light weight language. Basic requirements
> that
> > >we expect from new improvement are, high usability, flexible, light
> weight
> > >and real time monitoring support. As you can see above requirements are
> > >not
> > >direct comes with workflow languages but we need workflow language which
> > >help to support above requirements.
> > >
> > >After reading few papers and googling, initially i have come up with
> > >following three existing languages,
> > >1. YAWL 
> > >2. WS-BPEL
> > >​3. SIDL
> > >
> > >
> > >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> > >also uses slightly modified version of SIDL. Other than above three
> > >languages we can come up with simple workflow language base on json(or
> > >yaml) which support all our requirements for some extends.
> > >
> > >It would be grate if I can get more input regarding the $Subject form
> the
> > >airavata community. You all are more than welcome to provide any type of
> > >suggestions.
> > >
> > >Thanks,
> > >Shameera.
> > >
> > >​
> > >
> > >--
> > >Best Regards,
> > >Shameera Rathnayaka.
> > >
> > >email: shameera AT apache.org , shameerainfo AT gmail.com
> > >Blog : http://shameerarathnayaka.blogspot.com/
> >
> >
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Bruce Barkstrom]

Workflows for either producing or using scientific data
may be very simple (observer looks at a thermometer
and records a temperature using a browser form - and
used to be writes down the temperature in a log book)
to very complex (cf the article in the newest CACM:
Greengard, S., 2014: Weathering a New Era of Big
Data, CACM, 57, No. 9, 12-14 which provides some
statistics on the number of data sources and such
that weather forecasting uses).

Unfortunately, the social groupings of practitioners
are often isolated - leading to misleading views of
how simple the workflows are.  One of the critical
issues in dealing with this realistically is to figure
out when the production workflows create hierarchical
collections of data, usually because of different
versions in the production code.  Bioinformatics
has almost no reprocessing and not much in the
way of versioning.  Operational data collection
(weather and related things) has intermittent version
"glitches" - but not much reporcessing (reanalysis
projects take years).  Climate data records involving
satellite instruments can have VERY complex
collection processes and versioning.

It's also important to be careful about the disciplinary
groupings of different kinds of data users.  For example,
geological data archives can have rock cores turned
in by drilling teams in response to legal requirements
(which means prospectors may want property records
stated in English units and with local political jurisdictions),
while geologists may expect English units (if they're from
American petroleum firms), whereas academic geologists
may want MKS.

I expect choosing a language is going to be a long
and very complex social negotiation process.

Bruce B.

On Thu, Sep 18, 2014 at 1:15 PM, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
>
> -Original Message-
> From: Shameera Rathnayaka 
> Reply-To: "architect...@airavata.apache.org"
> 
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architect...@airavata.apache.org" ,
> dev 
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Hi All,
> >
> >As we all know Airavata has its own workflow language call XWF. When XWF
> >was introduced, main focus points are interoperability and convertibility.
> >But with years of experience it is convinced that above requirements are
> >not really useful when we come to real world use cases. And XWF is XML
> >based bulky language where we attache WSDLs and Workflow image it self.
> >But
> >with the recent changes WSDL part is being removed from XWF.
> >
> >It is worth to evaluate handy Scientific workflow languages in industry
> >and
> >find out pros and cons, at the end of this evaluation we need to come up
> >with idea how we should improve Airavata workflow language, either we can
> >improve existing XWF language, totally change to a new language available
> >in industry or write a new light weight language. Basic requirements that
> >we expect from new improvement are, high usability, flexible, light weight
> >and real time monitoring support. As you can see above requirements are
> >not
> >direct comes with workflow languages but we need workflow language which
> >help to support above requirements.
> >
> >After reading few papers and googling, initially i have come up with
> >following three existing languages,
> >1. YAWL 
> >2. WS-BPEL
> >​3. SIDL
> >
> >
> >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >also uses slightly modified version of SIDL. Other than above three
> >languages we can come up with simple workflow language base on json(or
> >yaml) which support all our requirements for some extends.
> >
> >It would be grate if I can get more input regarding the $Subject form the
> >airavata community. You all are more than welcome to 

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Suresh Marru]

Hi Chris,

Great to hear OODT community will be interested in adopting a JSON based 
workflow language and potentially a web based composer as well. Airavata 
previously had BPEL support initially through a home grown implementation [1] 
by Alek Slominski and later through Apache ODE [2]. Also a white paper [3] by 
Alek on this topic is an interesting read. 

I am of the same opinion that we should adopt something more modern as the 
challenges from scientific workflows seems to be converging with the data flow 
patterns in business workflows. 

It will be great if we can all compile a list of potential candidates and hack 
them through. 

Suresh
[1] - http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
[2] - 
http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_runtime_for_eScience_workflows
[3] - 
http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a326.pdf


On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) 
 wrote:

> Hi Guys,
> 
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
> 
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me. 
> 
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
> 
> Cheers,
> Chris
> 
> 
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
> 
> 
> 
> 
> 
> 
> -Original Message-
> From: Shameera Rathnayaka 
> Reply-To: "architect...@airavata.apache.org"
> 
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architect...@airavata.apache.org" ,
> dev 
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
> 
>> Hi All,
>> 
>> As we all know Airavata has its own workflow language call XWF. When XWF
>> was introduced, main focus points are interoperability and convertibility.
>> But with years of experience it is convinced that above requirements are
>> not really useful when we come to real world use cases. And XWF is XML
>> based bulky language where we attache WSDLs and Workflow image it self.
>> But
>> with the recent changes WSDL part is being removed from XWF.
>> 
>> It is worth to evaluate handy Scientific workflow languages in industry
>> and
>> find out pros and cons, at the end of this evaluation we need to come up
>> with idea how we should improve Airavata workflow language, either we can
>> improve existing XWF language, totally change to a new language available
>> in industry or write a new light weight language. Basic requirements that
>> we expect from new improvement are, high usability, flexible, light weight
>> and real time monitoring support. As you can see above requirements are
>> not
>> direct comes with workflow languages but we need workflow language which
>> help to support above requirements.
>> 
>> After reading few papers and googling, initially i have come up with
>> following three existing languages,
>> 1. YAWL 
>> 2. WS-BPEL
>> ​3. SIDL 
>> 
>> 
>> In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
>> also uses slightly modified version of SIDL. Other than above three
>> languages we can come up with simple workflow language base on json(or
>> yaml) which support all our requirements for some extends.
>> 
>> It would be grate if I can get more input regarding the $Subject form the
>> airavata community. You all are more than welcome to provide any type of
>> suggestions.
>> 
>> Thanks,
>> Shameera.
>> 
>> ​
>> 
>> -- 
>> Best Regards,
>> Shameera Rathnayaka.
>> 
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[BW]

Is there a list of graphical BEL workflow tools?

On Thursday, September 18, 2014, Mattmann, Chris A (3980) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> Hi Guys,
>
> I've been interested in this too - we don't per have a specific
> OODT workflow language, but we specific workflows using XML, and
> other configuration (we are also thinking of moving to JSON for
> this).
>
> In the past I've also looked at YAWL and BPEL - both seem complex
> to me.
>
> I wonder at the end of the day if we should adopt something more
> modern like PIG or some other data flow type of language (PIG
> is really neat).
>
> Cheers,
> Chris
>
>
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov 
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>
>
>
>
>
>
> -Original Message-
> From: Shameera Rathnayaka >
> Reply-To: "architect...@airavata.apache.org "
> >
> Date: Thursday, September 18, 2014 8:26 AM
> To: "architect...@airavata.apache.org " <
> architect...@airavata.apache.org >,
> dev >
> Subject: Evaluate Suitable Scientific Workflow Language for Airavata.
>
> >Hi All,
> >
> >As we all know Airavata has its own workflow language call XWF. When XWF
> >was introduced, main focus points are interoperability and convertibility.
> >But with years of experience it is convinced that above requirements are
> >not really useful when we come to real world use cases. And XWF is XML
> >based bulky language where we attache WSDLs and Workflow image it self.
> >But
> >with the recent changes WSDL part is being removed from XWF.
> >
> >It is worth to evaluate handy Scientific workflow languages in industry
> >and
> >find out pros and cons, at the end of this evaluation we need to come up
> >with idea how we should improve Airavata workflow language, either we can
> >improve existing XWF language, totally change to a new language available
> >in industry or write a new light weight language. Basic requirements that
> >we expect from new improvement are, high usability, flexible, light weight
> >and real time monitoring support. As you can see above requirements are
> >not
> >direct comes with workflow languages but we need workflow language which
> >help to support above requirements.
> >
> >After reading few papers and googling, initially i have come up with
> >following three existing languages,
> >1. YAWL 
> >2. WS-BPEL
> >​3. SIDL
> >
> >
> >In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
> >also uses slightly modified version of SIDL. Other than above three
> >languages we can come up with simple workflow language base on json(or
> >yaml) which support all our requirements for some extends.
> >
> >It would be grate if I can get more input regarding the $Subject form the
> >airavata community. You all are more than welcome to provide any type of
> >suggestions.
> >
> >Thanks,
> >Shameera.
> >
> >​
> >
> >--
> >Best Regards,
> >Shameera Rathnayaka.
> >
> >email: shameera AT apache.org , shameerainfo AT gmail.com
> >Blog : http://shameerarathnayaka.blogspot.com/
>
>

Re: Evaluate Suitable Scientific Workflow Language for Airavata.

[Mattmann, Chris A (3980)]

Hi Guys,

I've been interested in this too - we don't per have a specific
OODT workflow language, but we specific workflows using XML, and
other configuration (we are also thinking of moving to JSON for
this).

In the past I've also looked at YAWL and BPEL - both seem complex
to me. 

I wonder at the end of the day if we should adopt something more
modern like PIG or some other data flow type of language (PIG
is really neat).

Cheers,
Chris


++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++






-Original Message-
From: Shameera Rathnayaka 
Reply-To: "architect...@airavata.apache.org"

Date: Thursday, September 18, 2014 8:26 AM
To: "architect...@airavata.apache.org" ,
dev 
Subject: Evaluate Suitable Scientific Workflow Language for Airavata.

>Hi All,
>
>As we all know Airavata has its own workflow language call XWF. When XWF
>was introduced, main focus points are interoperability and convertibility.
>But with years of experience it is convinced that above requirements are
>not really useful when we come to real world use cases. And XWF is XML
>based bulky language where we attache WSDLs and Workflow image it self.
>But
>with the recent changes WSDL part is being removed from XWF.
>
>It is worth to evaluate handy Scientific workflow languages in industry
>and
>find out pros and cons, at the end of this evaluation we need to come up
>with idea how we should improve Airavata workflow language, either we can
>improve existing XWF language, totally change to a new language available
>in industry or write a new light weight language. Basic requirements that
>we expect from new improvement are, high usability, flexible, light weight
>and real time monitoring support. As you can see above requirements are
>not
>direct comes with workflow languages but we need workflow language which
>help to support above requirements.
>
>After reading few papers and googling, initially i have come up with
>following three existing languages,
>1. YAWL 
>2. WS-BPEL
>​3. SIDL 
>
>
>In my opinion SIDL is more familiar with scientific domain, Radical-SAGA
>also uses slightly modified version of SIDL. Other than above three
>languages we can come up with simple workflow language base on json(or
>yaml) which support all our requirements for some extends.
>
>It would be grate if I can get more input regarding the $Subject form the
>airavata community. You all are more than welcome to provide any type of
>suggestions.
>
>Thanks,
>Shameera.
>
>​
>
>-- 
>Best Regards,
>Shameera Rathnayaka.
>
>email: shameera AT apache.org , shameerainfo AT gmail.com
>Blog : http://shameerarathnayaka.blogspot.com/