Hello,
I am sorry, I am bit late on this thread. But when reading through this
thread I simply got lost, what this thread is discussing. I have few
questions.
1. @Shameera : Is XWF actually a language to define workflow ? To my
understanding it was an intermediate representation to convert workflow
defined in UI to java object model. Was XWF ever used by any airavata user
to define a workflow graph ?
>From initial description what I understood is we are looking for a improved
"intermediate representation", not a language which describes workflows.
2. So what is the exact purpose of this proposed language ?
- Is it to hide parallelism in workflows ?
- Is it to replace the XBAYA functionalities ? (i hope not)
3. What are we trying to achieve by this proposed language which we cannot
achieved through workflow UI tool ?
4. Who is going to use this language ?
5. Why would a user prefer (assuming intended audience of proposed language
is end users) a language over a Work Flow UI tool such as XBAYA ? (In other
words what are the things we can do with language which we cannot do with
UI)
Sorry, if above questions are in-appropriate, just wanted to understand
what exactly needed.
Thanks
-Thejaka Amila
On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva <supu...@gmail.com>
wrote:
> I think I'm not suggesting to create a Workflow interpreter using Python
> etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
> and move it to a more higher level component. The more I think about it,
> the model I'm suggesting is similar to what Hadoop, Storm etc has done for
> distributed system computations. This model is proven to be successful over
> the years.
>
> Keeping what Airavata does at its core can help you to build a more robust
> system. If we look at Airavata as middleware to execute applications on
> computing resources we can simplify what Airavata does and focus on
> improving the core functionality. All the successful systems have thrived
> on defining what it does at its core and keeping it simple and being
> excellent at what it does. In that regard keeping workflow aspect out of
> Airavata can help you to focus on the core problem. That is to execute an
> application on a remote computing resource in a fault tolerant and scalable
> manner.
>
> What I'm suggesting is to give the Orchestrator the capability to execute
> a Driver program that is specified by the user. (This program can be
> written in Python, Java or any other language). This driver program is
> similar to what you define in a Hadoop or Storm configuration. The driver
> program specifies the flow of the computation. It specifies what are the
> applications needs to be executed, how to manipulate input output. The
> driver program is the workflow for the user. Because the user specifies the
> program he can program it to handle workflow steering etc. Every time the
> user wants to execute this program he can tell Airavata to execute the
> Driver program.
>
> I'm also not 100% sure about all the details. But this can be a new way to
> look at how systems like Airavata should behave. Your thoughts and
> suggestions are more than welcome.
>
> Thanks,
> Supun..
>
>
>
>
>
>
> On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <
> shameerai...@gmail.com> wrote:
>
>> Hi Supun,
>>
>> I think we were in two context, because I as suggesting a way to serialize
>> and deserialize the workflow description while you are suggesting to
>> implement
>> some kind of workflow interpreter using Python, where Python client can
>> send
>> thrift calls to Airavata server to run the application. I can see with
>> your
>> suggested
>> approach we can control the workflow execution process from client side
>> which
>> make it easy to implement workflow debugger.As you mentioned this is a
>> major change
>> to Airavata. So we should neatly think as this will change our existing
>> architecture.
>>
>> Still if someone need to use different language java, php, JS etc ... to
>> run the same
>> workflow which generated by Python, we need a language independent
>> workflow
>> description.
>> My initial question was what is the best language for this?. But as I have
>> explained in
>> one of my previous reply, It is not matter what language we used Either we
>> can use
>> XML or JSON to write this description, what matters is how easy to
>> generate
>> workflow with the provided API. Hence it would be great to have set of
>> neat
>> APIs in
>> different languages.
>>
>> Thanks,
>> Shameera.
>>
>>
>> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supu...@gmail.com>
>> wrote:
>>
>> > Hi Shameera,
>> >
>> > Using python is a radical approach for workflow I think. But I believe
>> this
>> > is a very beautiful and new approach.Here is a possible scenario for
>> > implementation and running using a Python based library.
>> >
>> > The Python library facilitates the creation of Applications and
>> submitting
>> > them to Airavata for execution. The underlying functionality is exactly
>> > similar to what Java clients provides.The only difference is that,
>> Python
>> > library should have a more fluent API than Java for easy creation of
>> > workflows. We can generate the Python clients that talk to Airavata
>> server
>> > using Thrift.
>> >
>> > Here is an example off the top of my head to a Python script created by
>> > user for a Workflow. This is a very crude example and we need to come up
>> > with a much better API if we are going to go along this path. First we
>> need
>> > to write a Python script that can execute a workflow using Airavata.
>> >
>> > import airavata
>> >
>> > host = Host("localhost", ....)
>> > app1 = Application(host, ....)
>> > app2 = Application(host, ....)
>> >
>> > # we will connect these applications as a workflow using some topology
>> > builder or other constructs
>> >
>> > wb = WorkFlowBuilder()
>> > wb.setApp("name1", app1)
>> > # you can do a simple output transformation here etc
>> >
>> > # connects the input of app1 to app2 etc
>> > wb.setApp("name2", app2).connectInput("name1")
>> >
>> > wb.submit()
>> >
>> > Now we can load this Python script from XBaya. When XBaya loads this
>> script
>> > the Python script can output an XML configuration of the topology, XBaya
>> > can render. There are other ways like directly executing the Python
>> script
>> > from command line and connecting XBaya indirectly as well. Now you can
>> run
>> > the workflow from XBaya. Running the Workflow means just executing the
>> > Python script.
>> >
>> > XBaya gets the notifications through messaging and update the UI
>> > accordingly.
>> >
>> > The users need to write the Python script by hand. XBaya cannot create
>> the
>> > script. Because workflow language is an actual python program the
>> benefits
>> > are immense. For example user can do workflow steering in the workflow
>> > itself by subscribing to messages from Airavata.
>> >
>> > Thanks,
>> > Supun..
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
>> > shameerai...@gmail.com> wrote:
>> >
>> > > Hi Supun,
>> > >
>> > > I meant to say JS is a well-known client side scripting language i
>> have
>> > > messed scripting part. Even we use Python, ultimately we should
>> convert
>> > > this to java model in server side, somehow we need to serialized
>> python
>> > > representation to the language which java can parse and generate that
>> > > model. In this case we need to parse python script in java isn't it?
>> I am
>> > > not exactly clear how you suggesting to use python here. More details
>> on
>> > > how end system works if we use Python would be great help to clearly
>> > > understand your points.
>> > >
>> > > Thanks,
>> > > Shameera.
>> > >
>> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
>> > chris.mattm...@gmail.com>
>> > > wrote:
>> > >
>> > > > Have you guys considered using JCC [1] as a means
>> > > > to expose the workflow API currently in Java as a
>> > > > Python API?
>> > > >
>> > > > We are exploring its use in OODT, and we have already
>> > > > created a Tika [2] JCC-based python API.
>> > > >
>> > > > Cheers,
>> > > > Chris
>> > > >
>> > > > [1] http://lucene.apache.org/pylucene/jcc/
>> > > > [2] http://github.com/chrismattmann/tika-python/
>> > > >
>> > > > ------------------------
>> > > > Chris Mattmann
>> > > > chris.mattm...@gmail.com
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > -----Original Message-----
>> > > > From: Supun Kamburugamuva <supu...@gmail.com>
>> > > > Reply-To: <dev@oodt.apache.org>
>> > > > Date: Thursday, October 16, 2014 at 3:43 PM
>> > > > To: dev <d...@airavata.apache.org>
>> > > > Cc: "Alek Jones (Indiana)" <alek...@gmail.com>, Suresh Marru
>> > > > <sma...@apache.org>, "architect...@airavata.apache.org"
>> > > > <architect...@airavata.apache.org>, "dev@oodt.apache.org"
>> > > > <dev@oodt.apache.org>
>> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
>> > Airavata.
>> > > >
>> > > > >Once we had an offline discussion about the Airavata Workflow
>> language
>> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
>> came
>> > > out
>> > > > >was why we need to invent a different language when a simple
>> library
>> > > like
>> > > > >Python will full fill of Airavata requirements.
>> > > > >
>> > > > >There are many benefits in using a Python library as the API for
>> > > > >controlling Airavata workflows.
>> > > > >
>> > > > >1. It is a library, gives the ultimate control over the execution
>> and
>> > it
>> > > > >can be simpler than any domain specific language that we can come
>> with
>> > > > >like
>> > > > >XML, JSON etc
>> > > > >2. Most people use python and can learn it easily than any Domain
>> > > specific
>> > > > >language
>> > > > >3. You can easily create a Python library for Airavata because all
>> the
>> > > > >APIs
>> > > > >are thrift based.
>> > > > >4. If you design the constructs correctly you can plug an XBaya.
>> > > > >
>> > > > >Any thoughts?
>> > > > >
>> > > > >Thanks,
>> > > > >Supun..
>> > > > >
>> > > > >
>> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
>> > supu...@gmail.com
>> > > >
>> > > > >wrote:
>> > > > >
>> > > > >> Hi Shameera,
>> > > > >>
>> > > > >> Why you prefer JavaScript over a language like Python?
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Supun..
>> > > > >>
>> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
>> > > > >> shameerai...@gmail.com> wrote:
>> > > > >>
>> > > > >>> ​Hi,
>> > > > >>>
>> > > > >>> First of all thanks everyone for giving valuable inputs. After
>> > doing
>> > > > >>>some
>> > > > >>> background search and talking to different people in the
>> University
>> > > > >>>who has
>> > > > >>> used different workflow languages, I myself convinced that
>> > > introducing
>> > > > >>>an
>> > > > >>> another workflow language is not what actually they need. By
>> > changing
>> > > > >>> exiting workflow language to another will not solve problems.
>> What
>> > > they
>> > > > >>> asking is a easy way to construct the workflows. Indirectly what
>> > they
>> > > > >>> asking for a sort of API which they can use to generate the
>> > workflows
>> > > > >>>and
>> > > > >>> run it. Correct me if i am wrong here.
>> > > > >>>
>> > > > >>> As most of above replies depict, if we can get a simple API,
>> as an
>> > > > >>> example, for a web based application, JavaScript API would be a
>> > good
>> > > > >>> solution, and probably JSON would be a good candidate for
>> language,
>> > > > >>>instead
>> > > > >>> of XML.
>> > > > >>>
>> > > > >>> Airavata community already have started to implement web base
>> GUI.
>> > > > >>>Hence
>> > > > >>> introducing a JSON base JavaScript API would be great help.
>> WDYT?
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Shameera.
>> > > > >>>
>> > > > >>>
>> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
>> > > > >>> alek...@gmail.com> wrote:
>> > > > >>>
>> > > > >>>> Hi,
>> > > > >>>>
>> > > > >>>> it is not dataflow instead focused on orchestrating REST
>> services
>> > > but
>> > > > >>>> you may find it useful datapoint - we created worfklow service
>> > that
>> > > > >>>>uses
>> > > > >>>> natively JavaScript and JSON to describe what happens during
>> > > workflow
>> > > > >>>> execution:
>> > > > >>>>
>> > > >
>> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
>> > > > >>>>
>> > > > >>>> HTH,
>> > > > >>>>
>> > > > >>>> Alek
>> > > > >>>>
>> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
>> sma...@apache.org>
>> > > > >>>>wrote:
>> > > > >>>>
>> > > > >>>>> Hi Chris,
>> > > > >>>>>
>> > > > >>>>> Great to hear OODT community will be interested in adopting a
>> > JSON
>> > > > >>>>> based workflow language and potentially a web based composer
>> as
>> > > well.
>> > > > >>>>> Airavata previously had BPEL support initially through a home
>> > grown
>> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
>> ODE
>> > > > >>>>>[2]. Also
>> > > > >>>>> a white paper [3] by Alek on this topic is an interesting
>> read.
>> > > > >>>>>
>> > > > >>>>> I am of the same opinion that we should adopt something more
>> > modern
>> > > > >>>>>as
>> > > > >>>>> the challenges from scientific workflows seems to be
>> converging
>> > > with
>> > > > >>>>>the
>> > > > >>>>> data flow patterns in business workflows.
>> > > > >>>>>
>> > > > >>>>> It will be great if we can all compile a list of potential
>> > > candidates
>> > > > >>>>> and hack them through.
>> > > > >>>>>
>> > > > >>>>> Suresh
>> > > > >>>>> [1] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
>> > > > >>>>> [2] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
>> > > > >>>>>time_for_eScience_workflows
>> > > > >>>>> [3] -
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
>> > > > >>>>>6.pdf
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
>> > > > >>>>> chris.a.mattm...@jpl.nasa.gov> wrote:
>> > > > >>>>>
>> > > > >>>>> > Hi Guys,
>> > > > >>>>> >
>> > > > >>>>> > I've been interested in this too - we don't per have a
>> specific
>> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
>> > and
>> > > > >>>>> > other configuration (we are also thinking of moving to JSON
>> for
>> > > > >>>>> > this).
>> > > > >>>>> >
>> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
>> > complex
>> > > > >>>>> > to me.
>> > > > >>>>> >
>> > > > >>>>> > I wonder at the end of the day if we should adopt something
>> > more
>> > > > >>>>> > modern like PIG or some other data flow type of language
>> (PIG
>> > > > >>>>> > is really neat).
>> > > > >>>>> >
>> > > > >>>>> > Cheers,
>> > > > >>>>> > Chris
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Chris Mattmann, Ph.D.
>> > > > >>>>> > Chief Architect
>> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
>> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> > > > >>>>> > Office: 168-519, Mailstop: 168-527
>> > > > >>>>> > Email: chris.a.mattm...@nasa.gov
>> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
>> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
>> > > > >>>>> >
>> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> >
>> > > > >>>>> > -----Original Message-----
>> > > > >>>>> > From: Shameera Rathnayaka <shameerai...@gmail.com>
>> > > > >>>>> > Reply-To: "architect...@airavata.apache.org"
>> > > > >>>>> > <architect...@airavata.apache.org>
>> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
>> > > > >>>>> > To: "architect...@airavata.apache.org" <
>> > > > >>>>> architect...@airavata.apache.org>,
>> > > > >>>>> > dev <d...@airavata.apache.org>
>> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
>> > > > >>>>>Airavata.
>> > > > >>>>> >
>> > > > >>>>> >> Hi All,
>> > > > >>>>> >>
>> > > > >>>>> >> As we all know Airavata has its own workflow language call
>> > XWF.
>> > > > >>>>>When
>> > > > >>>>> XWF
>> > > > >>>>> >> was introduced, main focus points are interoperability and
>> > > > >>>>> convertibility.
>> > > > >>>>> >> But with years of experience it is convinced that above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not really useful when we come to real world use cases. And
>> > XWF
>> > > is
>> > > > >>>>> XML
>> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
>> image
>> > > it
>> > > > >>>>> self.
>> > > > >>>>> >> But
>> > > > >>>>> >> with the recent changes WSDL part is being removed from
>> XWF.
>> > > > >>>>> >>
>> > > > >>>>> >> It is worth to evaluate handy Scientific workflow
>> languages in
>> > > > >>>>> industry
>> > > > >>>>> >> and
>> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
>> need
>> > to
>> > > > >>>>> come up
>> > > > >>>>> >> with idea how we should improve Airavata workflow language,
>> > > either
>> > > > >>>>> we can
>> > > > >>>>> >> improve existing XWF language, totally change to a new
>> > language
>> > > > >>>>> available
>> > > > >>>>> >> in industry or write a new light weight language. Basic
>> > > > >>>>>requirements
>> > > > >>>>> that
>> > > > >>>>> >> we expect from new improvement are, high usability,
>> flexible,
>> > > > >>>>>light
>> > > > >>>>> weight
>> > > > >>>>> >> and real time monitoring support. As you can see above
>> > > > >>>>>requirements
>> > > > >>>>> are
>> > > > >>>>> >> not
>> > > > >>>>> >> direct comes with workflow languages but we need workflow
>> > > language
>> > > > >>>>> which
>> > > > >>>>> >> help to support above requirements.
>> > > > >>>>> >>
>> > > > >>>>> >> After reading few papers and googling, initially i have
>> come
>> > up
>> > > > >>>>>with
>> > > > >>>>> >> following three existing languages,
>> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
>> > > > >>>>> >> 2. WS-BPEL
>> > > > >>>>> >> ​3. SIDL
>> > > > >>>>> >> <
>> > > http://computation.llnl.gov/casc/components/index.html#page=home
>> > > > >
>> > > > >>>>> >>
>> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
>> > > > >>>>> Radical-SAGA
>> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
>> above
>> > > > >>>>>three
>> > > > >>>>> >> languages we can come up with simple workflow language
>> base on
>> > > > >>>>> json(or
>> > > > >>>>> >> yaml) which support all our requirements for some extends.
>> > > > >>>>> >>
>> > > > >>>>> >> It would be grate if I can get more input regarding the
>> > $Subject
>> > > > >>>>> form the
>> > > > >>>>> >> airavata community. You all are more than welcome to
>> provide
>> > any
>> > > > >>>>> type of
>> > > > >>>>> >> suggestions.
>> > > > >>>>> >>
>> > > > >>>>> >> Thanks,
>> > > > >>>>> >> Shameera.
>> > > > >>>>> >>
>> > > > >>>>> >> ​
>> > > > >>>>> >>
>> > > > >>>>> >> --
>> > > > >>>>> >> Best Regards,
>> > > > >>>>> >> Shameera Rathnayaka.
>> > > > >>>>> >>
>> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> The best way to predict the future is to invent it - Alan Kay
>> > > > >>>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> --
>> > > > >>> Best Regards,
>> > > > >>> Shameera Rathnayaka.
>> > > > >>>
>> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Supun Kamburugamuva
>> > > > >> Member, Apache Software Foundation; http://www.apache.org
>> > > > >> E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
>> > > > >> Blog: http://supunk.blogspot.com
>> > > > >>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >--
>> > > > >Supun Kamburugamuva
>> > > > >Member, Apache Software Foundation; http://www.apache.org
>> > > > >E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
>> > > > >Blog: http://supunk.blogspot.com
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best Regards,
>> > > Shameera Rathnayaka.
>> > >
>> > > email: shameera AT apache.org , shameerainfo AT gmail.com
>> > > Blog : http://shameerarathnayaka.blogspot.com/
>> > >
>> >
>> >
>> >
>> > --
>> > Supun Kamburugamuva
>> > Member, Apache Software Foundation; http://www.apache.org
>> > E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
>> > Blog: http://supunk.blogspot.com
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org , shameerainfo AT gmail.com
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>
>
>
> --
> Supun Kamburugamuva
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
> Blog: http://supunk.blogspot.com
>
>
