I think I'm not suggesting to create a Workflow interpreter using Python
etc. What I'm suggesting is to remove the Worflow aspect from core Airavata
and move it to a more higher level component. The more I think about it,
the model I'm suggesting is similar to what Hadoop, Storm etc has done for
distributed system computations. This model is proven to be successful over
the years.
Keeping what Airavata does at its core can help you to build a more robust
system. If we look at Airavata as middleware to execute applications on
computing resources we can simplify what Airavata does and focus on
improving the core functionality. All the successful systems have thrived
on defining what it does at its core and keeping it simple and being
excellent at what it does. In that regard keeping workflow aspect out of
Airavata can help you to focus on the core problem. That is to execute an
application on a remote computing resource in a fault tolerant and scalable
manner.
What I'm suggesting is to give the Orchestrator the capability to execute a
Driver program that is specified by the user. (This program can be written
in Python, Java or any other language). This driver program is similar to
what you define in a Hadoop or Storm configuration. The driver program
specifies the flow of the computation. It specifies what are the
applications needs to be executed, how to manipulate input output. The
driver program is the workflow for the user. Because the user specifies the
program he can program it to handle workflow steering etc. Every time the
user wants to execute this program he can tell Airavata to execute the
Driver program.
I'm also not 100% sure about all the details. But this can be a new way to
look at how systems like Airavata should behave. Your thoughts and
suggestions are more than welcome.
Thanks,
Supun..
On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka <shameerai...@gmail.com
> wrote:
> Hi Supun,
>
> I think we were in two context, because I as suggesting a way to serialize
> and deserialize the workflow description while you are suggesting to
> implement
> some kind of workflow interpreter using Python, where Python client can
> send
> thrift calls to Airavata server to run the application. I can see with your
> suggested
> approach we can control the workflow execution process from client side
> which
> make it easy to implement workflow debugger.As you mentioned this is a
> major change
> to Airavata. So we should neatly think as this will change our existing
> architecture.
>
> Still if someone need to use different language java, php, JS etc ... to
> run the same
> workflow which generated by Python, we need a language independent workflow
> description.
> My initial question was what is the best language for this?. But as I have
> explained in
> one of my previous reply, It is not matter what language we used Either we
> can use
> XML or JSON to write this description, what matters is how easy to generate
> workflow with the provided API. Hence it would be great to have set of neat
> APIs in
> different languages.
>
> Thanks,
> Shameera.
>
>
> On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva <supu...@gmail.com>
> wrote:
>
> > Hi Shameera,
> >
> > Using python is a radical approach for workflow I think. But I believe
> this
> > is a very beautiful and new approach.Here is a possible scenario for
> > implementation and running using a Python based library.
> >
> > The Python library facilitates the creation of Applications and
> submitting
> > them to Airavata for execution. The underlying functionality is exactly
> > similar to what Java clients provides.The only difference is that, Python
> > library should have a more fluent API than Java for easy creation of
> > workflows. We can generate the Python clients that talk to Airavata
> server
> > using Thrift.
> >
> > Here is an example off the top of my head to a Python script created by
> > user for a Workflow. This is a very crude example and we need to come up
> > with a much better API if we are going to go along this path. First we
> need
> > to write a Python script that can execute a workflow using Airavata.
> >
> > import airavata
> >
> > host = Host("localhost", ....)
> > app1 = Application(host, ....)
> > app2 = Application(host, ....)
> >
> > # we will connect these applications as a workflow using some topology
> > builder or other constructs
> >
> > wb = WorkFlowBuilder()
> > wb.setApp("name1", app1)
> > # you can do a simple output transformation here etc
> >
> > # connects the input of app1 to app2 etc
> > wb.setApp("name2", app2).connectInput("name1")
> >
> > wb.submit()
> >
> > Now we can load this Python script from XBaya. When XBaya loads this
> script
> > the Python script can output an XML configuration of the topology, XBaya
> > can render. There are other ways like directly executing the Python
> script
> > from command line and connecting XBaya indirectly as well. Now you can
> run
> > the workflow from XBaya. Running the Workflow means just executing the
> > Python script.
> >
> > XBaya gets the notifications through messaging and update the UI
> > accordingly.
> >
> > The users need to write the Python script by hand. XBaya cannot create
> the
> > script. Because workflow language is an actual python program the
> benefits
> > are immense. For example user can do workflow steering in the workflow
> > itself by subscribing to messages from Airavata.
> >
> > Thanks,
> > Supun..
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 12:41 PM, Shameera Rathnayaka <
> > shameerai...@gmail.com> wrote:
> >
> > > Hi Supun,
> > >
> > > I meant to say JS is a well-known client side scripting language i have
> > > messed scripting part. Even we use Python, ultimately we should convert
> > > this to java model in server side, somehow we need to serialized python
> > > representation to the language which java can parse and generate that
> > > model. In this case we need to parse python script in java isn't it? I
> am
> > > not exactly clear how you suggesting to use python here. More details
> on
> > > how end system works if we use Python would be great help to clearly
> > > understand your points.
> > >
> > > Thanks,
> > > Shameera.
> > >
> > > On Fri, Oct 17, 2014 at 1:00 AM, Chris Mattmann <
> > chris.mattm...@gmail.com>
> > > wrote:
> > >
> > > > Have you guys considered using JCC [1] as a means
> > > > to expose the workflow API currently in Java as a
> > > > Python API?
> > > >
> > > > We are exploring its use in OODT, and we have already
> > > > created a Tika [2] JCC-based python API.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > [1] http://lucene.apache.org/pylucene/jcc/
> > > > [2] http://github.com/chrismattmann/tika-python/
> > > >
> > > > ------------------------
> > > > Chris Mattmann
> > > > chris.mattm...@gmail.com
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Supun Kamburugamuva <supu...@gmail.com>
> > > > Reply-To: <dev@oodt.apache.org>
> > > > Date: Thursday, October 16, 2014 at 3:43 PM
> > > > To: dev <d...@airavata.apache.org>
> > > > Cc: "Alek Jones (Indiana)" <alek...@gmail.com>, Suresh Marru
> > > > <sma...@apache.org>, "architect...@airavata.apache.org"
> > > > <architect...@airavata.apache.org>, "dev@oodt.apache.org"
> > > > <dev@oodt.apache.org>
> > > > Subject: Re: Evaluate Suitable Scientific Workflow Language for
> > Airavata.
> > > >
> > > > >Once we had an offline discussion about the Airavata Workflow
> language
> > > > >(with Milinda, Saliya and Shameera). In that discussion one thing
> came
> > > out
> > > > >was why we need to invent a different language when a simple library
> > > like
> > > > >Python will full fill of Airavata requirements.
> > > > >
> > > > >There are many benefits in using a Python library as the API for
> > > > >controlling Airavata workflows.
> > > > >
> > > > >1. It is a library, gives the ultimate control over the execution
> and
> > it
> > > > >can be simpler than any domain specific language that we can come
> with
> > > > >like
> > > > >XML, JSON etc
> > > > >2. Most people use python and can learn it easily than any Domain
> > > specific
> > > > >language
> > > > >3. You can easily create a Python library for Airavata because all
> the
> > > > >APIs
> > > > >are thrift based.
> > > > >4. If you design the constructs correctly you can plug an XBaya.
> > > > >
> > > > >Any thoughts?
> > > > >
> > > > >Thanks,
> > > > >Supun..
> > > > >
> > > > >
> > > > >On Thu, Oct 16, 2014 at 6:30 PM, Supun Kamburugamuva <
> > supu...@gmail.com
> > > >
> > > > >wrote:
> > > > >
> > > > >> Hi Shameera,
> > > > >>
> > > > >> Why you prefer JavaScript over a language like Python?
> > > > >>
> > > > >> Thanks,
> > > > >> Supun..
> > > > >>
> > > > >> On Thu, Oct 16, 2014 at 6:25 PM, Shameera Rathnayaka <
> > > > >> shameerai...@gmail.com> wrote:
> > > > >>
> > > > >>> ​Hi,
> > > > >>>
> > > > >>> First of all thanks everyone for giving valuable inputs. After
> > doing
> > > > >>>some
> > > > >>> background search and talking to different people in the
> University
> > > > >>>who has
> > > > >>> used different workflow languages, I myself convinced that
> > > introducing
> > > > >>>an
> > > > >>> another workflow language is not what actually they need. By
> > changing
> > > > >>> exiting workflow language to another will not solve problems.
> What
> > > they
> > > > >>> asking is a easy way to construct the workflows. Indirectly what
> > they
> > > > >>> asking for a sort of API which they can use to generate the
> > workflows
> > > > >>>and
> > > > >>> run it. Correct me if i am wrong here.
> > > > >>>
> > > > >>> As most of above replies depict, if we can get a simple API, as
> an
> > > > >>> example, for a web based application, JavaScript API would be a
> > good
> > > > >>> solution, and probably JSON would be a good candidate for
> language,
> > > > >>>instead
> > > > >>> of XML.
> > > > >>>
> > > > >>> Airavata community already have started to implement web base
> GUI.
> > > > >>>Hence
> > > > >>> introducing a JSON base JavaScript API would be great help. WDYT?
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Shameera.
> > > > >>>
> > > > >>>
> > > > >>> On Fri, Sep 19, 2014 at 5:01 PM, Aleksander Slominski (NY) <
> > > > >>> alek...@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> it is not dataflow instead focused on orchestrating REST
> services
> > > but
> > > > >>>> you may find it useful datapoint - we created worfklow service
> > that
> > > > >>>>uses
> > > > >>>> natively JavaScript and JSON to describe what happens during
> > > workflow
> > > > >>>> execution:
> > > > >>>>
> > > >
> https://www.ng.bluemix.net/docs/#services/workflow/index.html#coewf002
> > > > >>>>
> > > > >>>> HTH,
> > > > >>>>
> > > > >>>> Alek
> > > > >>>>
> > > > >>>> On Thu, Sep 18, 2014 at 1:54 PM, Suresh Marru <
> sma...@apache.org>
> > > > >>>>wrote:
> > > > >>>>
> > > > >>>>> Hi Chris,
> > > > >>>>>
> > > > >>>>> Great to hear OODT community will be interested in adopting a
> > JSON
> > > > >>>>> based workflow language and potentially a web based composer as
> > > well.
> > > > >>>>> Airavata previously had BPEL support initially through a home
> > grown
> > > > >>>>> implementation [1] by Alek Slominski and later through Apache
> ODE
> > > > >>>>>[2]. Also
> > > > >>>>> a white paper [3] by Alek on this topic is an interesting read.
> > > > >>>>>
> > > > >>>>> I am of the same opinion that we should adopt something more
> > modern
> > > > >>>>>as
> > > > >>>>> the challenges from scientific workflows seems to be converging
> > > with
> > > > >>>>>the
> > > > >>>>> data flow patterns in business workflows.
> > > > >>>>>
> > > > >>>>> It will be great if we can all compile a list of potential
> > > candidates
> > > > >>>>> and hack them through.
> > > > >>>>>
> > > > >>>>> Suresh
> > > > >>>>> [1] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://link.springer.com/chapter/10.1007%2F978-1-84628-757-2_14#page-1
> > > > >>>>> [2] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.academia.edu/1485773/Experience_with_adapting_a_WS-BPEL_run
> > > > >>>>>time_for_eScience_workflows
> > > > >>>>> [3] -
> > > > >>>>>
> > > > >>>>>
> > > >
> http://www.computer.org/csdl/proceedings/services/2010/4129/00/4129a32
> > > > >>>>>6.pdf
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sep 18, 2014, at 1:15 PM, Mattmann, Chris A (3980) <
> > > > >>>>> chris.a.mattm...@jpl.nasa.gov> wrote:
> > > > >>>>>
> > > > >>>>> > Hi Guys,
> > > > >>>>> >
> > > > >>>>> > I've been interested in this too - we don't per have a
> specific
> > > > >>>>> > OODT workflow language, but we specific workflows using XML,
> > and
> > > > >>>>> > other configuration (we are also thinking of moving to JSON
> for
> > > > >>>>> > this).
> > > > >>>>> >
> > > > >>>>> > In the past I've also looked at YAWL and BPEL - both seem
> > complex
> > > > >>>>> > to me.
> > > > >>>>> >
> > > > >>>>> > I wonder at the end of the day if we should adopt something
> > more
> > > > >>>>> > modern like PIG or some other data flow type of language (PIG
> > > > >>>>> > is really neat).
> > > > >>>>> >
> > > > >>>>> > Cheers,
> > > > >>>>> > Chris
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Chris Mattmann, Ph.D.
> > > > >>>>> > Chief Architect
> > > > >>>>> > Instrument Software and Science Data Systems Section (398)
> > > > >>>>> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > >>>>> > Office: 168-519, Mailstop: 168-527
> > > > >>>>> > Email: chris.a.mattm...@nasa.gov
> > > > >>>>> > WWW: http://sunset.usc.edu/~mattmann/
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> > Adjunct Associate Professor, Computer Science Department
> > > > >>>>> > University of Southern California, Los Angeles, CA 90089 USA
> > > > >>>>> >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> >
> > > > >>>>> > -----Original Message-----
> > > > >>>>> > From: Shameera Rathnayaka <shameerai...@gmail.com>
> > > > >>>>> > Reply-To: "architect...@airavata.apache.org"
> > > > >>>>> > <architect...@airavata.apache.org>
> > > > >>>>> > Date: Thursday, September 18, 2014 8:26 AM
> > > > >>>>> > To: "architect...@airavata.apache.org" <
> > > > >>>>> architect...@airavata.apache.org>,
> > > > >>>>> > dev <d...@airavata.apache.org>
> > > > >>>>> > Subject: Evaluate Suitable Scientific Workflow Language for
> > > > >>>>>Airavata.
> > > > >>>>> >
> > > > >>>>> >> Hi All,
> > > > >>>>> >>
> > > > >>>>> >> As we all know Airavata has its own workflow language call
> > XWF.
> > > > >>>>>When
> > > > >>>>> XWF
> > > > >>>>> >> was introduced, main focus points are interoperability and
> > > > >>>>> convertibility.
> > > > >>>>> >> But with years of experience it is convinced that above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not really useful when we come to real world use cases. And
> > XWF
> > > is
> > > > >>>>> XML
> > > > >>>>> >> based bulky language where we attache WSDLs and Workflow
> image
> > > it
> > > > >>>>> self.
> > > > >>>>> >> But
> > > > >>>>> >> with the recent changes WSDL part is being removed from XWF.
> > > > >>>>> >>
> > > > >>>>> >> It is worth to evaluate handy Scientific workflow languages
> in
> > > > >>>>> industry
> > > > >>>>> >> and
> > > > >>>>> >> find out pros and cons, at the end of this evaluation we
> need
> > to
> > > > >>>>> come up
> > > > >>>>> >> with idea how we should improve Airavata workflow language,
> > > either
> > > > >>>>> we can
> > > > >>>>> >> improve existing XWF language, totally change to a new
> > language
> > > > >>>>> available
> > > > >>>>> >> in industry or write a new light weight language. Basic
> > > > >>>>>requirements
> > > > >>>>> that
> > > > >>>>> >> we expect from new improvement are, high usability,
> flexible,
> > > > >>>>>light
> > > > >>>>> weight
> > > > >>>>> >> and real time monitoring support. As you can see above
> > > > >>>>>requirements
> > > > >>>>> are
> > > > >>>>> >> not
> > > > >>>>> >> direct comes with workflow languages but we need workflow
> > > language
> > > > >>>>> which
> > > > >>>>> >> help to support above requirements.
> > > > >>>>> >>
> > > > >>>>> >> After reading few papers and googling, initially i have come
> > up
> > > > >>>>>with
> > > > >>>>> >> following three existing languages,
> > > > >>>>> >> 1. YAWL <http://www.yawlfoundation.org/>
> > > > >>>>> >> 2. WS-BPEL
> > > > >>>>> >> ​3. SIDL
> > > > >>>>> >> <
> > > http://computation.llnl.gov/casc/components/index.html#page=home
> > > > >
> > > > >>>>> >>
> > > > >>>>> >> In my opinion SIDL is more familiar with scientific domain,
> > > > >>>>> Radical-SAGA
> > > > >>>>> >> also uses slightly modified version of SIDL. Other than
> above
> > > > >>>>>three
> > > > >>>>> >> languages we can come up with simple workflow language base
> on
> > > > >>>>> json(or
> > > > >>>>> >> yaml) which support all our requirements for some extends.
> > > > >>>>> >>
> > > > >>>>> >> It would be grate if I can get more input regarding the
> > $Subject
> > > > >>>>> form the
> > > > >>>>> >> airavata community. You all are more than welcome to provide
> > any
> > > > >>>>> type of
> > > > >>>>> >> suggestions.
> > > > >>>>> >>
> > > > >>>>> >> Thanks,
> > > > >>>>> >> Shameera.
> > > > >>>>> >>
> > > > >>>>> >> ​
> > > > >>>>> >>
> > > > >>>>> >> --
> > > > >>>>> >> Best Regards,
> > > > >>>>> >> Shameera Rathnayaka.
> > > > >>>>> >>
> > > > >>>>> >> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>>>> >> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> The best way to predict the future is to invent it - Alan Kay
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best Regards,
> > > > >>> Shameera Rathnayaka.
> > > > >>>
> > > > >>> email: shameera AT apache.org , shameerainfo AT gmail.com
> > > > >>> Blog : http://shameerarathnayaka.blogspot.com/
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Supun Kamburugamuva
> > > > >> Member, Apache Software Foundation; http://www.apache.org
> > > > >> E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
> > > > >> Blog: http://supunk.blogspot.com
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > >--
> > > > >Supun Kamburugamuva
> > > > >Member, Apache Software Foundation; http://www.apache.org
> > > > >E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
> > > > >Blog: http://supunk.blogspot.com
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Shameera Rathnayaka.
> > >
> > > email: shameera AT apache.org , shameerainfo AT gmail.com
> > > Blog : http://shameerarathnayaka.blogspot.com/
> > >
> >
> >
> >
> > --
> > Supun Kamburugamuva
> > Member, Apache Software Foundation; http://www.apache.org
> > E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
> > Blog: http://supunk.blogspot.com
> >
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
