Build fails in gfac-bes with a fresh m2 repo
[INFO] Airavata GFac GRAM implementation .. SUCCESS [ 0.713 s] [INFO] Airavata GFac BES implementation ... FAILURE [ 1.917 s] [INFO] Airavata Security Implementation ... SKIPPED [INFO] Airavata Credential Store Service .. SKIPPED [INFO] airavata-credential-store-webapp ... SKIPPED [INFO] Airavata Orchestrator Client SDK ... SKIPPED [INFO] Airavata Tools . SKIPPED [INFO] registry-tool .. SKIPPED [INFO] Airavata Standalone Server . SKIPPED [INFO] Airavata Test Suite SKIPPED [INFO] distribution ... SKIPPED [INFO] Airavata server distribution ... SKIPPED [INFO] Airavata Client Distribution Parent SKIPPED [INFO] Airavata Client Java Distribution .. SKIPPED [INFO] Airavata release artifacts . SKIPPED [INFO] Airavata Integration Tests . SKIPPED [INFO] Airavata XBaya . SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 01:25 min [INFO] Finished at: 2014-10-20T13:42:35-04:00 [INFO] Final Memory: 121M/1408M [INFO] [ERROR] Failed to execute goal on project airavata-gfac-bes: Could not resolve dependencies for project org.apache.airavata:airavata-gfac-bes:jar:0.14-SNAPSHOT: The following artifacts could not be resolved: net.sf.saxon:saxon:jar:9.1.0.8, net.sf.saxon:saxon-dom:jar:9.1.0.8, net.sf.saxon:saxon-xpath:jar:9.1.0.8: Could not find artifact net.sf.saxon:saxon:jar:9.1.0.8 in central ( http://repo1.maven.org/maven2) - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :airavata-gfac-bes [airavata@156-56-179-160 airavata]$ mvn clean install -Dm -- Research Assistant Science Gateways Group Indiana University
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hello, I am sorry, I am bit late on this thread. But when reading through this thread I simply got lost, what this thread is discussing. I have few questions. 1. @Shameera : Is XWF actually a language to define workflow ? To my understanding it was an intermediate representation to convert workflow defined in UI to java object model. Was XWF ever used by any airavata user to define a workflow graph ? From initial description what I understood is we are looking for a improved intermediate representation, not a language which describes workflows. 2. So what is the exact purpose of this proposed language ? - Is it to hide parallelism in workflows ? - Is it to replace the XBAYA functionalities ? (i hope not) 3. What are we trying to achieve by this proposed language which we cannot achieved through workflow UI tool ? 4. Who is going to use this language ? 5. Why would a user prefer (assuming intended audience of proposed language is end users) a language over a Work Flow UI tool such as XBAYA ? (In other words what are the things we can do with language which we cannot do with UI) Sorry, if above questions are in-appropriate, just wanted to understand what exactly needed. Thanks -Thejaka Amila On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva supu...@gmail.com wrote: I think I'm not suggesting to create a Workflow interpreter using Python etc. What I'm suggesting is to remove the Worflow aspect from core Airavata and move it to a more higher level component. The more I think about it, the model I'm suggesting is similar to what Hadoop, Storm etc has done for distributed system computations. This model is proven to be successful over the years. Keeping what Airavata does at its core can help you to build a more robust system. If we look at Airavata as middleware to execute applications on computing resources we can simplify what Airavata does and focus on improving the core functionality. All the successful systems have thrived on defining what it does at its core and keeping it simple and being excellent at what it does. In that regard keeping workflow aspect out of Airavata can help you to focus on the core problem. That is to execute an application on a remote computing resource in a fault tolerant and scalable manner. What I'm suggesting is to give the Orchestrator the capability to execute a Driver program that is specified by the user. (This program can be written in Python, Java or any other language). This driver program is similar to what you define in a Hadoop or Storm configuration. The driver program specifies the flow of the computation. It specifies what are the applications needs to be executed, how to manipulate input output. The driver program is the workflow for the user. Because the user specifies the program he can program it to handle workflow steering etc. Every time the user wants to execute this program he can tell Airavata to execute the Driver program. I'm also not 100% sure about all the details. But this can be a new way to look at how systems like Airavata should behave. Your thoughts and suggestions are more than welcome. Thanks, Supun.. On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi Supun, I think we were in two context, because I as suggesting a way to serialize and deserialize the workflow description while you are suggesting to implement some kind of workflow interpreter using Python, where Python client can send thrift calls to Airavata server to run the application. I can see with your suggested approach we can control the workflow execution process from client side which make it easy to implement workflow debugger.As you mentioned this is a major change to Airavata. So we should neatly think as this will change our existing architecture. Still if someone need to use different language java, php, JS etc ... to run the same workflow which generated by Python, we need a language independent workflow description. My initial question was what is the best language for this?. But as I have explained in one of my previous reply, It is not matter what language we used Either we can use XML or JSON to write this description, what matters is how easy to generate workflow with the provided API. Hence it would be great to have set of neat APIs in different languages. Thanks, Shameera. On Sat, Oct 18, 2014 at 12:09 PM, Supun Kamburugamuva supu...@gmail.com wrote: Hi Shameera, Using python is a radical approach for workflow I think. But I believe this is a very beautiful and new approach.Here is a possible scenario for implementation and running using a Python based library. The Python library facilitates the creation of Applications and submitting them to Airavata for execution. The underlying functionality is exactly similar to what Java clients provides.The only difference is that, Python library should have a
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Amila, Please see my comments inline. On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara thejaka.am...@gmail.com wrote: Hello, I am sorry, I am bit late on this thread. But when reading through this thread I simply got lost, what this thread is discussing. I have few questions. 1. @Shameera : Is XWF actually a language to define workflow ? To my understanding it was an intermediate representation to convert workflow defined in UI to java object model. Was XWF ever used by any airavata user to define a workflow graph ? Yes, XWF is the language defined and used by Airavata to explain the workflows but it is not well documented. Both server and client sides read this description language to generate runtime java representation. so when you used XBaya to create a workflow with multiple applications, under the hood XBaya generate xwf which describe that workflow to server. From initial description what I understood is we are looking for a improved intermediate representation, not a language which describes workflows. 2. So what is the exact purpose of this proposed language ? - Is it to hide parallelism in workflows ? - Is it to replace the XBAYA functionalities ? (i hope not) Actually initial idea was to identify good, well-defined and scientific domain friendly language. Whole purpose of this effort is reduce the entry barrier of the end user. But later it is understood that introducing a new language won't fix our issue. 3. What are we trying to achieve by this proposed language which we cannot achieved through workflow UI tool ? 4. Who is going to use this language ? As I explained, our direction has been changed.By introducing a new language we are get nothing but nice description file. No functional improvements etc ... The current language should use all airavata client(Currently we have only XBaya) side applications to explain the workflow to the server side. 5. Why would a user prefer (assuming intended audience of proposed language is end users) a language over a Work Flow UI tool such as XBAYA ? (In other words what are the things we can do with language which we cannot do with UI) Let's say you are going to write a new web base client for Airavata which generate workflows and launch it. What you need to do is do some magic with UI and finally come up with description language and send it to server side. Here you need to learn how to write a valid XWF file and write your own JS code to generate it. But if airavata has a JavaScript API which can be used to generate XWF for you by getting JSON objects as input then it would be great help for client side developers. Thank, Shameera. Sorry, if above questions are in-appropriate, just wanted to understand what exactly needed. Thanks -Thejaka Amila On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva supu...@gmail.com wrote: I think I'm not suggesting to create a Workflow interpreter using Python etc. What I'm suggesting is to remove the Worflow aspect from core Airavata and move it to a more higher level component. The more I think about it, the model I'm suggesting is similar to what Hadoop, Storm etc has done for distributed system computations. This model is proven to be successful over the years. Keeping what Airavata does at its core can help you to build a more robust system. If we look at Airavata as middleware to execute applications on computing resources we can simplify what Airavata does and focus on improving the core functionality. All the successful systems have thrived on defining what it does at its core and keeping it simple and being excellent at what it does. In that regard keeping workflow aspect out of Airavata can help you to focus on the core problem. That is to execute an application on a remote computing resource in a fault tolerant and scalable manner. What I'm suggesting is to give the Orchestrator the capability to execute a Driver program that is specified by the user. (This program can be written in Python, Java or any other language). This driver program is similar to what you define in a Hadoop or Storm configuration. The driver program specifies the flow of the computation. It specifies what are the applications needs to be executed, how to manipulate input output. The driver program is the workflow for the user. Because the user specifies the program he can program it to handle workflow steering etc. Every time the user wants to execute this program he can tell Airavata to execute the Driver program. I'm also not 100% sure about all the details. But this can be a new way to look at how systems like Airavata should behave. Your thoughts and suggestions are more than welcome. Thanks, Supun.. On Sat, Oct 18, 2014 at 7:03 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi Supun, I think we were in two context, because I as suggesting a way to serialize and deserialize the
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Shameera, I think the whole point of introducing thrift is to support language bindings for gateway developers and I do not think we should limit this freedom by introducing a python client just to build a workflow. I propose to represent a workflow as a thrift data-structure (I think by using thrift's list objects we can create an adjacency list and of course it will be tricky because of thrift unsupport for inheritance and other limitations but I think it will be worth the effort assuming thrift IDL will be improved a lot in future). If we can represent a workflow as a thrift object model in the client side all the workflow related CRUD operations can be generated using thrift, so we can support all the languages which thrift support. To submit a workflow we can introduce another thrift operation which accept the big workflow thrift object and workflow interpreter can parse this and execute the workflow. Internally we can use XWF and we can convert thrift model to XWF and XWF to a thrift object. Regards Lahiru On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi Amila, Please see my comments inline. On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara thejaka.am...@gmail.com wrote: Hello, I am sorry, I am bit late on this thread. But when reading through this thread I simply got lost, what this thread is discussing. I have few questions. 1. @Shameera : Is XWF actually a language to define workflow ? To my understanding it was an intermediate representation to convert workflow defined in UI to java object model. Was XWF ever used by any airavata user to define a workflow graph ? Yes, XWF is the language defined and used by Airavata to explain the workflows but it is not well documented. Both server and client sides read this description language to generate runtime java representation. so when you used XBaya to create a workflow with multiple applications, under the hood XBaya generate xwf which describe that workflow to server. From initial description what I understood is we are looking for a improved intermediate representation, not a language which describes workflows. 2. So what is the exact purpose of this proposed language ? - Is it to hide parallelism in workflows ? - Is it to replace the XBAYA functionalities ? (i hope not) Actually initial idea was to identify good, well-defined and scientific domain friendly language. Whole purpose of this effort is reduce the entry barrier of the end user. But later it is understood that introducing a new language won't fix our issue. 3. What are we trying to achieve by this proposed language which we cannot achieved through workflow UI tool ? 4. Who is going to use this language ? As I explained, our direction has been changed.By introducing a new language we are get nothing but nice description file. No functional improvements etc ... The current language should use all airavata client(Currently we have only XBaya) side applications to explain the workflow to the server side. 5. Why would a user prefer (assuming intended audience of proposed language is end users) a language over a Work Flow UI tool such as XBAYA ? (In other words what are the things we can do with language which we cannot do with UI) Let's say you are going to write a new web base client for Airavata which generate workflows and launch it. What you need to do is do some magic with UI and finally come up with description language and send it to server side. Here you need to learn how to write a valid XWF file and write your own JS code to generate it. But if airavata has a JavaScript API which can be used to generate XWF for you by getting JSON objects as input then it would be great help for client side developers. Thank, Shameera. Sorry, if above questions are in-appropriate, just wanted to understand what exactly needed. Thanks -Thejaka Amila On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva supu...@gmail.com wrote: I think I'm not suggesting to create a Workflow interpreter using Python etc. What I'm suggesting is to remove the Worflow aspect from core Airavata and move it to a more higher level component. The more I think about it, the model I'm suggesting is similar to what Hadoop, Storm etc has done for distributed system computations. This model is proven to be successful over the years. Keeping what Airavata does at its core can help you to build a more robust system. If we look at Airavata as middleware to execute applications on computing resources we can simplify what Airavata does and focus on improving the core functionality. All the successful systems have thrived on defining what it does at its core and keeping it simple and being excellent at what it does. In that regard keeping workflow aspect out of Airavata can help you to focus on the core problem. That is to execute an application on a remote
Re: Evaluate Suitable Scientific Workflow Language for Airavata.
Hi Shameera, Thanks for the reply. I wouldnt call XWF a language. It is not a language in any sense but an intermediate representation of a workflow graph. To my understanding what you need is not a language but an efficient portable intermediate representation. Seems you also have realized it. JSON might be a good candidate provided most of the frontends are web based. Thanks -Thejaka On Mon, Oct 20, 2014 at 10:35 PM, Shameera Rathnayaka shameerai...@gmail.com wrote: Hi Amila, Please see my comments inline. On Mon, Oct 20, 2014 at 3:04 PM, Amila Jayasekara thejaka.am...@gmail.com wrote: Hello, I am sorry, I am bit late on this thread. But when reading through this thread I simply got lost, what this thread is discussing. I have few questions. 1. @Shameera : Is XWF actually a language to define workflow ? To my understanding it was an intermediate representation to convert workflow defined in UI to java object model. Was XWF ever used by any airavata user to define a workflow graph ? Yes, XWF is the language defined and used by Airavata to explain the workflows but it is not well documented. Both server and client sides read this description language to generate runtime java representation. so when you used XBaya to create a workflow with multiple applications, under the hood XBaya generate xwf which describe that workflow to server. From initial description what I understood is we are looking for a improved intermediate representation, not a language which describes workflows. 2. So what is the exact purpose of this proposed language ? - Is it to hide parallelism in workflows ? - Is it to replace the XBAYA functionalities ? (i hope not) Actually initial idea was to identify good, well-defined and scientific domain friendly language. Whole purpose of this effort is reduce the entry barrier of the end user. But later it is understood that introducing a new language won't fix our issue. 3. What are we trying to achieve by this proposed language which we cannot achieved through workflow UI tool ? 4. Who is going to use this language ? As I explained, our direction has been changed.By introducing a new language we are get nothing but nice description file. No functional improvements etc ... The current language should use all airavata client(Currently we have only XBaya) side applications to explain the workflow to the server side. 5. Why would a user prefer (assuming intended audience of proposed language is end users) a language over a Work Flow UI tool such as XBAYA ? (In other words what are the things we can do with language which we cannot do with UI) Let's say you are going to write a new web base client for Airavata which generate workflows and launch it. What you need to do is do some magic with UI and finally come up with description language and send it to server side. Here you need to learn how to write a valid XWF file and write your own JS code to generate it. But if airavata has a JavaScript API which can be used to generate XWF for you by getting JSON objects as input then it would be great help for client side developers. Thank, Shameera. Sorry, if above questions are in-appropriate, just wanted to understand what exactly needed. Thanks -Thejaka Amila On Sun, Oct 19, 2014 at 2:29 PM, Supun Kamburugamuva supu...@gmail.com wrote: I think I'm not suggesting to create a Workflow interpreter using Python etc. What I'm suggesting is to remove the Worflow aspect from core Airavata and move it to a more higher level component. The more I think about it, the model I'm suggesting is similar to what Hadoop, Storm etc has done for distributed system computations. This model is proven to be successful over the years. Keeping what Airavata does at its core can help you to build a more robust system. If we look at Airavata as middleware to execute applications on computing resources we can simplify what Airavata does and focus on improving the core functionality. All the successful systems have thrived on defining what it does at its core and keeping it simple and being excellent at what it does. In that regard keeping workflow aspect out of Airavata can help you to focus on the core problem. That is to execute an application on a remote computing resource in a fault tolerant and scalable manner. What I'm suggesting is to give the Orchestrator the capability to execute a Driver program that is specified by the user. (This program can be written in Python, Java or any other language). This driver program is similar to what you define in a Hadoop or Storm configuration. The driver program specifies the flow of the computation. It specifies what are the applications needs to be executed, how to manipulate input output. The driver program is the workflow for the user. Because the
