[jira] [Commented] (AIRAVATA-2717) [GSoC] Resurrect User-Defined Airavata Workflows
[ https://issues.apache.org/jira/browse/AIRAVATA-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409595#comment-16409595 ] Dimuthu Upeksha commented on AIRAVATA-2717: --- Hi Yasas, It is bit outdated but you can get an idea by looking at the document. Recently we have developed a task execution framework on top of Apache Helix which satisfies most of the requirements mentioned in the document. You can refer to the discussion thread [1] under the subject "Evaluating Helix as the task execution framework" to have a better understanding of the design. You can see currently implemented tasks from [2]. However currently, we statically bind the tasks into a workflow [3]. Workflow is just a sequence of tasks. Order of these tasks is pre defined and embedded into the current orchestrator code. For example we run in the order of Environment setup tasks -> Input Data Staging Tasks -> Job Submission Task so on. In future we should be able to have more flexibility. Rather than statically defining order of tasks inside the orchestrator code, we should be able give that order from the outside. That's why we need a workflow language to interpret it. Ideally the flow would be Create a workflow description -> submit to orchestrator -> orchestrator parses the workflow -> Submit to task execution engine To create the workflow you can simply write the workflow in a text file, or use a GUI to generate it. If you follow the mail thread that I have mentioned, there is a image that illustrate such GUI tool. If you need more information, let's move to dev mailing list [1] [http://mail-archives.apache.org/mod_mbox/airavata-dev/201711.mbox/thread] [2] [https://github.com/apache/airavata/tree/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task] [3] [https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/workflow/PreWorkflowManager.java#L83] Thanks Dimuthu > [GSoC] Resurrect User-Defined Airavata Workflows > - > > Key: AIRAVATA-2717 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2717 > Project: Airavata > Issue Type: Epic >Affects Versions: 0.17 >Reporter: Suresh Marru >Priority: Major > Labels: gsoc2018 > > Airavata used to support user-defined workflows using an interface XBaya to > drag and drop application components to a workspace and define data flow and > control flow dependencies among the application nodes. Airavata's workflow > system was used for composing, executing, and monitoring workflow graphs > primarly web service components. The workflow description was high level > abstraction and used to be converted to lower level execution run times like > BPEL, SCUFL and Python scripts. > > Airavata has evolved significantly and the current development version is > being built-over Apache Helix for DAG orchestration. This provides an > opportunity to resurrect workflow capabilities in Airavata. > This GSoC project involves finalizing a Airavata Workflow Language; modify > the orchestrator to parse user described workflow and translate to equivalent > Helix DAG's; execute and monitor the worklfows; develop a simple UI to > demonstrate the capabilities. > To describe the workflows, you can build on this - > [https://docs.google.com/document/d/1eh7BV8CHupxyM2jeqcM2tUG5MnXFt7hNDX4PQDfxCcM/edit] > or follow other discussions like - > https://issues.apache.org/jira/browse/AIRAVATA-2555 and > User community & Impact of the software: Airavata is primarily targeted to > build science gateways using computational resources from various > disciplines. The initial targeted set of gateways include projects supporting > research and education in chemistry, biophysics, and geosciences . The goal > of airavata is to enhance productivity of these gateways to utilize > cyberinfrastructure of resources (e.g., local lab resources, the Extreme > Science and Engineering Discovery Environment (XSEDE), University Clusters, > Academic and Commercial Computational Clouds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2718) [GSoC] Re-architect Output Data Parsing into Airavata core
[ https://issues.apache.org/jira/browse/AIRAVATA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409608#comment-16409608 ] Dimuthu Upeksha commented on AIRAVATA-2718: --- Hi Lahiru, Thanks for your interest. One possible architecture for generalizing data parsers can be found from [1]. However you are free to come up with your own design but try to utilize the current task execution framework. You can have a good insight of the task framework by referring to my comment in [2]. If you need further clarifications, let's discuss on dev list [1] [https://docs.google.com/presentation/d/1CiPLE6Ht9ynNC9R9Bk0U7yHlsqw2g8ONTDxxDs6R_MY/edit?usp=sharing] [2] https://issues.apache.org/jira/browse/AIRAVATA-2717 Thanks Dimuthu > [GSoC] Re-architect Output Data Parsing into Airavata core > -- > > Key: AIRAVATA-2718 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2718 > Project: Airavata > Issue Type: Epic >Reporter: Suresh Marru >Priority: Major > > As discussed in this paper [1] Airavata based SEAGrid gateway has prototyped > a data catalog system [2]. [3] and [4] are also related references. The new > airavata execution architecture in develop branch is based on Apache Helix. > This provides an opportunity to re-architect the data catalog and build it on > new Helix DAG based execution within Airavata. > This project involves > * the data parsers as Airavata tasks and execute them as Helix DAG's. > * Incorporate the MongoDB based search and catalog registry and explore > Thrift API's. > * Modify the current simple UI into the new Django portal. > * Generalize the data catalog. > * Publish a paper [optional] > [1] - > [https://pdfs.semanticscholar.org/2938/686c5c7eecb1b82ce8064b30555298bd649e.pdf] > [2] - https://github.com/SciGaP/seagrid-data > [[3] - > https://www.researchgate.net/profile/Suresh_Marru/publication/275948320_Scientific_Data_Cataloging_System/links/554a05680cf2e859ce18afb4.pdf|https://www.researchgate.net/profile/Suresh_Marru/publication/275948320_Scientific_Data_Cataloging_System/links/554a05680cf2e859ce18afb4.pdf] > [[4] - > https://www.researchgate.net/profile/Dilum_Bandara/publication/282989239_Schema-independent_scientific_data_cataloging_framework/links/5653a40508aeafc2aabb59e8/Schema-independent-scientific-data-cataloging-framework.pdf|https://www.researchgate.net/profile/Dilum_Bandara/publication/282989239_Schema-independent_scientific_data_cataloging_framework/links/5653a40508aeafc2aabb59e8/Schema-independent-scientific-data-cataloging-framework.pdf] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2607) Change all thrift clients to use ThriftClientPool
[
https://issues.apache.org/jira/browse/AIRAVATA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409770#comment-16409770
]
ASF GitHub Bot commented on AIRAVATA-2607:
--
DImuthuUpe commented on a change in pull request #183: [AIRAVATA-2607] Change
all thrift clients to use ThriftClientPool
URL: https://github.com/apache/airavata/pull/183#discussion_r176476902
##
File path:
airavata-services/profile-service/profile-service-server/src/main/java/org/apache/airavata/service/profile/handlers/IamAdminServicesHandler.java
##
@@ -272,8 +274,8 @@ private PasswordCredential
getTenantAdminPasswordCredential(String tenantId) thr
csClientPool.returnResource(csClient);
registryClientPool.returnResource(regClient);
return TenantAdminPassword;
-}catch (Exception e){
-logger.error("Error while getting the tenant admin password", e);
+} catch (Exception e){
+logger.error("Error while getting the tenant admin password for
the tenant id "+tenantId+"", e);
Review comment:
Keep spaces before and after +. id " + tenantId. Remove trailing "" as it is
useless
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change all thrift clients to use ThriftClientPool
> -
>
> Key: AIRAVATA-2607
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2607
> Project: Airavata
> Issue Type: Improvement
>Reporter: Suresh Marru
>Assignee: Sachin Kariyattin
>Priority: Major
>
> Currently, the API server uses ThriftClientPool but all other thrift servers
> use a regular client.
> We probably should move the current ThriftClientPool into the commons library
> and have all thrift servers use this instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2607) Change all thrift clients to use ThriftClientPool
[
https://issues.apache.org/jira/browse/AIRAVATA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409773#comment-16409773
]
ASF GitHub Bot commented on AIRAVATA-2607:
--
DImuthuUpe commented on a change in pull request #183: [AIRAVATA-2607] Change
all thrift clients to use ThriftClientPool
URL: https://github.com/apache/airavata/pull/183#discussion_r176477086
##
File path:
airavata-services/profile-service/profile-service-server/src/main/java/org/apache/airavata/service/profile/handlers/IamAdminServicesHandler.java
##
@@ -196,6 +196,8 @@ public void updateUserProfile(AuthzToken authzToken,
UserProfile userDetails) th
}
if (!username.equals(userDetails.getUserId())) {
+String msg = "userId in user profile
"+userDetails.getUserId()+" doesn't match authorization token "+username+"";
Review comment:
Same comment as below
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change all thrift clients to use ThriftClientPool
> -
>
> Key: AIRAVATA-2607
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2607
> Project: Airavata
> Issue Type: Improvement
>Reporter: Suresh Marru
>Assignee: Sachin Kariyattin
>Priority: Major
>
> Currently, the API server uses ThriftClientPool but all other thrift servers
> use a regular client.
> We probably should move the current ThriftClientPool into the commons library
> and have all thrift servers use this instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRAVATA-2719) Refactoring Replica Catalog
Sneha Tilak created AIRAVATA-2719: - Summary: Refactoring Replica Catalog Key: AIRAVATA-2719 URL: https://issues.apache.org/jira/browse/AIRAVATA-2719 Project: Airavata Issue Type: Improvement Components: Registry API Reporter: Sneha Tilak Assignee: Sneha Tilak -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2607) Change all thrift clients to use ThriftClientPool
[ https://issues.apache.org/jira/browse/AIRAVATA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409931#comment-16409931 ] ASF GitHub Bot commented on AIRAVATA-2607: -- DImuthuUpe commented on issue #183: [AIRAVATA-2607] Change all thrift clients to use ThriftClientPool URL: https://github.com/apache/airavata/pull/183#issuecomment-375388690 @karankotz Can you add some unit tests that cover this implementation so that I can have a look at the functionality This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Change all thrift clients to use ThriftClientPool > - > > Key: AIRAVATA-2607 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2607 > Project: Airavata > Issue Type: Improvement >Reporter: Suresh Marru >Assignee: Sachin Kariyattin >Priority: Major > > Currently, the API server uses ThriftClientPool but all other thrift servers > use a regular client. > We probably should move the current ThriftClientPool into the commons library > and have all thrift servers use this instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
