[jira] [Commented] (AIRAVATA-1652) Implement initial framework
[ https://issues.apache.org/jira/browse/AIRAVATA-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384495#comment-14384495 ] ASF subversion and git services commented on AIRAVATA-1652: --- Commit f5a2eac628291bd218c872eeb6798149441d6a9d in airavata's branch refs/heads/master from [~chathuri] [ https://git-wip-us.apache.org/repos/asf?p=airavata.git;h=f5a2eac ] adding gateways, ssh credentials, projects and compute resources - AIRAVATA-1652 > Implement initial framework > --- > > Key: AIRAVATA-1652 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1652 > Project: Airavata > Issue Type: Sub-task >Reporter: Chathuri Wimalasena >Assignee: Chathuri Wimalasena > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRAVATA-1646) [GSoC] Brainstorm Airavata Data Management Needs
[ https://issues.apache.org/jira/browse/AIRAVATA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383969#comment-14383969 ] Suresh Marru commented on AIRAVATA-1646: Hi Pankaj, initially i was looking at these aspects from data organization and retrieval aspects. But I am embracing more of what you are proposing here. Yes, this will be a good enhancement to add processing capabilities to airavata data catalogs. That way we could let users derive insights from their experiments. > [GSoC] Brainstorm Airavata Data Management Needs > > > Key: AIRAVATA-1646 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1646 > Project: Airavata > Issue Type: Brainstorming >Reporter: Suresh Marru > Labels: gsoc, gsoc2015,, mentor > > Currently Airavata focuses on Execution Management and the Registry > Sub-System (with app, resource and experiment catalogs) capture metadata > about applications and executions. There were few efforts (primarily from > student projects) to explore this void. It will be good to concretely propose > data management solutions to for input data registration, input and generated > retrieval, data transfers and replication management. > Metadata Catalog: In addition current metadata management is based on > shredding thrift data models into mysql/derby schema. This is described in > [1]. We have discussed extensively on using Object Store data bases with a > conclusion of understanding the requirements more systematically. A good > stand alone task would be to understand current metadata management and > propose alternative solutions with proof of concept implementations. Once the > community is convinced, we can then plan on implementing them into > production. > Provenance: Airavata could be enhanced to capture provenance to organize the > data for reuse, discovery, comparison and sharing. This is a well explored > field. There might be good compelling third party solutions. Especially it > will be good to explore in the bigdata space and identify leverages (either > concepts, or even better implementations). > Auditing and Traceability: As Airavata mediates executions on behalf of > gateways, it has to strike a balance between abstracting the compute resource > interactions at the same time providing transparent execution trace. This > will bloat the amount of data to be catalogued. A good effort will be to > understand the current extent of airavata audits and provide suggestions. > BigData Leverage: Airavata needs to leverage the influx of tools in this > space. Any suggestions on relevant tools which will enhance Airavata > experience will be a good fit. > [1] - > https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Data+Models+0.12 > [2] - http://markmail.org/thread/4lguliiktjohjmsd -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRAVATA-1644) Develop a framework to test release features
[ https://issues.apache.org/jira/browse/AIRAVATA-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Marru updated AIRAVATA-1644: --- Description: As briefly discussed on the dev list [1], develop a test framework which can validate the advertised release features. The initial focus of the framework will be to test multi-tenancy aspects of Airavata [1] - http://markmail.org/thread/t5muj5smfocpwtin Epic Name: Airavata Test Framework (was: Develop a test framework to load test airavata) Summary: Develop a framework to test release features (was: Airavata needs to be tested with multiple gateways with multiple users) > Develop a framework to test release features > > > Key: AIRAVATA-1644 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1644 > Project: Airavata > Issue Type: Epic >Reporter: Chathuri Wimalasena > > As briefly discussed on the dev list [1], develop a test framework which can > validate the advertised release features. > The initial focus of the framework will be to test multi-tenancy aspects of > Airavata > [1] - http://markmail.org/thread/t5muj5smfocpwtin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRAVATA-1650) Come up with a document to gather requirements and define the architecture
[ https://issues.apache.org/jira/browse/AIRAVATA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383943#comment-14383943 ] Suresh Marru commented on AIRAVATA-1650: Hi Chathuri, thank you for jumping on this topic, this should be incredibly useful to authoritatively declare Airavata is ready for production. I have following suggestion: * I see that the goal is in general to come up with a test framework for Airavata agnostic to a particular feature (like multi tenancy). For each release this goal might change based on the targeted features for that release. * If a particular feature is tested with this framework, we could mark it as a supported. * The requirement list of good for the current target of multi-tenancy. The steps specified in the document look good. I have a meta comment: We probably should clearly have three category of tests: Acceptance Tests, Scalability Tests and Reliability Tests * Acceptance Tests - These test through the targeted advertised feature with both positive and negative scenarios. This could include automatic or manual fault injection. * Scalability Tests - While Acceptance tests ensures few instances of tests pass successfully, scalability tests can focus on running the same tests multiple times. These should bring out sporadic resource usages (memory, i/o, cpu, threads, sockets, file handles), race conditions and concurrency limits of a single instance. This information should be very helpful for load balancing and replication of services. Reliability Tests - The focus can be to ensure there are no resource leaks, and as much as possible the system is auto-recovering. A service goes down, load is reshuffled or restarted after a component is back up. > Come up with a document to gather requirements and define the architecture > -- > > Key: AIRAVATA-1650 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1650 > Project: Airavata > Issue Type: Sub-task >Reporter: Chathuri Wimalasena >Assignee: Chathuri Wimalasena > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRAVATA-1646) [GSoC] Brainstorm Airavata Data Management Needs
[ https://issues.apache.org/jira/browse/AIRAVATA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383801#comment-14383801 ] pankaj saha commented on AIRAVATA-1646: --- Hi Suresh, one of the outcomes is a software pipeline. So, for application X, the recommended pipeline could be input -> read from mongoDB processing -> via MapReduce implementation (Hadoop) ouput -> write to Cassandra Thanks Pankaj > [GSoC] Brainstorm Airavata Data Management Needs > > > Key: AIRAVATA-1646 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1646 > Project: Airavata > Issue Type: Brainstorming >Reporter: Suresh Marru > Labels: gsoc, gsoc2015,, mentor > > Currently Airavata focuses on Execution Management and the Registry > Sub-System (with app, resource and experiment catalogs) capture metadata > about applications and executions. There were few efforts (primarily from > student projects) to explore this void. It will be good to concretely propose > data management solutions to for input data registration, input and generated > retrieval, data transfers and replication management. > Metadata Catalog: In addition current metadata management is based on > shredding thrift data models into mysql/derby schema. This is described in > [1]. We have discussed extensively on using Object Store data bases with a > conclusion of understanding the requirements more systematically. A good > stand alone task would be to understand current metadata management and > propose alternative solutions with proof of concept implementations. Once the > community is convinced, we can then plan on implementing them into > production. > Provenance: Airavata could be enhanced to capture provenance to organize the > data for reuse, discovery, comparison and sharing. This is a well explored > field. There might be good compelling third party solutions. Especially it > will be good to explore in the bigdata space and identify leverages (either > concepts, or even better implementations). > Auditing and Traceability: As Airavata mediates executions on behalf of > gateways, it has to strike a balance between abstracting the compute resource > interactions at the same time providing transparent execution trace. This > will bloat the amount of data to be catalogued. A good effort will be to > understand the current extent of airavata audits and provide suggestions. > BigData Leverage: Airavata needs to leverage the influx of tools in this > space. Any suggestions on relevant tools which will enhance Airavata > experience will be a good fit. > [1] - > https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Data+Models+0.12 > [2] - http://markmail.org/thread/4lguliiktjohjmsd -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (AIRAVATA-1646) [GSoC] Brainstorm Airavata Data Management Needs
[ https://issues.apache.org/jira/browse/AIRAVATA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pankaj saha updated AIRAVATA-1646: -- Comment: was deleted (was: Hi Suresh, one of the outcomes is a software pipeline. So, for application X, the recommended pipeline could be input -> read from mongoDB processing -> via MapReduce implementation (Hadoop) ouput -> write to Cassandra Thanks Pankaj) > [GSoC] Brainstorm Airavata Data Management Needs > > > Key: AIRAVATA-1646 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1646 > Project: Airavata > Issue Type: Brainstorming >Reporter: Suresh Marru > Labels: gsoc, gsoc2015,, mentor > > Currently Airavata focuses on Execution Management and the Registry > Sub-System (with app, resource and experiment catalogs) capture metadata > about applications and executions. There were few efforts (primarily from > student projects) to explore this void. It will be good to concretely propose > data management solutions to for input data registration, input and generated > retrieval, data transfers and replication management. > Metadata Catalog: In addition current metadata management is based on > shredding thrift data models into mysql/derby schema. This is described in > [1]. We have discussed extensively on using Object Store data bases with a > conclusion of understanding the requirements more systematically. A good > stand alone task would be to understand current metadata management and > propose alternative solutions with proof of concept implementations. Once the > community is convinced, we can then plan on implementing them into > production. > Provenance: Airavata could be enhanced to capture provenance to organize the > data for reuse, discovery, comparison and sharing. This is a well explored > field. There might be good compelling third party solutions. Especially it > will be good to explore in the bigdata space and identify leverages (either > concepts, or even better implementations). > Auditing and Traceability: As Airavata mediates executions on behalf of > gateways, it has to strike a balance between abstracting the compute resource > interactions at the same time providing transparent execution trace. This > will bloat the amount of data to be catalogued. A good effort will be to > understand the current extent of airavata audits and provide suggestions. > BigData Leverage: Airavata needs to leverage the influx of tools in this > space. Any suggestions on relevant tools which will enhance Airavata > experience will be a good fit. > [1] - > https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Data+Models+0.12 > [2] - http://markmail.org/thread/4lguliiktjohjmsd -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRAVATA-1646) [GSoC] Brainstorm Airavata Data Management Needs
[ https://issues.apache.org/jira/browse/AIRAVATA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383800#comment-14383800 ] pankaj saha commented on AIRAVATA-1646: --- Hi Suresh, one of the outcomes is a software pipeline. So, for application X, the recommended pipeline could be input -> read from mongoDB processing -> via MapReduce implementation (Hadoop) ouput -> write to Cassandra Thanks Pankaj > [GSoC] Brainstorm Airavata Data Management Needs > > > Key: AIRAVATA-1646 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1646 > Project: Airavata > Issue Type: Brainstorming >Reporter: Suresh Marru > Labels: gsoc, gsoc2015,, mentor > > Currently Airavata focuses on Execution Management and the Registry > Sub-System (with app, resource and experiment catalogs) capture metadata > about applications and executions. There were few efforts (primarily from > student projects) to explore this void. It will be good to concretely propose > data management solutions to for input data registration, input and generated > retrieval, data transfers and replication management. > Metadata Catalog: In addition current metadata management is based on > shredding thrift data models into mysql/derby schema. This is described in > [1]. We have discussed extensively on using Object Store data bases with a > conclusion of understanding the requirements more systematically. A good > stand alone task would be to understand current metadata management and > propose alternative solutions with proof of concept implementations. Once the > community is convinced, we can then plan on implementing them into > production. > Provenance: Airavata could be enhanced to capture provenance to organize the > data for reuse, discovery, comparison and sharing. This is a well explored > field. There might be good compelling third party solutions. Especially it > will be good to explore in the bigdata space and identify leverages (either > concepts, or even better implementations). > Auditing and Traceability: As Airavata mediates executions on behalf of > gateways, it has to strike a balance between abstracting the compute resource > interactions at the same time providing transparent execution trace. This > will bloat the amount of data to be catalogued. A good effort will be to > understand the current extent of airavata audits and provide suggestions. > BigData Leverage: Airavata needs to leverage the influx of tools in this > space. Any suggestions on relevant tools which will enhance Airavata > experience will be a good fit. > [1] - > https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Data+Models+0.12 > [2] - http://markmail.org/thread/4lguliiktjohjmsd -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRAVATA-1635) [GSoC] Integrate Airavata Java Client SDK with GridChem Client
[ https://issues.apache.org/jira/browse/AIRAVATA-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383553#comment-14383553 ] Dimuthu Upeksha commented on AIRAVATA-1635: --- I tried GridChem client by putting some jobs on the servers and getting outputs. Input files that we push has a format like this. %chk=water.chk %nprocshared=1 %mem=500MB #P RHF/6-31g* opt pop=reg gfinput gfprint iop(6/7=3) SCF=direct Gaussian Test Job 00 Water with archiving 0 1 O H 1 0.96 H 1 0.96 2 109.471221 Is this format some standard way of passing jobs or specific format for GridChem? Does client directly pass this file to middleware or parse this and pass only necessary data? > [GSoC] Integrate Airavata Java Client SDK with GridChem Client > --- > > Key: AIRAVATA-1635 > URL: https://issues.apache.org/jira/browse/AIRAVATA-1635 > Project: Airavata > Issue Type: Epic >Reporter: Suresh Marru > Labels: gsoc, gsoc2015, mentor > > GridChem is a Science Gateway enables users to run computational experiments > on multiple supercomputing resources. Currently GridChem, a java swing based > webstart client [1] uses a Axis2 based Middleware Service [2] which brokers > users actions into computational jobs. > This project needs to understand the Client [1] and port it to use Apache > Airavata java client SDK. The project has following components: > * Integrate GridChem client with Airavata User Store (implemented by WSO2 > Identity Server) > * Integrate with Airavata API for application executions. > * Integrate with Atlassian JIRA + Confluence for user error reporting and > status notifications. > [1] - https://github.com/SciGaP/GridChem-Client > [2] - https://github.com/SciGaP/GridChem-Middleware-Service -- This message was sent by Atlassian JIRA (v6.3.4#6332)
