Re: Use of GitHub Discussion

[Suresh Marru]

Thanks everyone for weighing in. I created an INFRA ticket to request GitHub 
discussions for Custos repo - https://issues.apache.org/jira/browse/INFRA-26087

Suresh

> On Sep 3, 2024, at 12:55 AM, Eroma Abeysinghe  
> wrote:
> 
> +1 for the initiative, Suresh.
> 
> Best,
> Eroma Abeysinghe
> 
> On Mon, Sep 2, 2024 at 10:49 PM Praneeth Kumar Chityala 
> mailto:praneethchityal...@gmail.com>> wrote:
>> Hi Suresh,
>> 
>> Sounds good to me too. I see it helps in managing everything related to 
>> developers at one place.
>> 
>> Best,
>> Praneeth
>> 
>> On Mon, Sep 2, 2024 at 10:44 AM Lahiru Jayathilake 
>> mailto:lahirujayathil...@gmail.com>> wrote:
>>> Hi Suresh, 
>>> 
>>> Sounds great! This would make it much easier to access and manage 
>>> discussions efficiently.
>>>   
>>> Thanks,
>>> Lahiru
>>> 
>>> 
>>> On Mon, Sep 2, 2024 at 10:24 AM Pamidighantam, Sudhakar V 
>>> mailto:spami...@illinois.edu>> wrote:
>>>> +1 fully agree that one place make it convenient and efficient.
>>>> 
>>>>  
>>>> 
>>>> Thanks,
>>>> 
>>>> Sudhakar.
>>>> 
>>>>  
>>>> 
>>>> From: DImuthu Upeksha >>> <mailto:dimuthu...@apache.org>>
>>>> Date: Monday, September 2, 2024 at 9:29 AM
>>>> To: dev@airavata.apache.org <mailto:dev@airavata.apache.org> 
>>>> mailto:dev@airavata.apache.org>>
>>>> Subject: Re: Use of GitHub Discussion
>>>> 
>>>> Hi Suresh,
>>>> 
>>>>  
>>>> 
>>>> +1 for this. It is great for developers to stay on one platform for both 
>>>> code and discussion contributions. 
>>>> 
>>>>  
>>>> 
>>>> Thanks
>>>> 
>>>> Dimuthu
>>>> 
>>>>  
>>>> 
>>>> On Sun, Sep 1, 2024 at 11:00 PM Ganning Xu >>> <mailto:gannin...@gmail.com>> wrote:
>>>> 
>>>> Hi Suresh,
>>>> 
>>>>  
>>>> 
>>>> I agree, this sounds like a great idea. This also allows us to link pull 
>>>> requests to specific discussions, and members to be assigned to specific 
>>>> issues/discussions. 
>>>> 
>>>>  
>>>> 
>>>> Thanks!
>>>> 
>>>> Ganning Xu
>>>> 
>>>>  
>>>> 
>>>> On Sun, Sep 1, 2024 at 10:57 PM Piyush Nalawade >>> <mailto:nalawadepiy...@gmail.com>> wrote:
>>>> 
>>>> Hey Suresh,
>>>> 
>>>> Sounds like a great visibility initiative. This will benefit developers 
>>>> who want to contribute but are not added on the mailing list. They can 
>>>> also have visibility regarding repository-specific discussions. 
>>>> 
>>>>  
>>>> 
>>>> On Sun, Sep 1, 2024 at 3:42 PM Suresh Marru >>> <mailto:sma...@apache.org>> wrote:
>>>> 
>>>> Hi All,
>>>> 
>>>> We have stalled our mailing list usage, discussions are spread across 
>>>> GitHub Issues, and some are buried in PRs and Slack. How about we have 
>>>> repository-specific discussions on GitHub? For instance, we can use the 
>>>> Custos repository as an example trial to see of how it goes.
>>>> 
>>>> We can see other projects, and I am referencing Airflow as an example - 
>>>> https://airflow.apache.org/community/ 
>>>> <https://urldefense.com/v3/__https:/airflow.apache.org/community/__;!!DZ3fjg!-iDMWxvub26AETif4uUbAaN8jFN0hsBqbd_srSP5Ipj5O5FwweJw1EGteRUAzCYaGETLOrYEEY4jPRPWbJJDm4g$>
>>>> 
>>>> Thoughts?
>>>> 
>>>> Suresh
>>>> 
> 
> 
> --
> Thank You,
> Best Regards,
> Eroma

Use of GitHub Discussion

[Suresh Marru]

Hi All,

We have stalled our mailing list usage, discussions are spread across GitHub 
Issues, and some are buried in PRs and Slack. How about we have 
repository-specific discussions on GitHub? For instance, we can use the Custos 
repository as an example trial to see of how it goes.

We can see other projects, and I am referencing Airflow as an example - 
https://airflow.apache.org/community/

Thoughts?

Suresh

GSoC 2024 projects on Wiki

[Suresh Marru]

Ganning, Shivang,

Can both of you create a WIKI page and attach your proposal and the final 
reports, links to code and so on to this wiki - 
https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2024

Create a child page with your projects. 

Cheers,
Suresh

[ANNOUNCE] Welcome Lahiru Jayathilake as Airavata PMC member/committer

[Suresh Marru]

Hi All,

The Project Management Committee (PMC) for Apache Airavata has asked Lahiru 
Jayathilake to become a committer and PMC member based on his contributions to 
the project. We are pleased to announce that he has accepted.

Being a committer enables easier contribution to the project since there is no 
need to go via the patch submission process. This should enable better 
productivity. Being a PMC member enables assistance with the management and to 
guide the direction of the project.

Please join me in welcoming Lahiru to Airavata.

Suresh
(On Behalf of Apache Airavata PMC)

Re: CyberShuttle | Updates on orchestration agent and consul

[Suresh Marru]

Hi Praneeth,

This is very nice work. Can you consider submitting this as a journal article 
to Journal of Open Source Software - https://joss.theoj.org/

Suresh

> On Aug 26, 2023, at 9:10 PM, Praneeth Kumar Chityala 
>  wrote:
> 
> Hi Suresh,
> 
> Please find the link below to access the summary of the project.
> 
> GSoC-2023-Praneeth_Chityala 
> <https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment>
>  
> (https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment)
> 
> I will be working on Medium article and demo video over the next few weeks.
> 
> Best,
> Praneeth
> 
> 
> On Fri, Aug 25, 2023 at 4:41 PM Suresh Marru  <mailto:sma...@apache.org>> wrote:
>> Also, both of you feel free to publish Medium articles on your work or let 
>> me know if you want them to get highlighted in other Apache Blogs.
>> 
>> Suresh
>> 
>>> On Aug 25, 2023, at 4:39 PM, Praneeth Kumar Chityala 
>>> mailto:praneethchityal...@gmail.com>> wrote:
>>> 
>>> Hi Suresh,
>>> 
>>> Sure, I am working on the last piece of integration of the orchestration 
>>> agent to consul. I will create a clear article and update it over the 
>>> weekend.
>>> 
>>> Best,
>>> Praneeth
>>> 
>>> On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru >> <mailto:sma...@apache.org>> wrote:
>>>> Hi Praneeth,
>>>> 
>>>> Can you create a child page here and summarize your project  - 
>>>> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>>>> 
>>>> Suresh
>>>> 
>>>>> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala 
>>>>> mailto:praneethchityal...@gmail.com>> 
>>>>> wrote:
>>>>> 
>>>>> Dear All,
>>>>> 
>>>>> I wanted to post an update of the orchestration agent and consul 
>>>>> components of the architecture.
>>>>> 
>>>>> Orchestration Agent:
>>>>> After multiple considerations came to a conclusion to use nomad as the 
>>>>> container orchestration agent
>>>>> Advantages of using nomad are:
>>>>> It can execute docker/container applications with simple json 
>>>>> configuration file
>>>>> It can also execute java applications (so we can use MFT without 
>>>>> containerizing it)
>>>>> It works with SDKs in many languages and also it has simple http client 
>>>>> which makes it web friendly agent
>>>>> Present orchestration agent could connect to Nomad, start/stop job, fetch 
>>>>> existing jobs
>>>>> Orchestration agent in available in cybershuttle-server - 
>>>>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>>>>> Consul (message passing):
>>>>> Inspired from the architecture of MFT, used consul as the main service in 
>>>>> communications from server to orchestration agent
>>>>> Advantages of consul in this use case:
>>>>> We can easily keep track of all available orchestrations agents (service 
>>>>> discovery and availability)
>>>>> We can control access of each orchestration agent with controlled KV 
>>>>> store access
>>>>> It makes our server and orchestration agent decoupled which makes huge 
>>>>> scope for horizontal scaling when required
>>>>> Consul configurations as in this PR - 
>>>>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
>>>>> ConsulClient of cybershuttle server in this PR - 
>>>>> https://github.com/cyber-shuttle/cybershuttle-server/pull/4
>>>>> As of now consul to server integration is completed
>>>>> 
>>>>> Next steps to complete the whole integration:
>>>>> Code integration of consul and orchestration agent
>>>>> Final end to end testing of operation from desktop to server to consul to 
>>>>> orchestration agent
>>>>> 
>>>>> Please feel free to comment or suggest.
>>>>> 
>>>>> Best Regards,
>>>>> Praneeth Chityala
>>>> 
>> 

Re: CyberShuttle | Updates on orchestration agent and consul

[Suresh Marru]

Also, both of you feel free to publish Medium articles on your work or let me 
know if you want them to get highlighted in other Apache Blogs.

Suresh

> On Aug 25, 2023, at 4:39 PM, Praneeth Kumar Chityala 
>  wrote:
> 
> Hi Suresh,
> 
> Sure, I am working on the last piece of integration of the orchestration 
> agent to consul. I will create a clear article and update it over the weekend.
> 
> Best,
> Praneeth
> 
> On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru  <mailto:sma...@apache.org>> wrote:
>> Hi Praneeth,
>> 
>> Can you create a child page here and summarize your project  - 
>> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>> 
>> Suresh
>> 
>>> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala 
>>> mailto:praneethchityal...@gmail.com>> wrote:
>>> 
>>> Dear All,
>>> 
>>> I wanted to post an update of the orchestration agent and consul components 
>>> of the architecture.
>>> 
>>> Orchestration Agent:
>>> After multiple considerations came to a conclusion to use nomad as the 
>>> container orchestration agent
>>> Advantages of using nomad are:
>>> It can execute docker/container applications with simple json configuration 
>>> file
>>> It can also execute java applications (so we can use MFT without 
>>> containerizing it)
>>> It works with SDKs in many languages and also it has simple http client 
>>> which makes it web friendly agent
>>> Present orchestration agent could connect to Nomad, start/stop job, fetch 
>>> existing jobs
>>> Orchestration agent in available in cybershuttle-server - 
>>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>>> Consul (message passing):
>>> Inspired from the architecture of MFT, used consul as the main service in 
>>> communications from server to orchestration agent
>>> Advantages of consul in this use case:
>>> We can easily keep track of all available orchestrations agents (service 
>>> discovery and availability)
>>> We can control access of each orchestration agent with controlled KV store 
>>> access
>>> It makes our server and orchestration agent decoupled which makes huge 
>>> scope for horizontal scaling when required
>>> Consul configurations as in this PR - 
>>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
>>> ConsulClient of cybershuttle server in this PR - 
>>> https://github.com/cyber-shuttle/cybershuttle-server/pull/4
>>> As of now consul to server integration is completed
>>> 
>>> Next steps to complete the whole integration:
>>> Code integration of consul and orchestration agent
>>> Final end to end testing of operation from desktop to server to consul to 
>>> orchestration agent
>>> 
>>> Please feel free to comment or suggest.
>>> 
>>> Best Regards,
>>> Praneeth Chityala
>> 

Re: CyberShuttle | Updates on orchestration agent and consul

[Suresh Marru]

Hi Praneeth,

Can you create a child page here and summarize your project  - 
https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023

Suresh

> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala 
>  wrote:
> 
> Dear All,
> 
> I wanted to post an update of the orchestration agent and consul components 
> of the architecture.
> 
> Orchestration Agent:
> After multiple considerations came to a conclusion to use nomad as the 
> container orchestration agent
> Advantages of using nomad are:
> It can execute docker/container applications with simple json configuration 
> file
> It can also execute java applications (so we can use MFT without 
> containerizing it)
> It works with SDKs in many languages and also it has simple http client which 
> makes it web friendly agent
> Present orchestration agent could connect to Nomad, start/stop job, fetch 
> existing jobs
> Orchestration agent in available in cybershuttle-server - 
> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
> Consul (message passing):
> Inspired from the architecture of MFT, used consul as the main service in 
> communications from server to orchestration agent
> Advantages of consul in this use case:
> We can easily keep track of all available orchestrations agents (service 
> discovery and availability)
> We can control access of each orchestration agent with controlled KV store 
> access
> It makes our server and orchestration agent decoupled which makes huge scope 
> for horizontal scaling when required
> Consul configurations as in this PR - 
> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
> ConsulClient of cybershuttle server in this PR - 
> https://github.com/cyber-shuttle/cybershuttle-server/pull/4
> As of now consul to server integration is completed
> 
> Next steps to complete the whole integration:
> Code integration of consul and orchestration agent
> Final end to end testing of operation from desktop to server to consul to 
> orchestration agent
> 
> Please feel free to comment or suggest.
> 
> Best Regards,
> Praneeth Chityala

GSoC Mid-term Progress Report

[Suresh Marru]

Praneeth, Saurav,

To evaluate your GSoC midterm process, can you submit a report to the dev list 
with the following:

* Original Proposal Summary.

* Progress you made so far on the project.

* Pull requests you submitted so far — this is the key one, your progress will 
be measured on the contributions. 

* Changed you had to make which may have deviations from the original proposal. 

* Plans going forward.

Cheers,
Suresh

Re: airavata api endpoint handler/controller implementation

[Suresh Marru]

Hi Saurav,

You have to look into Thrift API descriptions  
https://github.com/apache/airavata/tree/develop/thrift-interface-descriptions - 
follow the handlers from - 
https://github.com/apache/airavata/blob/develop/airavata-api/airavata-api-server/src/main/java/org/apache/airavata/api/server/handler/AiravataServerHandler.java

If you haven’t done it, I encourage you to build a simple Apache Thrift example 
with Java server and Python clients so it will give you the right context to 
follow the code.

Suresh

> On Jul 10, 2023, at 1:19 PM, saurav kumar jha  
> wrote:
> 
> Hi Community,
> 
> For an api request made by airavata-django-portal , for example on 
> /api/groups ( https://testdrive.airavata.org 
> /api/groups/?limit=10&offset=10 ) , where 
> can I find corresponding controller or handler function in airavata code 
>  where it is being exposed 
> as an endpoint or handled?
> 
> I need this because for "User groups created within a given period",  I want 
> to load the creation date of groups along with other details using an api 
> call.
> In one of the implementations, the date is not being fetched, so I wanted to 
> look at the implementation. 
> 
> Thanks & Regards,
> Saurav Kumar Jha

Re: Minutes of meeting with Sudhakar on Dashboards to get quick statistics

[Suresh Marru]

Thanks for sharing this update Saurav.

These goals look good. 

Suresh

> On Jun 16, 2023, at 12:46 PM, saurav kumar jha  
> wrote:
> 
> Hi Community,
> I got a wonderful opportunity to meet Sudhakar in person. Main agenda of our 
> meeting was to get myself a better understanding of airavata to be able to 
> make progress on Dashboards to get quick statistics  
>  .
> 
> Here are what we discussed:
> 
> Overview of Details, Interface and Deployment within Django portal.
> Demonstration of creation of experiment and submission of job.
> Discussion on allocation and  Resources .
> Discussion on Roles of Scientific Developers, Gateway admins and gateway 
> users.
> Discussion on journey of job from submission, queuing, running and producing 
> output leveraging the high performance computing resources.
> User and admin roles.
> How the user can give inputs.
> Discussion on gateways, modules, user groups, current admin dashboard for 
> statistics.
> 
> We also discussed each and every subtask mentioned in this jira 
> .
> Sudhakar also gave me some ideas on how the final ui can look like.
> 
> I am sure this meeting will help me progress.
> 
> Thanks & Regards,
> Saurav Kumar Jha

Re: Proposal for GSOC 2023 - Dashboards to get Quick Statistics

[Suresh Marru]

Hi Saurav,

Thanks for your proposal. Are you applying for a medium project or a large 
project?

Suresh

> On Apr 1, 2023, at 4:27 PM, saurav kumar jha  
> wrote:
> 
> Hi Everyone,
> 
> I hope this email finds you well. I'm writing to share my proposal for the 
> "Dashboards to get Quick Statistics" project idea for GSOC 2023. I have 
> attached the proposal for your review and feedback.
> 
> I would appreciate it if you could take some time to go through my proposal 
> and provide any suggestions or modifications that you think would help make 
> it better.
> 
> Thank you for your time and consideration, and I look forward to hearing your 
> thoughts.
> 
> Best regards,
> Saurav Kumar Jha 
> 

Re: Status of airavata-mft-portal ?

[Suresh Marru]

Hi Jason,

We envisioned a tool with specific capabilities when we initially 
conceptualized MFT (and the portal). But since MFT has evolved into a 
framework, we have used various user interfaces (including a CLI) based on the 
built applications. 

An example is a transfer portal built by NASA to move to orchestrate and 
monitor petabytes of data between an OpenStack cloud in Europe to AWS S3 
(https://esa-nasa-data-transfers.cybershuttle.org/). This portal requires a 
login, but I can demo it if you are interested. 

If you have a particular application/use case for MFT, we may rethink the MFT 
portal strategy. Google Summer of Code aspirants are looking for ideas, and may 
be we can conceptualize something if we can act fast. 

Thanks,
Suresh

> On Mar 22, 2023, at 5:20 PM, Jason Levitt  wrote:
> 
> Hi folks,
> 
>   I'm wondering about the status of the airavata-mft-portal repo on Github.
> Is this an active project that is going to be updated and integrated
> with airavata-mft ?
> 
> Thanks!
> 
> Jason

Re: Airavata-MFT | Local transport implementation

[Suresh Marru]

Hi Praneeth,

Thank you for sharing a detailed report. This is very encouraging, and KUDOs 
for a significant contribution. 

In future work, you mentioned you planned to move the data from a local machine 
to Jetstream2. Do you plan to push it over SSH/SCP or Swift protocol? Including 
the SCP in your benchmarks will be great as it is a very common use case for 
Airavata users to move from a local machine to a remote HPC cluster. 

Cheers,
Suresh

> On Mar 17, 2023, at 12:22 AM, Chityala, Praneeth  wrote:
> 
> Dear All,
>  
> This is Praneeth Chityala currently pursuing master in Computer Science at 
> Indiana University Bloomington. As part of my independent study I took up MFT 
> as the research area and starting understanding the architecture.
>  
> As many of you know MFT uses Agent to transfer data from one cloud storage to 
> other cloud storage. These agents can be deployed on any compute machines. If 
> the machine in which agent is deployed might have data files which needs to 
> be uploaded to cloud storage, that’s where my involvement in the project came 
> in. I worked on implementing the below extensions:
> Implemented the Local transport extension to allow agent to transfer data 
> from its host machine given storage – Local transport extension 
> 
> Transport has three variations – streaming, chunked file transfer and chunked 
> streaming
> Implemented the CLI for configuring local agent – Local agent CLI 
> 
>  
> Performance testing results:
>  
> After successfully testing from my local machine to AWS S3 storage, I have 
> deployed agent in AWS EC2 machine and performed multiple tests for compare 
> it’s performance with rclone and AWS cli.
> Below charts indicates the average transfer speeds from our analysis.
>  
> 
>  
>  
> For files from 100MB to 1GB, MFT is more than 60% faster than rclone and more 
> than 150% faster than AWS cli.
>  
> Configurations of the testing:
>  
> Local Machine: It’s Ubuntu EC2 VM on AWS (instance type c5.9xlarge) with 18 
> cores, 10Gbps dedicated network speed and 1GBps read/write speed to disk.
>  
> Cloud Storage: AWS S3 bucket in the same region as above VM.
>  
> Test sets: From x-axis labels of the graph, 10m_1000 means a test set of 1000 
> 10MB files. All other test sets follow similar naming convention.
>  
> Testing trails: Each test is run for 5 times on each transfer method.
>  
> Testing presets: Before each test caching of VM is cleared so none of the 
> tests get advantage of higher read speeds using page caching. This is done to 
> simulate worst possible conditions while reading data.
>  
> MFT configuration: I used chunked streaming with
> 20MB as chunk size
> 32 concurrent transfers
> 32 concurrent chunked threads
>  
> rclone configuration: After exploring many possible optimizations available 
> for rclone I used following settings:
> --s3-chunk-size 128000
> --buffer-size 128000
> —s3-upload-cutoff 0
> --s3-upload-concurrency 32
> --multi-thread-streams 32
> --multi-thread-cutoff 0
> --s3-disable-http2
> --no-check-dest
> --transfers 32
> --fast-list
>  
> AWS cli configuration: I used native AWS cli to transfer as it doesn’t have 
> much dedicated optimizations in our findings
>  
> Observations:
> For local transport I used BufferedStreaming which helped MFT to get the max 
> read speeds from local disk without hitting the max IOPS.
>  
> Future plans for testing:
> Jetstream2: Planning to replace AWS EC2 with Jetstream2 virtual machine and 
> perform similar tests
> Emulab: Simulate same testing using Emulab VMs and custom configurations with 
> help of Dimuthu.
> Azure: Perform local to Azure cloud storages testing with MFT, rclone and 
> Azure cli
> GCP: Perform local to GCS testing with MFT, rclone and GCP cli
> I have different implementation of MFT local transport for system which 
> support DMA (Direct Memory Access), we also plan to test on such systems with 
> DMA, the present EC2 system doesn’t support DMA.
>  
> Further Improvements of MFT:
> As we noticed MFT is lagging speeds vs rclone for files less than or equal to 
> 1MB, we plan to stress analyze the whole system and improve speeds for 
> smaller files
>  
> Acknowledgement: I thank Dimuthu Wannipurage for clearing many doubts about 
> MFT and providing guidance when needed.
>  
> Thank you and please let us know your comments or thoughts. 
>  
> Best,
> Praneeth Chityala

Re: Cybershuttle Replica Catalog API

[Suresh Marru]

Hi Jayan,

Can you contribute a PR to the data catalog repo so we can keep the feedback on 
that issue?

Thanks for your contribution,
Suresh

> On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana 
>  wrote:
> 
> Hi All,
> 
> I have updated the draft code base[1] with a simple workflow of adding data 
> to replica catalog. Still services are not yet finalized and will be enhanced 
> with the workflow. 
> 
> [1] https://github.com/Jayancv/airavata-replica-catalog 
> 
> 
> Thanks.
> 
> On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana 
> mailto:jcvidanapathir...@gmail.com>> wrote:
> Hi Dimuthu and Marcus,
> 
> Thank you both for checking my PoC and providing valuable feedback.
> 
> Dimuthu,
> Im agree with you regarding Replica location categories. It should be a data 
> catalog level attribute. 
> To manage replica data access permissions don't we need user information at 
> Replica catalog level ? I'm a bit confused on the permission management side 
> of this catalog. 
> ReplicaListEntry  - Added to expose the list DataReplicaLocation s with basic 
> details in AllDataReplicaGetResponse which provide all the replica items for 
> the given product_id. However, here I was not considering that hierarchical 
> structure. ReplicaGroupEntry is actually a one product replica which holds 
> the file structure of the replica data. According to your suggestion we can 
> model that AllDataReplicaGetResponse as follows,
> message AllDataReplicaGetResponse {
>   data_product_id = 1
>   repeated ReplicaGroupEntry replica_list = 2;
> }
> 
> message ReplicaGroupEntry {
>   string replica_group_id = 1
>   repeated ReplicaGroupEntry directories = 2;
>   repeated DataReplicaLocation files = 3;
> }
> 
> Marcus, 
> Yes, I will remove replica_id  from the data catalog diagram. 
> I added that parent_data_product_id to replica data by considering full 
> context with replica catalog and data catalog relation. But within replica 
> catalog context there is no such paranet product relationship. Therefore we 
> can rename it to data_product_id. Thanks for pointing this out. 
> 
> Thanks.
> 
> On Thu, Feb 23, 2023 at 2:48 AM Christie, Marcus Aaron  > wrote:
> Hi Jayan,
> 
> I would like to echo Dimuthu and say that this looks great and I appreciate 
> the effort in your pulling this all together.  I have some feedback to share.
> 
> The high-level architecture diagram shows the replica id being stored in the 
> data catalog. That was an initial idea that we had, but we decided that the 
> replica catalog would store the data product id. That seems reflected in your 
> API design so I think you already know this, but I wanted to point it out 
> since the diagram might be a little confusing for others.
> 
> In the ReplicaCatalogAPI.proto the name of the data product id field is 
> "parent_data_product_id". I would suggest calling it "data_product_id" 
> instead. "parent_data_product_id" means "the id of the parent data product of 
> this data product" in the data catalog. It might be confusing to use the same 
> name in ReplicaCatalogAPI.proto.
> 
> 
> Thanks,
> 
> Marcus
> 
> > On Feb 18, 2023, at 3:09 PM, Jayan Vidanapathirana 
> > mailto:jcvidanapathir...@gmail.com>> wrote:
> > 
> > Hi All, 
> > 
> > As a new contributor to the Cybershuttle project, I have been actively 
> > involved in implementing the Data Replica Catalog. This new catalog is 
> > designed to interface with both the Apache Airavata Data Catalog [1] and 
> > Airavata MFT[2]. This replica catalog should be able to store each replica 
> > resource storage details and secret/credential details specific to the 
> > storage type. The proposed high-level architecture will be as follows:
> > 
> > 
> > 
> > I will mainly work on the highlighted area (red color box) and as an 
> > initial step started defining APIs which communicate with Replica Catalog. 
> > This API calls will be gRPC APIs and following methods will be implement,
> > 
> > Replica Registration
> > 
> >   • registerReplicaLocation(DataReplicaCreateRequest createRequest)
> >   • updateReplicaLocation(DataReplicaCreateRequest updateRequest)
> >   • DataReplicaLocationModel getReplicaLocation(DataReplicaGetRequest 
> > getReplicaRequest)
> >   • removeReplicaLocation(DataReplicaDeleteRequest deleteReplicaRequest)
> >   • getAllReplicaLocations(AllDataReplicaGetRequest allDataGetRequest)
> >   • removeAllReplicaLocations(AllDataReplicaDeleteRequest 
> > allDataDeleteRequest)
> > 
> > Storage Registration
> > 
> > registerSecretForStorage(SecretForStorage request)
> > deleteSecretsForStorage(SecretForStorageDeleteRequest request)
> > getSecretForStorage(SecretForStorageGetRequest request)
> > searchStorages(StorageSearchRequest request)
> > listStorages(StorageListRequest request)
> > resolveStorageType (StorageTypeResolveRequest request)
> > 
> > Storage - Internal APIs
> > 
> > S3StorageListResponse listS

Re: A small bug in collaborations.html of the Airavata Webpage

[Suresh Marru]

You should now have a JIRA account - druvidoshi

Suresh

> On Feb 28, 2023, at 11:02 AM, DHRUVI DOSHI  wrote:
> 
> Hello,
> Here is a screenshot of a small yet visibly significant bug on collaborations 
> page . 
> 
> I don't have a JIRA account as of now. Could someone please create a jira 
> ticket for the same or get me a jira account for me? 
> I would like to happily contribute towards fixing the same too.
> 
> BR,
> Dhruvi

Re: Expressing interest for GSOC 2023

[Suresh Marru]

Hi Dhruvi,

Yes you are on the right track, you can setup Django Portal locally and try 
various tutorials - 
https://apache-airavata-django-portal.readthedocs.io/en/latest/ 


If you can get to this one - 
https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_django_app/
 

 we might have some projects for you.

Suresh

> On Feb 28, 2023, at 10:49 AM, DHRUVI DOSHI  wrote:
> 
> Hello Airavata Community,
> 
> I hope this email finds you well.
> I'm Dhruvi Doshi, a final year undergrad pursuing Computer Engineering at 
> VJTI, India. I express my sincere interest in contributing to the Airavata. 
> With the GSOC program, I wish to commence my contributions to this community. 
> I do have a good hold over python. I have been a GSOC 2022 mentee where I 
> have worked on python and debian packages. Please find my attached resume.
> 
> I have been reading about the project and getting an overview of it. I would 
> like to request for some guidelines to get started. Starter tasks would be 
> great too. 
> Until then I will be setting up Airavata Django Portal locally. 
> 
> Thank You,
> BR,
> Dhruvi Doshi.
> 
> 

Re: Airawata MFT setup guide

[Suresh Marru]

Hi Uday,

You can start with the MFT client from the README Sudhakar mentioned. You can 
also try the docker version. 
https://github.com/apache/airavata-mft/tree/master/docker 


To understand the inner workings, starting from API and slowly digging into 
implementation might help. You can use this as a starting point - 
https://github.com/apache/airavata-mft/blob/master/api/stub/src/main/proto/MFTTransferApi.proto
 


Suresh

> On Feb 25, 2023, at 3:51 PM, Pamidighantam, Sudhakar  wrote:
> 
> Hi Uday:
>  
> Did you try to follow the README.md at all?
>  
> Thanks,
> Sudhakar.
>  
> From: Uday Kumar Bommala 
> Reply-To: "dev@airavata.apache.org" 
> Date: Saturday, February 25, 2023 at 3:37 PM
> To: "dev@airavata.apache.org" 
> Subject: Airawata MFT setup guide
>  
> Hi Team,
>  
> I am trying to setup Airawata MFT on my local machine to understand the 
> internal working. 
>  
> Can someone guide me with the steps to locally setup Airawata MFT?
>  
> Regards
> Uday Kumar Bommala

Re: SEAGrid Data Catalog

[Suresh Marru]

Hi Lahiru,

The two dependencies, a Django-grpc fork 
(https://github.com/socotecio/django-socio-grpc/ 
<https://github.com/socotecio/django-socio-grpc/>) and 
https://github.com/grpc/grpc-web <https://github.com/grpc/grpc-web> are 
reasonably ok. So building on them may not be a bad idea. But if you are 
hitting too frequent roadblocks, it may be wise to switch to 
Django-rest-framework and take your approach 1. Sometimes the downsides of 
depending on not-so-actively maintained dependencies outweigh the technical 
advantages. 

So + 1 to proceed with grpc, but if you stumble, revert to the REST-based 
approach. 

Suresh

> On Feb 16, 2023, at 3:24 AM, Lahiru Jayathilake  
> wrote:
> 
> Hi Marcus,
> 
> Thanks for the suggestions and the heads-up. Sure, I will do more 
> investigation on that and get back to you with the details.
> 
> Thanks,
> Lahiru
> 
> On Wed, Feb 15, 2023 at 8:36 PM Christie, Marcus Aaron  <mailto:machr...@iu.edu>> wrote:
> Hi Lahiru,
> 
> Thanks for putting together this investigation. I'm not 100% sure but it 
> looks like gRPC-JS only works with Node.js since it uses Node.js APIs. I 
> think you'll need gRPC-Web to make gRPC calls from a browser. My 
> understanding is that that requires an Envoy proxy on the server side. 
> (Rereading your email, I think you probably already know this, but just in 
> case I thought I would point this out.)
> 
> It looks like django-grpc-framework isn't an active project [1], so I agree 
> with your concern about depending on it. One issue with using gRPC in Django, 
> I think, is that the integration that we've done with the Django framework 
> would need to be re-implemented, things like middleware and authentication.  
> It's probably doable, just something to keep in mind.
> 
> It would be good if the gRPC server could run on the same HTTP port as the 
> Django server, but I'm not sure how that would work.  From the client, 
> accessing the Django server or the gRPC server should both be over SSL, on 
> the same port. Maybe on the backend they run on different ports but with the 
> proxy it looks like from the client's perspective they run on the same port.
> 
> The django-grpc-framework project may be good to mine for some ideas. I like 
> that it follows django-rest-framework conventions. We use 
> django-rest-framework in the Airavata Django Portal.
> 
> Thanks,
> 
> Marcus
> 
> [1] https://github.com/fengsp/django-grpc-framework/issues/34 
> <https://github.com/fengsp/django-grpc-framework/issues/34>
> 
> > On Feb 14, 2023, at 1:17 PM, Lahiru Jayathilake 
> > mailto:lahirujayathil...@gmail.com>> wrote:
> > 
> > Hi Suresh,
> > 
> > Thank you for the feedback. The other library that can be used to 
> > facilitate browser communication with gRPC services is gRPC-JS 
> > (https://github.com/grpc/grpc-node/tree/master/packages/grpc-js 
> > <https://github.com/grpc/grpc-node/tree/master/packages/grpc-js>). However, 
> > in terms of browser support, gRPC-Web is specifically designed for use in 
> > web browsers, and it supports all major browsers including Chrome, Firefox, 
> > Safari, and Edge. In contrast, gRPC-JS is designed to work with both web 
> > browsers and Node.js, and it may require additional configuration to work 
> > correctly in web browsers and it is a bit cumbersome.
> > 
> > @machr...@iu.edu <mailto:machr...@iu.edu> I had a chat with Suresh and 
> > wanted to clarify a few points with you.
> > 
> > 1. In the second approach what I have done is spinup up a gRPC server in 
> > the background (inside Django App). When I was doing that I came across a 
> > framework called django-grpc-framework [1][2]. I did not proceed with that 
> > framework because it is coming from a personal repository. What do you 
> > think? Is it good to go with this?
> > 
> > 2. Any suggestions or comments on the approach of using gRPC (gRPC-web) to 
> > establish communications with the frontend?
> > 
> > I'd be really happy to hear your thoughts and suggestions on these.
> > 
> > [1] - https://github.com/fengsp/django-grpc-framework 
> > <https://github.com/fengsp/django-grpc-framework>
> > [2] - https://pypi.org/project/djangogrpcframework/ 
> > <https://pypi.org/project/djangogrpcframework/>
> > 
> > Thanks, 
> > Lahiru
> > 
> > 
> > 
> > On Tue, Feb 14, 2023 at 7:47 PM Suresh Marru  > <mailto:sma...@apache.org>> wrote:
> > Hi Lahiru,
> > 
> > Thank you for summarizing both of these, and your POCs of both approaches 
> > are helpful. The second opt

Re: SEAGrid Data Catalog

[Suresh Marru]

Hi Lahiru,

Thank you for summarizing both of these, and your POCs of both approaches are 
helpful. The second option, if feasible, will be preferable. You already 
mentioned the performance. In addition, you will also get forward/backward 
compatibility if the underlying protobuff structures are maintained with some 
discipline. 

The big plus side of your 1st approach is the REST-compatible javascript 
libraries. Other than grpc-web (https://github.com/grpc/grpc-web 
) have you seen broader support? I see you 
are building on grpc-js how is that experience?

Suresh

> On Feb 13, 2023, at 2:49 PM, Lahiru Jayathilake  
> wrote:
> 
> Hi All,
> 
> I have been engaging with the SMILES project to implement the Gateway and its 
> necessary components. Just to give you a brief introduction, the SMILES 
> project has three types of data that need to be combined for publication: 
> Computational DB, Literature DB, and Experiment DB. There should be a 
> frontend to filter, create, and delete data products, with a Django app as 
> the backend that will communicate with Apache Airavata Data Catalog [1].
> 
> Mainly, I have been exploring two approaches.
> 
> 1. 
> 
> 
> The frontend will communicate with the Django app via REST, and the Django 
> app will manage the manipulation of data products through gRPC calls to the 
> Data Catalog API. Django models will be used to represent the Computational, 
> Literature, and Experiment data products, without storing the data. In the 
> end, these data products will reside in the Data Catalog, following its 
> established conventions.
> 
> POC - https://github.com/lahirujayathilake/SEAGrid 
>  
> This has been implemented to cover the data product creation
> 
> 2. 
> 
> 
> In this approach, the distinction will be a gRPC server operating within the 
> Django app. To represent the three data products, protobufs will be defined 
> that extend the DataCatalog proto messages [2]. The frontend will communicate 
> using gRPC calls.
> The gRPC API can be used to manipulate data from other clients, resulting in 
> improved performance.
> 
> POC - https://github.com/lahirujayathilake/SEAGrid/tree/with-grpc 
> 
> (The frontend is inprogress)
> 
> I would like to hear your thoughts and feedback on the designs to improve and 
> to go with the right approach. 
> 
> 
> [1] - https://github.com/apache/airavata-data-catalog 
> 
> [2] - 
> https://github.com/apache/airavata-data-catalog/blob/main/data-catalog-api/stubs/src/main/proto/DataCatalogAPI.proto
>  
> 
> 
> Cheers!
> Lahiru

[jira] [Updated] (AIRAVATA-1436) Provide meta scheduling capabilities within Airavata

[Suresh Marru (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRAVATA-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Marru updated AIRAVATA-1436:
---
Labels: gsoc2022 gsoc2023  (was: gsoc2022)

> Provide meta scheduling capabilities within Airavata
> 
>
> Key: AIRAVATA-1436
> URL: https://issues.apache.org/jira/browse/AIRAVATA-1436
> Project: Airavata
>  Issue Type: Epic
>  Components: Airavata Orchestrator, GFac
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, gsoc2023
> Fix For: WISHLIST
>
>
> As discussed on the architecture mailing list [1] and summarized at [2], 
> Airavata will need to develop a metascheduler. In the short term, a user 
> request (demeler, gobert) is to have airavata throttle jobs to resources. In 
> the future more informed scheduling strategies needs to be integrated. 
> Hopefully, the actual scheduling algorithms can be borrowed from third party 
> implementations. 
> [1] - http://markmail.org/message/tdae5y3togyq4duv
> [2] - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Metascheduler



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (AIRAVATA-3595) Airavata Rich Client based on ElectronJS

[Suresh Marru (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRAVATA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Marru updated AIRAVATA-3595:
---
Labels: gsoc2022 gsoc2023 mentor  (was: gsoc2022 mentor)

> Airavata Rich Client based on ElectronJS 
> -
>
> Key: AIRAVATA-3595
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3595
> Project: Airavata
>  Issue Type: Epic
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, gsoc2023, mentor
>
> Using SEAGrid Rich Client as an example, develop a native application based 
> on electronJS to mimic Airavata Django Portal.
> Reference example - [https://github.com/SciGaP/seagrid-rich-client] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (AIRAVATA-3598) Airavata Jupyter Platform Services

[Suresh Marru (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRAVATA-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Marru updated AIRAVATA-3598:
---
Labels: gsoc2023 mentor  (was: gsoc2022 mentor)

> Airavata Jupyter Platform Services
> --
>
> Key: AIRAVATA-3598
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3598
> Project: Airavata
>  Issue Type: Epic
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2023, mentor
> Attachments: image-2022-04-18-12-46-29-459.png
>
>
> # UI Framework 
>  ## To host the jupyter environment we will need to envolop the notebooks in 
> a user interface and connect it with Apache Airavata services 
>  ## Leverage Airavata communications from within the Django Portal - 
> [https://github.com/apache/airavata-django-portal] 
>  ## Explore if the platform is better to be developed as VSCode extensions 
> leveraging jupyter extensions like - 
> [https://github.com/Microsoft/vscode-jupyter]
>  ## Alternatively, explore developing a standalone native application using 
> ElectronJS
>  # Draft up a platform architecture - Airavata based infrastructure with 
> functionality similar to collab. 
>  # Authenticate with Airavata Custos Framework - 
> [https://github.com/apache/airavata-custos] 
>  # Extend Notebook filesystem using the virtual file system approaching 
> integration with Airavata based storage and catalog
>  # Make the notebooks registered with Airavata app catalog and experiment 
> catalog. 
>  
> Advanced Possibilities:
> Explore Multi-tenanted JupyterHub 
>  * Can K8 namespace isolation accomplish?
>  * Make deployment of Jupyter support as part of the default core
>  * Data and the user-level tenancy can be assumed, how to make sure 
> infrastructure can isolate them, like not one gateway crashing a hosting 
> environment.
>  # How to leverage computational resources jupypter hub



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [External] [DISCUSS] New name for MFT

[Suresh Marru]

Good, we are getting options. Let's gather a few more and then do a trademark 
search, filter, and vote on the final 3.

Suresh

> On Jan 31, 2023, at 12:59 PM, Thejaka Amila J Kanewala 
>  wrote:
> 
> Finding a name is harder than developing the product :D.
> 
> I think we already have nice suggestions.
> 
> Here are a few more:
> 1. Javelin : https://en.wikipedia.org/wiki/Javelin_throw 
> <https://en.wikipedia.org/wiki/Javelin_throw>
> 2. Catapult -- https://en.wikipedia.org/wiki/Catapult 
> <https://en.wikipedia.org/wiki/Catapult>
> 3. Dionysius -- they engineered the first version of catapult -- 
> https://www.hellenicaworld.com/Greece/Technology/en/Catapults.html 
> <https://www.hellenicaworld.com/Greece/Technology/en/Catapults.html>
> 4. Trebuchet -- https://en.wikipedia.org/wiki/Trebuchet 
> <https://en.wikipedia.org/wiki/Trebuchet>
> 
> As you can see I am a little obsessed with catapults :-).
> 
> Best Regards,
> Thejaka Amila Kanewala, PhD
> https://github.com/thejkane/agm <https://github.com/thejkane/agm>
> http://valagamba.net/ <http://valagamba.net/>
> 
> 
> On Tue, Jan 31, 2023 at 6:36 AM Lahiru Jayathilake 
> mailto:lahirujayathil...@gmail.com>> wrote:
> Three suggestions from me as well,
> 
> 1. SkyBridge
> 
> 2. Hammurabi - is a Babylonian king. The "Code of Hammurabi" is considered to 
> be one of the first written legal codes in history with the idea of fair and 
> orderly transfer of information. And the name exhibits stability, 
> reliability, and order in the transfer of information
> 
> 3. Akkad - was an ancient empire that was recognized for its advanced 
> administration and communication systems. The name exhibits stability and 
> efficiency in the transfer of large-scale data
> 
> Thanks,
> Lahiru
> 
> On Tue, Jan 31, 2023 at 7:48 PM Pierce, Marlon  <mailto:marpi...@iu.edu>> wrote:
> How about a ship theme? Here’s a list of types of ships: 
> https://en.wikipedia.org/wiki/List_of_ship_types 
> <https://en.wikipedia.org/wiki/List_of_ship_types>.
> 
>  
> 
> A couple of ancient types of ships: Kerkouros, Corbita. 
> 
>  
> 
> Marlon
> 
>  
> 
>  
> 
> From: Suresh Marru mailto:sma...@apache.org>>
> Date: Monday, January 30, 2023 at 5:59 PM
> To: Airavata Dev mailto:dev@airavata.apache.org>>
> Subject: [External] [DISCUSS] New name for MFT
> 
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> 
> Hi All,
> 
>  
> 
> We chose MFT as a code name as we carved out all file transfer use cases from 
> core Airavata. Now MFT has evolved significantly [1] and is most notably used 
> for large scale data transfers between NASA and ESA HLS Data [2] Transfers 
> between Openstack and AWS and IBM clouds. 
> 
>  
> 
> If we choose the name Custos for the security spinoff [3] [4], it is probably 
> time to choose a name for MFT. Please keep your creative naming thoughts 
> flowing?
> 
>  
> 
> What would you name a more generic MFT without naming technology terms like 
> Transfers? 
> 
>  
> 
> Cheers,
> 
> Suresh
> 
>  
> 
> [1] - https://github.com/apache/airavata-mft 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairavata-mft&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vjh6zmtXAfBtawFjibKuuVsC32iVdlN2NumHt3mdzoM%3D&reserved=0>
>  
> 
> [2] - https://hls.gsfc.nasa.gov/ 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhls.gsfc.nasa.gov%2F&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ounc1BVwq9t8Hw4V097SVopWc0sdUqUD2W1Ta0CVFoo%3D&reserved=0>
>  
> 
> [3] - https://airavata.apache.org/custos/ 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairavata.apache.org%2Fcustos%2F&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=htgm0fUgxiHCnhdmObJ1AIrllH5iSIF7ZNq%2F1hsaJHs%3D&reserved=0>
>  
> 
> [4] - https://github.com/apache/airavata-custos 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairavata-custos&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1wTvuC1z43ycfa273ILfE8PRyS1G5CUpBCS3POWSR0o%3D&reserved=0>
>  
> 

Re: [External] [DISCUSS] New name for MFT

[Suresh Marru]

Both Hauler and Ship names are good suggestions. Other thoughts? 

Suresh

> On Jan 31, 2023, at 9:18 AM, Pierce, Marlon  wrote:
> 
> How about a ship theme? Here’s a list of types of ships: 
> https://en.wikipedia.org/wiki/List_of_ship_types 
> <https://en.wikipedia.org/wiki/List_of_ship_types>.
>  
> A couple of ancient types of ships: Kerkouros, Corbita.  
>  
> Marlon
>  
>  
> From: Suresh Marru mailto:sma...@apache.org>>
> Date: Monday, January 30, 2023 at 5:59 PM
> To: Airavata Dev mailto:dev@airavata.apache.org>>
> Subject: [External] [DISCUSS] New name for MFT
> 
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Hi All,
>  
> We chose MFT as a code name as we carved out all file transfer use cases from 
> core Airavata. Now MFT has evolved significantly [1] and is most notably used 
> for large scale data transfers between NASA and ESA HLS Data [2] Transfers 
> between Openstack and AWS and IBM clouds. 
>  
> If we choose the name Custos for the security spinoff [3] [4], it is probably 
> time to choose a name for MFT. Please keep your creative naming thoughts 
> flowing?
>  
> What would you name a more generic MFT without naming technology terms like 
> Transfers? 
>  
> Cheers,
> Suresh
>  
> [1] - https://github.com/apache/airavata-mft 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairavata-mft&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vjh6zmtXAfBtawFjibKuuVsC32iVdlN2NumHt3mdzoM%3D&reserved=0>
>  
> [2] - https://hls.gsfc.nasa.gov/ 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhls.gsfc.nasa.gov%2F&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ounc1BVwq9t8Hw4V097SVopWc0sdUqUD2W1Ta0CVFoo%3D&reserved=0>
>  
> [3] - https://airavata.apache.org/custos/ 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairavata.apache.org%2Fcustos%2F&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=htgm0fUgxiHCnhdmObJ1AIrllH5iSIF7ZNq%2F1hsaJHs%3D&reserved=0>
>  
> [4] - https://github.com/apache/airavata-custos 
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairavata-custos&data=05%7C01%7Cmarpierc%40iu.edu%7C1d584c45eb9f4bb4c87f08db03159350%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C638107163426479694%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1wTvuC1z43ycfa273ILfE8PRyS1G5CUpBCS3POWSR0o%3D&reserved=0>

[DISCUSS] New name for MFT

[Suresh Marru]

Hi All,

We chose MFT as a code name as we carved out all file transfer use cases from 
core Airavata. Now MFT has evolved significantly [1] and is most notably used 
for large scale data transfers between NASA and ESA HLS Data [2] Transfers 
between Openstack and AWS and IBM clouds. 

If we choose the name Custos for the security spinoff [3] [4], it is probably 
time to choose a name for MFT. Please keep your creative naming thoughts 
flowing?

What would you name a more generic MFT without naming technology terms like 
Transfers? 

Cheers,
Suresh

[1] - https://github.com/apache/airavata-mft 
 
[2] - https://hls.gsfc.nasa.gov/  
[3] - https://airavata.apache.org/custos/  
[4] - https://github.com/apache/airavata-custos 
 

Re: Airavata MFT - Google Cloud Storage (GCS) transfer

[Suresh Marru]

Hi Jayan,

Thanks for your contributions. Can you please point to PR’s and issues related 
to these contributions?

Suresh

> On Jan 16, 2023, at 12:56 AM, Jayan Vidanapathirana 
>  wrote:
> 
> Hi All,
> 
> The existing Airavata Managed File Transfer system is now capable of handling 
> Google Cloud Storage as well. Also to improve the user experience with file 
> transferring, Airavata MFT uses Google Cloud CLI to load default user 
> credentials and python-based MFT CLI to interact easily with file 
> transferring and file exploring. Furthermore, there are some improvements 
> that need to be done to performance wise and user experience.  
> 
> Next steps,
> Perform a load test on GCS transfers
> Implement file chuck transferring for GCS  
> 
> Special thanks go to Dimuthu Upeksha for guiding me on these tasks. 
> 
> Thanks. 
> -- 
> Best Regards
> 
> Jayan Vidanapathirana
>  

Re: Regarding Pre and Post job commands in Airavata

[Suresh Marru]

Hi All,

We need to fork this discussion into three threads:

1. Implicit Workflow: The pre, post, and job submitter commands at the 
conceptual level are all within one process and within a task of that process. 
In this approach, we do not have granular monitoring and fault tolerance or 
reaction to the success/failure of these individual steps (for instance, of the 
pre-job fails, the job script is still executed). 

2. Explicit and dependent Workflow: All pre, post, and job steps can be made 
into separate tasks within a single process. In this approach, we can react to 
and control the flow between these steps based on the execution status. 

3. Classic Workflows: In these, we create individual processes which have 
subtasks and are treated at a higher level, like workflow engines or 
orchestrators. 

Hi Dinuka, 

I'd like to point out that the response to your question depends on how you are 
looking at it. For instance, the SLURM implements category 1 (above) in a 
single execution file. You can implement HTC pre and post-job scenarios using 
HTConfor DAG without any changes to the framework by making them individual 
tasks within HTCondor. Still, for Airavata, they will be one task. The obvious 
downside is you will loose granular monitoring and reaction to failures of 
individual steps. If you approach it as category 2 or 3, that will lead to a 
broader discussion and changes to the framework.

Suresh

> On Jan 5, 2023, at 10:28 AM, Pamidighantam, Sudhakar V 
>  wrote:
> 
> Dimuthu:
>  
> For gateway with specific applications the application or executable needs to 
> be transferred along with the user inputs to the condor pool, which may 
> require unpacking a tar ball or other mechanisms and identify the actual 
> executable by a driver script. The driver script is the current executable 
> application in the gateway but the actual executable or a container (the 
> tarball above) also need to transferred before the execution can be set up. 
> These steps require a pre job script or a way to add additional 
> transfer_input_files instructions at the connecting HT condor host.
>  
> Of course dags are quite diverse and can be used for various other usecases 
> such as parameter sweeps with different inputs (which again could be packaged 
> in tarballs and need extraction after staging) etc..
>  
> Thanks,
> Sudhakar. 
>  
> From: DImuthu Upeksha 
> Reply-To: "dev@airavata.apache.org" 
> Date: Thursday, January 5, 2023 at 10:14 AM
> To: "dev@airavata.apache.org" 
> Subject: Re: Regarding Pre and Post job commands in Airavata
>  
> Hi Dinuka,
>  
> Sorry for the late reply. It is great to explore options to integrate Dag 
> capabilities into the job submitter. We already have some form of HTCondor 
> support in Airavata. Can you summarize the difference between what we already 
> have for HTCondor and this Dag feature? I am specifically looking for 
> practical usages instead of technical differences.
>  
> Thanks
> Dimuthu 
>  
> On Sun, Jan 1, 2023 at 10:35 AM Dinuka De Silva  > wrote:
>> Hi,
>>  
>> The current implementation of mapping these (Pre, Post, etc. job commands) 
>> to the job scheduler script has assumed the job scheduler script to be a 
>> type of shell script. So, the order of the execution is based on the order 
>> of the commands listed in the script which is as below.
>>  
>> - Module Commands
>> - Pre Job Commands
>> - Job Submitter Command
>> - Post Job Commands
>>  
>> The scheduler script of SLURM, FORK, LSF, UGE, and PBS are shell scripts 
>> while HTCondor and maybe some other job schedulers have different file 
>> types. The script grammar in HTCondor does not support appending shell 
>> scripts inside. Now, we are needing to support Pre, Post, and other commands 
>> for HTCondor realizing the current design doesn't support it. 
>>  
>> In HTCondor there's an option [1] to configure pre and post-scripts to be 
>> executed at the worker instance. But then the script has to be Dag and the 
>> pre-script, post-script and job-script are to be separate files. So, I tried 
>> a sample and planning to put this to airavata.
>>  
>> [1] 
>> https://htcondor.readthedocs.io/en/latest/users-manual/dagman-workflows.html 
>> 
>>  
>> Thanks & Regards,
>> Dinuka

Re: How to setup the latest pga portal

[Suresh Marru]

Hi Dinuka,

These libraries are already within Airavata repo, so do not commit them, but 
rather fix the ide-integration to copy them into the right place on-demand. See:

https://github.com/apache/airavata-php-gateway/tree/master/app/libraries 
 

 
https://github.com/apache/airavata/tree/master/airavata-api/airavata-client-sdks/airavata-php-sdk/src/main/resources/lib
 


The first link is indeed copied from generated thrift libraries from Airavata, 
so copying them back will make it redundant. 

Suresh

> On Jan 5, 2023, at 8:00 AM, Dinuka De Silva  wrote:
> 
> Hi,
> 
> I found one workaround. 
> 
> Instead of copying everything from airavata-php-gateway, I copied only 
> "airavata-php-gateway/app/libraries/Airavata" to 
> "airavata/modules/ide-integration/src/main/containers/pga/airavata-php-gateway/app/libraries"
>  and then it works even though I'm not certain of the exact issue.
> 
> However, now I could get the latest pga code to work within airavata 
> repository, should I commit this code? I assume it's not meant to commit but 
> if I do whoever wants to set up this wouldn't have to go through what I did.
> 
> Thanks & Regards,
> Dinuka
> 
> On Wed, Jan 4, 2023 at 10:59 PM Dinuka De Silva  > wrote:
> Hi,
> 
> I'm trying to set up the latest code from airavata-php-portal [1] on my local 
> machine. I need that for creating and testing some HTCondor compute 
> resources. Here's how I tried.
> 
> I copied the latest source from airavata-php-portal to 
> "airavata/modules/ide-integration/src/main/containers/pga" path in the 
> airavata repository and ran "docker compose up" on the same directory. And I 
> end up with the following errors on the command line.
> 
> (base) dinukadesilva@Dinukas-MacBook-Pro pga % docker compose up
> [+] Running 1/0
>  ⠿ Container pga-pga-1  Created   
>   
>0.0s
> Attaching to pga-pga-1
> pga-pga-1  | 
> pga-pga-1  |   
>   
> pga-pga-1  |   [UnexpectedValueException]  
> 
> pga-pga-1 exited with code 1
> (base) dinukadesilva@Dinukas-MacBook-Pro pga % 
> 
> 
> But, if I do the same steps without copying the latest airavata-php-portal, 
> it works fine, and pga portal up and runs perfectly. 
> 
> Thanks & Regards,
> Dinuka
> 
> [1] https://github.com/apache/airavata-php-gateway 
> 

Re: Custos baremetal deployment - need domain names for jetstream2 VMs

[Suresh Marru]

Hi Abhinav,

I creating the following mapping, you should be able to use lets encrypt now to 
create the SSL certificates:

bm-dev-1.usecustos.org - 149.165.152.226
bm-dev-2.usecustos.org - 149.165.154.45
bm-dev-3.usecustos.org - 149.165.152.76

Suresh

> On Jul 20, 2022, at 9:33 PM, Abhinav Sinha  wrote:
> 
> Hi Eroma,
>  
> I am looking to generate SSL certificates for three vms needed to test Custos 
> bare-metal deployment. To do this, I need to register domain names for these 
> three jetstream2 ubuntu vms:
>  
> 149.165.152.226
> 149.165.154.45
> 149.165.152.76
>  
> Could you pls help me with this?
>  
> Thanks,
> Abhinav

Re: Regarding seagrid desktop client molecule viewer inetgration

[Suresh Marru]

Hi Aishwarya,

I echo it as well. Please do not wait for the code to be perfect, try to make 
it a habit to commit early and often to a PR. A big commit will delay reviews 
or demotivate reviews (due to lack of time), smaller and focused chucks make it 
faster. 

Suresh

> On Jul 1, 2022, at 1:56 PM, Aishwarya Sinhasane 
>  wrote:
> 
> Hii Dimuthu 
> 
> I will share document and code next week need some changes to do. 
> 
> Thanks and regards
> Aishwarya Sinhasane 
> 
> On Fri, Jul 1, 2022, 1:27 PM DImuthu Upeksha  > wrote:
> Hi Aishwarya, 
> 
> This is a very solid improvement. Is there a way we can try these out? Do you 
> have a working fork?
> 
> Thanks
> Dimuthu
> 
> On Fri, Jul 1, 2022 at 12:59 PM Pamidighantam, Sudhakar  > wrote:
> Aishwarya:
> 
>  
> 
> Looks good. Keep this and then look at JS molecular Editors or third party 
> editors such as Avagadro, or Ovito. Let us plan to meet next week to look at 
> these and discuss further.
> 
>  
> 
> Thanks,
> 
> Sudhakar.
> 
>  
> 
> From: Aishwarya Sinhasane  >
> Reply-To: "dev@airavata.apache.org " 
> mailto:dev@airavata.apache.org>>
> Date: Friday, July 1, 2022 at 12:02 PM
> To: "dev@airavata.apache.org " 
> mailto:dev@airavata.apache.org>>
> Subject: Regarding seagrid desktop client molecule viewer inetgration
> 
>  
> 
> Hello Sudhakar
> 
>  
> 
> I was working on integrating a molecule viewer in seagrid desktop client. I 
> used the NGL Viewer. I tried various other molecule viewers developed in 
> javascript but some of them are not working properly so I choose NGLViewer. 
> This molecule viewer supports mmCIF, PDB, mol2, SDF, MMTF structures. 
> 
>  
> 
> I am trying to implement a JSMol molecule viewer. Please refer to the 
> attached screenshots. 
> 

Re: Clarification on dashboard for quick statistics development

[Suresh Marru]

Hi Shubham,

See below: 

> On Jun 28, 2022, at 12:39 PM, Bipin Kumar, Shubham  wrote:
> 
> Hi Marcus,
> 
> I had the following question on dashboard developments . Can you please 
> clarify these:
> Can I use React for frontend development or is it compulsory to use VueJs?

As such we would not have opinion, but in this case please use VueJS since rest 
of the apps are built in it and It. 

> Should the dashboard be an independent application, or it should be 
> integrated as a page in Django portal?
It should be a Django app to be deployed within the Airavata Django Framework. 

> For getting the data like user details, experiment details or resources 
> details should we establish direct connections to respective databases or use 
> the Airavata API's to fetch those details. In case there aren't any API's to 
> support the data needed what should be the course of action?
Always only use API’s we suggest database interactions do not go beyond the 
respective components. So if you need to get any data Client -> Airavata 
API->Component CPI -> Database

Suresh

> 
> Regards,
> Shubham

Re: Custos - looking for collaborator access to Custos github

[Suresh Marru]

Hi Abhinav,

I already see a branch - 
https://github.com/apache/airavata-custos/tree/berametal 
<https://github.com/apache/airavata-custos/tree/berametal> can you use it? 

Suresh

> On Jun 20, 2022, at 12:23 PM, Abhinav Sinha  wrote:
> 
> Understood, thanks Suresh!
> 
> Hi Isuru,
> 
> I want to create a branch from the berametal branch in Custos. Could you 
> create a baremetal-integration-services branch for me?
> 
> Thanks,
> Abhinav
> 
> 
> On Mon, Jun 20, 2022, 12:17 PM Suresh Marru  <mailto:sma...@apache.org>> wrote:
> Hi Abhinav,
> 
> All Custos GitHub access control is reserved for Apache Airavata committers. 
> So you will have to request one of us to create a branch and from there on 
> submit anything as a PR.
> 
> Suresh
> 
>> On Jun 20, 2022, at 12:15 PM, Abhinav Sinha > <mailto:abhinav7.si...@gmail.com>> wrote:
>> 
>> Hi Isuru, Dev,
>>  
>> I’m looking to create a new branch in Custos GitHub repo. Could you provide 
>> me with collaborator access?
>> My GitHub username is – abhinav7sinha
>> Email – abhinav7si...@gmail.com <mailto:abhinav7si...@gmail.com>
>>  
>> Thanks,
>> Abhinav
> 

Re: SMILES Proto Schema

[Suresh Marru]

I suggest we use a system generated unique ID as the primary key and SMILES 
Strings as an optional queryable file.

Yes a call to demonstrate is fine. Let's coordinate on a time on Airavata slack 
channel.

Suresh

> On Jun 20, 2022, at 10:22 AM, Pamidighantam, Sudhakar V 
>  wrote:
> 
> Bhavesh:
>  
> We need to pick a primary key. SMILES string could be a good one but is not 
> absolutely needed to be. Also sometimes it is difficult to auto-generate 
> SMILES strings for molecules. We can choose another one such as name but we 
> need to consistently and uniquely use the name across data models.
> The filtering should get all the records and we can have pagination to 
> control how many are shown. Some options in the filters could include ranges 
> for example, Absorption Max between 500-560nm.
>  
> Thanks,
> Sudhakar.
> From: Bhavesh Asanabada  >
> Date: Monday, June 20, 2022 at 9:16 AM
> To: dev@airavata.apache.org  
> mailto:dev@airavata.apache.org>>
> Subject: Re: SMILES Proto Schema
> 
> Hi Sudhakar,
>  
> I have a few doubts;
> I don't find the primary key (the SMILES string) in other proto files. Do I 
> need to include it and access the data with the SMILES string?
> In the filtering options, Are there any predefined thresholds for the 
> quantities?
> I also request to have a meeting to confirm with my SMILES query code.
>  
> Thanks & Regards
> Bhavesh Asanabada

Re: Custos - looking for collaborator access to Custos github

[Suresh Marru]

Hi Abhinav,

All Custos GitHub access control is reserved for Apache Airavata committers. So 
you will have to request one of us to create a branch and from there on submit 
anything as a PR.

Suresh

> On Jun 20, 2022, at 12:15 PM, Abhinav Sinha  wrote:
> 
> Hi Isuru, Dev,
>  
> I’m looking to create a new branch in Custos GitHub repo. Could you provide 
> me with collaborator access?
> My GitHub username is – abhinav7sinha
> Email – abhinav7si...@gmail.com 
>  
> Thanks,
> Abhinav

Re: Changes in Django tutorial page

[Suresh Marru]

Hi Bhavesh,

This is good that you are suggesting changes, can you take the next step and 
submit a pull request with the changes you are suggesting against - 
https://github.com/apache/airavata-django-portal/edit/master/docs/tutorial/custom_ui_tutorial.md
 


Suresh

> On Jun 11, 2022, at 10:27 AM, Bhavesh Asanabada  
> wrote:
> 
> Hi everyone,
> 
> To the one who is looking at Airavata Django Portal - Tutorial 
> ,
>  please note the following changes;
> 
> 1. On the tutorial page, it is mentioned to install the Python of version 
> 3.9. But working with Python 3.9 throws an error in building grpcio wheel. 
> Therefore, downgrade the python version to 3.8.3 to run the application 
> locally.
> 
> 2. In the tutorial exercise of "Create a custom Django App", following step-5 
> of Adding a list of "Hello" greetings note the following changes in code.
> 
> 
> Thanks & Regards
> Bhavesh Asanabada
> 

Re: GSOC - 2022: Access to Jetstream VMs

[Suresh Marru]

Hi Abhinav,

I realized the class allocation expiration, will add you to a new allocation, 
will follow up offline and getting you Jetstream Cloud Access.

Suresh

> On Jun 8, 2022, at 10:14 AM, Abhinav Sinha  wrote:
> 
> Hi Dev,
>  
> I was looking to run some deploy Custos on a Jetstream VM. Is there a 
> Jetstream project allocation that I can use for this?
> @Suresh Marru <mailto:sma...@apache.org> The ADS group seems to have been 
> disabled.
>  
> Thanks,
> Abhinav 

[GSoC] - Actions Items - Sprint documents

[Suresh Marru]

Hi Bhavesh, Abhinav, Shubham, Aishwarya, Himanshu,

We had a below par community bonding period and we need all of you step up and 
shift into high frequency contribution mode. Please do the following before 
June 10th: 

* Create a wiki entry of your proposal. I created empty placeholders for each 
of your projects, edit them - 
https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+2022 
<https://cwiki.apache.org/confluence/display/AIRAVATA/GSOC+2022> 

* Create weekly sprints following this example - Create your next two weeks of 
sprints 1 and 2 as child documents to your wiki entry - 
https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+1 
<https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+1> 

Cheers,
Suresh

> On Jun 2, 2022, at 6:54 PM, Suresh Marru  wrote:
> 
> GSoC Students,
> 
> We haven't seen any communications on the mailing lists from you all. I 
> remind again GSoC is not a programming task but you need to be proactive and 
> get involved with everything on the community (Airavata in your case). So get 
> started on submitting PR’s. Suggest improvements to website, documentation, 
> newcomer tutorials. Essentially make contributions to anything which makes 
> Airavata community even more welcoming. 
> 
> Suresh

[GSoC] Community Bonding Period

[Suresh Marru]

GSoC Students,

We haven't seen any communications on the mailing lists from you all. I remind 
again GSoC is not a programming task but you need to be proactive and get 
involved with everything on the community (Airavata in your case). So get 
started on submitting PR’s. Suggest improvements to website, documentation, 
newcomer tutorials. Essentially make contributions to anything which makes 
Airavata community even more welcoming. 

Suresh

Review of DataLake Architecture and Database choices

[Suresh Marru]

Hi All,

We need to review Data Lake architecture - 
https://github.com/apache/airavata-data-lake 


Can Isuru or others please describe the current data base decisions, 
experiences with early deployments so we can discuss improvements. 

Thanks,
Suresh

[jira] [Commented] (AIRAVATA-3593) SMILES data Models

[Suresh Marru (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRAVATA-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526949#comment-17526949
 ] 

Suresh Marru commented on AIRAVATA-3593:


Suggested tasks to demonstrate required skills:
 * Deploy a local MongoDB
 * Store JSON-LD examples from [https://github.com/MolSSI/QCSchema] and 
[https://github.com/stuchalk/scidata] into the database
 * Do not directly expose the database to the UI but create an API to query 
data. For reference, you can look into [https://github.com/SciGaP/seagrid-data] 
 * Develop a Django App with VueJS front end to query the data and display the 
data and deploy it to a stand-alone instance of Airavata Django Portal
 * Refer to Airavata Data Lake and EmCenter UI for reference - 
[https://github.com/SciGaP/emcenter-gateway] 
 * Integrate with Custos Security [https://airavata.apache.org/custos/] 

> SMILES data Models
> --
>
> Key: AIRAVATA-3593
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3593
> Project: Airavata
>  Issue Type: Epic
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, mentor
> Attachments: Proposal Draft-1.pdf, Proposal architecture draft-1.png
>
>
> Extend Airavata Data Catalog to record metadata extracted from experimental 
> and computational data in support of the small-molecule ionic isolation 
> lattices SMILES data.
> Suggested flow:
> VueJS user interfaces -> Django App -> API Server -> Data Orchestrator -> 
> Data Lake
> Refer to [https://github.com/apache/airavata-data-lake]
> The data models should be developed in JSON-LD [https://json-ld.org/] 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: GSOC - Apache data lake

[Suresh Marru]

Hi Sreesha,

I am looking at the proposals and I do not see any submissions from you? Just 
making sure we did not miss any. 

Suresh

> On Apr 6, 2022, at 4:07 PM, Sreesha Srinivasan Kuruvadi  
> wrote:
> 
> Hi,
> I am interested in the Arivata data lake project. I am also interested in 
> Kubernetes and Docker based projects. 
> 
> I feel apache atlas like architecture is much more powerful for the airavata 
> data lake project.
> -- 
> Sreesha S Kuruvadi

Re: [GSoC] Apache Airavata GSOC Application instructions

[Suresh Marru]

Very important - Make sure you submitted your GSOC proposal on the official 
website - https://summerofcode.withgoogle.com/

Suresh

> On Apr 13, 2022, at 1:02 PM, Suresh Marru  wrote:
> 
> GSoC Aspirants,
> 
> Here are some helpful pointers as you prepare your applications for Airavata 
> GSOC projects:
> 
> * Understand GSOC Student Guide - 
> https://google.github.io/gsocguides/student/ 
> <https://google.github.io/gsocguides/student/> 
> 
> * Airavata GSoC Ideas - 
> https://cwiki.apache.org/confluence/display/COMDEV/GSoC+2022+Ideas+list#GSoC2022Ideaslist-Airavata
>  
> <https://cwiki.apache.org/confluence/display/COMDEV/GSoC+2022+Ideas+list#GSoC2022Ideaslist-Airavata>
>  
> 
> * Review general do’s and don’ts - 
> https://opensource.googleblog.com/2011/03/dos-and-donts-of-google-summer-of-code.html
>  
> <https://opensource.googleblog.com/2011/03/dos-and-donts-of-google-summer-of-code.html>
> 
> * Python foundation helpful tips - 
> https://wiki.python.org/moin/SummerOfCode/Expectations 
> <https://wiki.python.org/moin/SummerOfCode/Expectations>
> 
> * Sample proposal from a previous successful project - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGSoC+Proposal%5D+Securing+Airavata+API
>  
> <https://cwiki.apache.org/confluence/display/AIRAVATA/[GSoC+Proposal]+Securing+Airavata+API>
> 
> * Finally, follow the application template listed here - 
> https://community.apache.org/gsoc.html 
> <https://community.apache.org/gsoc.html>
> 
> Most importantly, do ask for feedback on your proposal early and often on 
> this list. Yes your proposal content can be open and public and may be your 
> first task embracing open source approaches. 
> 
> Suresh

[jira] [Commented] (AIRAVATA-3593) SMILES data Models

[Suresh Marru (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRAVATA-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522287#comment-17522287
 ] 

Suresh Marru commented on AIRAVATA-3593:


Hi Bhavesh,

>From a high level perspective, yes there are Airavata pipelines that will 
>integrate with these data systems. But for the scope of the GSOC project, I 
>think you can focus on this problem and build a plan. For reference, you can 
>see the data lake architecture - 
>[https://github.com/apache/airavata-data-lake] but it is ok to confine your 
>project to stand alone development. 

> SMILES data Models
> --
>
> Key: AIRAVATA-3593
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3593
> Project: Airavata
>      Issue Type: Epic
>Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, mentor
> Attachments: Proposal architecture draft-1.png
>
>
> Extend Airavata Data Catalog to record metadata extracted from experimental 
> and computational data in support of the small-molecule ionic isolation 
> lattices SMILES data.
> Suggested flow:
> VueJS user interfaces -> Django App -> API Server -> Data Orchestrator -> 
> Data Lake
> Refer to [https://github.com/apache/airavata-data-lake]
> The data models should be developed in JSON-LD [https://json-ld.org/] 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GSoC] Apache Airavata GSOC Application instructions

[Suresh Marru]

GSoC Aspirants,

Here are some helpful pointers as you prepare your applications for Airavata 
GSOC projects:

* Understand GSOC Student Guide - https://google.github.io/gsocguides/student/ 
 

* Airavata GSoC Ideas - 
https://cwiki.apache.org/confluence/display/COMDEV/GSoC+2022+Ideas+list#GSoC2022Ideaslist-Airavata
 

 

* Review general do’s and don’ts - 
https://opensource.googleblog.com/2011/03/dos-and-donts-of-google-summer-of-code.html
 


* Python foundation helpful tips - 
https://wiki.python.org/moin/SummerOfCode/Expectations 


* Sample proposal from a previous successful project - 
https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGSoC+Proposal%5D+Securing+Airavata+API
 


* Finally, follow the application template listed here - 
https://community.apache.org/gsoc.html 

Most importantly, do ask for feedback on your proposal early and often on this 
list. Yes your proposal content can be open and public and may be your first 
task embracing open source approaches. 

Suresh

Re: Google Summer of Code

[Suresh Marru]

Hi Himanshu,

With your background, may be you can explore a project to integrate data 
catalog - https://github.com/SciGaP/seagrid-data 
 into Airavata Data Lake - 
https://github.com/apache/airavata-data-lake 


I can have a call and describe the problem so you can prepare a proposal. You 
can ping me on Airavata Slack - 
https://join.slack.com/t/airavata/shared_invite/zt-178t82i1g-bu7CQRvcFBOnZVNgYGxATA
 

 

Suresh

> On Apr 12, 2022, at 6:34 PM, Hansaria, Himanshu  wrote:
> 
> Hello All,
> 
> I'm Himanshu Hansaria. I'm doing my Masters in Data Science from IUB. Prior 
> to joining IUB, I have worked as a Software Engineer at HSBC for 3 years and 
> 1.5 years as a Machine Learning Engineer where I got experience working with 
> consuming REST API's and using SDK's to autofill data in a form from images. 
> I was also involved in creating visualization dashboards from customer 
> finance data using Javascript.
> Working for the client of Eli Lilly during my course at IUB, I got an 
> experience working with graph databases as well.
> 
> Currently, I'm looking at these two projects :- AIRAVATA-3593 Smiles Data 
> Models and AIRAVATA-3595 Airavata Rich client based on ElectronJS.
> 
> I'm excited and looking forward to applying for the Google Summer of Code 
> program with Apache Airavata.
> Can someone help me in understanding these problems in depth and provide any 
> starting pointers if possible for coming up with a proposal.
> 
> Thanking you,
> Kind regards,
> Himanshu

[jira] [Updated] (AIRAVATA-3595) Airavata Rich Client based on ElectronJS

[Suresh Marru (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRAVATA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Marru updated AIRAVATA-3595:
---
difficulty-level: Hard  (was: Medium)

> Airavata Rich Client based on ElectronJS 
> -
>
> Key: AIRAVATA-3595
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3595
> Project: Airavata
>  Issue Type: Epic
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, mentor
>
> Using SEAGrid Rich Client as an example, develop a native application based 
> on electronJS to mimic Airavata Django Portal.
> Reference example - [https://github.com/SciGaP/seagrid-rich-client] 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Applying for Google Summer Code

[Suresh Marru]

Hi Ashwarya,

You can follow this issue and see if it interests you - 
https://issues.apache.org/jira/projects/AIRAVATA/issues/AIRAVATA-3595?filter=allopenissues
 


Suresh

> On Apr 12, 2022, at 4:05 PM, Aishwarya Sinhasane 
>  wrote:
> 
> Hello 
> 
> I am Aishwarya Sinhasane. I am currently pursuing a Masters in Computer 
> Science. Before coming to the USA, I was working as a software engineer 
> Intern at Accenture. I was responsible for building Rest APIs to extract data 
> for different SKU's such as sales per month, Inventory reports. I have also 
> worked on Java Spring boot during my undergraduate. I have some experience 
> working on deep learning models for image classification. After taking an 
> applied distributed systems course I got more interested in distributed 
> systems. 
> 
> I'd like to take part in the Google Summer Code program. I request you to 
> please share the project details as soon as possible. 
> 
> Thank you 
> Aishwarya Sinhasane 

[jira] [Commented] (AIRAVATA-3593) SMILES data Models

[Suresh Marru (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRAVATA-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519281#comment-17519281
 ] 

Suresh Marru commented on AIRAVATA-3593:


Some reference examples to look at - [https://stuchalk.github.io/scidata/] 
[https://github.com/stuchalk/scidata] 

[https://github.com/MolSSI/QCSchema] 

> SMILES data Models
> --
>
> Key: AIRAVATA-3593
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3593
> Project: Airavata
>  Issue Type: Epic
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, mentor
>
> Extend Airavata Data Catalog to record metadata extracted from experimental 
> and computational data in support of the small-molecule ionic isolation 
> lattices SMILES data.
> Suggested flow:
> VueJS user interfaces -> Django App -> API Server -> Data Orchestrator -> 
> Data Lake
> Refer to [https://github.com/apache/airavata-data-lake]
> The data models should be developed in JSON-LD [https://json-ld.org/] 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (AIRAVATA-3593) SMILES data Models

[Suresh Marru (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRAVATA-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Marru updated AIRAVATA-3593:
---
Description: 
Extend Airavata Data Catalog to record metadata extracted from experimental and 
computational data in support of the small-molecule ionic isolation lattices 
SMILES data.

Suggested flow:

VueJS user interfaces -> Django App -> API Server -> Data Orchestrator -> Data 
Lake

Refer to [https://github.com/apache/airavata-data-lake]

The data models should be developed in JSON-LD [https://json-ld.org/] 

  was:Extend Airavata Data Catalog to record metadata extracted from 
experimental and computational data in support of the small-molecule ionic 
isolation lattices SMILES data.


> SMILES data Models
> --
>
> Key: AIRAVATA-3593
> URL: https://issues.apache.org/jira/browse/AIRAVATA-3593
> Project: Airavata
>  Issue Type: Epic
>Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022, mentor
>
> Extend Airavata Data Catalog to record metadata extracted from experimental 
> and computational data in support of the small-molecule ionic isolation 
> lattices SMILES data.
> Suggested flow:
> VueJS user interfaces -> Django App -> API Server -> Data Orchestrator -> 
> Data Lake
> Refer to [https://github.com/apache/airavata-data-lake]
> The data models should be developed in JSON-LD [https://json-ld.org/] 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [airavata-user-docs] branch main created (now 55a1907)

[Suresh Marru]

+ 1, I think we need that separation currently they are tangled.

Suresh

> On Apr 1, 2022, at 4:57 PM, Christie, Marcus Aaron  wrote:
> 
> Hi Dev,
> 
> Eroma is working on a project to move all of the user level documentation to 
> a separate repo/site. The plan is to keep airavata-docs for 
> developer-oriented documentation and airavata-user-docs for user 
> documentation.
> 
> Thanks,
> 
> Marcus
> 
>> On Apr 1, 2022, at 11:35 AM, machris...@apache.org wrote:
>> 
>> This is an automated email from the ASF dual-hosted git repository.
>> 
>> machristie pushed a change to branch main
>> in repository 
>> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Fairavata-user-docs.git&data=04%7C01%7Cmachrist%40iu.edu%7Cab8902144ba44288680808da13f56049%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637844241842549600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=V87PGfTF5IchcCY63%2FnKDmGywqKJY3HpTc9dwG1RSTQ%3D&reserved=0.
>> 
>> 
>> at 55a1907  Moving 0.18 to Previous Releases section
>> 
>> This branch includes the following new commits:
>> 
>>new 79b9984  adding a doc index file to trigger github integrations
>>new eed63a2  adding airavata documentation contributed by Eroma on github 
>> scigap docs folder
>>new 3fd4bb9  Added WSO2 IS related configurations in pga_config.php
>>new e36bf37  adding IS config configuration in pga
>>new 7e01591  Disambiguating scigap and airavata
>>new 3ea5a69  Modifications to generate DBs just as its presented in 
>> airavata-server.properties. This will reduce the user confusion and 
>> simplifies the steps.
>>new 441e3cb  Merge remote-tracking branch 'origin/master'
>>new 8816c10  Fixed the Ubuntu dependency installation link
>>new 73599cd  Fixed an error in one of the MYSQL commands
>>new 6a8e54b  Fixed some of the links in the documentation
>>new 297c904  Added a link button for Ubuntu for easy navigation
>>new d598581  Removed extra Compute Resource link
>>new b682138  Modifications to some sections to make moe user friendly
>>new e21a583  Modifications to SQL commands to make them appear in a 
>> single line
>>new b251db6  Changed the color of 'Question' and 'Answer'
>>new 9448468  Adding WSO2 IS installation Steps
>>new 5872766  Adding WSO2 IS installation Steps
>>new d51a39d  Moving IS installation under installation menu
>>new 4bfb96a  Moving IS installation under installation menu
>>new 5bd89a1  Minor modifications to the heading size, and sub heading 
>> size through out documentation and also added Apache Airavata in place on 
>> Airavata at heading level
>>new 0e848b6  Added mysql.jar download path
>>new ecdb94b  Added mysql.jar download path
>>new 4350e89  Modifications to Airavata Introduction
>>new 787932d  Adding API methods for Admin Dashboard Functions
>>new cfb04ee  Fixed a Typo on API documentation
>>new acf1994  Adding descriptions to API methods and linking with 
>> generated API documentation
>>new 0a9bdfe  Adding descriptions to Admin Dashboard API methods
>>new 68bd2d7  Adding descriptions to Admin Dashboard API methods
>>new 390a7f4  Adding descriptions to Admin Dashboard API methods
>>new 514b4c0  Corrections and clarifications to the instructions.
>>new 7462f68  Fixing a error in WSO2 claim configuration documentation
>>new af6cefe  Fixing a error in WSO2 claim configuration documentation
>>new b441bcd  Gateway Admin Maintenance Steps
>>new 9d0e808  Modified a Typo
>>new 5200bb1  Corrected the WSO2 IS configuration steps in 
>> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwso2.org%2Fclaims&data=04%7C01%7Cmachrist%40iu.edu%7Cab8902144ba44288680808da13f56049%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637844241842549600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Cp68AQPVSEyBJKFLZH8EOXbOli%2FPtU%2FGG4%2B%2BPtlNCZU%3D&reserved=0
>>  and 
>> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwso2.org%2Foidc%2Fclaim&data=04%7C01%7Cmachrist%40iu.edu%7Cab8902144ba44288680808da13f56049%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637844241842549600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hcQpDKLKS0Ur43VlW%2BjJk77PAmF8upMbXF9wkxYzGLE%3D&reserved=0
>>new 707208b  Common Gateway maintenance scenarios
>>new f95ec0e  Adding Maintenance Instruction Pages
>>new 487f77e  Adding PGA upgrade steps
>>new bfeeeb4  Adding PGA upgrade steps
>>new 1b4410f  Adding Airavata upgrade steps page
>>new 493bc25   Added stesp to upgrade Airavata
>>new 74a4125   Added stesp to upgrade PGA
>>new 2cbcc70  Added link to Airavata update page and typo fix
>>new d4adcf4  Added link to Airavata update page and typo fix
>>new d2e791e  Modifications to 

[jira] [Updated] (AIRAVATA-1436) Provide meta scheduling capabilities within Airavata

[Suresh Marru (Jira)]

 [ 
https://issues.apache.org/jira/browse/AIRAVATA-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Marru updated AIRAVATA-1436:
---
  difficulty-level: Medium
Labels: gsoc2022  (was: gsoc2019)
  potential-mentor: Dimuthu, Suresh Marru
required-knowledge: Distributed Systems
  useful-knowledge: Job Scheduling

> Provide meta scheduling capabilities within Airavata
> 
>
> Key: AIRAVATA-1436
> URL: https://issues.apache.org/jira/browse/AIRAVATA-1436
> Project: Airavata
>  Issue Type: Epic
>  Components: Airavata Orchestrator, GFac
>    Reporter: Suresh Marru
>Priority: Major
>  Labels: gsoc2022
> Fix For: WISHLIST
>
>
> As discussed on the architecture mailing list [1] and summarized at [2], 
> Airavata will need to develop a metascheduler. In the short term, a user 
> request (demeler, gobert) is to have airavata throttle jobs to resources. In 
> the future more informed scheduling strategies needs to be integrated. 
> Hopefully, the actual scheduling algorithms can be borrowed from third party 
> implementations. 
> [1] - http://markmail.org/message/tdae5y3togyq4duv
> [2] - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Metascheduler



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: GSOC 2022: Airavata meta scheduler Documentation

[Suresh Marru]

Hi Shubham,

You can probably review the email list discussions and brief wiki referenced 
from here - https://issues.apache.org/jira/browse/AIRAVATA-1436 


Suresh

> On Mar 29, 2022, at 11:11 AM, Bipin Kumar, Shubham  wrote:
> 
> Hello Suresh,
> 
> I was going through the Apache Airavata documentation 
> 👎https://cwiki.apache.org/confluence/display/AIRAVATA/Apache+Airavata+Overview
>  
> 
> 
> and also, the tutorial links provided by Dinuka :- 
> https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/custom_ui_tutorial/
>  
> 
> 
> Further I was looking for the Airavata meta scheduler Documentation so that I 
> can explore all the areas before deciding on one. Can you please point me in 
> the right direction.
> 
> 
> Regards,
> Shubham

Re: Regarding GSoC 2022 - Custos architecture review

[Suresh Marru]

Hi Abhinav,

Here are few papers and tutorials which might provide you the details:

https://dl.acm.org/doi/pdf/10.1145/3311790.3396635 


https://arxiv.org/pdf/2107.04172.pdf 

https://cwiki.apache.org/confluence/display/CUSTOS/eScience21+Custos+Tutorial 
 

Suresh

> On Mar 29, 2022, at 9:56 AM, Abhinav Sinha  wrote:
> 
> Hello Suresh!
> 
> Good morning and I hope you're doing well! I am going through the Custos 
> Architecture documentation on Confluence and trying to understand where I 
> could contribute.
> https://cwiki.apache.org/confluence/display/CUSTOS/Custos+Architecture+and+Internals
>  
> 
> 
> I am just reaching out to ask if you have any supplementary information that 
> I can refer to as well?
> 
> Thanks,
> Abhinav
> 
> 
> 
>  

Re: Completion of Tutorial

[Suresh Marru]

Hi Bhavesh,

Please summarize the gsoc project idea conversation you had with Sudhakar. 

Meanwhile here are some related work:

https://jcheminf.biomedcentral.com/articles/10.1186/s13321-016-0168-9 
<https://jcheminf.biomedcentral.com/articles/10.1186/s13321-016-0168-9>

https://docs.open-reaction-database.org/en/latest/schema.html 
<https://docs.open-reaction-database.org/en/latest/schema.html>

Suresh

> On Mar 15, 2022, at 10:51 AM, Bhavesh Asanabada  
> wrote:
> 
> Hi Suresh,
> 
> I had completed the setup of Airavata Data Lake and checked the codebase.
> 
> 
> Thanks & Regards
> Bhavesh Asanabada
> 
> On Mon, Mar 14, 2022 at 8:43 AM Suresh Marru  <mailto:sma...@apache.org>> wrote:
> Hi Bhavesh,
> 
> Thats very nice, kudos on your first step. Given that you expressed interest 
> to contribute Open Inventory like capabilities to Airavata, couple of these 
> steps might help you get more familiar:
> 
> Review Airavata Data Lake code and set it up locally - 
> https://github.com/apache/airavata-data-lake 
> <https://github.com/apache/airavata-data-lake>
> 
> Review the codebase https://github.com/SciGaP/seagrid-data 
> <https://github.com/SciGaP/seagrid-data>
> 
> Review similar GSoC projects in the past - 
> https://medium.com/@lahiru_j/gsoc-2018-re-architect-output-data-parsing-into-airavata-core-81da4b37057e
>  
> <https://medium.com/@lahiru_j/gsoc-2018-re-architect-output-data-parsing-into-airavata-core-81da4b37057e>
> 
> Suresh 
> 
>> On Mar 13, 2022, at 10:51 PM, Bhavesh Asanabada > <mailto:asanabhav...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> I have completed the Airavata Django Portal - Custom UI Tutorial.
>> Heartly thanks to Dinuka, for helping me with this tutorial.
>> 
>> Thanks & Regards
>> Bhavesh Asanabada
>> 
> 

Re: Completion of Tutorial

[Suresh Marru]

Hi Bhavesh,

Thats very nice, kudos on your first step. Given that you expressed interest to 
contribute Open Inventory like capabilities to Airavata, couple of these steps 
might help you get more familiar:

Review Airavata Data Lake code and set it up locally - 
https://github.com/apache/airavata-data-lake 


Review the codebase https://github.com/SciGaP/seagrid-data 


Review similar GSoC projects in the past - 
https://medium.com/@lahiru_j/gsoc-2018-re-architect-output-data-parsing-into-airavata-core-81da4b37057e
 


Suresh 

> On Mar 13, 2022, at 10:51 PM, Bhavesh Asanabada  
> wrote:
> 
> Hi,
> 
> I have completed the Airavata Django Portal - Custom UI Tutorial.
> Heartly thanks to Dinuka, for helping me with this tutorial.
> 
> Thanks & Regards
> Bhavesh Asanabada
> 

Re: Access to run an application (Gaussian16).

[Suresh Marru]

You should now have permissions. Refresh your screen.

> On Mar 10, 2022, at 9:24 AM, Bhavesh Asanabada  wrote:
> 
> Hi,
> 
> I have installed all the requisites mentioned in Custom UI Tutorial and 
> created the Airavata Test Drive account. I can find many applications in the 
> dashboard but none has an access to use, also it shows a message as a 
> notification is sent to the administrator to grant privileges. Waiting to get 
> access from the administrator.
> 
> Thank you.

Re: Implementing ELN model under GSoC.

[Suresh Marru]

Also, reading this will help you get the context again - 
https://google.github.io/gsocguides/student/ 
<https://google.github.io/gsocguides/student/>

Suresh

> On Mar 8, 2022, at 7:50 AM, Suresh Marru  wrote:
> 
> Hi Bhavesh,
> 
> Good to see your interest in GSoC and Apache Airavata. Try this tutorial on 
> developing a custom Django App as a starter task - 
> https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/custom_ui_tutorial/
>  
> <https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/custom_ui_tutorial/>
> 
> We will soon post Airavata GSoC ideas, but this should help you get started.
> 
> Suresh
> 
>> On Mar 7, 2022, at 10:06 PM, Bhavesh Asanabada > <mailto:asanabhav...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> I'm Mr. Bhavesh Asanabada, studying final year of B. Tech in the Department 
>> of Electronics and Communication Engineering. With my interest in Artificial 
>> intelligence and Machine Learning, I started to learn Python programming and 
>> thereby learned many technologies to find the purpose of AI in each field. 
>> I've attached my resume for your kind consideration.
>> 
>> I'm excited on hearing about Google Summer on Code(GSoC) and the benefits I 
>> can gain from it. Also, I would like to express my gratitude to Mr. Suresh 
>> for introducing me to this and sharing an open-source project.
>> 
>> After studying the open-inventory document and the website, I feel this 
>> product is much helpful for students, research scholars, and also the future 
>> development of ELN for a particular purpose will maximize their productivity 
>> and stretch their goals. In the last meeting we had, I came to know about 
>> this product in detail and was happy to develop the product under Apache 
>> Foundation.
>> 
>> It would give me great pleasure to find my work profile in this project 
>> scope and with at most sincerity, I'll work on this project and carry out 
>> the results on each due date. 
>> 
>> Thank you
>> With regards.
>> ‌
>> 
> 

Re: Implementing ELN model under GSoC.

[Suresh Marru]

Hi Bhavesh,

Good to see your interest in GSoC and Apache Airavata. Try this tutorial on 
developing a custom Django App as a starter task - 
https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/custom_ui_tutorial/
 


We will soon post Airavata GSoC ideas, but this should help you get started.

Suresh

> On Mar 7, 2022, at 10:06 PM, Bhavesh Asanabada  wrote:
> 
> Hi,
> 
> I'm Mr. Bhavesh Asanabada, studying final year of B. Tech in the Department 
> of Electronics and Communication Engineering. With my interest in Artificial 
> intelligence and Machine Learning, I started to learn Python programming and 
> thereby learned many technologies to find the purpose of AI in each field. 
> I've attached my resume for your kind consideration.
> 
> I'm excited on hearing about Google Summer on Code(GSoC) and the benefits I 
> can gain from it. Also, I would like to express my gratitude to Mr. Suresh 
> for introducing me to this and sharing an open-source project.
> 
> After studying the open-inventory document and the website, I feel this 
> product is much helpful for students, research scholars, and also the future 
> development of ELN for a particular purpose will maximize their productivity 
> and stretch their goals. In the last meeting we had, I came to know about 
> this product in detail and was happy to develop the product under Apache 
> Foundation.
> 
> It would give me great pleasure to find my work profile in this project scope 
> and with at most sincerity, I'll work on this project and carry out the 
> results on each due date. 
> 
> Thank you
> With regards.
> ‌
> 

Airavata Jupyter Kernel Repo

[Suresh Marru]

Hi All,

We have been scattering Jupyter support within Airavata in various repo’s and 
sandbox. I created a new repo [1[ to pull together Airavata data and execution 
support to enable from Jupyter. All major kernels are listed at [2], we can 
probably model against the Spark kernel support incubated as a new ASF project 
Toree [3]  [4].

Cheers,
Suresh

[1] - https://github.com/apache/airavata-jupyter-kernel 

[2] - https://github.com/jupyter/jupyter/wiki/Jupyter-kernels 

[3] - https://toree.apache.org/  
[4] - https://github.com/apache/incubator-toree 
 

Re: Adding user API for Airavata

[Suresh Marru]

Hi Derek,

The security modules including user management are pulled into a separate 
service Custos [1] and relevant API’s are here [2]. Custos services will 
provide a Authorization Token which will then need to be used to authenticate 
and authorize for all Airavata API’s [3]. 

Cheers,
Suresh

[1] - https://github.com/apache/airavata-custos 

[2] - 
https://cwiki.apache.org/confluence/display/CUSTOS/Use+Custos+REST+Endpoints 
 
[3] - 
http://airavata.apache.org/api-docs/0.16/security_model.html#Struct_AuthzToken 

 

> On Nov 19, 2021, at 5:08 PM, Derek Weitzel  wrote:
> 
> Hi,
> 
> In the API documentation 
> (https://docs.airavata.org/en/latest/technical-documentation/airavata-api/), 
> I don’t see an API call to add a user.  Is it documented anywhere?
> 
> - Derek
> 
> 
> 

Re: [External] Request for a GSoC project

[Suresh Marru]

Hi Chirag,

The deadline is tomorrow 2pm EST. Go-ahead and submit the application at 
https://summerofcode.withgoogle.com/ <https://summerofcode.withgoogle.com/> 

Suresh

> On Apr 12, 2021, at 3:48 AM, Shankar Indi, Chirag  wrote:
> 
> Hi Professor,
> 
> I just wanted to clarify a few things regarding the problem statement. Please 
> correct me if I am wrong.
> We are trying to create visualizations for a user dashboard with Superset for 
> the AppCatalog variables like the Compute Resource, Storage Resource, User 
> Resources etc.
>  Similarly, what are items of interest to visualize in Custos?
> 
> I have created a draft proposal with an estimated timeline. Please let me 
> know what changes are required.
> 
> Thank you,
> 
> Chirag
> From: Shankar Indi, Chirag 
> Sent: Sunday, April 11, 2021 8:10 PM
> To: dev@airavata.apache.org ; sma...@apache.org 
> 
> Subject: Re: [External] Re: Request for a GSoC project
>  
> Hi Professor,
> 
> I've gone through it and I think I understand the goals, although I still 
> have doubts regarding the implementation.
> 
> I will send a proposal draft in a few hours, which I kindly request you to 
> review and let me know if any changes are needed.
> 
> Thank you,
> 
> Chirag 
> 
> From: Suresh Marru 
> Sent: Sunday, April 11, 2021, 8:06 PM
> To: Airavata Dev; Shankar Indi, Chirag
> Subject: [External] Re: Request for a GSoC project
> 
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Any question on this project? 
> 
>> On Apr 9, 2021, at 12:38 PM, Suresh Marru > <mailto:sma...@apache.org>> wrote:
>> 
>> Hi Chirag,
>> 
>> Please call me Suresh.
>> 
>> Here is a project idea you can attempt to write a proposal to - 
>> https://issues.apache.org/jira/browse/AIRAVATA-3436 
>> <https://issues.apache.org/jira/browse/AIRAVATA-3436>
>> 
>> You can start by looking to Airavata registries and understand the database 
>> structures and integrate a dashboard - 
>> https://github.com/apache/airavata/tree/master/modules/registry 
>> <https://github.com/apache/airavata/tree/master/modules/registry>
>> 
>> For GSOC specific stuff, here are some guidelines - 
>> https://community.apache.org/gsoc.html 
>> <https://community.apache.org/gsoc.html>
>> 
>> The key is to engage with this dev list extensively and keep asking 
>> questions until you feel you understood the problem. 
>> 
>> Suresh
>> 
>>> On Apr 9, 2021, at 10:39 AM, Shankar Indi, Chirag >> <mailto:chs...@iu.edu>> wrote:
>>> 
>>> Hi Professor Marru,
>>> 
>>> I am interested in participating in GSoC 2021 and I am requesting a project.
>>> 
>>> Looking forward to hearing from you.
>>> 
>>> Thank you!
>>> 
>>> Chirag Shankar Indi
>> 
> 
> 
> 

Re: Request for a GSoC project

[Suresh Marru]

Any question on this project? 

> On Apr 9, 2021, at 12:38 PM, Suresh Marru  wrote:
> 
> Hi Chirag,
> 
> Please call me Suresh.
> 
> Here is a project idea you can attempt to write a proposal to - 
> https://issues.apache.org/jira/browse/AIRAVATA-3436 
> <https://issues.apache.org/jira/browse/AIRAVATA-3436>
> 
> You can start by looking to Airavata registries and understand the database 
> structures and integrate a dashboard - 
> https://github.com/apache/airavata/tree/master/modules/registry 
> <https://github.com/apache/airavata/tree/master/modules/registry>
> 
> For GSOC specific stuff, here are some guidelines - 
> https://community.apache.org/gsoc.html 
> <https://community.apache.org/gsoc.html>
> 
> The key is to engage with this dev list extensively and keep asking questions 
> until you feel you understood the problem. 
> 
> Suresh
> 
>> On Apr 9, 2021, at 10:39 AM, Shankar Indi, Chirag > <mailto:chs...@iu.edu>> wrote:
>> 
>> Hi Professor Marru,
>> 
>> I am interested in participating in GSoC 2021 and I am requesting a project.
>> 
>> Looking forward to hearing from you.
>> 
>> Thank you!
>> 
>> Chirag Shankar Indi
> 

Re: Request for a GSoC project

[Suresh Marru]

Hi Chirag,

Please call me Suresh.

Here is a project idea you can attempt to write a proposal to - 
https://issues.apache.org/jira/browse/AIRAVATA-3436 


You can start by looking to Airavata registries and understand the database 
structures and integrate a dashboard - 
https://github.com/apache/airavata/tree/master/modules/registry 


For GSOC specific stuff, here are some guidelines - 
https://community.apache.org/gsoc.html 

The key is to engage with this dev list extensively and keep asking questions 
until you feel you understood the problem. 

Suresh

> On Apr 9, 2021, at 10:39 AM, Shankar Indi, Chirag  wrote:
> 
> Hi Professor Marru,
> 
> I am interested in participating in GSoC 2021 and I am requesting a project.
> 
> Looking forward to hearing from you.
> 
> Thank you!
> 
> Chirag Shankar Indi

Airavata style guides

[Suresh Marru]

Hi All,

I am trying to compile a list of Style Guides we use in Airavata code bases, so 
we can make the code uniform. As an example, I am trying to normalize the 
naming conventions for proto files and picked google one. Please suggest 
alternatives if you have other favorites.

https://cwiki.apache.org/confluence/display/AIRAVATA/Style+Guide

Cheers,
Suresh

Catching up on releases

[Suresh Marru]

Hi All,

We are quite behind on releases. I am starting to capture updated release 
instructions in the wiki - 
https://cwiki.apache.org/confluence/display/AIRAVATA/Apache+Airavata+Release+Management
 


We have releasable tags of 0.17, 0.18 and 0.19 branches. Once we catch up, may 
be we should all brainstorm what should constitute in a 1.0 release and make 
plans for it.

I will follow up with 0.17 release,

Cheers,
Suresh

Metadata Catalog Evaluations - Hackethons

[Suresh Marru]

Hi All,

To build on prior discussion on Airavata Data Lake [1], [2], [3], our next big 
step is to make implementation choices. Looks like Apache Airflow [4]  is a 
unparalleled choice. If any of you are interested will be happy to provide a 
detailed breakdown of this evaluation. 

On the contrary, a choice is metadata catalog is tricky given the overwhelming 
number of competing options and all have their own strengths. Looks like the 
best way forward is for us to document the capabilities which are important to 
airavata and do a hackethon exploring each of the choices and settle on one. 
Magda [5], and Atlas [6] both looked promising but do not natively support 
multi-tenancy. Can we all explore together DataHub [7], Amundsen [8] and 
Metacat [9]. There are more options, but I listed the ones with wide 
contribution base. 

Thoughts,

Cheers,
Suresh

[1] - https://markmail.org/thread/cjasb2m5ag6hb7y6 
 
[2] - https://markmail.org/thread/z2arxbby6xxb57pq 
 
[3] - https://github.com/apache/airavata-data-lake 

[4] - https://airflow.apache.org/  
[5] - https://magda.io/  
[6] - https://atlas.apache.org/#/ 
[7] - https://github.com/linkedin/datahub  
[8] -https://github.com/amundsen-io/amundsen 
 
[9] - https://github.com/Netflix/metacat  

Re: [External] Data Catalog into its own repo

[Suresh Marru]

Also a pointer to a potential JSON-LD java implementation to consider - 
https://github.com/jsonld-java/jsonld-java 
<https://github.com/jsonld-java/jsonld-java>

Suresh

> On Nov 10, 2020, at 1:06 PM, Suresh Marru  wrote:
> 
> Thank you all for weighing in. I bootstrapped the repo with some basic 
> information, please contribute to set the goals for this refactored sub 
> system - https://github.com/apache/airavata-data-lake 
> <https://github.com/apache/airavata-data-lake>
> 
> I was doing a literature and software survey on any open source metadata and 
> provenance systems we can integrate with, I found this survey paper useful - 
> https://www.researchgate.net/profile/Carlos_Saenz-Adan/publication/323242431_A_systematic_review_of_provenance_systems/links/5b34ae1caca2720785effb1a/A-systematic-review-of-provenance-systems.pdf
>  
> <https://www.researchgate.net/profile/Carlos_Saenz-Adan/publication/323242431_A_systematic_review_of_provenance_systems/links/5b34ae1caca2720785effb1a/A-systematic-review-of-provenance-systems.pdf>
> 
> Seems like we can build fairly flexible and yet sophisticated capabilities 
> using the schema.org <http://schema.org/> JSON-LD schema - 
> https://schema.org/ <https://schema.org/> , thoughts?
> 
> Please contribute any other pointers we should brainstorm before proceeding. 
> 
> Cheers,
> Suresh
> 
>> On Nov 9, 2020, at 4:22 PM, Pierce, Marlon > <mailto:marpi...@iu.edu>> wrote:
>> 
>> +1 for this refactoring. 
>> 
>> On 11/9/20, 9:47 AM, "Pamidighantam, Sudhakar" > <mailto:pamid...@iu.edu>> wrote:
>> 
>>This message was sent from a non-IU address. Please exercise caution when 
>> clicking links or opening attachments from external sources.
>>---
>> 
>>+2. 
>> 
>>Thanks,
>>Sudhakar.
>> 
>>On 11/9/20, 9:37 AM, "Marru, Suresh" > <mailto:sma...@iu.edu>> wrote:
>> 
>>Hi All,
>> 
>>Airavata Experiment catalog evolved over time and though the replica 
>> catalog and data product models are stand alone, they are buried to use them 
>> outside the experiment context. Any objections to refactor experiment 
>> catalog and make data catalog a first class repo, in the lines of Custos and 
>> MFT?
>> 
>>Cheers,
>>Suresh
>> 
> 

Re: [External] Data Catalog into its own repo

[Suresh Marru]

Thank you all for weighing in. I bootstrapped the repo with some basic 
information, please contribute to set the goals for this refactored sub system 
- https://github.com/apache/airavata-data-lake 


I was doing a literature and software survey on any open source metadata and 
provenance systems we can integrate with, I found this survey paper useful - 
https://www.researchgate.net/profile/Carlos_Saenz-Adan/publication/323242431_A_systematic_review_of_provenance_systems/links/5b34ae1caca2720785effb1a/A-systematic-review-of-provenance-systems.pdf
 


Seems like we can build fairly flexible and yet sophisticated capabilities 
using the schema.org JSON-LD schema - https://schema.org/  
, thoughts?

Please contribute any other pointers we should brainstorm before proceeding. 

Cheers,
Suresh

> On Nov 9, 2020, at 4:22 PM, Pierce, Marlon  wrote:
> 
> +1 for this refactoring. 
> 
> On 11/9/20, 9:47 AM, "Pamidighantam, Sudhakar"  wrote:
> 
>This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
>---
> 
>+2. 
> 
>Thanks,
>Sudhakar.
> 
>On 11/9/20, 9:37 AM, "Marru, Suresh"  wrote:
> 
>Hi All,
> 
>Airavata Experiment catalog evolved over time and though the replica 
> catalog and data product models are stand alone, they are buried to use them 
> outside the experiment context. Any objections to refactor experiment catalog 
> and make data catalog a first class repo, in the lines of Custos and MFT?
> 
>Cheers,
>Suresh
> 

Re: [AIRAVATA-3379] airavata-ruby-sdk

[Suresh Marru]

Thank you Eric, we will follow up on the PR.

Suresh

> On Oct 22, 2020, at 4:13 PM, Franz, Eric  wrote:
> 
> Suresh,
>  
> I posted a comment here 
> https://github.com/apache/airavata/pull/265#issuecomment-714734200 
> <https://github.com/apache/airavata/pull/265#issuecomment-714734200>
>  
> And am happy to help however I can.
>  
> Thanks,
> Eric
>  
> ---
> Eric Franz, Gateways Lead Engineer
> Ohio Supercomputer Center
> An Ohio Technology Consortium (OH-TECH) Member
> 1224 Kinnear Road
> Columbus, OH 43212
> email: efr...@osc.edu <mailto:efr...@osc.edu>
>  
> From: Suresh Marru 
> Date: Thursday, October 22, 2020 at 12:18 PM
> To: Airavata Dev , "Franz, Eric" 
> Subject: Re: [AIRAVATA-3379] airavata-ruby-sdk
>  
> Hi Eric,
>  
> Do you have suggestions on how ruby SDK’s should be distributed? Once Dinuka 
> packages thrift generated ruby libraries and adds some higher order 
> simplification API’s, can you please advise on what is the best practice for 
> creating a gem and publishing it to a gem repo?
>  
> Thanks,
> Suresh
> 
> 
>> On Oct 21, 2020, at 3:02 PM, Dinuka Desilva > <mailto:l.dinukadesi...@gmail.com>> wrote:
>>  
>> Hi Everyone,
>>  
>> Following the other client sdks [1] available, I'm going to enable the ruby 
>> sdk [2] for the airavata api. Meanwhile, I have a couple of points to 
>> discuss.
>>  
>> 1) Should we follow a similar pattern for additional code (apart from what's 
>> generated from thrift) introduced to the clients regardless of language? The 
>> feasibility is also one aspect to be discussed.
>>  
>> 2) What's the process of publishing the clients to public registries?
>>  
>> Regards,
>> Dinuka
>>  
>> [1] 
>> https://github.com/apache/airavata/tree/master/airavata-api/airavata-client-sdks
>>  
>> <https://urldefense.com/v3/__https:/github.com/apache/airavata/tree/master/airavata-api/airavata-client-sdks__;!!KGKeukY!kF3Lp5XEjGrUd3Mn1_MHMskmcOjmNBqg3n3lDCUX8Va4jJ8H3oowtrIcirtiPA$>
>> [2] https://github.com/apache/airavata/pull/265 
>> <https://urldefense.com/v3/__https:/github.com/apache/airavata/pull/265__;!!KGKeukY!kF3Lp5XEjGrUd3Mn1_MHMskmcOjmNBqg3n3lDCUX8Va4jJ8H3oowtrImbZefOQ$>

Re: [AIRAVATA-3379] airavata-ruby-sdk

[Suresh Marru]

Hi Eric,

Do you have suggestions on how ruby SDK’s should be distributed? Once Dinuka 
packages thrift generated ruby libraries and adds some higher order 
simplification API’s, can you please advise on what is the best practice for 
creating a gem and publishing it to a gem repo?

Thanks,
Suresh

> On Oct 21, 2020, at 3:02 PM, Dinuka Desilva  wrote:
> 
> Hi Everyone,
> 
> Following the other client sdks [1] available, I'm going to enable the ruby 
> sdk [2] for the airavata api. Meanwhile, I have a couple of points to discuss.
> 
> 1) Should we follow a similar pattern for additional code (apart from what's 
> generated from thrift) introduced to the clients regardless of language? The 
> feasibility is also one aspect to be discussed.
> 
> 2) What's the process of publishing the clients to public registries?
> 
> Regards,
> Dinuka
> 
> [1] 
> https://github.com/apache/airavata/tree/master/airavata-api/airavata-client-sdks
>  
> 
> [2] https://github.com/apache/airavata/pull/265 
> 
> 

Re: HTCondor Extension Help

[Suresh Marru]

Hi Daniel,

Thanks for this nice summary for keeping the dev list apprised of your 
progress. 

One additional item is to discuss and plan on job monitoring approaches, you 
can look at current references here - 
https://github.com/apache/airavata/tree/master/modules/job-monitor 

Suresh

> On Oct 1, 2020, at 4:52 PM, Daniel Shchur  
> wrote:
> 
> Hi all,
>  
> We are working on the HTCondor extension for Airavata. We have gone through 
> the code and started to implement additions based on the other job workflow 
> systems such as SLURM. We forked 
>  the project and 
> created a new branch (htcondor) to host our changes. We will make a pull 
> request into the main Airavata repository for it once we have finished the 
> implementation. A relevant issue has also been created here 
> .
>  
> We would like help to know exactly where additions need to be made in case we 
> missed a few spots. At this point, we know that:
> A groovy template needs to be made (finished)
> A job configuration class needs to be made (WIP)
> A job parser class needs to be made (WIP)
> And enums need to be modified throughout the project so that systems know of 
> HTCondor as an option. (finished)
>  
> If any of you could direct us to the right places, that would be greatly 
> appreciated.
>  
> Also, if there’s more documentation or information about the code, such as 
> how the command map should work or should be implemented (configuration 
> class?), that would be appreciated.
>  
> Thank you!
>  
> - Daniel Shchur

Re: Recursive entity fetching in registry

[Suresh Marru]

Hi Dimuthu,

I have been wondering if GraphQL will be worthwhile to explore for such bulk 
retrieval’s? I think this will be a low hanging fruit for us to try to 
integrating with something like Hasura[4] it is dominantly used for Postgres 
but they have been actively supporting MySql as well [5] 

I am not sure if this will address you point out though, since GraphQL spec 
doesn’t support recursive nested objects [6], simple one to many seems to be 
supported [7] but not bi directional.

Suresh

[4] - https://github.com/hasura/graphql-engine 

[5] - https://hasura.io/docs/1.0/graphql/core/guides/mysql-preview.html 
 
[6] - https://github.com/graphql/graphql-spec/issues/91 

[7] - 
https://hasura.io/docs/1.0/graphql/core/schema/table-relationships/database-modelling/one-to-many.html
 

 

> On Sep 22, 2020, at 11:01 AM, DImuthu Upeksha  
> wrote:
> 
> Hi Folks,
> 
> I just noticed that the entity relations that have bidirectional definitions 
> (OneToMany and ManyToOne) between 2 tables cause recursive object fetching 
> and it is causing a significant performance drop in database querying. For 
> example [1] and [2]. When we fetch an experiment output, I can see a stack of 
> experiment -> experiment outputs -> experiment 
> 
> Same issue was mentioned in [3] but the solution is for spring boot json 
> parsing. As we don't use spring boot, we might have to figure out some other 
> way to stop this recursive stack creation.
> 
> [1] 
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L88
>  
> 
> [2] 
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputEntity.java#L80
>  
> 
> [3] 
> https://medium.com/@udith.indrakantha/issue-related-with-infinite-recursive-fetching-of-data-from-relationships-between-entity-classes-ffc5fac6c816
>  
> 
> 
> Thanks
> Dimuthu

Re: Dev Environment - Registering Compute Resources?

[Suresh Marru]

Hi Eugene,

Apache Airavata web portal transitioned from a PHP (laravel) based code base to 
Python (Django) based code. What you are citing is one the last features which 
have not yet been ported.  Larval code base  is here 
https://github.com/apache/airavata-php-gateway 
<https://github.com/apache/airavata-php-gateway>

We need tp provide you a work around on registering your compute resources. I 
will contact you off line to help with that.

Suresh

> On Jul 26, 2020, at 1:57 PM, Eugene Walker  
> wrote:
> 
> Hi Suresh,
> 
> Thank you for the quick response!  Are you suggesting I try the 
> ansible-playbooks, but using master branch? If so I can definitely try that.
> 
> I have the IntellJ environment + Airavata Django Portal setup now, BUT I am 
> not seeing where one goes to register a NEW compute resource in the Django 
> Portal. I only see a default compute resource of 
> js-156-93.jetstream-cloud.org <http://js-156-93.jetstream-cloud.org/>. Is 
> there no interface for registering compute resources using Airavata Django 
> Portal? Is that interface excluded from the open source distribution of 
> Airavat Django Portal?
> 
> Thank you!
> 
>  
> 
> On Sun, Jul 26, 2020 at 9:01 AM Suresh Marru  <mailto:sma...@apache.org>> wrote:
> Hi Eugene,
> 
> Sorry for lack of clarity in documentation. Yes, you are right 
> ode-integration instructions are most upto date if you are setting a local 
> development instance. But give you mention the use of hosted SciGaP services, 
> we need to understand your setup more to provide appropriate guidance. 
> 
> Firstly, can you please try using the master branch instead of 0.17 release?
> 
> Please refer to to this gateway administrator guide for configuring compute 
> resources and application descriptions - 
> https://docs.airavata.org/en/master/user-documentation/admin-tutorials/ 
> <https://docs.airavata.org/en/master/user-documentation/admin-tutorials/>
> 
> If you are using SciGaP services, you need not setup core Airavata services 
> locally and only deploy the portal. 
> 
> If you are adding a new application to integrate with existing portal this 
> tutorial might be of help -  
> https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/gateways2019_tutorial/#tutorial-exercise-create-a-custom-django-app
>  
> <https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/gateways2019_tutorial/#tutorial-exercise-create-a-custom-django-app>
> 
> Suresh
> 
>> On Jul 24, 2020, at 9:38 PM, Eugene Walker > <mailto:eugenesunsetwal...@gmail.com>> wrote:
>> 
>> I followed the instructions for setting up a development environment with 
>> IntelliJ and using airavat-django-portal for frontend:
>> https://github.com/apache/airavata-django-portal 
>> <https://github.com/apache/airavata-django-portal>
>> https://github.com/apache/airavata/tree/master/modules/ide-integration 
>> <https://github.com/apache/airavata/tree/master/modules/ide-integration>
>> 
>> Where can I find out how to register compute resources in this setup? The 
>> only compute preference I see out of the box is for:
>> js-156-93.jetstream-cloud.org <http://js-156-93.jetstream-cloud.org/> And I 
>> don't see where to add another.
>> 
>> I have been going through the documentation I can find and a lot of it is 
>> from 2014, 2015, and 2016. I found a guide for deploying Airavata 0.17 using 
>> ansible playbooks, but that is also from many years ago, and the ansible 
>> scripts use master branch and rely on artifacts which are no longer 
>> generated as the output of building with those branches. 
>> 
>> It seems like the ide-integration instructions are the most up-to-date 
>> documentation. Is this right? Are there more up-to-date instructions / 
>> documentation available? I am trying to help my team set up a development 
>> environment for the purpose of integrating our software with an application 
>> currently deployed via the hosted SciGap service. In order to test we will 
>> need to be able to register compute resources, is this possible using the 
>> development environment setup described in ide-integration link above?
>> 
>> Also, is anyone aware of up-to-date Ansible playbooks available? Something 
>> other than for Airavata 0.17 ?
> 

Re: Dev Environment - Registering Compute Resources?

[Suresh Marru]

Hi Eugene,

Sorry for lack of clarity in documentation. Yes, you are right ode-integration 
instructions are most upto date if you are setting a local development 
instance. But give you mention the use of hosted SciGaP services, we need to 
understand your setup more to provide appropriate guidance. 

Firstly, can you please try using the master branch instead of 0.17 release?

Please refer to to this gateway administrator guide for configuring compute 
resources and application descriptions - 
https://docs.airavata.org/en/master/user-documentation/admin-tutorials/ 


If you are using SciGaP services, you need not setup core Airavata services 
locally and only deploy the portal. 

If you are adding a new application to integrate with existing portal this 
tutorial might be of help -  
https://apache-airavata-django-portal.readthedocs.io/en/latest/tutorial/gateways2019_tutorial/#tutorial-exercise-create-a-custom-django-app
 


Suresh

> On Jul 24, 2020, at 9:38 PM, Eugene Walker  
> wrote:
> 
> I followed the instructions for setting up a development environment with 
> IntelliJ and using airavat-django-portal for frontend:
> https://github.com/apache/airavata-django-portal 
> 
> https://github.com/apache/airavata/tree/master/modules/ide-integration 
> 
> 
> Where can I find out how to register compute resources in this setup? The 
> only compute preference I see out of the box is for:
> js-156-93.jetstream-cloud.org  And I 
> don't see where to add another.
> 
> I have been going through the documentation I can find and a lot of it is 
> from 2014, 2015, and 2016. I found a guide for deploying Airavata 0.17 using 
> ansible playbooks, but that is also from many years ago, and the ansible 
> scripts use master branch and rely on artifacts which are no longer generated 
> as the output of building with those branches. 
> 
> It seems like the ide-integration instructions are the most up-to-date 
> documentation. Is this right? Are there more up-to-date instructions / 
> documentation available? I am trying to help my team set up a development 
> environment for the purpose of integrating our software with an application 
> currently deployed via the hosted SciGap service. In order to test we will 
> need to be able to register compute resources, is this possible using the 
> development environment setup described in ide-integration link above?
> 
> Also, is anyone aware of up-to-date Ansible playbooks available? Something 
> other than for Airavata 0.17 ?

GitHub Issues enabled on MFT Repos

[Suresh Marru]

Hi All,

Looks like the use of GitHub issues is working well for Custos repos. I now 
enabled it for MFT and MFT Portal repos as well. Let me know if anyone rather 
prefer to use JIRA instead.

Cheers,
Suresh

Re: [Discussion] Securing single-page-application using JWT token

[Suresh Marru]

Hi Isuru,

The images got dropped, can you please add them to wiki and share a link?

Thanks,
Suresh

> On Jul 8, 2020, at 4:55 PM, Isuru Ranawaka  wrote:
> 
> Hi Shivam, Anuj
> 
> thanks for your comments on this.  Please find and below description and
> comments, it might helpful for further understanding of the message flow of
> authentication and come up with a good design.
> 
> 
> The first figure depicts endpoint types for external applications to
> integrate with Custos services. Non browser applications can directly
> integrate with Custos using Python, Java, or Rest endpoints as preferred.
> 
> Integrating browser applications is not straightforward with python SDK or
> JAVA SDK, because it might require some framework to cross the bridge
> between frontend client side code to server side code. (e.g Django
> framework). But, browser apps can directly use Rest endpoints for
> integration.
> 
> *How Authentication Works*
> 
>   All clients are confidential clients for Custos. The minimum
> authentication mechanism for APIs is to use basic auth with base64 encoded
> Custos credentials. Application should securely store Custos secret and use
> that for basic authentication. Since, this is static, I  guess storing that
> won’t be an issue.
> 
> Some Custos APIs and application APIs require user access tokens to
> authenticate.
> 
> At first login, Access token , ID token and Refresh token are issued in the
> response body in JWT. Subsequent calls to API require an access token of
> the logged user to authenticate.
> 
> We need to finalize methods to store access token, refresh token and Id
> token. As Shivam , anuj suggested  can use local storage,  cookies or  API
> Gateway  approach. Any ideas  on
>   - How to handle silent refresh for invalidated tokens
>   -  Synchronizing logouts for multiple tabs
> 
> On Tue, Jul 7, 2020 at 9:32 PM Suresh Marru  <mailto:sma...@apache.org>> wrote:
> 
>> Thanks Anuj for your insights.
>> 
>> Hi Isuru,
>> 
>> It might help draw up an architecture of front end UI components to
>> backend security gateway to brainstorm this and arrive at a consensus of
>> handling API security for UI’s.
>> 
>> Suresh
>> 
>> On Jul 7, 2020, at 7:52 PM, anuj bhandar  wrote:
>> 
>> I agree with your research, Options [1] & [2] are *not* recommended.
>> 
>> We use AWS Amplify (https://aws.amazon.com/amplify/) to handle that
>> scenario, our tech stack comprises React front-end, AWS Amplify integrated
>> with our company's SSO provider, and an Rails API backend.
>> 
>> Using a third party like AWS Amplify will not handle all the security
>> concerns, you will have to pay special attention to Token Validity. JWT
>> claims offer a good amount of control if set up properly (
>> https://tools.ietf.org/html/rfc7519).
>> 
>> I am not sure if this was helpful. Happy to help wherever I can.
>> 
>> Thanks,
>> *Anuj Bhandar*
>> Intuit Inc.
>> 
>> On Tue, Jul 7, 2020 at 3:52 PM Rastogi, Shivam  wrote:
>> 
>>> Hi Everyone,
>>> 
>>> 
>>> Just wanted to check if anyone has any experience in securing Single Page
>>> Webapps using JWT tokens.
>>> 
>>> 
>>> 
>>> *Challenge*
>>> 
>>> The challenge we are facing is how to store the JWT token at the
>>> client-side in the browser which will be used to securely call the back-end
>>> APIs.
>>> 
>>> 
>>> *Background*
>>> 
>>> We are working on the custos UI portal and one of the design options that
>>> we are considering is to directly call the custos backend APIs from
>>> client-side javascript. The basic flow would look like:
>>> 
>>> 
>>> 1. User logins using username and password from the login screen using
>>> login API
>>> 
>>> 2. JWT token and basic user-information is returned to the client in
>>> response.
>>> 
>>> 3. In subsequent API requests, user will send this authentication token.
>>> But,how to store this token at client-side?
>>> 
>>> 
>>> *Options*
>>> 
>>>   1. *Store it in localStorage* - This is not considered to be secure
>>>   as localstorage is vulnerable to XSS attacks-
>>>   https://dev.to/rdegges/please-stop-using-local-storage-1i04 
>>> <https://dev.to/rdegges/please-stop-using-local-storage-1i04> -
>>>   2. *Store the JWT token in httpOnly cookie -* Browser adds the cookie
>>>   to all the API calls - vulnerable to CSRF attacks and n

Re: [Discussion] Securing single-page-application using JWT token

[Suresh Marru]

Thanks Anuj for your insights. 

Hi Isuru,

It might help draw up an architecture of front end UI components to backend 
security gateway to brainstorm this and arrive at a consensus of handling API 
security for UI’s.

Suresh

> On Jul 7, 2020, at 7:52 PM, anuj bhandar  wrote:
> 
> I agree with your research, Options [1] & [2] are not recommended.
> 
> We use AWS Amplify (https://aws.amazon.com/amplify/ 
> ) to handle that scenario, our tech stack 
> comprises React front-end, AWS Amplify integrated with our company's SSO 
> provider, and an Rails API backend.
> 
> Using a third party like AWS Amplify will not handle all the security 
> concerns, you will have to pay special attention to Token Validity. JWT 
> claims offer a good amount of control if set up properly 
> (https://tools.ietf.org/html/rfc7519 ).
> 
> I am not sure if this was helpful. Happy to help wherever I can.
> 
> Thanks,
> Anuj Bhandar
> Intuit Inc.
> 
> On Tue, Jul 7, 2020 at 3:52 PM Rastogi, Shivam  > wrote:
> Hi Everyone,
> 
> 
> 
> Just wanted to check if anyone has any experience in securing Single Page 
> Webapps using JWT tokens. 
> 
> 
> 
> 
> Challenge
> 
> The challenge we are facing is how to store the JWT token at the client-side 
> in the browser which will be used to securely call the back-end APIs.
> 
> 
> 
> ​Background
> 
> We are working on the custos UI portal and one of the design options that we 
> are considering is to directly call the custos backend APIs from client-side 
> javascript. The basic flow would look like:
> 
> 
> 
> 1. User logins using username and password from the login screen using login 
> API
> 
> 2. JWT token and basic user-information is returned to the client in 
> response. 
> 
> 3. In subsequent API requests, user will send this authentication token. 
> But,how to store this token at client-side?
> 
> 
> 
> ​Options
> 
> Store it​ in localStorage - This is not considered to be secure as 
> localstorage is vulnerable to XSS attacks- ​
> https://dev.to/rdegges/please-stop-using-local-storage-1i04 
>  - 
> Store the JWT token in httpOnly cookie - Browser adds the cookie to all the 
> API calls - vulnerable to CSRF attacks and need to be protected using CSRF 
> token or in some other manner. Do you have any experience implementing these 
> securely?
> Have an authentication middleware(API gateway) - All user requests are routed 
> to API-gateway which authenticates the users using sessionId and then 
> re-routes the requests to the API by including the users JWT token or session 
> information. More of a traditional way of doing things, but some big 
> companies use this approach like Airbnb as per this article - 
> https://medium.com/airbnb-engineering/building-services-at-airbnb-part-2-142be1c5d506
>  
> ​
> 
> 
> It would be really helpful if you can provide any inputs or share your 
> experience with SPAs. 
> 
> 
> 
> Thanks and regards,
> 
> Shivam Rastogi
> 

GSoC Mid-Term Reports

[Suresh Marru]

GSOC Students,

Can you immediately put your proposal, progress and plans on Airavata Wiki - 
https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2020 


It is ok to work on them on google docs, but we cannot grade them and only will 
grade what is on the Airavata Wiki.

Suresh

Archiving Airavata PHP Gateway Repository

[Suresh Marru]

Hi All,

Since all active development is on Airavata Django Portal, any objections to 
change the PGA repo [1] to archiving and read-only[2]?

Cheers,
Suresh

[1] - https://github.com/apache/airavata-php-gateway 

[2] - 
https://help.github.com/en/github/creating-cloning-and-archiving-repositories/archiving-repositories
 

Re: [External] Errors while creating experiments

[Suresh Marru]

Hi Vivek,

Probably you want to to login as your admin account and give permissions to 
your user account (by adding to gateway-users group), these instructions might 
help - 
https://docs.airavata.org/en/master/user-documentation/admin-tutorials/#GtwyAccess
 
<https://docs.airavata.org/en/master/user-documentation/admin-tutorials/#GtwyAccess>

Suresh

> On May 15, 2020, at 4:43 PM, Bandaru, Vivek Shresta  wrote:
> 
> Hi Professor,
>  
> Thank you for the response.
> Yes, I have Airavata and Django portal setup locally.
> ‘Are you also using a local Keycloak?’ I do have a keycloak docker instance 
> running in my local and I’m assuming my code is connecting to that. Please 
> find attached a screenshot of all the docker instances running in my local.
> 
> Regards,
> Vivek.
>  
> From: Suresh Marru mailto:sma...@apache.org>>
> Reply-To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org>" 
> mailto:dev@airavata.apache.org>>
> Date: Friday, May 15, 2020 at 4:19 PM
> To: Airavata Dev mailto:dev@airavata.apache.org>>
> Subject: [External] Re: Errors while creating experiments
>  
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Hi Vivek, 
>  
> Can you describe more about your environment? Did you setup both Airavata and 
> Django Portal locally? Are you also using a local Keycloak? Or are you 
> connecting to a development instance? 
>  
> Suresh
> 
> 
> On May 14, 2020, at 6:45 PM, Bandaru, Vivek Shresta  <mailto:vivb...@iu.edu>> wrote:
>  
> Still looking for help. How do I mitigate this issue? Is this expected from 
> the default-admin user? Am I supposed to create a new user with permissions 
> to either create or use an experiment? If yes, how do I do that?
> Any help is appreciated.
>  
> Regards,
> Vivek.
>  
> From: "Bandaru, Vivek Shresta" mailto:vivb...@iu.edu>>
> Date: Wednesday, May 13, 2020 at 6:43 PM
> To: "dev@airavata.apache.org <mailto:dev@airavata.apache.org>" 
> mailto:dev@airavata.apache.org>>
> Subject: Errors while creating experiments
>  
> Hi All,
>  
> I’m trying to understand the flow of how an experiment is created and stored.
> When I try to create an experiment using the default-admin user, I’m getting 
> AuthorizationExceptions in my Django-portal and I see ‘Failed to fetch’ 
> errors on UI.
> And the UI shows an error when I choose the ONLY computing resource 
> available, ‘js-156-93.jetstream-cloud.org 
> <http://js-156-93.jetstream-cloud.org/>’.
> Please find attached the png of the UI and the logs of Django-portal and 
> Airavata.
>  
> I did try creating a new user, but looks like a new user requires permission 
> from admin to create or access experiments.
>  
> And finally, trying to run the Django Portal on Debug mode in PyCharm just 
> exits returning a Segmentation fault. Stackoverflow answers which support 
> this claim can be found here 
> <https://stackoverflow.com/questions/45016449/pycharm-debugger-instantly-exits-with-139-code>
>  and here 
> <https://stackoverflow.com/questions/40203572/pycharm-debug-segmentation-fault-signal-11>.
> Not sure if this is the right place to ask this question, but do we have a 
> workaround for this?
> 
> Any help is greatly appreciated.
>  
> Regards,
> Vivek.
>  
> 

Re: [External] Airavata database connection issue

[Suresh Marru]

Hi Bhakti,

Looks like your issues are generic port and database. May be you should search 
for issues on stack overflow and places like that. The error message is clear, 
the service externally is unable to connect to the database. So you may want to 
ignore airavata for a while and try to connect to the database container 
(externally) and verify before you switch back to bring up airavata services.

Suresh

> On May 14, 2020, at 11:49 PM, Narvekar, Bhakti Sunil  wrote:
> 
> Hi,
>  
> I’m still not able to resolve this issue. It would be great if someone can 
> help me out.
>  
> Thanks.
>  
> Sincerely,
> Bhakti
>  
> From: Narvekar, Bhakti Sunil 
> Sent: Wednesday, May 13, 2020 4:14 PM
> To: dev@airavata.apache.org 
> Subject: RE: [External] Re: Airavata database connection issue
>  
> Hi Aarushi,
>  
> I tried stopping the local MySQL instance and also I tried uninstalling the 
> instance as well but I’m still facing the same issue. All the containers are 
> up but now I’m not able to access localhost:13306 as well.
>  
> I have attached the StackTrace and logs for MySQL. 
>  
> Thanks,
> Bhakti
>  
> From: Bisht, Aarushi 
> Sent: Wednesday, May 13, 2020 3:45 PM
> To: dev@airavata.apache.org 
> Subject: RE: [External] Re: Airavata database connection issue
>  
> Hi Bhakti,
> 
> Since you have a local mysql instance running that is why you are able to 
> connect to localhost 13306. Can you try to stop the mysql instance running 
> locally and only start the docker containers. If the container has 
> successfully started mysql should be available on localhost:3306. You can see 
> all the running containers using docker-compose ps.
> 
> Thanks,
> Aarushi Bisht
> 
> On May 13, 2020 1:03 PM, "Narvekar, Bhakti Sunil"  wrote:
> Hi Vivek,
>  
> I feel this is a different issue. I have attached the StackTrace as well.
> Please let me know if you think it is similar.
>  
> Thanks,
> Bhakti
>  
> From: Bandaru, Vivek Shresta 
> Sent: Wednesday, May 13, 2020 12:04 PM
> To: dev@airavata.apache.org 
> Subject: Re: [External] Re: Airavata database connection issue
>  
> Hi Bhakti,
>  
> This looks very similar to the database connection error a couple of us faced 
> few weeks ago.
> Without the full stacktrace I can’t say for sure, but you can try the fix 
> mentioned in this mail: 
> https://markmail.org/message/oojdzfmczss2gepr?q=Airavata+database+connection+error+while+starting
>  
> 
>  
> Regards,
> Vivek.
>  
> From: "Narvekar, Bhakti Sunil" 
> Reply-To: "dev@airavata.apache.org" 
> Date: Wednesday, May 13, 2020 at 11:52 AM
> To: "dev@airavata.apache.org" 
> Subject: RE: [External] Re: Airavata database connection issue
>  
> Hi Aarushi,
>  
> The services are up and running. I have attached the screenshots for the 
> same. The localhost:13306 is reachable but localhost:3306 is not.
> I have attached the result of the telnet for both the ports 3306 and 13306.
> Also, I have MySQL server installed on my local machine, would that be an 
> issue.
>  
> Thanks & Regards,
> Bhakti Narvekar
>  
> From: Bisht, Aarushi 
> Sent: Wednesday, May 13, 2020 4:15 AM
> To: dev@airavata.apache.org 
> Subject: RE: [External] Re: Airavata database connection issue
>  
> Hi Bhakti, 
>  
> From the logs it seems you are not able to connect to database on localhost: 
> 13306.
> Make sure db container is up and running. Try using nc or some other command 
> to make sure that you are able to connect to this port.
>  
> - Aarushi Bisht
>  
> On May 13, 2020 12:05 AM, "Narvekar, Bhakti Sunil"  wrote:
> Hi Akhil,
> 
>  
> Yes, I tried deleting the ide-integration/resources/database_data folder, 
> stopping all the services, and restarting them again. But I'm still facing 
> this issue.
> 
>  
> Regards,
> 
> Bhakti
> 
>  
> From: akhil nagulavancha 
> Sent: Tuesday, May 12, 2020 9:43 PM
> To: dev@airavata.apache.org 
> Subject: [External] Re: Airavata database connection issue
> 
>  
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
>  
> Hi Bhakti , 
> 
>  
> Did you try the note mentioned in start airavata server step of cwiki ? It 
> has helped me to solve a similar database related error. 
> 
>  
> Regards 
> 
> Akhil 
> 
>  
>  
>  
> On Tue, May 12, 2020 at 8:41 PM Narvekar, Bhakti Sunil  > wrote:
> 
> Hi,
> 
>  
> On starting APIServerStarter, I ran into an exception - 
> SQLNonTransientConnectionException. I have attached the StackTrace and 
> database logs.
> 
> Could anyone please help me with this issue. 
> 
>  
>  
> Thanks & Regards,
> 
> Bh

Re: GSoC proposal

[Suresh Marru]

> On May 14, 2020, at 10:08 PM, Bandaru, Vivek Shresta  wrote:
> 
> Hi All,
>  
> As part of GSoC, I will be working on the EPIC: 
> https://issues.apache.org/jira/browse/AIRAVATA-3315 
> 
>  
> I have a few questions about the deliverables and it would be really helpful 
> if someone can answer them.
>  
> The last time I had a discussion about the proposal, I was told about the use 
> case where a user can add his/her external storage (S3/Azure/GDrive etc) if 
> they feel the storage quota assigned by the admin isn’t enough.
> I do not see this use case in the EPIC description though. Is this a nice to 
> have or do we have another EPIC to address this use case?

Lets simplify this goal. Let us for a while forget multiple gateways share a 
storage and there is a notion of internal vs external storage. Lets think about 
it in simple terms as a Gateway gets created and it chooses its target storage, 
it could be local to the web server where portal is deployed or it could be on 
a cloud storage (ignoring the fact how portal access the data). The goal of 
this GSoC project should be how multiple users within this single gateways gets 
a default quota (configurable in the portal) and to provide information for 
users to see storage usage and percentage burn rate against the quota. You can 
take inspirations from Google Drive like examples. When you login you will see 
used/total storage, something like that. 

Do not restrict your GSoC to one single EPIC. Normally GSoC proposals get 
developed in the community over 3 months and things are clear. Given these 
happened in a rush now is the time to expand the details and create new epics 
and issues where needed. 

> Add user interfaces to display user-specific quota and current usage. - 
> There's no Mock UI attached to the the epic like this EPIC 
>  does. Am I supposed to 
> follow a similar pattern of the mock UI mentioned in the aforementioned epic?

Not every Airavata new feature has the luxury of having a mock UI. So I suggest 
you directly work through programming API’s and code examples. But if you think 
mock UI’s help frame your thinking you can propose them too. 

> Use Airavata MFT service to fetch current information on users storage to 
> cross validate. – I’m not really sure what this line means. Is this point 
> anyway related to my first point, i.e. a user fetching his current usage 
> stats from his external storage(Box/S3 etc)?

Again do not need to take these too literally and you can step back and look at 
the goal and propose your interpretations of the larger problem. I was just 
trying to jump ahead and think on how the storage usage can be calculated. One 
way is to keep track of the files going into storage and record increment the 
sum in a database. Otherway is to fetch the used storage space by queuing a 
service like MFT. The cross-validation above is a mix of both. You can 
pro-actively track and validate with actual used storage. I think I was 
over-complicating it here, so you can propose a simpler solution for a user 
knowing their usage. 

>  
> Any pointers are appreciated.
>  
> Regards,
> Vivek.

Re: Errors while creating experiments

[Suresh Marru]

Hi Vivek,

Can you describe more about your environment? Did you setup both Airavata and 
Django Portal locally? Are you also using a local Keycloak? Or are you 
connecting to a development instance? 

Suresh

> On May 14, 2020, at 6:45 PM, Bandaru, Vivek Shresta  wrote:
> 
> Still looking for help. How do I mitigate this issue? Is this expected from 
> the default-admin user? Am I supposed to create a new user with permissions 
> to either create or use an experiment? If yes, how do I do that?
> Any help is appreciated.
>  
> Regards,
> Vivek.
>  
> From: "Bandaru, Vivek Shresta" 
> Date: Wednesday, May 13, 2020 at 6:43 PM
> To: "dev@airavata.apache.org" 
> Subject: Errors while creating experiments
>  
> Hi All,
>  
> I’m trying to understand the flow of how an experiment is created and stored.
> When I try to create an experiment using the default-admin user, I’m getting 
> AuthorizationExceptions in my Django-portal and I see ‘Failed to fetch’ 
> errors on UI.
> And the UI shows an error when I choose the ONLY computing resource 
> available, ‘js-156-93.jetstream-cloud.org’.
> Please find attached the png of the UI and the logs of Django-portal and 
> Airavata.
>  
> I did try creating a new user, but looks like a new user requires permission 
> from admin to create or access experiments.
>  
> And finally, trying to run the Django Portal on Debug mode in PyCharm just 
> exits returning a Segmentation fault. Stackoverflow answers which support 
> this claim can be found here 
> 
>  and here 
> .
> Not sure if this is the right place to ask this question, but do we have a 
> workaround for this?
> 
> Any help is greatly appreciated.
>  
> Regards,
> Vivek.

Re: Welcome 2020 GSoC Students

[Suresh Marru]

Hi All,

Hope you are progressing in working on updated proposals (due this Friday). 
Please read this workflow to follow test driven development and a suggested 
workflow - https://google.github.io/gsocguides/mentor/workflow 
<https://google.github.io/gsocguides/mentor/workflow> especially pay attention 
to the anti-pattern at the end of the document.

Suresh

> On May 4, 2020, at 2:25 PM, Suresh Marru  wrote:
> 
> Hi Vivek, Nick, Akhil, Nayeem,
> 
> Congratulations on your GSoC acceptance!! And welcome to Apache Airavata. 
> 
> I am sure you have pointers to GSoC guides in your acceptance emails, in 
> addition to student guide (https://google.github.io/gsocguides/student/ 
> <https://google.github.io/gsocguides/student/>), review the mentor guide as 
> well (https://google.github.io/gsocguides/mentor/community-basics 
> <https://google.github.io/gsocguides/mentor/community-basics>). Do follow the 
> readings on “How to ask good questions”. 
> 
> A key GSoC aspect to remember: community engagement.  Above and beyond summer 
> coding which you are anyway expected to contribute significantly, more 
> importantly you should be very active in mailing lists and ask and answer 
> questions. All of you have scored below par on mailing list communications 
> and hoping you will fix this during the community bonding period and beyond. 
> 
> Your first mid-term evaluation is June 29th, so few logistic things will be 
> required from all of you (try to get these by May 15th):
> 
> * Post your updated/modified proposal to Airavata Wiki - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2020 
> <https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2020> include the 
> following as a minimum:
> * Create a detailed and clear description of the problem you will tackle 
> between now and June 29th. Yes, lets break this into three phase. Second one 
> due July 27th and final one due August 24th. It is totally fine if your 
> problem changes from your original proposal. Lets document the updated goal. 
> * Crete a good description on how your solution to the above problem will 
> look like.
> * Also discuss of any alternative solutions and featured you considered.
> * Describe how to evaluate/test your solution.
> 
> It is expected that your understanding (as well as the communities 
> understanding) of the problem and feasible solutions will evolve over emmer. 
> Thats fine, but keep this document updated as you go.
> 
> Lastly, write weekly sprint updates as child pages to the wiki and post them 
> to this list as well. Here is a good example project to follow:
> 
> Proposal - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGSoC+Proposal%5D+Securing+Airavata+API
>  
> <https://cwiki.apache.org/confluence/display/AIRAVATA/[GSoC+Proposal]+Securing+Airavata+API>
> Design document (submitted in community bonding period, notice the detailed 
> updates since proposal) - 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851790 
> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851790>
> Weekly sprints - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+1 
> <https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+1> and 
> https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+2 
> <https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+2>
> Mid-term evaluation (your June 29th evaluation will be based on such a 
> report) - 
> https://cwiki.apache.org/confluence/display/AIRAVATA/Update+for+the+mid+evaluation
>  
> <https://cwiki.apache.org/confluence/display/AIRAVATA/Update+for+the+mid+evaluation>
> 
> Welcome on board and happy coding,
> 
> Cheers,
> Suresh

Re: Issue while doing Airavata setup

[Suresh Marru]

Hi Bhakti,

Can you search in mailing list archives for others who faced similar issues? 
For instance this thread - https://markmail.org/message/bcivwy6y4d4omf5w 


Suresh

> On May 11, 2020, at 11:20 PM, Narvekar, Bhakti Sunil  wrote:
> 
> Hi All,
>  
> I’m trying to do the setup for Airavata and I’m facing an issue for quite 
> sometime now. I’m comparatively new to this and I’m not sure where I’m making 
> mistakes. I have attached the error log.
>  
> Here are the steps that I performed:
> Clone Airavata repository
> Checkout to the develop branch
> Build the develop branch using Maven -> BUILD SUCCESS
> Add entry into hosts file.
> Go to directory src/main/resources and run docker-compose up -d
> Go to org.apache.airavata.ide.integration.APIServerStarter and run the 
> APIServerStarter -> Error in RabbitMQ
>  
> Could anyone please help me with this issue. Thanking you in advance.
>  
>  
> Regards,
> Bhakti Narvekar
> 

Welcome 2020 GSoC Students

[Suresh Marru]

Hi Vivek, Nick, Akhil, Nayeem,

Congratulations on your GSoC acceptance!! And welcome to Apache Airavata. 

I am sure you have pointers to GSoC guides in your acceptance emails, in 
addition to student guide (https://google.github.io/gsocguides/student/ 
), review the mentor guide as 
well (https://google.github.io/gsocguides/mentor/community-basics 
). Do follow the 
readings on “How to ask good questions”. 

A key GSoC aspect to remember: community engagement.  Above and beyond summer 
coding which you are anyway expected to contribute significantly, more 
importantly you should be very active in mailing lists and ask and answer 
questions. All of you have scored below par on mailing list communications and 
hoping you will fix this during the community bonding period and beyond. 

Your first mid-term evaluation is June 29th, so few logistic things will be 
required from all of you (try to get these by May 15th):

* Post your updated/modified proposal to Airavata Wiki - 
https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2020 
 include the 
following as a minimum:
* Create a detailed and clear description of the problem you will tackle 
between now and June 29th. Yes, lets break this into three phase. Second one 
due July 27th and final one due August 24th. It is totally fine if your problem 
changes from your original proposal. Lets document the updated goal. 
* Crete a good description on how your solution to the above problem will look 
like.
* Also discuss of any alternative solutions and featured you considered.
* Describe how to evaluate/test your solution.

It is expected that your understanding (as well as the communities 
understanding) of the problem and feasible solutions will evolve over emmer. 
Thats fine, but keep this document updated as you go.

Lastly, write weekly sprint updates as child pages to the wiki and post them to 
this list as well. Here is a good example project to follow:

Proposal - 
https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGSoC+Proposal%5D+Securing+Airavata+API
 

Design document (submitted in community bonding period, notice the detailed 
updates since proposal) - 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851790 

Weekly sprints - https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+1 
 and 
https://cwiki.apache.org/confluence/display/AIRAVATA/Sprint+2 

Mid-term evaluation (your June 29th evaluation will be based on such a report) 
- 
https://cwiki.apache.org/confluence/display/AIRAVATA/Update+for+the+mid+evaluation
 


Welcome on board and happy coding,

Cheers,
Suresh

Re: Python SDK for MFT

[Suresh Marru]

On Apr 30, 2020, at 10:26 PM, Isuru Ranawaka  wrote:
> 
> Hi Rishabh,
> 
> Some suggestions, try out a maven plugin (e.g protbuf-maven-plugin) to
> generate stubs and copy them into relevant folders or write a script (e.g
> shell script) and execute via maven phase.

Here is an example shell script we use for generation of Thrift SDK’s. You 
could follow this example and write equivalent protobuff one and link it maven 
phase as Isuru suggested:

https://github.com/apache/airavata/blob/master/thrift-interface-descriptions/generate-thrift-stubs.sh
 


Suresh

> 
> thanks
> Isuru
> 
> 
> 
> 
> On Thu, Apr 30, 2020 at 10:03 PM Rishabh Gajra 
> wrote:
> 
>> Hi,
>> This is related to custos python client that is present in the develop
>> branch. I believe all the classes that are present in the module are
>> generated from the proto buff. I wanted to address the issue where there is
>> a change in the protobuff file on the server-side. Is there any method
>> where we can avoid manually putting the class file in every client that we
>> build on. Also, our team is planning to write the Python SDK for Airavata
>> MFT (Possibly extend to GOLANG too if time permits).
>> Specifically, I want to address the above issue. Any suggestion on how to
>> go about this is appreciated. Thank you.
>> 
>> 
>> --
>> Regards,
>> Rishabh.
>> 
> 
> 
> -- 
> Research Software Engineer
> Indiana University, IN

Peer Review of MFT PR's

[Suresh Marru]

Contributions to MFT related issues:

Firstly thank you for these fine contributions. Please note that you are 
contributing to a code base which has a quick path to production, so we need to 
ensure quality over quantity. Seems like there are contributions but are 
waiting for Dimuthu’s review. Looking at the PR’s, most of the reviews seems to 
be for basic code quality and best practices as opposed to deeper architectural 
concerns. Instead of waiting for Dimuthu, may be you guys can help each other 
out and record these reviews as a contribution part of your projects?

Here are PR’s which need review, please volunteer:

https://github.com/apache/airavata-mft/pull/10 

https://github.com/apache/airavata-mft/pull/11 

https://github.com/apache/airavata-mft/pull/12 


In addition, please consider contributing the following:

* Documentation on how to configure various transports. Because of the variance 
of the cloud storages and multiple ways to work with them, it is important to 
capture which approach is being used in the contributed implementation.
* Examples on using each of these transports as a stand alone as well as 
examples covering multiple transports.

Lastly, please do not restrict yourself to the transport you contributed, do 
try examples with other contributions, thats a good way to review and critique 
as well.

Thanks for all your efforts, MFT is shaping nicely and now all of you should be 
proud of being part of its developer community,
Suresh

Re: Airavata MFT- Gdrive transport implementation

[Suresh Marru]

A big kudos for all the work on transports, these are nicely evolving. 

We do not need to drive these with IU decisions, but good point these will lead 
to potential use cases. On that note, looks like Google Drive will be used for 
non-sensitive data and one drive for PHI data. So an eventual plan for MFT to 
support both will be nice. I added it to the spreadsheet. 

Suresh

> On Apr 24, 2020, at 12:45 PM, DImuthu Upeksha  
> wrote:
> 
> Because of the IUs decision of migration of Box data to Google Drive, I 
> expect the use cases will come into MFT to handle drive transports. So from 
> the usability perspective, I'd prefer gdrive over one drive. But it's up to 
> you to decide. Once you have finalized on one, please update it in [1]
> 
> [1] 
> https://docs.google.com/spreadsheets/d/1M7-Reda-pCi1l-TSSstI6Yi1pSbtINUqlBFcy5UrOW0/edit#gid=0
>  
> 
>  
> 
> On Fri, Apr 24, 2020 at 11:43 AM Jejurkar, Shivali  > wrote:
> Hello Dimuthu,
> 
> I have set up MFT locally and was able to transfer file from S3 to S3 and S3 
> to local. I am interested in implementing Gdrive transfers. Would MFT also 
> extend support for other cloud storage platforms like OneDrive? Can you share 
> your feedback or any challenges that you see if my team pick Gdrive and 
> Onedrive up?​
> 
> 
> Thanks,
> Shivali
> 
> 

Re: How to generate the structured js:es6 client stub for airavata api

[Suresh Marru]

Hi Dinuka,

We have not successfully used Thrift generated JS previously (its possible but 
do not have that experience within Airavata). Django portal uses the python 
generated code and exposes them as REST API’s using DRF 
(https://www.django-rest-framework.org/ 
). The Vue.js UI components communicate 
to these REST API’s. I wonder if you can have electronJS talk to the same API’s 
instead of directly to Airavata API.

Also, we would like to move from Thrift to Protobuf and gRPC. I wonder if REST 
support can be more seamless once the migration is done. 

Suresh

> On Apr 5, 2020, at 4:17 PM, Dinuka Desilva  wrote:
> 
> Hi,
> 
> I'm trying to generate the es6 client stub for airavata api using the 
> following script.
> 
> thrift -r --gen js:es6 
> ../../airavata/thrift-interface-descriptions/airavata-apis/airavata_api.thrift
>  
> 
> But, I'm not getting it correctly I guess. I'm getting a list of files in a 
> folder called gen-js. Instead what I need is a structured code as there in 
> the airavata-django-portal. 
> 
> I'm also not sure whether what's on the portal is a generated code. Please 
> advise.
> 
> 
> 
> Can you help me?
> 
> Regards,
> Dinuka

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[Suresh Marru]

Hi Isuru,

I wonder if it is related to renew tokens? Any insights?

Suresh

> On Apr 3, 2020, at 6:48 AM, Dinuka Desilva  wrote:
> 
> Hi,
> 
> I got the airava and airavata-django-portal running on localhost by executing 
> following script on 4 terminals one after the other. 
> docker-compose down && rm -rf database_data/ && docker-compose up
> mvn exec:java 
> -Dexec.mainClass="org.apache.airavata.ide.integration.APIServerStarter"
> . ./build_js.sh
> source ./venv/bin/activate && rm -rf ./tmp/ && rm -rf db.sqlite3 && python 
> manage.py migrate && python manage.py load_default_gateway && python 
> manage.py runserver
> 
> With this I could access the localhost server at http://localhost:8000 
> . But, only for less than half an hour it worked and 
> then it's ended up with the following error. It has the logged in user though 
> an error occurred. 
> 
> 
> 
> Neither restarting the servers nor clearing cache didn't solve this issue. 
> And I had to execute above four script again which recreates the database. 
> For kind of each 20-30 minutes, I had to do this. I'm doubtful whether I'm 
> doing anything wrong. 
> 
> Also, below I have attached the settings_local.py.
> 
> Thanka & Regards,
> Dinuka
> 

Re: [External] Apache Airavata MFT - Dashboard

[Suresh Marru]

Hi Sharanya,

For aligning with lest of Airavata UI architectiure, we would like to stick to 
Django (as a UI backend framework and bridging to MFT API’s) and on use VueJS 
and bootstrap for the frontends.

Suresh

> On Apr 4, 2020, at 10:30 PM, Ravichandran, Sharanya  wrote:
> 
> Hello Professor , 
> 
> Could you please let us know if it would be ok if we implemented the 
> dashboard using React.js instead of Django ? 
> 
> Thanks ,
>  Sharanya R.
> 
> From: Suresh Marru 
> Sent: Saturday, April 4, 2020 9:01 PM
> To: Airavata Dev
> Subject: [External] Re: Apache Airavata MFT - Dashboard
>  
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Hi Sharanya,
> 
> This JIRA has a straw design for some components - 
> https://issues.apache.org/jira/browse/AIRAVATA-3314 
> <https://issues.apache.org/jira/browse/AIRAVATA-3314>
> 
> Do not take these designs too literally, as you understand the goals and 
> motivations for MFT, please feel free to put your own thoughts in proposing a 
> design. 
> 
> Suresh
> 
>> On Apr 4, 2020, at 7:23 PM, Ravichandran, Sharanya > <mailto:shar...@iu.edu>> wrote:
>> 
>> Hi,
>> 
>> We were looking to extend the airavata mft project to be a standalone 
>> application by setting up a dashboard and an api gateway to interact with 
>> the mft system. Could we get the existing design of the proposed dashboard 
>> so that we could get an idea about what it is supposed to look like ?
>> 
>> Thanks,
>> Sharanya R.
> 

Re: Apache Airavata MFT - AWS/GCS support

[Suresh Marru]

Hi Aravind,

I have to catch up with the code, but you may want to look at the S3 
implementation and extend it to Azure, GCP or other cloud services like Box, 
Dropbox and so on. 

There could be many use cases, here is an idea:

* Compute a job on a supercomputer with SCP access and push the outputs to a 
Cloud storage. 

Suresh

> On Apr 4, 2020, at 8:09 PM, Aravind Ramalingam  wrote:
> 
> Hello,
> 
> We set up the MFT project on local system and tested out SCP transfer between 
> JetStream VMs, we were wondering how the support can be extended for AWS/GCS.
> 
> As per our understanding, the current implementation has support for two 
> protocols i.e. local-transport and scp-transport. Would we have to modify/add 
> to the code base to extend support for AWS/GCS clients? 
> 
> Could you please provide suggestions for this use case. 
> 
> Thank you
> Aravind Ramalingam

Re: Question regarding Resource and Secret service Backend in Production

[Suresh Marru]

Hi Akshay,

For production deployments I suggest to integrate with Airavata Custos services 
- https://github.com/apache/airavata-custos/tree/develop 


Please ask questions as you explore these. 

Suresh

> On Apr 4, 2020, at 7:29 PM, Rajvanshi, Akshay  wrote:
> 
> Hello,
>  
> Currently we did the local setup for the airavata-mft and copied a file 
> within our VM, but currently we have an implementation for resource and 
> service backend with file based backend. How do we go about implementing at 
> the production level ? Are we expected to add the support for the service we 
> plan to integrate with separately?
>  
> Thanks

Re: Apache Airavata MFT - Dashboard

[Suresh Marru]

Hi Sharanya,

This JIRA has a straw design for some components - 
https://issues.apache.org/jira/browse/AIRAVATA-3314 


Do not take these designs too literally, as you understand the goals and 
motivations for MFT, please feel free to put your own thoughts in proposing a 
design. 

Suresh

> On Apr 4, 2020, at 7:23 PM, Ravichandran, Sharanya  wrote:
> 
> Hi,
> 
> We were looking to extend the airavata mft project to be a standalone 
> application by setting up a dashboard and an api gateway to interact with the 
> mft system. Could we get the existing design of the proposed dashboard so 
> that we could get an idea about what it is supposed to look like ?
> 
> Thanks,
> Sharanya R.

Re: Backend calls for

[Suresh Marru]

Hi Vivek,

You seem to be looking at Airavata client SDK’s. If you want to follow server 
side create experiment flow you should look into the API handler code for 
create experiment - 
https://github.com/apache/airavata/blob/master/airavata-api/airavata-api-server/src/main/java/org/apache/airavata/api/server/handler/AiravataServerHandler.java#L1400
 


But you quickly realize static code analysis will not take you too far since 
the communications happen between these components via messaging layer.

If you are looking for how experiment folder is created here is the flow:

The client (for instance Django Portal) create a unique working directory for 
the authenticated user and stages user provided input file into this folder. 
Then it sets this as the experiment Data Directory and passes the location to 
the server - 
https://github.com/apache/airavata/blob/master/thrift-interface-descriptions/data-models/experiment-catalog-models/experiment_model.thrift#L69
 


The serve code after the application execution is complete stages the output 
data to the folder specified above. All the required security credentials and 
communication protocols to the above storage resource should be pre-configired. 
If you are looking for the class where this transfer happens, this might help - 
https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/staging/OutputDataStagingTask.java
 


Suresh

> On Mar 30, 2020, at 10:45 PM, Bandaru, Vivek Shresta  wrote:
> 
> Hi All,
>  
> I’m trying to understand how an experiment and its respective folder is 
> created by basic static code analysis.
> I don’t think I get the whole picture.
> 
> I believe in workspace/views.py, through apache thrift, Django-portal 
> delegates the creation of experiment to backend services.
>  
> Then, I believe the code for creating an experiement is in Java: 
> org.apache.airavata.client.samples.CreateLaunchExperiment#createExperimentGROMACSStampede
>  (Multiple create experiment functions in this class)
>  
> What I don’t understand is, I find createExperiments functions in other 
> languages which make a thrift call to create an experiment.
> Php :  
> pga/airavata-php-gateway/app/libraries/Airavata/API/Airavata.php:6274
> Cpp :  lib/airavata/Airavata.h:7844 (Multiple create experiments)
> 
> When does the Php and Cpp code bases come into picture? Any pointers are very 
> much appreciated.
> 
> Regards,
> Vivek.

[GSoC] Draft proposals and deadlines

[Suresh Marru]

Hi All,

Note that GSoC proposals are due tomorrow at 2pm US Eastern time. Airavata 
community (this dev list) needs to look at your draft before you submit, so do 
not wait for long and within next 6 to 8 hours we would like to see a draft 
version. 

GSoC application will require other aspect like validating your age, school 
enrollment and so forth. So do not wait until tomorrow to get started. Right 
away get online to the GSOC portal and finish up other logistics and then open 
up the google document draft, make it public and post on this list while you 
work on it.

Suresh

Re: Storage module

[Suresh Marru]

Hi Vivek,

Please see below 

> On Mar 29, 2020, at 12:22 PM, Bandaru, Vivek Shresta  wrote:
> 
> Hi All,
> 
> I would like to know more about the following modules: 
> 1. Aggregation of the data and analytics of all the tenants in a unified 
> portal.

This JIRA has details of the project and its break down - 
https://issues.apache.org/jira/browse/AIRAVATA-3200 

> 2. Setting a memory cap for every individual or a tenant in the S3 bucket. I 
> believe this involves delineating all the memory used by the individual.

A background of MFT will help you understand some goals - 
https://docs.google.com/document/d/1zrO4Z1dn7ENhm1RBdVCw-dDpWiebaZEWy66ceTWoOlo/edit?usp=sharing
 


A related paper from last years GSOC student might help - 
https://docs.google.com/document/d/1gemKVec_6Ou_c6QKysHTKQzd1RJlJE3w_60A4I0pQjI/edit?usp=sharing
 


Suresh

>  
> What work is going to be involved in developing these modules?
> 
> Regards,
> Vivek.

Re: [GSoC] Call to brainstorm Airavata project ideas

[Suresh Marru]

Hi All,

As promised we will have the call shortly (at 11 am Eastern time) on Zoom - 
https://iu.zoom.us/j/938427151

During the call after a brief introduction, we will brainstorm each of your 
interests and expertise and finalize project ideas. Follow this wiki link for 
application template and project ideas (to be populated after the call) - 
https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2020 
<https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2020>

Suresh

> On Mar 28, 2020, at 2:36 PM, Suresh Marru  wrote:
> 
> GSoC Aspirants,
> 
> Since there is a surge of last minute interest, lets have a video call to 
> walk through the project ideas and proposal template. Lets do it tomorrow 
> (Sunday March 29th) at 11 am US Eastern Time. If any one is planing to join 
> the call and this time does not work, let me know and we can choose a 
> different time. I will send a Zoom url to join the call tomorrow morning. 
> 
> Cheers,
> Suresh

[GSoC] Call to brainstorm Airavata project ideas

[Suresh Marru]

GSoC Aspirants,

Since there is a surge of last minute interest, lets have a video call to walk 
through the project ideas and proposal template. Lets do it tomorrow (Sunday 
March 29th) at 11 am US Eastern Time. If any one is planing to join the call 
and this time does not work, let me know and we can choose a different time. I 
will send a Zoom url to join the call tomorrow morning. 

Cheers,
Suresh