Re: Django Portal README and Auth App Webpack

[Christie, Marcus Aaron]

Thanks Jackson, I just merged the PR. I think this is a great contribution. 
We're always looking for Windows developers to help make sure the instructions 
work well on Windows!

> On Aug 11, 2023, at 11:33 AM, Jackson Crantford 
>  wrote:
> 
> Hi everyone!
> 
> I am working on issue 3699 Vue3 migration. I noticed a couple things while 
> getting to know the source code and running it locally.
> I created a pull request with the minor changes.
> 
> Please let me know if this is helpful. This is my first ever contributing 
> pull request. Any feedback is welcome and appreciated!
> 
> Thanks,
> 
> Jackson



smime.p7s
Description: S/MIME cryptographic signature

Re: questions regarding dashboards to get quick statistics

[Christie, Marcus Aaron]

Hi Saurav,

This is a little tricky to answer. I believe what you would need to do is look 
at JOB_STATUS table for a given experiment/process. There will be a SUBMITTED 
status, an ACTIVE status and then either a COMPLETED or FAILED or CANCELED 
status.

You would want to get the time different between the ACTIVE and the 
COMPLETED/FAILED/CANCELED status. This would give the elapsed time of the job.

Then you would need to multiple this time, in hours, with the number of cpus 
used in the job (USER_CONFIGURATION_DATA.TOTAL_CPU_COUNT).

Someone on the list might have a better idea, but I think that's the general 
gist of how you would get cpu hours used.

Thanks,

Marcus

> On Aug 13, 2023, at 1:52 PM, saurav kumar jha  
> wrote:
> 
> Hi Community,
> I have a question regarding this jira.
> 
> 10. The total cpu-hours used 1. by a user, 2. sorted, 3. quarterly, 4. 
> plotted over a period of time
> 11. The total cpu-hours consumed by application, sorted, quarterly, plotted 
> over a period of time
> 
> These are the two requirements with respect to cpu hours. 
> 
> Although the COMPUTE_RESOURCE_SCHEDULING table has some of the information, I 
> couldn't find any table which stores cpu hours used information.
> 
> Can anyone please tell me where I can get this info? 
> 
> Thanks & Regards,
> Saurav Kumar Jha



smime.p7s
Description: S/MIME cryptographic signature

Re: Dashboard additions pull request

[Christie, Marcus Aaron]

Hi Saurav,

I would recommend that you stick to the current design, given the GSOC 
timeline. We can add accordions or tabs later.

However, please refactor to use a common date selector. I think that will be a 
good change.

Thanks,

Marcus

> On Aug 13, 2023, at 1:53 PM, saurav kumar jha  
> wrote:
> 
> Hi Marcus,
> Thanks a lot for your review and comments.
> 
> I agree that the page will become longer gradually and I think accordions 
> would be intuitive for users. Should I go ahead and put each of the 
> statistics in accordions?
> 
> I agree with you about having a common date selector for all stats, I will 
> make changes in my pull request accordingly.
> 
> No, I didn't think much about UX design till now. I followed the current 
> design of the portal.
> 
> Thanks & Regards,
> Saurav Kumar Jha
> 
> On Fri, Aug 11, 2023 at 3:57 AM Christie, Marcus Aaron  
> wrote:
> Hi Saurav,
> 
> I've reviewed your pull request and it looks good to me.
> 
> One concern I have is with the UI layout. I think with this many forms it 
> will make the statistics page rather long. I wonder if maybe a tab-based 
> layout or an accordion style layout might work better?
> 
> Another concern is that the date filtering is repeated for each of these 
> statistics views. I wonder, why not just have them all utilize the main date 
> range?
> 
> Are you working from a UX design?
> 
> Thanks,
> 
> Marcus
> 
> > On Aug 3, 2023, at 12:44 PM, saurav kumar jha  
> > wrote:
> > 
> > Hi Community,
> > 
> > I am updating my commits for AIRAVATA-3594 on django-portal in this pull 
> > request .
> > Any Comments or Recommendations will be very helpful.
> > 
> > Thanks & Regards,
> > Saurav Kumar Jha
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Dashboard additions pull request

[Christie, Marcus Aaron]

Hi Saurav,

I've reviewed your pull request and it looks good to me.

One concern I have is with the UI layout. I think with this many forms it will 
make the statistics page rather long. I wonder if maybe a tab-based layout or 
an accordion style layout might work better?

Another concern is that the date filtering is repeated for each of these 
statistics views. I wonder, why not just have them all utilize the main date 
range?

Are you working from a UX design?

Thanks,

Marcus

> On Aug 3, 2023, at 12:44 PM, saurav kumar jha  
> wrote:
> 
> Hi Community,
> 
> I am updating my commits for AIRAVATA-3594 on django-portal in this pull 
> request .
> Any Comments or Recommendations will be very helpful.
> 
> Thanks & Regards,
> Saurav Kumar Jha



smime.p7s
Description: S/MIME cryptographic signature

Re: airavata api endpoint handler/controller implementation

[Christie, Marcus Aaron]

Hi Saurav, 

The REST API is mapped to the Airavata API in 
django_airavata/apps/api/views.py. For the example you mentioned, you can see 
in the following link the implementation of GET/POST/PUT/DELETE /api/groups: 

https://github.com/apache/airavata-django-portal/blob/master/django_airavata/apps/api/views.py#L79-L141
 

 


From: saurav kumar jha 
Date: Wednesday, July 12, 2023 at 1:48 PM
To: dev@airavata.apache.org 
Subject: Re: airavata api endpoint handler/controller implementation 

Hi Suresh, 

Thanks for the response, it was very helpful. 


Now, I am able to understand interface definitions, handler function 
implementations etc., but couldn't understand how an endpoint (for example 
/api/groups) is mapped to a particular function (for example getGatewayGroups) 
? 

Thanks & Regards, 


Saurav Kumar Jha 




On Mon, Jul 10, 2023 at 11:15 PM Suresh Marru mailto:sma...@apache.org>> wrote: 

Hi Saurav, 


You have to look into Thrift API descriptions 
https://github.com/apache/airavata/tree/develop/thrift-interface-descriptions 
<_blank> - follow the handlers from - 
https://github.com/apache/airavata/blob/develop/airavata-api/airavata-api-server/src/main/java/org/apache/airavata/api/server/handler/AiravataServerHandler.java
 <_blank> 



If you haven’t done it, I encourage you to build a simple Apache Thrift example 
with Java server and Python clients so it will give you the right context to 
follow the code. 



Suresh 



On Jul 10, 2023, at 1:19 PM, saurav kumar jha > wrote: 


Hi Community, 


For an api request made by airavata-django-portal , for example on /api/groups 
( https://testdrive.airavata.org <_blank>/api/groups/?limit=10=10 ) , 
where can I find corresponding controller or handler function in airavata code 
<_blank> where it is being exposed as an endpoint or handled?

I need this because for "User groups created within a given period", I want to 
load the creation date of groups along with other details using an api call.
In one of the implementations, the date is not being fetched, so I wanted to 
look at the implementation. 

Thanks & Regards, 

Saurav Kumar Jha 















smime.p7s
Description: S/MIME cryptographic signature

Re: Error while running Airavata locally

[Christie, Marcus Aaron]

Hi Saurav, 

I suspect that the database didn’t get initialized correctly, but then you said 
you only get this for experiment statistics api calls. 

The error is a little misleading, experiment_catalog.EXPERIMENT_SUMMARY is a 
view, not an actual table. 

You can try creating it manually with this script: 
https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/resources/expcatalog-mysql.sql#L232-L238
 
<https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/resources/expcatalog-mysql.sql#L232-L238>
 

Thanks, 

Marcus 

From: saurav kumar jha 
Date: Thursday, July 13, 2023 at 2:26 AM
To: Christie, Marcus Aaron 
Cc: dev@airavata.apache.org 
Subject: Error while running Airavata locally 

Hi Marcus, 


I am getting this error while running airavata locally when an api call for 
experiment statistics is made ,

Table 'experiment_catalog.EXPERIMENT_SUMMARY' doesn't exist

Here is the full stack trace, 



124 [TThreadPoolServer WorkerProcess-0] ERROR 
org.apache.airavata.registry.core.repositories.AbstractRepository {} - Failed 
to execute transaction 

org.apache.openjpa.persistence.ArgumentException: Failed to execute query 
"SELECT count(ES.experimentId) FROM ExperimentSummaryEntity ES WHERE 
ES.gatewayId = :gatewayId AND ES.creationTime BETWEEN :fromDate AND :toDate". 
Check the query syntax for correctness. See nested exception for details. 

at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:921) 
~[openjpa-3.1.0.jar:3.1.0] 

at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:843) 
~[openjpa-3.1.0.jar:3.1.0] 

at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:601) 
~[openjpa-3.1.0.jar:3.1.0] 

at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:297) 
~[openjpa-3.1.0.jar:3.1.0] 

at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:314) 
~[openjpa-3.1.0.jar:3.1.0] 

at org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:343) 
~[openjpa-3.1.0.jar:3.1.0] 

at 
org.apache.airavata.registry.core.repositories.expcatalog.ExperimentSummaryRepository.lambda$getExperimentStatisticsCountForState$0(ExperimentSummaryRepository.java:362)
 ~[classes/:?] 

at 
org.apache.airavata.registry.core.repositories.AbstractRepository.execute(AbstractRepository.java:141)
 ~[classes/:?] 

at 
org.apache.airavata.registry.core.repositories.expcatalog.ExperimentSummaryRepository.getExperimentStatisticsCountForState(ExperimentSummaryRepository.java:356)
 ~[classes/:?] 

at 
org.apache.airavata.registry.core.repositories.expcatalog.ExperimentSummaryRepository.getAccessibleExperimentStatistics(ExperimentSummaryRepository.java:280)
 ~[classes/:?] 

at 
org.apache.airavata.registry.api.service.handler.RegistryServerHandler.getExperimentStatistics(RegistryServerHandler.java:433)
 ~[classes/:?] 

at 
org.apache.airavata.registry.api.RegistryService$Processor$getExperimentStatistics.getResult(RegistryService.java:17739)
 ~[classes/:?] 

at 
org.apache.airavata.registry.api.RegistryService$Processor$getExperimentStatistics.getResult(RegistryService.java:1)
 ~[classes/:?] 

at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40) 
~[libthrift-0.18.1.jar:0.18.1] 

at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) 
~[libthrift-0.18.1.jar:0.18.1] 

at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:257)
 ~[libthrift-0.18.1.jar:0.18.1] 

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
~[?:?] 

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
~[?:?] 

at java.lang.Thread.run(Thread.java:829) ~[?:?] 

Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: (conn:268) Table 
'experiment_catalog.EXPERIMENT_SUMMARY' doesn't exist {prepstmnt 900574728 




Can you please help me in getting it resolved?

Thanks & Regards, 

Saurav Kumar Jha 









smime.p7s
Description: S/MIME cryptographic signature

Re: Architecture for Cybershuttle Orchestration App

[Christie, Marcus Aaron]

Hi Praneeth,

Thanks for putting this together. This looks really good to me.

Thanks,

Marcus

> On Jun 26, 2023, at 7:39 PM, Praneeth Kumar Chityala 
>  wrote:
> 
> You don't often get email from praneethchityal...@gmail.com. Learn why this 
> is important
> Dear Marcus and All,
> 
> Thank you for the valuable feedback. I did incorporate the services as 
> suggested.
> 
> Below is the design choices I finally picked and implemented a couple of 
> services:
>   • For the desktop app I went ahead with the electronJS+Vue3 setup. As 
> electronJS is more widely adopted and easier to migrate electronJS apps to 
> web apps, electronJS is selected over Tauri.
>   • For the client side communication used grpc-web, which has a new 
> implementation of gRPC which can be used on the browser side as well.
>   • Electronjs app code can be accessed with 
> https://github.com/cyber-shuttle/cybershuttle-agent/tree/electron/cybershuttle-app
>   • @Marcus As suggested by you used provide-inject with the 
> userService pointing to UserServiceGrpc, which can updated to other services 
> as needed
>   • UserServiceGrpc service: 
> https://github.com/cyber-shuttle/cybershuttle-agent/blob/electron/cybershuttle-app/src/api/grpc/auth.js
>   • provide grpc service in main: 
> https://github.com/cyber-shuttle/cybershuttle-agent/blob/electron/cybershuttle-app/src/main.js#L13
>   • injection grpc service in a view: 
> https://github.com/cyber-shuttle/cybershuttle-agent/blob/electron/cybershuttle-app/src/components/Signup.vue#L97C1-L97C71
>   • For now I went ahead and built the total app with JavaScript 
> as official grpc-web has good support for it.
>   • @Marcus please let me know if this implementation looks good 
> to start with.
>   • Server side is built with Springboot and java (server: 
> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/app-server . 
> It for not reads clients request and respond with static data)
>   • As grpc-web sends request with http1 and gRPC server works on http2, 
> used an envoy proxy server to mediate the communication between client and 
> server (envoy proxy - 
> https://github.com/cyber-shuttle/cybershuttle-server/blob/main/envoy.yaml)
> 
> I will be working on building the Appview where users will be able to see 
> available applications/projects to launch from the desktop app.
> 
> Best,
> Praneeth
> 
> 
> On Fri, Jun 23, 2023 at 9:27 AM Christie, Marcus Aaron  
> wrote:
> Hi Praneeth,
> 
> This looks good. I've been thinking about how we can reuse UI components that 
> are developed for this local app in a web browser-based context. I think a 
> good approach is a to create a service layer and then provide an 
> implementation for those services when running in the desktop app. These 
> implementations will communicate with a local gRPC client. But we can also 
> create implementations of the same service interfaces that are implemented 
> based on a REST proxy to the same gRPC services (or whatever will work within 
> a web browser).
> 
> Vue 3 and Typescript have a dependency injection mechanism that make this 
> pretty easy to work with. I put together a little demo app here: 
> https://github.com/machristie/service-dependency-injection
> 
> I was able to create a service interface for "users":
> 
> export default interface UserService {
>   getAll(): Promise
> }
> 
> And a couple different implementations (A and B) of this service, one of 
> which is 
> 
> export default class UserServiceA implements UserService {
>   getAll() {
> return Promise.resolve([
>   {
> firstName: 'Ann',
> lastName: 'Armstrong',
> email: 'aarms...@example.com'
>   },
>   {
> firstName: 'Alex',
> lastName: 'Applegate',
> email: 'alexa...@example.com'
>   },
>   {
> firstName: 'Arnold',
> lastName: 'Avanoff',
> email: 'arnav...@example.com'
>   }
> ])
>   }
> }
> 
> 
> In the entry point for the app, I can "provide" either the A implementation 
> or the B implementation:
> 
> const app = createApp(App)
> 
> app.provide(UserServiceKey, new UserServiceA())
> // app.provide(UserServiceKey, new UserServiceB())
> 
> 
> Then it's easy to inject the implementation anywhere in the Vue app. What I 
> did was inject the user service into the user store:
> 
> 
> export const useUserStore = defineStore('user', () => {
>   const users = ref()
>   const userService = inject(UserServiceKey)
> 
>   async function loadUsers() {
> us

Re: Architecture for Cybershuttle Orchestration App

[Christie, Marcus Aaron]

Hi Praneeth,

This looks good. I've been thinking about how we can reuse UI components that 
are developed for this local app in a web browser-based context. I think a good 
approach is a to create a service layer and then provide an implementation for 
those services when running in the desktop app. These implementations will 
communicate with a local gRPC client. But we can also create implementations of 
the same service interfaces that are implemented based on a REST proxy to the 
same gRPC services (or whatever will work within a web browser).

Vue 3 and Typescript have a dependency injection mechanism that make this 
pretty easy to work with. I put together a little demo app here: 
https://github.com/machristie/service-dependency-injection

I was able to create a service interface for "users":

export default interface UserService {
  getAll(): Promise
}

And a couple different implementations (A and B) of this service, one of which 
is 

export default class UserServiceA implements UserService {
  getAll() {
return Promise.resolve([
  {
firstName: 'Ann',
lastName: 'Armstrong',
email: 'aarms...@example.com'
  },
  {
firstName: 'Alex',
lastName: 'Applegate',
email: 'alexa...@example.com'
  },
  {
firstName: 'Arnold',
lastName: 'Avanoff',
email: 'arnav...@example.com'
  }
])
  }
}


In the entry point for the app, I can "provide" either the A implementation or 
the B implementation:

const app = createApp(App)

app.provide(UserServiceKey, new UserServiceA())
// app.provide(UserServiceKey, new UserServiceB())


Then it's easy to inject the implementation anywhere in the Vue app. What I did 
was inject the user service into the user store:


export const useUserStore = defineStore('user', () => {
  const users = ref()
  const userService = inject(UserServiceKey)

  async function loadUsers() {
users.value = await userService?.getAll()
  }
  return {
users,
loadUsers
  }
})


More details are here: 

- https://vuejs.org/guide/components/provide-inject.html
- https://vuejs.org/guide/typescript/composition-api.html#typing-provide-inject

I don't want to dictate technology choices here, rather I'm suggesting that the 
UI layer should use some sort of dependency injection to inject the components 
that make network requests, and this is just an example of the general concept.


Thanks,

Marcus


> On Jun 14, 2023, at 12:42 AM, Praneeth Kumar Chityala 
>  wrote:
> 
> You don't often get email from praneethchityal...@gmail.com. Learn why this 
> is important
> Dear All,
> 
> I have been working on automating MFT agent deployment over compute 
> resources. As an extension to it I am also working on creating an UI for a 
> desktop application which can do the orchestration of MFT agent and 
> Cybershuttle services.
> 
> As MFT and cybershuttle are predominantly using protobufs and gRPC for 
> communication between the respective micro-services, I propose Tauri 
> (Frontend - JavaScript, HTML, CSS and Backend - Rust) into the architecture 
> to build the desktop UI application. I have also explored electronJS but 
> proposing Tauri for 3 key below reasons:
>   • electronJS uses NodeJS as backend which doesn't have integration with 
> gRPC yet, where as the Tauri used Rust which has production ready extension 
> to gRPC via Tonic
>   • Tauri in lighter (ex.180MB electonJS app vs 8MB Tauri app for just 
> "hello world" app) and faster when compared to electronJS
>   • As this proposed UI matures in future we might have to take some 
> heavy lifting tasks, Rust handles them much faster when compared to NodeJS
> Other quick comparisons of electronJS and Tauri are as below:
> 
> Feature   ElectronTauri
> Runtime   Chromium and Node.jsWebAssembly and Rust
> App size  Larger  Smaller
> Startup time  Slower  Faster
> Maturity  More mature Less mature
> Community Larger communitySmaller community
> Security  Less secure More secure
> Potential High potential  High potential
> 
> The architecture would look something like below:
> 
> 
> 
> As we are brainstorming this architecture, any suggestions or comments are 
> welcomed.
> 
> Best Regards,
> Praneeth Chityala
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: questions regarding dashboards to get quick statistics.

[Christie, Marcus Aaron]

Hi Saurav, 

Yeah, I’ve seen this issue where it takes a few tries to get it to work. What I 
would suggest is that you see what Entities aren’t enhanced, then go into that 
module and do a “mvn install -DskipTests” in just that module and try again. 

For example, if ExperimentEntity isn’t enhanced, go into 
modules/registry/registry-core and run “mvn install -DskipTests”. 

That will be faster than rebuilding the whole thing. But again, you might have 
to do it a couple of times. 

From: saurav kumar jha 
Date: Monday, June 12, 2023 at 10:45 PM
To: Christie, Marcus Aaron 
Cc: dev@airavata.apache.org 
Subject: Re: questions regarding dashboards to get quick statistics. 

Hi Marcus, 
I turned off the Autobuild : Enabled option. 

Did a clean install, cloned the repo again but getting the same openJpa error 
in all cases. 



Thanks & Regards, 

Saurav Kumar Jha 







On Tue, Jun 13, 2023 at 2:08 AM Christie, Marcus Aaron mailto:machr...@iu.edu>> wrote: 

Hi Saurav,

> On Jun 11, 2023, at 10:15 AM, saurav kumar jha  <_blank>> wrote:
> 
> I use VS Code ide but I couldn't find any option "Auto Build" . It will be 
> very helpful if you can elaborate on where I can find this option.
> 

Go into Settings, and search for "Auto Build". Turn off the "Java > Autobuild: 
Enabled" option. 






smime.p7s
Description: S/MIME cryptographic signature

Re: questions regarding dashboards to get quick statistics.

[Christie, Marcus Aaron]

Hi Saurav,

> On Jun 11, 2023, at 10:15 AM, saurav kumar jha  
> wrote:
> 
> I use VS Code ide but I couldn't find any option "Auto Build" . It will be 
> very helpful if you can elaborate on where I can find this option.
> 

Go into Settings, and search for "Auto Build". Turn off the "Java > Autobuild: 
Enabled" option.

smime.p7s
Description: S/MIME cryptographic signature

Re: questions regarding dashboards to get quick statistics.

[Christie, Marcus Aaron]

Hi Saurav,

> On Jun 7, 2023, at 10:34 AM, saurav kumar jha  
> wrote:
> 
> 2. Fixing 1 fixed the second one as you said. but there is another exception 
> related to expcatalog database:
> 

I just went through the instructions to make sure I could bring up the API 
server and django portal locally.

One thing I'll mention, you may want to disable any sort of "auto build" option 
in your IDE (in VS Code it's called "Auto Build"). The reason is that the auto 
build will tend to overwrite the enhanced entity classes.

> Exception in thread "main" java.lang.RuntimeException: Failed to 
> post-initialize the expcatalog database
> at 
> org.apache.airavata.registry.core.utils.ExpCatalogDBInitConfig.postInit(ExpCatalogDBInitConfig.java:81)
> at 
> org.apache.airavata.common.utils.DBInitializer.initializeDB(DBInitializer.java:46)
> at 
> org.apache.airavata.registry.api.service.RegistryAPIServer.StartRegistryServer(RegistryAPIServer.java:69)
> at 
> org.apache.airavata.registry.api.service.RegistryAPIServer.start(RegistryAPIServer.java:151)
> at 
> org.apache.airavata.ide.integration.APIServerStarter.main(APIServerStarter.java:23)
> Caused by: java.lang.RuntimeException: Failed to get EntityManager
> at 
> org.apache.airavata.registry.core.repositories.AbstractRepository.execute(AbstractRepository.java:137)
> at 
> org.apache.airavata.registry.core.repositories.AbstractRepository.get(AbstractRepository.java:75)
> at 
> org.apache.airavata.registry.core.repositories.AbstractRepository.isExists(AbstractRepository.java:113)
> at 
> org.apache.airavata.registry.core.repositories.expcatalog.GatewayRepository.isGatewayExist(GatewayRepository.java:99)
> at 
> org.apache.airavata.registry.core.utils.ExpCatalogDBInitConfig.postInit(ExpCatalogDBInitConfig.java:62)
> ... 4 more
> Caused by:  user error> org.apache.openjpa.persistence.ArgumentException: This 
> configuration disallows runtime optimization, but the following listed types 
> were not enhanced at build time or at class load time with a javaagent: "
> 
> 3. I copied the django_airavata/settings_local.py.ide to my settings_local.py 
> file and still the issue exists. I have attached the settings_local.py file 
> with this email.
> 

I was able to use the settings_local.py.ide successfully. If you are still 
getting errors after fixing #2 above, then make sure you are using a Python 
version from 3.6 to 3.10.

Thanks,

Marcus

smime.p7s
Description: S/MIME cryptographic signature

Re: questions regarding dashboards to get quick statistics.

[Christie, Marcus Aaron]

Hi Saurav,

> On Jun 7, 2023, at 10:34 AM, saurav kumar jha  
> wrote:
> 
> Hi Marcus,
> Thanks for the help!
> 1. Applying database migrations worked as you said to look at running 
> containers using docker ps  I observed that the name of the container was  
> containers_db_1 and not containers-db-1 as shown in documentation here. I 
> think we can update it in documentation.

What version of docker-compose are you using? v2 names containers with hyphens 
whereas v1 uses underscores. I suspect you are using version 1 of docker 
compose. I would recommend version 2.

We should then update the documentation to clarify either way.

> 
> 2. Fixing 1 fixed the second one as you said. but there is another exception 
> related to expcatalog database:
> 
> Exception in thread "main" java.lang.RuntimeException: Failed to 
> post-initialize the expcatalog database
> at 
> org.apache.airavata.registry.core.utils.ExpCatalogDBInitConfig.postInit(ExpCatalogDBInitConfig.java:81)
> at 
> org.apache.airavata.common.utils.DBInitializer.initializeDB(DBInitializer.java:46)
> at 
> org.apache.airavata.registry.api.service.RegistryAPIServer.StartRegistryServer(RegistryAPIServer.java:69)
> at 
> org.apache.airavata.registry.api.service.RegistryAPIServer.start(RegistryAPIServer.java:151)
> at 
> org.apache.airavata.ide.integration.APIServerStarter.main(APIServerStarter.java:23)
> Caused by: java.lang.RuntimeException: Failed to get EntityManager
> at 
> org.apache.airavata.registry.core.repositories.AbstractRepository.execute(AbstractRepository.java:137)
> at 
> org.apache.airavata.registry.core.repositories.AbstractRepository.get(AbstractRepository.java:75)
> at 
> org.apache.airavata.registry.core.repositories.AbstractRepository.isExists(AbstractRepository.java:113)
> at 
> org.apache.airavata.registry.core.repositories.expcatalog.GatewayRepository.isGatewayExist(GatewayRepository.java:99)
> at 
> org.apache.airavata.registry.core.utils.ExpCatalogDBInitConfig.postInit(ExpCatalogDBInitConfig.java:62)
> ... 4 more
> Caused by:  user error> org.apache.openjpa.persistence.ArgumentException: This 
> configuration disallows runtime optimization, but the following listed types 
> were not enhanced at build time or at class load time with a javaagent: "
> 


You need to run `mvn clean install -DskipTests` first as described in the 
README which will instrument the openjpa entities. You pretty much always have 
to rerun the build before starting the API server.

> 3. I copied the django_airavata/settings_local.py.ide to my settings_local.py 
> file and still the issue exists. I have attached the settings_local.py file 
> with this email.
> 

Yeah, your settings_local.py file looks fine. Do you have airavata.host in your 
/etc/hosts?

Thanks,

Marcus



smime.p7s
Description: S/MIME cryptographic signature

Re: questions regarding dashboards to get quick statistics.

[Christie, Marcus Aaron]

Hi Saurav,

> On Jun 6, 2023, at 12:51 PM, saurav kumar jha  
> wrote:
> 
> 1. While Following the steps mentioned at below link:
> airavata/README.md at develop · apache/airavata · GitHub
> 
> Apply any database migrations. Go to src/main/containers directory and run
>cat ./database_scripts/init/*-migrations.sql | docker exec -i 
> containers-db-1 mysql -p123456
> 
>Above command produces following error:
>Error response from daemon: No such container: containers-db-1
> 


You need to run docker-compose up first. Make sure that the 'containers-db-1' 
container is running, maybe by doing 'docker ps'.

> 2. Running the APIServerStarter produces following error:
> 
> Caused by: org.apache.openjpa.persistence.ArgumentException: Table 
> "COMPUTE_RESOURCE_SCHEDULING" given for 
> "org.apache.airavata.registry.core.entities.expcatalog.ComputationalResourceSchedulingEntity"
>  does not exist.
> 

Fixing #1 will fix this one.

> 3. Running Django-Portal works and app frontend is loaded but when I try to 
> login, it gives SSL version related error:
> 
> 

I'm not sure about this one. Did you use the 
django_airavata/settings_local.py.ide as your settings_local.py file? If you 
are still having issues, please share your settings_local.py file.

smime.p7s
Description: S/MIME cryptographic signature

Re: Cybershuttle Replica Catalog API

[Christie, Marcus Aaron]

Hi Jayan,

This looks really good. Thanks for putting together this test case. It makes it 
very clear how the API should be used.

Thanks,

Marcus

> On Apr 20, 2023, at 1:44 PM, Jayan Vidanapathirana 
>  wrote:
> 
> Hi All, 
> 
> A new test case [1] has been incorporated into the replica catalog, enabling 
> the addition and retrieval of replica data through the data catalog pull 
> request [2]. The current implementation allows users to operate with S3 and 
> Google Cloud Storage credentials and storage.
> 
> Ongoing implementation 
> How to model this replica locations as hierarchical system to support replica 
> groups
> 
> [1] org.apache.airavata.ReplicaCatalogAPIClientTest#testCase1
> [2] https://github.com/apache/airavata-data-catalog/pull/28
> 
> Thanks. 
> 
> On Sat, Apr 8, 2023 at 3:07 AM Christie, Marcus Aaron  wrote:
> Hi Jayan,
> 
>  
> 
> I think we can keep them loosely coupled. When a client is searching for 
> files, it will search the Data Catalog. That will yield data product ids. 
> Then the client needs to resolve those data product ids to replica locations. 
> So that means that the Replica Catalog needs to record the data product id 
> associated with each replica location.
> 
>  
> 
> For deployment plans, I haven’t though much about it but I think we’ll deploy 
> them as separate services. That is, separate JVM processes. They will likely 
> connect to the same backend database out of convenience, but I don’t think 
> they would have to do so, they could have separate databases. Does that 
> answer your question?
> 
>  
> 
> Thanks,
> 
>  
> 
> Marcus
> 
>  
> 
> From: Jayan Vidanapathirana 
> Date: Friday, April 7, 2023 at 2:00 PM
> To: dev@airavata.apache.org 
> Cc: Dimuthu Upeksha Wannipurage , Christie, 
> Marcus Aaron 
> Subject: Re: Cybershuttle Replica Catalog API
> 
> Hi Marcus, 
> 
> I sincerely appreciate you taking the time to review my changes. Actually, I 
> thought these two services needed to be operated independently when i'm 
> starting the API definition. 
> If we couple both services then users can simply use the Data-Catalog Data 
> Product  APIs and Data models. Then I can remove data product APIs and models 
> from the replica catalog. 
> 
>  
> 
> Can I know if there is any overall deployment plan for these 2 catalogs ?
> 
>  
> 
> Thanks. 
> 
>  
> 
> On Thu, Apr 6, 2023 at 7:44 PM Christie, Marcus Aaron  wrote:
> 
> Hi Jayan,
> 
> Thanks for sharing. One question, the airavata-data-catalog already has a 
> DATA_PRODUCT table and a way to store a data product's metadata. Could that 
> be used instead of adding a new table?
> 
> Or more generally my question is how does this replica catalog API relate to 
> the data catalog API/data model?
> 
> Thanks,
> 
> Marcus
> 
> > On Mar 31, 2023, at 4:11 PM, Jayan Vidanapathirana 
> >  wrote:
> > 
> > Hi All,
> > 
> > I have implemented basic flow(simple create and retrieve) of the replica 
> > catalog and drafted a pull request[1] to the Airavata data catalog as a new 
> > module. According to that implementation I have come to the following 
> > database structure for the replica catalog and I greatly appreciate your 
> > thoughts and feedback on the designs[2]. At this stage S3 storage type was 
> > considered as a sample. 
> > 
> > 
> > 
> > Also please refer to the following google doc[3] to review the implemented 
> > APIs.
> > 
> > [1] https://github.com/apache/airavata-data-catalog/pull/28
> > [2] 
> > https://drive.google.com/file/d/1KP-8IWdvpPvjSWUG2t41K7WQXW2f9_qN/view?usp=sharing
> > [3] 
> > https://docs.google.com/document/d/1U-ok1ICt_EmjjxR9UuACV6g6YYkgoADfm0ECJ9ZDI3k/edit?usp=sharing
> > 
> > Thank you.
> > 
> > On Mon, Mar 20, 2023 at 2:20 AM Suresh Marru  wrote:
> > Hi Jayan,
> > 
> > Can you contribute a PR to the data catalog repo so we can keep the 
> > feedback on that issue?
> > 
> > Thanks for your contribution,
> > Suresh
> > 
> >> On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana 
> >>  wrote:
> >> 
> >> Hi All,
> >> 
> >> I have updated the draft code base[1] with a simple workflow of adding 
> >> data to replica catalog. Still services are not yet finalized and will be 
> >> enhanced with the workflow. 
> >> 
> >> [1] https://github.com/Jayancv/airavata-replica-catalog
> >> 
> >> Thanks.
> >> 
> >> On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana 
> >>  wrote:
> >> Hi Dimuthu 

Re: Cybershuttle Replica Catalog API

[Christie, Marcus Aaron]

Hi Jayan, 

I think we can keep them loosely coupled. When a client is searching for files, 
it will search the Data Catalog. That will yield data product ids. Then the 
client needs to resolve those data product ids to replica locations. So that 
means that the Replica Catalog needs to record the data product id associated 
with each replica location. 

For deployment plans, I haven’t though much about it but I think we’ll deploy 
them as separate services. That is, separate JVM processes. They will likely 
connect to the same backend database out of convenience, but I don’t think they 
would have to do so, they could have separate databases. Does that answer your 
question? 

Thanks, 

Marcus 

From: Jayan Vidanapathirana 
Date: Friday, April 7, 2023 at 2:00 PM
To: dev@airavata.apache.org 
Cc: Dimuthu Upeksha Wannipurage , Christie, Marcus 
Aaron 
Subject: Re: Cybershuttle Replica Catalog API 

Hi Marcus, 

I sincerely appreciate you taking the time to review my changes. Actually, I 
thought these two services needed to be operated independently when i'm 
starting the API definition. 
If we couple both services then users can simply use the Data-Catalog Data 
Product APIs and Data models. Then I can remove data product APIs and models 
from the replica catalog. 


Can I know if there is any overall deployment plan for these 2 catalogs ?



Thanks. 



On Thu, Apr 6, 2023 at 7:44 PM Christie, Marcus Aaron mailto:machr...@iu.edu>> wrote:

Hi Jayan,

Thanks for sharing. One question, the airavata-data-catalog already has a 
DATA_PRODUCT table and a way to store a data product's metadata. Could that be 
used instead of adding a new table?

Or more generally my question is how does this replica catalog API relate to 
the data catalog API/data model?

Thanks,

Marcus

> On Mar 31, 2023, at 4:11 PM, Jayan Vidanapathirana 
> > wrote:
> 
> Hi All,
> 
> I have implemented basic flow(simple create and retrieve) of the replica 
> catalog and drafted a pull request[1] to the Airavata data catalog as a new 
> module. According to that implementation I have come to the following 
> database structure for the replica catalog and I greatly appreciate your 
> thoughts and feedback on the designs[2]. At this stage S3 storage type was 
> considered as a sample. 
> 
> 
> 
> Also please refer to the following google doc[3] to review the implemented 
> APIs.
> 
> [1] https://github.com/apache/airavata-data-catalog/pull/28 <_blank>
> [2] 
> https://drive.google.com/file/d/1KP-8IWdvpPvjSWUG2t41K7WQXW2f9_qN/view?usp=sharing
>  <_blank>
> [3] 
> https://docs.google.com/document/d/1U-ok1ICt_EmjjxR9UuACV6g6YYkgoADfm0ECJ9ZDI3k/edit?usp=sharing
>  <_blank>
> 
> Thank you.
> 
> On Mon, Mar 20, 2023 at 2:20 AM Suresh Marru > 
> wrote:
> Hi Jayan,
> 
> Can you contribute a PR to the data catalog repo so we can keep the feedback 
> on that issue?
> 
> Thanks for your contribution,
> Suresh
> 
>> On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana 
>> > wrote:
>> 
>> Hi All,
>> 
>> I have updated the draft code base[1] with a simple workflow of adding data 
>> to replica catalog. Still services are not yet finalized and will be 
>> enhanced with the workflow. 
>> 
>> [1] https://github.com/Jayancv/airavata-replica-catalog <_blank>
>> 
>> Thanks.
>> 
>> On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana 
>> > wrote:
>> Hi Dimuthu and Marcus,
>> 
>> Thank you both for checking my PoC and providing valuable feedback.
>> 
>> Dimuthu,
>> • Im agree with you regarding Replica location categories. It should be a 
>> data catalog level attribute. 
>> • To manage replica data access permissions don't we need user information 
>> at Replica catalog level ? I'm a bit confused on the permission management 
>> side of this catalog. 
>> • ReplicaListEntry - Added to expose the list DataReplicaLocation s with 
>> basic details in AllDataReplicaGetResponse which provide all the replica 
>> items for the given product_id. However, here I was not considering that 
>> hierarchical structure. ReplicaGroupEntry is actually a one product replica 
>> which holds the file structure of the replica data. According to your 
>> suggestion we can model that AllDataReplicaGetResponse as follows,
>> message AllDataReplicaGetResponse {
>> data_product_id = 1
>> repeated ReplicaGroupEntry replica_list = 2;
>> }
>> 
>> message ReplicaGroupEntry {
>> string replica_group_id = 1
>> repeated ReplicaGroupEntry directories = 2;
>> repeated DataReplicaLocation files = 3;
>> }
>> 
>> Marcus, 
>> • Yes, I will remove replica_id from the data catalog diagram. 
>&

Re: Cybershuttle Replica Catalog API

[Christie, Marcus Aaron]

Hi Jayan,

Thanks for sharing. One question, the airavata-data-catalog already has a 
DATA_PRODUCT table and a way to store a data product's metadata. Could that be 
used instead of adding a new table?

Or more generally my question is how does this replica catalog API relate to 
the data catalog API/data model?

Thanks,

Marcus

> On Mar 31, 2023, at 4:11 PM, Jayan Vidanapathirana 
>  wrote:
> 
> Hi All,
> 
> I have implemented basic flow(simple create and retrieve) of the replica 
> catalog and drafted a pull request[1] to the Airavata data catalog as a new 
> module. According to that implementation I have come to the following 
> database structure for the replica catalog and I greatly appreciate your 
> thoughts and feedback on the designs[2]. At this stage S3 storage type was 
> considered as a sample. 
> 
> 
> 
> Also please refer to the following google doc[3] to review the implemented 
> APIs.
> 
> [1] https://github.com/apache/airavata-data-catalog/pull/28
> [2] 
> https://drive.google.com/file/d/1KP-8IWdvpPvjSWUG2t41K7WQXW2f9_qN/view?usp=sharing
> [3] 
> https://docs.google.com/document/d/1U-ok1ICt_EmjjxR9UuACV6g6YYkgoADfm0ECJ9ZDI3k/edit?usp=sharing
> 
> Thank you.
> 
> On Mon, Mar 20, 2023 at 2:20 AM Suresh Marru  wrote:
> Hi Jayan,
> 
> Can you contribute a PR to the data catalog repo so we can keep the feedback 
> on that issue?
> 
> Thanks for your contribution,
> Suresh
> 
>> On Mar 19, 2023, at 12:55 PM, Jayan Vidanapathirana 
>>  wrote:
>> 
>> Hi All,
>> 
>> I have updated the draft code base[1] with a simple workflow of adding data 
>> to replica catalog. Still services are not yet finalized and will be 
>> enhanced with the workflow. 
>> 
>> [1] https://github.com/Jayancv/airavata-replica-catalog
>> 
>> Thanks.
>> 
>> On Sat, Feb 25, 2023 at 4:21 PM Jayan Vidanapathirana 
>>  wrote:
>> Hi Dimuthu and Marcus,
>> 
>> Thank you both for checking my PoC and providing valuable feedback.
>> 
>> Dimuthu,
>>  • Im agree with you regarding Replica location categories. It should be 
>> a data catalog level attribute. 
>>  • To manage replica data access permissions don't we need user 
>> information at Replica catalog level ? I'm a bit confused on the permission 
>> management side of this catalog. 
>>  • ReplicaListEntry  - Added to expose the list DataReplicaLocation s 
>> with basic details in AllDataReplicaGetResponse which provide all the 
>> replica items for the given product_id. However, here I was not considering 
>> that hierarchical structure. ReplicaGroupEntry is actually a one product 
>> replica which holds the file structure of the replica data. According to 
>> your suggestion we can model that AllDataReplicaGetResponse as follows,
>> message AllDataReplicaGetResponse {
>>   data_product_id = 1
>>   repeated ReplicaGroupEntry replica_list = 2;
>> }
>> 
>> message ReplicaGroupEntry {
>>   string replica_group_id = 1
>>   repeated ReplicaGroupEntry directories = 2;
>>   repeated DataReplicaLocation files = 3;
>> }
>> 
>> Marcus, 
>>  • Yes, I will remove replica_id  from the data catalog diagram. 
>>  • I added that parent_data_product_id to replica data by considering 
>> full context with replica catalog and data catalog relation. But within 
>> replica catalog context there is no such paranet product relationship. 
>> Therefore we can rename it to data_product_id. Thanks for pointing this out. 
>> 
>> Thanks.
>> 
>> On Thu, Feb 23, 2023 at 2:48 AM Christie, Marcus Aaron  
>> wrote:
>> Hi Jayan,
>> 
>> I would like to echo Dimuthu and say that this looks great and I appreciate 
>> the effort in your pulling this all together.  I have some feedback to share.
>> 
>> The high-level architecture diagram shows the replica id being stored in the 
>> data catalog. That was an initial idea that we had, but we decided that the 
>> replica catalog would store the data product id. That seems reflected in 
>> your API design so I think you already know this, but I wanted to point it 
>> out since the diagram might be a little confusing for others.
>> 
>> In the ReplicaCatalogAPI.proto the name of the data product id field is 
>> "parent_data_product_id". I would suggest calling it "data_product_id" 
>> instead. "parent_data_product_id" means "the id of the parent data product 
>> of this data product" in the data catalog. It might be confusing to use the 
>> same name in ReplicaCatalogAPI.proto.
>> 
>> 
>

Thrift upgrade to 0.18.1

[Christie, Marcus Aaron]

Hi all,

On the develop branch, I've just merged a PR [1] that upgrades our Thrift 
dependency to 0.18.1. This was done primarily so that I could upgrade the 
Python thrift dependency to a version that is Python 3.10+ compatible. [2]

One consequence of this Thrift version upgrade is that we can no longer use 
Docker images to run Thrift. The thrift Docker images are unsupported and 
haven't been updated in a while. See the deprecation notice on [3] for more 
information. I've removed the Docker container execution stuff from the 
generate stubs shell scripts in the PR.

So if you need to run Thrift to generate stubs, you'll need to install it 
natively. On macOS, it was as simple as `brew install thrift`.

If you have any questions or need any help with Thrift, let me know.

Thanks,

Marcus

[1] https://github.com/apache/airavata/pull/409
[2] 
https://issues.apache.org/jira/browse/AIRAVATA-3529?focusedCommentId=17645496=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17645496
[3] https://hub.docker.com/_/thrift

smime.p7s
Description: S/MIME cryptographic signature

Re: Installation error: "undefined subnets"

[Christie, Marcus Aaron]

Hi Dhruvi,

See 
https://github.com/apache/airavata/blob/master/dev-tools/ansible/inventories/scigap/production/group_vars/all/vars.yml#L212-L226
 for how we've defined the allowed subnets for a deployment at IU.

You could do something similar. Perhaps we should default zk_subnets, etc. to 
just be 0.0.0.0/0 to allow all traffic in the default case.

Thanks,

Marcus

> On Mar 18, 2023, at 5:53 AM, DHRUVI DOSHI  wrote:
> 
> Hello,
> I have encountered following errors on running the ansible playbooks.
> 1. ansible-playbook -i inventories/standalone database.yml
> TASK [database : allow only selected networks to access DB] 
> 
> fatal: [192.168.56.10]: FAILED! => {"msg": "'db_subnets' is undefined"}
> 2. ansible-playbook -i inventories/standalone airavata.yml
> TASK [zookeeper : open zookeeper port] 
> *
> fatal: [192.168.56.10]: FAILED! => {"msg": "'zk_subnets' is undefined"}
> 
> I'm really not getting how to go about it. Could anyone please point me in 
> the right direction?
> Thank you.
> BR, 
> Dhruvi



smime.p7s
Description: S/MIME cryptographic signature

Re: Airavata-MFT | Local transport implementation

[Christie, Marcus Aaron]

HI Praneeth,

This looks like a great contribution to MFT and I appreciate your write up here.

One question: these numbers are for uploading from the local EC2 instance to 
S3, correct? Did you do any analysis on the opposite, downloading from S3 to a 
local EC2 instance?

Thanks,

Marcus

> On Mar 17, 2023, at 12:22 AM, Chityala, Praneeth  wrote:
> 
> You don't often get email from pkchi...@iu.edu. Learn why this is important
> Dear All,
>  
> This is Praneeth Chityala currently pursuing master in Computer Science at 
> Indiana University Bloomington. As part of my independent study I took up MFT 
> as the research area and starting understanding the architecture.
>  
> As many of you know MFT uses Agent to transfer data from one cloud storage to 
> other cloud storage. These agents can be deployed on any compute machines. If 
> the machine in which agent is deployed might have data files which needs to 
> be uploaded to cloud storage, that’s where my involvement in the project came 
> in. I worked on implementing the below extensions:
>   • Implemented the Local transport extension to allow agent to transfer 
> data from its host machine given storage – Local transport extension
>   • Transport has three variations – streaming, chunked file 
> transfer and chunked streaming
>   • Implemented the CLI for configuring local agent – Local agent CLI
>  
> Performance testing results:
>  
> After successfully testing from my local machine to AWS S3 storage, I have 
> deployed agent in AWS EC2 machine and performed multiple tests for compare 
> it’s performance with rclone and AWS cli.
> Below charts indicates the average transfer speeds from our analysis.
>  
> 
>  
>  
> For files from 100MB to 1GB, MFT is more than 60% faster than rclone and more 
> than 150% faster than AWS cli.
>  
> Configurations of the testing:
>  
>   • Local Machine: It’s Ubuntu EC2 VM on AWS (instance type c5.9xlarge) 
> with 18 cores, 10Gbps dedicated network speed and 1GBps read/write speed to 
> disk.
>  
>   • Cloud Storage: AWS S3 bucket in the same region as above VM.
>  
>   • Test sets: From x-axis labels of the graph, 10m_1000 means a test set 
> of 1000 10MB files. All other test sets follow similar naming convention.
>  
>   • Testing trails: Each test is run for 5 times on each transfer method.
>  
>   • Testing presets: Before each test caching of VM is cleared so none of 
> the tests get advantage of higher read speeds using page caching. This is 
> done to simulate worst possible conditions while reading data.
>  
>   • MFT configuration: I used chunked streaming with
>   • 20MB as chunk size
>   • 32 concurrent transfers
>   • 32 concurrent chunked threads
>  
>   • rclone configuration: After exploring many possible optimizations 
> available for rclone I used following settings:
>   • --s3-chunk-size 128000
>   • --buffer-size 128000
>   • —s3-upload-cutoff 0
>   • --s3-upload-concurrency 32
>   • --multi-thread-streams 32
>   • --multi-thread-cutoff 0
>   • --s3-disable-http2
>   • --no-check-dest
>   • --transfers 32
>   • --fast-list
>  
>   • AWS cli configuration: I used native AWS cli to transfer as it 
> doesn’t have much dedicated optimizations in our findings
>  
> Observations:
>   • For local transport I used BufferedStreaming which helped MFT to get 
> the max read speeds from local disk without hitting the max IOPS.
>  
> Future plans for testing:
>   • Jetstream2: Planning to replace AWS EC2 with Jetstream2 virtual 
> machine and perform similar tests
>   • Emulab: Simulate same testing using Emulab VMs and custom 
> configurations with help of Dimuthu.
>   • Azure: Perform local to Azure cloud storages testing with MFT, rclone 
> and Azure cli
>   • GCP: Perform local to GCS testing with MFT, rclone and GCP cli
>   • I have different implementation of MFT local transport for system 
> which support DMA (Direct Memory Access), we also plan to test on such 
> systems with DMA, the present EC2 system doesn’t support DMA.
>  
> Further Improvements of MFT:
>   • As we noticed MFT is lagging speeds vs rclone for files less than or 
> equal to 1MB, we plan to stress analyze the whole system and improve speeds 
> for smaller files
>  
> Acknowledgement: I thank Dimuthu Wannipurage for clearing many doubts about 
> MFT and providing guidance when needed.
>  
> Thank you and please let us know your comments or thoughts. 
>  
> Best,
> Praneeth Chityala



smime.p7s
Description: S/MIME cryptographic signature

Re: MariaDB installation error on running ansible playbook

[Christie, Marcus Aaron]

Hi Dhruvi,

You might try the repo config for the latest 10. release for CentOS 7 from here:

https://mariadb.org/download/?t=repo-config=CentOS+7=10.11_m=acorn



> On Mar 8, 2023, at 9:26 AM, DHRUVI DOSHI  wrote:
> 
> Hello Marcus,
> I'm using this one. 
> 
> I believe Mariadb 10 is no longer available. I found the base url is in this 
> (/roles/database/files/MariaDB_yum_CentOS_7.repo) file. I tried changing the 
> url yet the changes aren't reflected while running ansible-playbook -i 
> inventories/standalone database.yml
> 
> On Wed, Mar 8, 2023 at 7:07 PM Christie, Marcus Aaron  wrote:
> Hi Dhruvi,
> 
> Which guide are you using?
> 
> > On Mar 7, 2023, at 3:08 AM, DHRUVI DOSHI  wrote:
> > 
> > You don't often get email from drdoshi_...@ce.vjti.ac.in. Learn why this is 
> > important
> > Hello developers,
> > I was locally setting up the django portal using this guide. While running 
> > the ansible playbook, it encountered an error in installing mariadb in the 
> > remote vm. I believe that particular mirror is no longer available. Where 
> > could I change the mirror link? 
> > 
> > I'll attach a screenshot of the same. 
> > To reproduce the error, run:
> > ansible-playbook -i inventories/standalone database.yml
> > Error:
> > 
> > Any help would be appreciated.
> > Thanking you in advance.
> > 
> > BR,
> > Dhruvi.
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: MariaDB installation error on running ansible playbook

[Christie, Marcus Aaron]

Hi Dhruvi,

Which guide are you using?

> On Mar 7, 2023, at 3:08 AM, DHRUVI DOSHI  wrote:
> 
> You don't often get email from drdoshi_...@ce.vjti.ac.in. Learn why this is 
> important
> Hello developers,
> I was locally setting up the django portal using this guide. While running 
> the ansible playbook, it encountered an error in installing mariadb in the 
> remote vm. I believe that particular mirror is no longer available. Where 
> could I change the mirror link? 
> 
> I'll attach a screenshot of the same. 
> To reproduce the error, run:
> ansible-playbook -i inventories/standalone database.yml
> Error:
> 
> Any help would be appreciated.
> Thanking you in advance.
> 
> BR,
> Dhruvi.



smime.p7s
Description: S/MIME cryptographic signature

Re: Cybershuttle Replica Catalog API

[Christie, Marcus Aaron]

Hi Jayan,

I would like to echo Dimuthu and say that this looks great and I appreciate the 
effort in your pulling this all together.  I have some feedback to share.

The high-level architecture diagram shows the replica id being stored in the 
data catalog. That was an initial idea that we had, but we decided that the 
replica catalog would store the data product id. That seems reflected in your 
API design so I think you already know this, but I wanted to point it out since 
the diagram might be a little confusing for others.

In the ReplicaCatalogAPI.proto the name of the data product id field is 
"parent_data_product_id". I would suggest calling it "data_product_id" instead. 
"parent_data_product_id" means "the id of the parent data product of this data 
product" in the data catalog. It might be confusing to use the same name in 
ReplicaCatalogAPI.proto.


Thanks,

Marcus

> On Feb 18, 2023, at 3:09 PM, Jayan Vidanapathirana 
>  wrote:
> 
> Hi All, 
> 
> As a new contributor to the Cybershuttle project, I have been actively 
> involved in implementing the Data Replica Catalog. This new catalog is 
> designed to interface with both the Apache Airavata Data Catalog [1] and 
> Airavata MFT[2]. This replica catalog should be able to store each replica 
> resource storage details and secret/credential details specific to the 
> storage type. The proposed high-level architecture will be as follows:
> 
> 
> 
> I will mainly work on the highlighted area (red color box) and as an initial 
> step started defining APIs which communicate with Replica Catalog. This API 
> calls will be gRPC APIs and following methods will be implement,
> 
> Replica Registration
> 
>   • registerReplicaLocation(DataReplicaCreateRequest createRequest)
>   • updateReplicaLocation(DataReplicaCreateRequest updateRequest)
>   • DataReplicaLocationModel getReplicaLocation(DataReplicaGetRequest 
> getReplicaRequest)
>   • removeReplicaLocation(DataReplicaDeleteRequest deleteReplicaRequest)
>   • getAllReplicaLocations(AllDataReplicaGetRequest allDataGetRequest)
>   • removeAllReplicaLocations(AllDataReplicaDeleteRequest 
> allDataDeleteRequest)
> 
> Storage Registration
> 
> registerSecretForStorage(SecretForStorage request)
> deleteSecretsForStorage(SecretForStorageDeleteRequest request)
> getSecretForStorage(SecretForStorageGetRequest request)
> searchStorages(StorageSearchRequest request)
> listStorages(StorageListRequest request)
> resolveStorageType (StorageTypeResolveRequest request)
> 
> Storage - Internal APIs
> 
> S3StorageListResponse listS3Storage(S3StorageListRequest request) 
> Optional getS3Storage(S3StorageGetRequest request) 
> S3Storage createS3Storage(S3StorageCreateRequest request) 
> boolean updateS3Storage(S3StorageUpdateRequest request) 
> boolean deleteS3Storage(S3StorageDeleteRequest request) 
> 
> AzureStorageListResponse listAzureStorage(AzureStorageListRequest request) 
> Optional getAzureStorage(AzureStorageGetRequest request) 
> AzureStorage createAzureStorage(AzureStorageCreateRequest request) 
> boolean updateAzureStorage(AzureStorageUpdateRequest request) 
> boolean deleteAzureStorage(AzureStorageDeleteRequest request) 
> 
> GCSStorageListResponse listGCSStorage(GCSStorageListRequest request) 
> Optional getGCSStorage(GCSStorageGetRequest request) 
> GCSStorage createGCSStorage(GCSStorageCreateRequest request) 
> boolean updateGCSStorage(GCSStorageUpdateRequest request) 
> boolean deleteGCSStorage(GCSStorageDeleteRequest request) 
> 
> Secret Registration
> 
> registerSecret(SecretRegistrationRequest request)
> deleteSecret(SecretDeleteRequest request)
> resolveStorageType (StorageTypeResolveRequest request)
> 
> Secret  - Internal APIs
> 
> Optional getS3Secret(S3SecretGetRequest request) 
> S3Secret createS3Secret(S3SecretCreateRequest request) 
> boolean updateS3Secret(S3SecretUpdateRequest request) 
> boolean deleteS3Secret(S3SecretDeleteRequest request) 
> 
> Optional getAzureSecret(AzureSecretGetRequest request) 
> AzureSecret createAzureSecret(AzureSecretCreateRequest request) 
> boolean updateAzureSecret(AzureSecretUpdateRequest request) 
> boolean deleteAzureSecret(AzureSecretDeleteRequest request) 
> 
> Optional getGCSSecret(GCSSecretGetRequest request) 
> GCSSecret createGCSSecret(GCSSecretCreateRequest request) 
> boolean updateGCSSecret(GCSSecretUpdateRequest request) 
> boolean deleteGCSSecret(GCSSecretDeleteRequest request) 
> 
> 
> Poc[3] : https://github.com/Jayancv/airavata-replica-catalog  (Defining API 
> calls)
> Draft APIs : refer the attachment replicaCatalogAPIsDocumentation.html[4] 
> which generated using the Poc [3]
> 
> I greatly appreciate your thoughts and feedback on the designs[5], as they 
> can help us improve and adopt a more generalized approach. Additionally, I 
> would like to identify any other factors that we should take into account to 
> minimize potential issues in the future. Are there any other considerations 
> that we should keep in 

Re: SEAGrid Data Catalog

[Christie, Marcus Aaron]

Hi Lahiru,

Thanks for putting together this investigation. I'm not 100% sure but it looks 
like gRPC-JS only works with Node.js since it uses Node.js APIs. I think you'll 
need gRPC-Web to make gRPC calls from a browser. My understanding is that that 
requires an Envoy proxy on the server side. (Rereading your email, I think you 
probably already know this, but just in case I thought I would point this out.)

It looks like django-grpc-framework isn't an active project [1], so I agree 
with your concern about depending on it. One issue with using gRPC in Django, I 
think, is that the integration that we've done with the Django framework would 
need to be re-implemented, things like middleware and authentication.  It's 
probably doable, just something to keep in mind.

It would be good if the gRPC server could run on the same HTTP port as the 
Django server, but I'm not sure how that would work.  From the client, 
accessing the Django server or the gRPC server should both be over SSL, on the 
same port. Maybe on the backend they run on different ports but with the proxy 
it looks like from the client's perspective they run on the same port.

The django-grpc-framework project may be good to mine for some ideas. I like 
that it follows django-rest-framework conventions. We use django-rest-framework 
in the Airavata Django Portal.

Thanks,

Marcus

[1] https://github.com/fengsp/django-grpc-framework/issues/34

> On Feb 14, 2023, at 1:17 PM, Lahiru Jayathilake  
> wrote:
> 
> Hi Suresh,
> 
> Thank you for the feedback. The other library that can be used to facilitate 
> browser communication with gRPC services is gRPC-JS 
> (https://github.com/grpc/grpc-node/tree/master/packages/grpc-js). However, in 
> terms of browser support, gRPC-Web is specifically designed for use in web 
> browsers, and it supports all major browsers including Chrome, Firefox, 
> Safari, and Edge. In contrast, gRPC-JS is designed to work with both web 
> browsers and Node.js, and it may require additional configuration to work 
> correctly in web browsers and it is a bit cumbersome.
> 
> @machr...@iu.edu I had a chat with Suresh and wanted to clarify a few points 
> with you.
> 
> 1. In the second approach what I have done is spinup up a gRPC server in the 
> background (inside Django App). When I was doing that I came across a 
> framework called django-grpc-framework [1][2]. I did not proceed with that 
> framework because it is coming from a personal repository. What do you think? 
> Is it good to go with this?
> 
> 2. Any suggestions or comments on the approach of using gRPC (gRPC-web) to 
> establish communications with the frontend?
> 
> I'd be really happy to hear your thoughts and suggestions on these.
> 
> [1] - https://github.com/fengsp/django-grpc-framework
> [2] - https://pypi.org/project/djangogrpcframework/
> 
> Thanks, 
> Lahiru
> 
> 
> 
> On Tue, Feb 14, 2023 at 7:47 PM Suresh Marru  wrote:
> Hi Lahiru,
> 
> Thank you for summarizing both of these, and your POCs of both approaches are 
> helpful. The second option, if feasible, will be preferable. You already 
> mentioned the performance. In addition, you will also get forward/backward 
> compatibility if the underlying protobuff structures are maintained with some 
> discipline. 
> 
> The big plus side of your 1st approach is the REST-compatible javascript 
> libraries. Other than grpc-web (https://github.com/grpc/grpc-web) have you 
> seen broader support? I see you are building on grpc-js how is that 
> experience?
> 
> Suresh
> 
>> On Feb 13, 2023, at 2:49 PM, Lahiru Jayathilake 
>>  wrote:
>> 
>> Hi All,
>> 
>> I have been engaging with the SMILES project to implement the Gateway and 
>> its necessary components. Just to give you a brief introduction, the SMILES 
>> project has three types of data that need to be combined for publication: 
>> Computational DB, Literature DB, and Experiment DB. There should be a 
>> frontend to filter, create, and delete data products, with a Django app as 
>> the backend that will communicate with Apache Airavata Data Catalog [1].
>> 
>> Mainly, I have been exploring two approaches.
>> 
>> 1. 
>> 
>> 
>> The frontend will communicate with the Django app via REST, and the Django 
>> app will manage the manipulation of data products through gRPC calls to the 
>> Data Catalog API. Django models will be used to represent the Computational, 
>> Literature, and Experiment data products, without storing the data. In the 
>> end, these data products will reside in the Data Catalog, following its 
>> established conventions.
>> 
>> POC - https://github.com/lahirujayathilake/SEAGrid 
>> This has been implemented to cover the data product creation
>> 
>> 2. 
>> 
>> 
>> In this approach, the distinction will be a gRPC server operating within the 
>> Django app. To represent the three data products, protobufs will be defined 
>> that extend the DataCatalog proto messages [2]. The frontend will 
>> communicate using gRPC calls.
>> The gRPC API 

Re: [External] [DISCUSS] New name for MFT

[Christie, Marcus Aaron]

Lots of good ideas here. When I think of MFT transfering data, it made me think 
of rivers. Here's a list of mythological rivers: 
https://en.wikipedia.org/wiki/Category:Mythological_rivers

One of those rivers is Iravati (https://en.wikipedia.org/wiki/Iravati), who is 
also the mother of Airavata.


> On Jan 31, 2023, at 1:14 PM, Suresh Marru  wrote:
> 
> Good, we are getting options. Let's gather a few more and then do a trademark 
> search, filter, and vote on the final 3.
> 
> Suresh
> 
>> On Jan 31, 2023, at 12:59 PM, Thejaka Amila J Kanewala 
>>  wrote:
>> 
>> Finding a name is harder than developing the product :D.
>> 
>> I think we already have nice suggestions.
>> 
>> Here are a few more:
>> 1. Javelin : https://en.wikipedia.org/wiki/Javelin_throw
>> 2. Catapult -- https://en.wikipedia.org/wiki/Catapult
>> 3. Dionysius -- they engineered the first version of catapult -- 
>> https://www.hellenicaworld.com/Greece/Technology/en/Catapults.html
>> 4. Trebuchet -- https://en.wikipedia.org/wiki/Trebuchet
>> 
>> As you can see I am a little obsessed with catapults :-).
>> 
>> Best Regards,
>> Thejaka Amila Kanewala, PhD
>> https://github.com/thejkane/agm
>> http://valagamba.net/
>> 
>> 
>> On Tue, Jan 31, 2023 at 6:36 AM Lahiru Jayathilake 
>>  wrote:
>> Three suggestions from me as well,
>> 
>> 1. SkyBridge
>> 
>> 2. Hammurabi - is a Babylonian king. The "Code of Hammurabi" is considered 
>> to be one of the first written legal codes in history with the idea of fair 
>> and orderly transfer of information. And the name exhibits stability, 
>> reliability, and order in the transfer of information
>> 
>> 3. Akkad - was an ancient empire that was recognized for its advanced 
>> administration and communication systems. The name exhibits stability and 
>> efficiency in the transfer of large-scale data
>> 
>> Thanks,
>> Lahiru
>> 
>> On Tue, Jan 31, 2023 at 7:48 PM Pierce, Marlon  wrote:
>> How about a ship theme? Here’s a list of types of ships: 
>> https://en.wikipedia.org/wiki/List_of_ship_types.
>> 
>>  
>> 
>> A couple of ancient types of ships: Kerkouros, Corbita. 
>> 
>>  
>> 
>> Marlon
>> 
>>  
>> 
>>  
>> 
>> From: Suresh Marru 
>> Date: Monday, January 30, 2023 at 5:59 PM
>> To: Airavata Dev 
>> Subject: [External] [DISCUSS] New name for MFT
>> 
>> This message was sent from a non-IU address. Please exercise caution when 
>> clicking links or opening attachments from external sources.
>> 
>> 
>> Hi All,
>> 
>>  
>> 
>> We chose MFT as a code name as we carved out all file transfer use cases 
>> from core Airavata. Now MFT has evolved significantly [1] and is most 
>> notably used for large scale data transfers between NASA and ESA HLS Data 
>> [2] Transfers between Openstack and AWS and IBM clouds. 
>> 
>>  
>> 
>> If we choose the name Custos for the security spinoff [3] [4], it is 
>> probably time to choose a name for MFT. Please keep your creative naming 
>> thoughts flowing?
>> 
>>  
>> 
>> What would you name a more generic MFT without naming technology terms like 
>> Transfers? 
>> 
>>  
>> 
>> Cheers,
>> 
>> Suresh
>> 
>>  
>> 
>> [1] - https://github.com/apache/airavata-mft 
>> 
>> [2] - https://hls.gsfc.nasa.gov/ 
>> 
>> [3] - https://airavata.apache.org/custos/ 
>> 
>> [4] - https://github.com/apache/airavata-custos 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Data Catalog Search API

[Christie, Marcus Aaron]

Hi Amila,

Thank you so much for your feedback! You asked a lot of good questions and made 
some great observations. I replied to your feedback in the google doc.

Thanks again,

Marcus

> On Jan 23, 2023, at 9:23 PM, Thejaka Amila J Kanewala 
>  wrote:
> 
> I added a few comments to the google doc. 
> Please let me know whether they make sense.
> 
> Thanks.
> Best Regards,
> Thejaka Amila Kanewala, PhD
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Data Catalog Search API

[Christie, Marcus Aaron]

> On Jan 18, 2023, at 12:26 PM, Thejaka Amila J Kanewala 
>  wrote:
> 
> Will be great if you could convert the design document into a google doc -- 
> it is easy to provide feedback in the google doc format.
> 

And here's the google doc link: 
https://docs.google.com/document/d/1itK9yGKlr1RSvCLyeujGM9Mts90h0TkUPPeIB5HsMT0/edit?usp=sharing

smime.p7s
Description: S/MIME cryptographic signature

Re: Data Catalog Search API

[Christie, Marcus Aaron]

ls/replica_catalog_models.thrift#L58
[2] https://about.zenodo.org/

> On Jan 18, 2023, at 12:26 PM, Thejaka Amila J Kanewala 
>  wrote:
> 
> You don't often get email from thejaka.am...@gmail.com. Learn why this is 
> important
> Hi Marcus,
> 
> Sorry for my lack of knowledge on this.
> 
> Just for my understanding, could you please define what a "data product" is ? 
> -- I can see that from the schema diagram it has an id and has a parent-child 
> relationship, but I would like to understand functionally what a "data 
> product" is.
> 
> In relation to the "Data product", when we do "createMetadataSchema(..., name 
> = "smilesdb")" -- does this create a "data product" named "smilesdb" ?
> Also, I am curious to know what motivated you to keep the metadata as a json 
> (other than PG's json indexing) ?
> 
> Cosmetic:
> Will be great if you could convert the design document into a google doc -- 
> it is easy to provide feedback in the google doc format.
> When i try to access 
> "https://raw.githubusercontent.com/apache/airavata-sandbox/master/gsoc2022/smilesdb/Migratio
> ns/data/molecule.json", i get a 404 error.
> 
> Thanks.
> Best Regards,
> Thejaka Amila Kanewala, PhD
> https://github.com/thejkane/agm
> http://valagamba.net/
> 
> 
> On Tue, Jan 17, 2023 at 9:42 AM Christie, Marcus Aaron  
> wrote:
> Hi All,
> 
> I've attached a design document for the search API of the redesigned Data 
> Catalog and I'm looking for some feedback on it.
> 
> Some context: for the Cybershuttle project, we're creating a redesigned data 
> catalog to store metadata about directories and files that may come from 
> several sources: user provided, instrument generated, generated as output 
> from a computation. Metadata about these data products may also come from 
> several sources.
> 
> The high-level requirements are:
> 
> - support searching and filtering for data products using schemaless metadata
> - supports bursts of writes, for example when scanning and registering all of 
> the files in a directory
> - basic CRUD operations on data products, including bulk operations
> - capture parent/child relationships (i.e., directory, sub-directory, file 
> relationships) and allow querying based on these relationships
> - basic CRUD operations on data product's metadata
> 
> Most of the basic CRUD operations are omitted from the design document. The 
> design document focuses on the search and querying API.
> 
> This redesign builds on Airavata's Replica Catalog and the DRMS Resource 
> Service [1] in airavata-data-lake.
> 
> The main difference in this design is that it uses the built-in JSON querying 
> and indexing capabilities of PostgreSQL. The goal is to support whatever 
> metadata is available as long as it is in JSON format and make it efficiently 
> searchable and filterable. Also, to make the API more developer friendly, the 
> API supports querying via SQL (which will be transformed to the actual 
> backend query using Apache Calcite).
> 
> Your feedback is most welcome.
> 
> Sincerely,
> 
> Marcus
> 
> 
> 
> [1] 
> https://github.com/apache/airavata-data-lake/blob/master/data-resource-management-service/drms-stubs/src/main/proto/resource/DRMSResourceService.proto
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Regarding airavata django portal error while running locally

[Christie, Marcus Aaron]

Hi Aishwarya,

Thanks for the screenshot. This was an issue with Node 18 (see 
https://issues.apache.org/jira/browse/AIRAVATA-3668). It is fixed on the 
'develop' branch but I've also just now merged it to the 'master' branch.

Can you please do a 'git pull' and then try again?

Thanks,

Marcus

> On Dec 14, 2022, at 3:22 AM, Aishwarya Sinhasane 
>  wrote:
> 
> Hello Marcus 
> 
> I am getting the following error while running build_js.bat file. I have 
> attached screenshot for your reference. 
> 
> Thanks & Regards
> Aishwarya Sinhasane
> 
> On Tue, 13 Dec 2022 at 09:25, Christie, Marcus Aaron  wrote:
> Hi Aishwarya,
> 
> Can you provide the output from running build_js.bat?
> 
> Thanks,
> 
> Marcus
> 
> > On Dec 12, 2022, at 6:07 PM, Aishwarya Sinhasane 
> >  wrote:
> > 
> > Hello Marcus
> > 
> > Thank you for your response. I have already run the build_js.bat file and 
> > it's saying that new_default_theme is loaded successfully. I think I have 
> > provided the screenshot for the same in previous email. If you think 
> > build_js.bat is not running properly then please let me know how to fix it. 
> > 
> > Thanks and Regards
> > Aishwarya Sinhasane 
> > 
> > On Mon, Dec 12, 2022, 5:33 PM Christie, Marcus Aaron  
> > wrote:
> > Hi Aishwarya,
> > 
> > You need to build the django portal frontend code. On Windows you'll need 
> > to run build_js.bat.
> > 
> > I realize that build_js.bat is missing from the README. I'll update it.
> > 
> > Thanks,
> > 
> > Marcus
> > 
> > > On Dec 11, 2022, at 4:12 AM, Aishwarya Sinhasane 
> > >  wrote:
> > > 
> > > Hello Everyone, 
> > > 
> > > I was running django portal locally for the seagrid desktop application. 
> > > I have to connect seagrid desktop application to create experiment page 
> > > from the molecule editor so that I can pass gaussian input file to the 
> > > gaussian experiment. But I am getting the following error. That there is 
> > > no web-pack.json file.I observed that there is no folder dist in the 
> > > common folder of static. I have attached the screenshots of the error 
> > > message. Can anybody please help me to resolve this issue if you are 
> > > available today?
> > > 
> > > Thanks and Regards
> > > Aishwarya Sinhasane
> > > 
> > 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Regarding airavata django portal error while running locally

[Christie, Marcus Aaron]

Hi Aishwarya,

Can you provide the output from running build_js.bat?

Thanks,

Marcus

> On Dec 12, 2022, at 6:07 PM, Aishwarya Sinhasane 
>  wrote:
> 
> Hello Marcus
> 
> Thank you for your response. I have already run the build_js.bat file and 
> it's saying that new_default_theme is loaded successfully. I think I have 
> provided the screenshot for the same in previous email. If you think 
> build_js.bat is not running properly then please let me know how to fix it. 
> 
> Thanks and Regards
> Aishwarya Sinhasane 
> 
> On Mon, Dec 12, 2022, 5:33 PM Christie, Marcus Aaron  wrote:
> Hi Aishwarya,
> 
> You need to build the django portal frontend code. On Windows you'll need to 
> run build_js.bat.
> 
> I realize that build_js.bat is missing from the README. I'll update it.
> 
> Thanks,
> 
> Marcus
> 
> > On Dec 11, 2022, at 4:12 AM, Aishwarya Sinhasane 
> >  wrote:
> > 
> > Hello Everyone, 
> > 
> > I was running django portal locally for the seagrid desktop application. I 
> > have to connect seagrid desktop application to create experiment page from 
> > the molecule editor so that I can pass gaussian input file to the gaussian 
> > experiment. But I am getting the following error. That there is no 
> > web-pack.json file.I observed that there is no folder dist in the common 
> > folder of static. I have attached the screenshots of the error message. Can 
> > anybody please help me to resolve this issue if you are available today?
> > 
> > Thanks and Regards
> > Aishwarya Sinhasane
> > 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Regarding airavata django portal error while running locally

[Christie, Marcus Aaron]

Hi Aishwarya,

You need to build the django portal frontend code. On Windows you'll need to 
run build_js.bat.

I realize that build_js.bat is missing from the README. I'll update it.

Thanks,

Marcus

> On Dec 11, 2022, at 4:12 AM, Aishwarya Sinhasane 
>  wrote:
> 
> Hello Everyone, 
> 
> I was running django portal locally for the seagrid desktop application. I 
> have to connect seagrid desktop application to create experiment page from 
> the molecule editor so that I can pass gaussian input file to the gaussian 
> experiment. But I am getting the following error. That there is no 
> web-pack.json file.I observed that there is no folder dist in the common 
> folder of static. I have attached the screenshots of the error message. Can 
> anybody please help me to resolve this issue if you are available today?
> 
> Thanks and Regards
> Aishwarya Sinhasane
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Custos Bare-metal deployment - Database Migration

[Christie, Marcus Aaron]

Hi Abhinav,

I wrote up some notes on making database schema migration scripts here: 
https://github.com/apache/airavata/blob/master/modules/registry/registry-core/README.md

I tried to use OpenJPA's capability to generate schema migration scripts. 
Unfortunately I ran into a lot of issues, and I wouldn't recommend that 
approach.

More recently, with a different project, I've had a lot of success using 
Liquibase to automate generating database schema migration scripts.

One key principle that's important to practice, is Expand/Migrate/Contract. See 
for more information: https://www.martinfowler.com/bliki/ParallelChange.html. 
The idea is to introduce new functionality and migrate all clients to it before 
removing the old functionality.  I've written up a draft of how to apply this 
principle to database migrations, see attached file. But this principle can 
apply to API evolution and deployments as well.



Airavata Deprecation_Migration Guidance.pdf
Description: Adobe PDF document


I hope this helps.

Thanks,

Marcus

> On Aug 4, 2022, at 5:58 PM, Abhinav Sinha  wrote:
> 
> Hello Marcus, all,
>  
> I am working on a plan to automate custos database migration for baremetal 
> deployment. Isuru pointed out that similar work was done for Airavata. I am 
> reaching out to ask if you could share with me how the migration was done for 
> Airavata – or maybe point me to the scripts that were used?
>  
> Thanks,
> Abhinav



smime.p7s
Description: S/MIME cryptographic signature

Ansible upgrade

[Christie, Marcus Aaron]

Hi All,

This is an FYI that the version of Ansible we're using has been updated to 
2.13.1. This version of Ansible requires at least version 3.8 of Python.

If you have an existing virtual environment for Ansible, you'll likely need to 
delete the ENV directory and recreate it with Python 3.8+. I've updated the 
README.md 
(https://github.com/apache/airavata/blob/master/dev-tools/ansible/README.md) 
with instructions on creating a virtual environment.


Thanks,

Marcus

smime.p7s
Description: S/MIME cryptographic signature

Re: [External] Error on setting up dev server using yarn

[Christie, Marcus Aaron]

Hi Shubham,

I just checked again and when I click on the Vue DevTools extension icon, I 
also get this same message. However, I can still access the Vue DevTools tab in 
Firefox dev tools. For whatever reason, this message seems to not be correct 
(could be a Vue 2 vs Vue 3 thing, not sure).

Can you try accessing the Vue DevTools tab in Firefox dev tools?

Thanks,

Marcus

> On Jul 25, 2022, at 8:14 PM, Bipin Kumar, Shubham  wrote:
> 
> Hi Marcus,
> 
> Thanks for the response. I did try that, like right now I'm building admin 
> and I can see whatever changes I make getting reflected on webapp but when I 
> click on my Vue DevTools extension icon it still says "Vue.js is detected on 
> this page.
> Devtools inspection is not available because it's in production mode or 
> explicitly disabled by the author."
> 
> 
> Regards,
> Shubham
> 
> 
> From: Christie, Marcus Aaron
> Sent: Monday, July 25, 2022 5:00 PM
> To: Airavata Dev
> Subject: Re: [External] Error on setting up dev server using yarn
> 
> Hi Shubham,
> 
> Only the Django app where you run yarn serve will be running in dev mode and 
> Vue DevTools should work for that app.
> 
> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/developing_frontend/
>  
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-airavata-django-portal.readthedocs.io%2Fen%2Flatest%2Fdev%2Fdeveloping_frontend%2F=05%7C01%7Cmachrist%40iu.edu%7C629b860bd0eb44dfe4b308da6e9be3e2%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637943913073174245%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=dUPbpjbNWhX19gdWt0Bsd%2ByiPTaymfBJ2uF5Kl0DqCY%3D=0>
> 
> Thanks,
> 
> Marcus
> 
>> On Jul 23, 2022, at 3:59 PM, Bipin Kumar, Shubham > <mailto:sbip...@iu.edu>> wrote:
>> 
>> I did try adding  Vue.config.devtools = true;  to main.js and store.js but 
>> still getting same error . I even tried running the python runserver command 
>> so that changes can take effect but to no avail.
>> 
>> please le me know if you want me to try something.
>> 
>> 
>> 
>> Regards,
>> Shubham
>> From: Bipin Kumar, Shubham mailto:sbip...@iu.edu>>
>> Sent: Saturday, July 23, 2022 3:24 PM
>> To: dev@airavata.apache.org <mailto:dev@airavata.apache.org> 
>> mailto:dev@airavata.apache.org>>
>> Subject: Re: [External] Re: Error on setting up dev server using yarn
>>  
>> Hi Dinuka,
>> 
>> Thanks for pointing that out . I can see the changes getting reflected on 
>> the 8000 port itself . One more question , I have installed vueDevTools to 
>> inspect in the browser but it throws an error "Vue.js is detected on this 
>> page.Devtools inspection is not available because it's in production mode or 
>> explicitly disabled by the author."
>> 
>> Can you please let me know if I have to make any changes for this.
>> 
>> 
>> Regards,
>> Shubham
>> From: Dinuka De Silva > <mailto:l.dinukadesi...@gmail.com>>
>> Sent: Saturday, July 23, 2022 2:36 PM
>> To: Airavata Dev mailto:dev@airavata.apache.org>>
>> Subject: Re: [External] Re: Error on setting up dev server using yarn
>>  
>> Hi Shubham,
>> 
>> About screenshot 1, there's nothing you have to do in 9000 port.
>> 
>> About screenshot 2, You are in the workspace. Click on the dropdown 
>> (Currently shows as "workspace:) in the app header and switch to settings 
>> then you will have the statistics menu.
>> 
>> Thanks & Regards,
>> Dinuka
>> 
>> On Sat, Jul 23, 2022 at 2:26 PM Bipin Kumar, Shubham > <mailto:sbip...@iu.edu>> wrote:
>> Hi Dinuka,
>> 
>> I'm using yarn and then yarn run serve . Django portal is active on 8000 
>> port.
>> Please refer screenshots for the same.
>> 
>> Also, I noticed admin privileges provided to my user id is gone . I'm no 
>> longer able to see statistics tab is local deployment after login.
>> 
>> 
>> Regards,
>> Shubham
>> From: Dinuka De Silva > <mailto:l.dinukadesi...@gmail.com>>
>> Sent: Saturday, July 23, 2022 1:56 PM
>> To: Airavata Dev mailto:dev@airavata.apache.org>>
>> Subject: [External] Re: Error on setting up dev server using yarn
>>  
>> This message was sent from a non-IU address. Please exercise caution when 
>> clicking links or opening attachments from external sources.
>> 
>> Hi Shubham,
>> 
>> Which commands are you using?
>> 
>> If you ran "yarn serve" on workspace directo

Re: [External] Error on setting up dev server using yarn

[Christie, Marcus Aaron]

Hi Shubham,

Only the Django app where you run yarn serve will be running in dev mode and 
Vue DevTools should work for that app.

https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/developing_frontend/
 


Thanks,

Marcus

> On Jul 23, 2022, at 3:59 PM, Bipin Kumar, Shubham  wrote:
> 
> I did try adding  Vue.config.devtools = true;  to main.js and store.js but 
> still getting same error . I even tried running the python runserver command 
> so that changes can take effect but to no avail.
> 
> please le me know if you want me to try something.
> 
> 
> 
> Regards,
> Shubham
> From: Bipin Kumar, Shubham 
> Sent: Saturday, July 23, 2022 3:24 PM
> To: dev@airavata.apache.org 
> Subject: Re: [External] Re: Error on setting up dev server using yarn
>  
> Hi Dinuka,
> 
> Thanks for pointing that out . I can see the changes getting reflected on the 
> 8000 port itself . One more question , I have installed vueDevTools to 
> inspect in the browser but it throws an error "Vue.js is detected on this 
> page.Devtools inspection is not available because it's in production mode or 
> explicitly disabled by the author."
> 
> Can you please let me know if I have to make any changes for this.
> 
> 
> Regards,
> Shubham
> From: Dinuka De Silva 
> Sent: Saturday, July 23, 2022 2:36 PM
> To: Airavata Dev 
> Subject: Re: [External] Re: Error on setting up dev server using yarn
>  
> Hi Shubham,
> 
> About screenshot 1, there's nothing you have to do in 9000 port.
> 
> About screenshot 2, You are in the workspace. Click on the dropdown 
> (Currently shows as "workspace:) in the app header and switch to settings 
> then you will have the statistics menu.
> 
> Thanks & Regards,
> Dinuka
> 
> On Sat, Jul 23, 2022 at 2:26 PM Bipin Kumar, Shubham  > wrote:
> Hi Dinuka,
> 
> I'm using yarn and then yarn run serve . Django portal is active on 8000 port.
> Please refer screenshots for the same.
> 
> Also, I noticed admin privileges provided to my user id is gone . I'm no 
> longer able to see statistics tab is local deployment after login.
> 
> 
> Regards,
> Shubham
> From: Dinuka De Silva  >
> Sent: Saturday, July 23, 2022 1:56 PM
> To: Airavata Dev mailto:dev@airavata.apache.org>>
> Subject: [External] Re: Error on setting up dev server using yarn
>  
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Hi Shubham,
> 
> Which commands are you using?
> 
> If you ran "yarn serve" on workspace directory, that's been watched by the 
> Django portal (8000). In that case you do not have to go to 9000 port. What 
> you see on 8000 port is the everything including the workspace (9000).
> 
> I hope this helps.
> 
> Thanks & Regards,
> Dinuka
> 
> On Sat, 23 Jul 2022, 13:47 Bipin Kumar, Shubham,  > wrote:
> Hi Marcus/Dinuka,
> 
> I was trying to debug using  the dev server setup for django portal which 
> comes up on port 9000 using yarn command . After following the instructions 
> I'm getting   Cannot GET / error on the  http://localhost:9000/ 
>   . I tried few solutions recommended on several 
> portals but couldn't get it working .Can you please point out to what I might 
> be doing wrong .
> 
> I do get this error in vscode "Vetur can't find `tsconfig.json` or 
> `jsconfig.json` in 
> /Users/shubhamkumar/Desktop/GSOC/django-portal/django_airavata/apps/workspace."
>   but I'm not sure if its related.
> 
> 
> 
> Regards,
> Shubham



smime.p7s
Description: S/MIME cryptographic signature

Re: Clarification on dashboard for quick statistics development

[Christie, Marcus Aaron]

Hi Shubham,

My apologies for taking so long to get back to you, and thanks to Suresh for 
following up.

I agree with everything that Suresh said. Also, for developing new Airavata 
APIs take a look at 
https://github.com/apache/airavata/tree/develop/modules/ide-integration 
 for 
setting up a dev environment where you have a local Airavata API server 
instance and local Django Portal instance.

Thanks,

Marcus

> On Jun 29, 2022, at 9:58 PM, Suresh Marru  wrote:
> 
> Hi Shubham,
> 
> See below: 
> 
>> On Jun 28, 2022, at 12:39 PM, Bipin Kumar, Shubham > > wrote:
>> 
>> Hi Marcus,
>> 
>> I had the following question on dashboard developments . Can you please 
>> clarify these:
>> Can I use React for frontend development or is it compulsory to use VueJs?
> 
> As such we would not have opinion, but in this case please use VueJS since 
> rest of the apps are built in it and It. 
> 
>> Should the dashboard be an independent application, or it should be 
>> integrated as a page in Django portal?
> It should be a Django app to be deployed within the Airavata Django 
> Framework. 
> 
>> For getting the data like user details, experiment details or resources 
>> details should we establish direct connections to respective databases or 
>> use the Airavata API's to fetch those details. In case there aren't any 
>> API's to support the data needed what should be the course of action?
> Always only use API’s we suggest database interactions do not go beyond the 
> respective components. So if you need to get any data Client -> Airavata 
> API->Component CPI -> Database
> 
> Suresh
> 
>> 
>> Regards,
>> Shubham
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: [airavata-user-docs] branch main created (now 55a1907)

[Christie, Marcus Aaron]

Hi Dev,

Eroma is working on a project to move all of the user level documentation to a 
separate repo/site. The plan is to keep airavata-docs for developer-oriented 
documentation and airavata-user-docs for user documentation.

Thanks,

Marcus

> On Apr 1, 2022, at 11:35 AM, machris...@apache.org wrote:
> 
> This is an automated email from the ASF dual-hosted git repository.
> 
> machristie pushed a change to branch main
> in repository 
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%2Fairavata-user-docs.gitdata=04%7C01%7Cmachrist%40iu.edu%7Cab8902144ba44288680808da13f56049%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637844241842549600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=V87PGfTF5IchcCY63%2FnKDmGywqKJY3HpTc9dwG1RSTQ%3Dreserved=0.
> 
> 
>  at 55a1907  Moving 0.18 to Previous Releases section
> 
> This branch includes the following new commits:
> 
> new 79b9984  adding a doc index file to trigger github integrations
> new eed63a2  adding airavata documentation contributed by Eroma on github 
> scigap docs folder
> new 3fd4bb9  Added WSO2 IS related configurations in pga_config.php
> new e36bf37  adding IS config configuration in pga
> new 7e01591  Disambiguating scigap and airavata
> new 3ea5a69  Modifications to generate DBs just as its presented in 
> airavata-server.properties. This will reduce the user confusion and 
> simplifies the steps.
> new 441e3cb  Merge remote-tracking branch 'origin/master'
> new 8816c10  Fixed the Ubuntu dependency installation link
> new 73599cd  Fixed an error in one of the MYSQL commands
> new 6a8e54b  Fixed some of the links in the documentation
> new 297c904  Added a link button for Ubuntu for easy navigation
> new d598581  Removed extra Compute Resource link
> new b682138  Modifications to some sections to make moe user friendly
> new e21a583  Modifications to SQL commands to make them appear in a 
> single line
> new b251db6  Changed the color of 'Question' and 'Answer'
> new 9448468  Adding WSO2 IS installation Steps
> new 5872766  Adding WSO2 IS installation Steps
> new d51a39d  Moving IS installation under installation menu
> new 4bfb96a  Moving IS installation under installation menu
> new 5bd89a1  Minor modifications to the heading size, and sub heading 
> size through out documentation and also added Apache Airavata in place on 
> Airavata at heading level
> new 0e848b6  Added mysql.jar download path
> new ecdb94b  Added mysql.jar download path
> new 4350e89  Modifications to Airavata Introduction
> new 787932d  Adding API methods for Admin Dashboard Functions
> new cfb04ee  Fixed a Typo on API documentation
> new acf1994  Adding descriptions to API methods and linking with 
> generated API documentation
> new 0a9bdfe  Adding descriptions to Admin Dashboard API methods
> new 68bd2d7  Adding descriptions to Admin Dashboard API methods
> new 390a7f4  Adding descriptions to Admin Dashboard API methods
> new 514b4c0  Corrections and clarifications to the instructions.
> new 7462f68  Fixing a error in WSO2 claim configuration documentation
> new af6cefe  Fixing a error in WSO2 claim configuration documentation
> new b441bcd  Gateway Admin Maintenance Steps
> new 9d0e808  Modified a Typo
> new 5200bb1  Corrected the WSO2 IS configuration steps in 
> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwso2.org%2Fclaimsdata=04%7C01%7Cmachrist%40iu.edu%7Cab8902144ba44288680808da13f56049%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637844241842549600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=Cp68AQPVSEyBJKFLZH8EOXbOli%2FPtU%2FGG4%2B%2BPtlNCZU%3Dreserved=0
>  and 
> https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwso2.org%2Foidc%2Fclaimdata=04%7C01%7Cmachrist%40iu.edu%7Cab8902144ba44288680808da13f56049%7C1113be34aed14d00ab4bcdd02510be91%7C0%7C0%7C637844241842549600%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000sdata=hcQpDKLKS0Ur43VlW%2BjJk77PAmF8upMbXF9wkxYzGLE%3Dreserved=0
> new 707208b  Common Gateway maintenance scenarios
> new f95ec0e  Adding Maintenance Instruction Pages
> new 487f77e  Adding PGA upgrade steps
> new bfeeeb4  Adding PGA upgrade steps
> new 1b4410f  Adding Airavata upgrade steps page
> new 493bc25   Added stesp to upgrade Airavata
> new 74a4125   Added stesp to upgrade PGA
> new 2cbcc70  Added link to Airavata update page and typo fix
> new d4adcf4  Added link to Airavata update page and typo fix
> new d2e791e  Modifications to airavata-server.properties detail section. 
> Fixe the URL for airavata-properties page.
> new df4e840  Formatting changes and minor corrections
> new 223e5da  Modifications to 

Re: [Airavata] Fetching intermediate outputs for an running experiment

[Christie, Marcus Aaron]

Hi Dimuthu,

Thanks for working on this and testing this functionality. I think this will be 
a great new feature.

On the PR I also added an API method to fetch the status of an intermediate 
output fetching process: 

ProcessStatus getIntermediateOutputProcessStatus(AuthzToken authzToken, 
String airavataExperimentId, List outputNames)

This will make it a little easier for clients to figure out the status of the 
process.

As for UI, you're right. Currently the output file UI only displays when an 
experiment is COMPLETED. I'm thinking of having a panel for each output file 
that will display once the experiment is EXECUTING. Each panel will have a 
button to fetch the current file. When the fetch is executing the button is 
disabled and a spinner displays with a message that the file is being fetched. 
Once the file is fetched, if it is a text file (and maybe within some max file 
size limit) then it will display in the panel and the user can scroll through 
the file. This is the main use case, retrieving the current value of a log file 
or STDOUT or something similar. If the file is too large or binary then instead 
of displaying the file directly there will be a download button.  That's the 
general idea.

Some other things, maybe lower priority:
- I need to think about what to do for a URI_COLLECTION (maybe a tabbed 
interface). 
- Output view providers, as long as they can work with partial files, should be 
able to be invoked on these intermediate files as well and so I also need to 
think about how to integrate those.

Thanks,

Marcus

> On Jan 2, 2022, at 11:03 PM, DImuthu Upeksha  
> wrote:
> 
> Hi All
> 
> Marcus and I were working on the feature of grabbing output files from an 
> already running experiment so that users can determine whether the experiment 
> is running as expected without waiting for it to complete. This will be 
> tremendously helpful for long-running and resource-intensive jobs in order to 
> determine possible early terminations in case of an input or convergence 
> issue. We added a new API [1] to invoke this feature and this will create a 
> separate process that is only running a subset of output data transfer 
> workflows of the parent process. A sample client code might look like below.
> List outputNames = new ArrayList<>();
> outputNames.add("Gaussian-Application-Output");
> apiClient.fetchIntermediateOutputs(token,"exp-id", 
> outputNames, 
> "PROCESS_98a8c45c-998c-4998-91e4-18ce915ed13e");
> Any failure to this new process will not affect the main experiment but users 
> can see those errors in the experiment statistics page under the new process 
> section. The attached screenshot shows how the new process might look like on 
> the statistics page. Output data will be placed in a separate directory named 
> 'intermediate' on the storage side. 
> 
> I merged this code to dev environment and it is ready for testing purposes. 
> This PR [2] will reflect all the file changes from API to Helix level.
> 
> Marcus, 
> We might need to add new buttons / UI components to make this available at 
> the portal level and we can discuss it here if that makes sense. 
> 
> Thanks and happy new year
> Dimuthu 
> 
> [1] 
> https://github.com/apache/airavata/blob/AIRAVATA-3549/thrift-interface-descriptions/airavata-apis/airavata_api.thrift#L989
>  
> 
> [2] https://github.com/apache/airavata/pull/276 
> 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: PyPI project ownership

[Christie, Marcus Aaron]

Looks like the invite expired. I've resent them.

> On Sep 2, 2021, at 5:55 PM, Christie, Marcus Aaron  wrote:
> 
> Hi Dimuthu, Isuru,
> 
> I think it's important that PyPI projects not be owned by only a single 
> Airavata committer so that updates can be pushed to PyPI even if one of us is 
> away.
> 
> To that end, I've invited you both to be co-owners of two PyPI projects I 
> created:
> 
> https://pypi.org/project/airavata-django-portal-sdk/
> https://pypi.org/project/airavata-python-sdk/
> 
> You should have received an invitation email.
> 
> Likewise, can I be made a co-owner of the custos SDK on PyPI?
> 
> Let me know if you have any questions.
> 
> 
> Thanks,
> 
> Marcus



smime.p7s
Description: S/MIME cryptographic signature

PyPI project ownership

[Christie, Marcus Aaron]

Hi Dimuthu, Isuru,

I think it's important that PyPI projects not be owned by only a single 
Airavata committer so that updates can be pushed to PyPI even if one of us is 
away.

To that end, I've invited you both to be co-owners of two PyPI projects I 
created:

https://pypi.org/project/airavata-django-portal-sdk/
https://pypi.org/project/airavata-python-sdk/

You should have received an invitation email.

Likewise, can I be made a co-owner of the custos SDK on PyPI?

Let me know if you have any questions.


Thanks,

Marcus

smime.p7s
Description: S/MIME cryptographic signature

Re: Django - CI Logon setup

[Christie, Marcus Aaron]

Hi Josh,

Here are the notes I have on adding CILogon as a Identity Provider in Keycloak:

• Log into the Keycloak and select the realm
• Create OIDC Identity Provider in Keycloak
• set the alias to something meaningful, like cilogo
• This alias will be used as the value of the kc_idp_hint query 
parameter to link directly to CILogon
• go to https://cilogon.org/oauth2/register
• the callback url to use is listed in Keycloak as the Redirect 
URI (for example: 
https://iam.scigap.org/auth/realms/seagrid/broker/cilogon/endpoint)
• home URL is the URL of the website (for example: 
https://seagrid.org)
• check all of the Scopes
• After submitting the registration you will get a page with a client 
id and secret. Enter these into the Keycloak page
• Also copy them to a secure location since you can't retrieve 
them later
• First login flow: first broker login
• Enable Trust Email
• Authorization URL: https://cilogon.org/authorize 
• Token URL: https://cilogon.org/oauth2/token
• Userinfo URL: https://cilogon.org/oauth2/userinfo 
• no logout URL
• Default scopes: openid email profile org.cilogon.userinfo
• Add the following attribute mappers
• family_name
• name: family_name
• Mapper Type: Attribute Importer
• claim: family_name
• User Attribute Name: lastName
• given_name
• name: given_name
• Mapper Type: Attribute Importer
• claim: given_name
• User Attribute Name: firstName
• Claim mapping documentation: 
http://www.keycloak.org/docs/2.5/server_admin/topics/identity-broker/mappers.html
 


I'll point out though that long term we're moving away from manually creating 
the CILogon client to automation provided by Airavata Custos [1] which 
automatically registers a CILogon client for tenants.

[1] https://airavata.apache.org/custos/ 


> On Feb 13, 2021, at 5:22 PM, Josh Seamans  
> wrote:
> 
> Hello, I am with the UNL Capstone project and I have been looking into how CI 
> Logon is set up on the Django Airavata gateway found here: Home 
> (airavata.org) 
>  
> I was wondering if there was any instructions that was found to set that up?



smime.p7s
Description: S/MIME cryptographic signature

Re: Django PyMol Output View

[Christie, Marcus Aaron]

Hi Evan,

Just to clarify, no PyMol viewer has been implemented as far as I know, that 
was just an idea of the kind of output viewer that could be created.

I'm glad that you are interested in implementing one and I'd be more than happy 
to answer your questions.

Thanks,

Marcus

> On Feb 3, 2021, at 12:44 PM, Evan Palmer  wrote:
> 
> Hello,
> 
> I am a student working with the University of Nebraska-Lincoln on the 
> development of a science gateway using the Airavata-Django Portal. We are 
> currently looking into implementing an interactive molecule viewer (i.e., 
> PyMol or JMol) as a custom output view for an application. While reading 
> through the Gateway tutorial 
> ,
>  I noticed that PyMol was an option that had been implemented before. I am 
> curious if there is anyone that we can contact to discuss some of our 
> questions regarding the implementation of PyMol, or other interactive 
> molecule viewers into the Django Portal.
> 
> Thank you,
> Evan Palmer



smime.p7s
Description: S/MIME cryptographic signature

Re: Data Catalog into its own repo

[Christie, Marcus Aaron]

+1

> On Nov 9, 2020, at 9:35 AM, Marru, Suresh  wrote:
> 
> Hi All,
> 
> Airavata Experiment catalog evolved over time and though the replica catalog 
> and data product models are stand alone, they are buried to use them outside 
> the experiment context. Any objections to refactor experiment catalog and 
> make data catalog a first class repo, in the lines of Custos and MFT?
> 
> Cheers,
> Suresh



smime.p7s
Description: S/MIME cryptographic signature

Re: [airavata-django-portal] Bugs in username and email validation

[Christie, Marcus Aaron]

Hi Dinuka,

Regarding #1, yes, definitely a bug.  However, you shouldn't use reset password 
when logging in with CILogon. You would change your password with the 
institution you use to login with CIlogon.

Regarding #2, yes, we allow users to use the same email address to signup.  
Going forward, and especially with Custos, we may want to rethink that or make 
it per-tenant customizable. Some gateways like Galaxy don't allow multiple 
accounts with the same email address.

> On Oct 30, 2020, at 10:43 AM, Dinuka Desilva  
> wrote:
> 
> Hi Everyone,
> 
> I was just playing around the django portal [1] and found the following 
> issues. 
> 
> 1) I created an account using the CI logon and then I tried to change the 
> password. I entered the email address as the username and it actually 
> validates the username not to have stops. I think that validation shouldn't 
> be there if the email is taken as the username. If not, I would like to know 
> what's taken as the username.
> 
> 
> One other thing, I think it makes sense to get the email address instead of 
> the username at the forgot password.
> 
> 2) At the same time, I signed up using the manual signup and it allowed me to 
> use the same email address to signup again. Now, I have two accounts with the 
> same email address.
> 
> Regards,
> Dinuka
> 
> [1] https://django.seagrid.org/ 


smime.p7s
Description: S/MIME cryptographic signature

Re: Regarding dockerization of Custos Admin Portal

[Christie, Marcus Aaron]

Hi Bhakti,

You might take a look at the airavata-django-portal Dockerfile [1]

This is really only used for development. It runs the dev server which isn't 
appropriate for production. But it's a start.

For production you probably want to look at using Docker with nginx and a wsgi 
server like Gunicorn [2] or uWSGI [3]. 

Might want to also think about how to handle Django's database. You can use 
MariaDB or some other database server. I think you are using a sqlite database 
file. That may be fine but you'll probably want to mount a volume to hold the 
file so it lives outside the container.

I hope that helps. Let me know if you have any questions.

Thanks,

Marcus


[1] https://github.com/apache/airavata-django-portal/blob/master/Dockerfile 

[2] https://docs.djangoproject.com/en/3.1/howto/deployment/wsgi/gunicorn/ 

[3] https://docs.djangoproject.com/en/3.1/howto/deployment/wsgi/uwsgi/ 




> On Nov 4, 2020, at 11:09 AM, Narvekar, Bhakti Sunil  wrote:
> 
> Hello,
>  
> We are trying to dockerize the Custos Admin Portal which is implemented using 
> Django and Vue.js. It would be great if anyone could give any suggestions on 
> this.
>  
>  
> Sincerely,
> Bhakti Narvekar



smime.p7s
Description: S/MIME cryptographic signature

Re: Airavata Djano login issue

[Christie, Marcus Aaron]

Hi Josh,

This is known issue: https://issues.apache.org/jira/browse/AIRAVATA-3376 


The current workaround is to use Python 3.6.

FYI, I'm planning on merging a fix to the develop branch this week.

Thanks,

Marcus

> On Oct 25, 2020, at 4:04 PM, Josh Seamans  
> wrote:
> 
> Hello,
> I am a part of the UNL team. I am working with the Django Airavata portal. My 
> group is trying to get logged into the /CMS portion but are having troubles 
> logging in. Aside from this when we try to run it outside of docker we are 
> experiencing issues building with the error: 
> “django.db.utils.OperationalError: Problem installing fixtures: no such 
> table: wagtailcore_page__old”
>  
> This occurs when running the command “python manage.py load_cms_data 
> new_default_theme”



smime.p7s
Description: S/MIME cryptographic signature

Re: [External] [AIRAVATA-3379] airavata-ruby-sdk

[Christie, Marcus Aaron]

Yeah, +1. This sounds similar to PyPI, you can delegate other users to be able 
to publish. As long as we have multiple dev team members with access I think 
we'll be fine.

> On Oct 26, 2020, at 12:40 PM, Marru, Suresh  wrote:
> 
> Sounds like a good plan Dinuka, +1.
> 
> Suresh
> 
>> On Oct 26, 2020, at 12:37 PM, Dinuka Desilva > > wrote:
>> 
>> Hi Suresh,
>> 
>> What I noticed was that there's no as one account even for apache [2] 
>> itself. Instead, individual users have published the packages. Whoever holds 
>> the name, owns it. Also, it looks possible to have multiple owners [2]. 
>> 
>> Maybe we could follow the same. I could publish first and then share the 
>> ownership with others.
>> 
>> Regards,
>> Dinuka
>> 
>> [2] https://rubygems.org/gems/thrift 
>> [3] https://rubygems.org/gems/whimsy-asf 
>> 
>> On Mon, Oct 26, 2020 at 9:54 PM Marru, Suresh > > wrote:
>> Hi Dinuka,
>> 
>> Can you please research which of the Apache Software Foundation projects 
>> https://projects.apache.org/  have ruby 
>> artifacts published and find the relevant documentation on how they are 
>> doing account management and official artifact publishing?
>> 
>> Thanks,
>> Suresh
>> 
>>> On Oct 26, 2020, at 12:17 PM, Dinuka Desilva >> > wrote:
>>> 
>>> This message was sent from a non-IU address. Please exercise caution when 
>>> clicking links or opening attachments from external sources.
>>> 
>>> Hi Suresh,
>>> 
>>> For publishing the packages, it's required to have an account [1]. I would 
>>> say having an account for the lab would be better than pushing them through 
>>> a personal account. What do you think?
>>> 
>>> One more thing I would like to clarify was the name of the package. I named 
>>> it just "airavata". And the default module is "Airavata" and inside that 
>>> there's a sub module called "Client". So, in a way it's extendible if 
>>> needed.
>>> 
>>> Example usage :- 
>>> 
>>> require "thrift"
>>> require "airavata"
>>> 
>>> transport = Thrift::BufferedTransport.new(Thrift::Socket.new('localhost', 
>>> 9930))
>>> protocol = Thrift::BinaryProtocol.new(transport)
>>> airavataApiClient = Airavata::Client.new(protocol)
>>> transport.open()
>>> 
>>> airavataApiClient.isUserExists(authzToken, gatewayId, userName)
>>> 
>>> 
>>> Regards,
>>> Dinuka
>>> 
>>> [1] https://rubygems.org 
>>> On Mon, Oct 26, 2020 at 8:10 PM Dinuka Desilva >> > wrote:
>>> Hi Eric,
>>> 
>>> Thanks for the feedback. I updated the PR accordingly and managed to test 
>>> it locally too.
>>> 
>>> Regards,
>>> Dinuka
>>> 
>>> On Fri, Oct 23, 2020 at 1:50 AM Suresh Marru >> > wrote:
>>> Thank you Eric, we will follow up on the PR.
>>> 
>>> Suresh
>>> 
 On Oct 22, 2020, at 4:13 PM, Franz, Eric >>> > wrote:
 
 Suresh,
  
 I posted a comment here 
 https://github.com/apache/airavata/pull/265#issuecomment-714734200 
 
  
 And am happy to help however I can.
  
 Thanks,
 Eric
  
 ---
 Eric Franz, Gateways Lead Engineer
 Ohio Supercomputer Center
 An Ohio Technology Consortium (OH-TECH) Member
 1224 Kinnear Road
 Columbus, OH 43212
 email: efr...@osc.edu 
  
 From: Suresh Marru mailto:sma...@apache.org>>
 Date: Thursday, October 22, 2020 at 12:18 PM
 To: Airavata Dev >>> >, "Franz, Eric" >>> >
 Subject: Re: [AIRAVATA-3379] airavata-ruby-sdk
  
 Hi Eric,
  
 Do you have suggestions on how ruby SDK’s should be distributed? Once 
 Dinuka packages thrift generated ruby libraries and adds some higher order 
 simplification API’s, can you please advise on what is the best practice 
 for creating a gem and publishing it to a gem repo?
  
 Thanks,
 Suresh
 
 
> On Oct 21, 2020, at 3:02 PM, Dinuka Desilva  > wrote:
>  
> Hi Everyone,
>  
> Following the other client sdks [1] available, I'm going to enable the 
> ruby sdk [2] for the airavata api. Meanwhile, I have a couple of points 
> to discuss.
>  
> 1) Should we follow a similar pattern for additional code (apart from 
> what's generated from thrift) introduced to the clients regardless of 
> language? The feasibility is also one aspect to be discussed.
>  
> 2) What's the process of publishing the clients to public registries?
>  
> Regards,
> Dinuka
>  
> [1] 
> https://github.com/apache/airavata/tree/master/airavata-api/airavata-client-sdks
>  
> 

Re: [em-center-gateway] MFT requirements for EMC portal

[Christie, Marcus Aaron]

> On Oct 20, 2020, at 9:00 AM, Dinuka Desilva  wrote:
> 
> Looking at the intention of MFT, I wonder if this is outside MFT scope. I 
> believe there's no permanent storage of the files in MFT. Instead, MFT is a 
> managed bridge for transferring files between two locations that could be 
> from the same or different protocols. 
> 

Hi Dinuka,

I'm wondering the same thing. If it is in scope for MFT to list directories and 
provide directory management in addition to getting and putting files, then we 
could use the user storage management UIs in the Django portal. Those UIs 
leverage the user_storage SDK module on the backend [1] so if the user_storage 
functions can be implemented using the MFT API it would work.

[1] 
https://airavata-django-portal-sdk.readthedocs.io/en/latest/#module-user_storage
 




smime.p7s
Description: S/MIME cryptographic signature

Re: Experiment Outputs -- Display Link?

[Christie, Marcus Aaron]

Hi Luke,

Glad you were able to get it working. Yes that issue has been pesky for us, but 
I have the code fixed on a branch. When we have bandwidth to do the testing, 
I'll get it merged.

> On Oct 18, 2020, at 2:49 PM, Luke Peyralans  wrote:
> 
> Hi Marcus--
> 
> Thank you! The settings file you sent worked perfectly. The only problem I 
> ran into was:
> 
> https://issues.apache.org/jira/browse/AIRAVATA-3376 
> <https://issues.apache.org/jira/browse/AIRAVATA-3376>
> 
> ... which I was able to resolve following:
> 
> https://issues.apache.org/jira/browse/AIRAVATA-3376 
> <https://issues.apache.org/jira/browse/AIRAVATA-3376>
> 
> Now that I have a local django-portal dev environment connected to the Phasta 
> SciGap I will work with my team on the Custom Output View Provider.
> 
> Thanks!
> Luke
> 
> 
> 
> From: Christie, Marcus Aaron
> Sent: Wednesday, October 14, 2020 3:25 PM
> To: Airavata Dev
> Subject: Re: Experiment Outputs -- Display Link?
> 
> Hi Luke,
> 
> I followed up with you via a separate email with a settings_local.py file you 
> can use to setup a local Phasta Django portal dev environment.
> 
> Please use the airavata-django-portal README to get setup. Then see 
> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/
>  
> <https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/>
>  for detailed documentation on creating a custom output view provider.
> 
> Let me know if you run into any issues. And yes we can definitely do a screen 
> share if you run into issues.
> 
> Thanks,
> 
> Marcus
> 
>> On Oct 12, 2020, at 5:28 PM, Luke Peyralans > <mailto:lpeyr...@uoregon.edu>> wrote:
>> 
>> Hi Marcus, that would be awesome if I could have some help setting up a 
>> local dev env. Would you recommend I just follow the instructions on 
>> airavata-django-portal github page? If I run into issues can we do a screen 
>> share? 
>> 
>> Thank you!
>> 
>>> On Oct 12, 2020, at 2:15 PM, Christie, Marcus Aaron >> <mailto:machr...@iu.edu>> wrote:
>>> 
>>> Hi Eugene,
>>> 
>>> Yes it can. The way it works is that the custom output view is packaged as 
>>> a Python package and then it can be installed into the hosted Django portal 
>>> virtual environment.
>>> 
>>> See 
>>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/
>>>  
>>> <https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/>
>>>  for more details.
>>> 
>>> I can also help you setup a local development environment that mirrors 
>>> phasta.scigap.org <http://phasta.scigap.org/> so you can quickly iterate on 
>>> the custom output view.
>>> 
>>> 
>>> - Marcus
>>> 
>>> 
>>>> On Oct 9, 2020, at 5:14 PM, Eugene Walker >>> <mailto:eugenesunsetwal...@gmail.com>> wrote:
>>>> 
>>>> Can this be utilized on the airavata hosted service, ie phasta.scigap.org 
>>>> <http://phasta.scigap.org/> ?
>>>> 
>>>>> On Oct 9, 2020, at 1:49 PM, Christie, Marcus Aaron >>>> <mailto:machr...@iu.edu>> wrote:
>>>>> 
>>>>> Hi Luke,
>>>>> 
>>>>> Not as an Airavata output, but our Django Portal supports a link type 
>>>>> custom output view. [1]  Essentially this involves creating a Python 
>>>>> function that takes the experiment metadata and the output file and 
>>>>> returns a label and a URL.
>>>>> 
>>>>> Let me know if you have any questions.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Marcus
>>>>> 
>>>>> [1] 
>>>>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link
>>>>>  
>>>>> <https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link>
>>>>> 
>>>>>> On Oct 6, 2020, at 12:49 PM, Luke Peyralans >>>>> <mailto:lpeyr...@uoregon.edu>> wrote:
>>>>>> 
>>>>>> Is it possible to include a link as an experiment output? I would like 
>>>>>> to upload some experiment data to a third-party hosted service that 
>>>>>> creates interactive visualizations, and then provide a *unique link* 
>>>>>> that would let a user access these visualizations upon conclusion of the 
>>>>>> experiment.



smime.p7s
Description: S/MIME cryptographic signature

Re: Question about generating and using Airavata Ruby client

[Christie, Marcus Aaron]

Hi Eric,

In addition to what Isuru said, I wanted to clear up a point about the 
JavaScript SDK. That's an SDK to the Airavata Django Portal REST API which is a 
kind of bridge to the Thrift API via the airavata-python-sdk. That is, the 
JavaScript SDK doesn't call directly to the Thrift API.

In ascii art:

JS SDK ---> Django Portal REST API endpoints ---> Airavata Python Thrift Stubs 
---> Airavata API

Just to give an example, the following from the JavaScript SDK:

> const loadAppInterface = 
> services.ApplicationInterfaceService.retrieve({lookup: appInterfaceId});
> 
> 

Makes a GET request to the /api/application-interfaces// REST 
endpoint. This endpoint is implemented in the backend by 
https://github.com/apache/airavata-django-portal/blob/master/django_airavata/apps/api/views.py#L701-L703
 


def get_instance(self, lookup_value):
return self.request.airavata_client.getApplicationInterface(
self.authz_token, lookup_value)

which as seen above calls the getApplicationInterface Airavata API method.

I hope that helps. As Isuru says, looking at the airavata-python-sdk and 
perhaps especially the samples folder will make it clearer how to use the API.

Thanks,

Marcus

> On Oct 16, 2020, at 11:58 AM, Isuru Ranawaka  wrote:
> 
> Hi Eric,
> 
> Glad to hear that you are trying out Airavata services. The way you followed 
> is correct. Please see the below steps on how to use Airavata thrift stubs.
> 
>  generate_thrift_stubs.sh 
> .
>  supports c++, Java, and Python by default. For the Ruby,   we need to add a 
> script to generate ruby files (I assume you correctly did It)
> Once you run, It should generate Ruby stubs and data models (Those are the 
> ones you listed above)
> Ruby stubs and data models are directly generated from thrift descriptions 
> and currently, we don't have a Ruby template project to automatically 
> organize them into a usable format.
> We need to copy those files into a separate Ruby project under  
> airavata-client-sdks 
> 
>as airavata-ruby- sdk and develop some ruby clients to utilize generated 
> stubs and data models to connect with the Airavata services.
> You might have to look into airavata-python-sdk 
> 
>  to get some idea. 
> We can help you with developing the Ruby SDK to use Airavata services and we 
> will initialize the Ruby project on the Airavata git repo and you can look 
> into that.
> 
> thanks
> Isuru
> 
> 
> 
> 
> 
>  
> 
> On Fri, Oct 16, 2020 at 11:19 AM Franz, Eric  > wrote:
> I was attempting to experiment with the testdrive Airavata instance from the 
> Gateways tutorial. Since we have already setup an echo app and there is an 
> example of using client sdk API library (in JavaScript) to both get a list of 
> the running experiments and create and launch a new experiment, I thought I’d 
> see if I could generate from the thrift idls the Ruby client library and 
> plugin the API keys required to execute the same actions using Ruby.
> 
>  
> 
> These are the steps I took. I cloned g...@github.com:apache/airavata.git and 
> modified thrift-interface-descriptions/generate-thrift-stubs.sh 
> 
>  to add the generation of Ruby stubs. I copied the generate_python_stubs 
> function and then change the --gen py to --gen rb and the location of the 
> generation directory. I just had the target for now be 
> airavata-api/airavata-client-sdks/airavata-ruby-sdk/airavata though I’d 
> imagine in a proper setup the resulting airavata-ruby-sdk would be structured 
> as a ruby gem with the stubs being added to the lib directory for that gem 
> like a lib/airavata/ directory.
> 
>  
> 
> Regardless the result was a list of Ruby files reprinted at the bottom of the 
> email. Presumably I would also need the Ruby thrift gem to install. However, 
> I realize I wasn’t actually sure where to go at this point, assuming that the 
> steps I had taken were reasonable.
> 
>  
> 
> I wasn’t sure what Python client sdk classes and methods corresponded to the 
> JavaScript example from the tutorial (relevant parts copied below):
> 
>  
> 
> 
> 
> 

Re: Experiment Outputs -- Display Link?

[Christie, Marcus Aaron]

Hi Luke,

I followed up with you via a separate email with a settings_local.py file you 
can use to setup a local Phasta Django portal dev environment.

Please use the airavata-django-portal README to get setup. Then see 
https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/
 
<https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/>
 for detailed documentation on creating a custom output view provider.

Let me know if you run into any issues. And yes we can definitely do a screen 
share if you run into issues.

Thanks,

Marcus

> On Oct 12, 2020, at 5:28 PM, Luke Peyralans  wrote:
> 
> Hi Marcus, that would be awesome if I could have some help setting up a local 
> dev env. Would you recommend I just follow the instructions on 
> airavata-django-portal github page? If I run into issues can we do a screen 
> share? 
> 
> Thank you!
> 
>> On Oct 12, 2020, at 2:15 PM, Christie, Marcus Aaron  wrote:
>> 
>> Hi Eugene,
>> 
>> Yes it can. The way it works is that the custom output view is packaged as a 
>> Python package and then it can be installed into the hosted Django portal 
>> virtual environment.
>> 
>> See 
>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/
>>  
>> <https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/>
>>  for more details.
>> 
>> I can also help you setup a local development environment that mirrors 
>> phasta.scigap.org <http://phasta.scigap.org/> so you can quickly iterate on 
>> the custom output view.
>> 
>> 
>> - Marcus
>> 
>> 
>>> On Oct 9, 2020, at 5:14 PM, Eugene Walker >> <mailto:eugenesunsetwal...@gmail.com>> wrote:
>>> 
>>> Can this be utilized on the airavata hosted service, ie phasta.scigap.org 
>>> <http://phasta.scigap.org/> ?
>>> 
>>>> On Oct 9, 2020, at 1:49 PM, Christie, Marcus Aaron >>> <mailto:machr...@iu.edu>> wrote:
>>>> 
>>>> Hi Luke,
>>>> 
>>>> Not as an Airavata output, but our Django Portal supports a link type 
>>>> custom output view. [1]  Essentially this involves creating a Python 
>>>> function that takes the experiment metadata and the output file and 
>>>> returns a label and a URL.
>>>> 
>>>> Let me know if you have any questions.
>>>> 
>>>> Thanks,
>>>> 
>>>> Marcus
>>>> 
>>>> [1] 
>>>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link
>>>>  
>>>> <https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link>
>>>> 
>>>>> On Oct 6, 2020, at 12:49 PM, Luke Peyralans >>>> <mailto:lpeyr...@uoregon.edu>> wrote:
>>>>> 
>>>>> Is it possible to include a link as an experiment output? I would like to 
>>>>> upload some experiment data to a third-party hosted service that creates 
>>>>> interactive visualizations, and then provide a *unique link* that would 
>>>>> let a user access these visualizations upon conclusion of the experiment. 
>>>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Experiment Outputs -- Display Link?

[Christie, Marcus Aaron]

Hi Eugene,

Yes it can. The way it works is that the custom output view is packaged as a 
Python package and then it can be installed into the hosted Django portal 
virtual environment.

See 
https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/
 
<https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/>
 for more details.

I can also help you setup a local development environment that mirrors 
phasta.scigap.org <http://phasta.scigap.org/> so you can quickly iterate on the 
custom output view.


- Marcus


> On Oct 9, 2020, at 5:14 PM, Eugene Walker  
> wrote:
> 
> Can this be utilized on the airavata hosted service, ie phasta.scigap.org ?
> 
>> On Oct 9, 2020, at 1:49 PM, Christie, Marcus Aaron  wrote:
>> 
>> Hi Luke,
>> 
>> Not as an Airavata output, but our Django Portal supports a link type custom 
>> output view. [1]  Essentially this involves creating a Python function that 
>> takes the experiment metadata and the output file and returns a label and a 
>> URL.
>> 
>> Let me know if you have any questions.
>> 
>> Thanks,
>> 
>> Marcus
>> 
>> [1] 
>> https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link
>>  
>> <https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link>
>> 
>>> On Oct 6, 2020, at 12:49 PM, Luke Peyralans >> <mailto:lpeyr...@uoregon.edu>> wrote:
>>> 
>>> Is it possible to include a link as an experiment output? I would like to 
>>> upload some experiment data to a third-party hosted service that creates 
>>> interactive visualizations, and then provide a *unique link* that would let 
>>> a user access these visualizations upon conclusion of the experiment. 
>> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Experiment Outputs -- Display Link?

[Christie, Marcus Aaron]

Hi Luke,

Not as an Airavata output, but our Django Portal supports a link type custom 
output view. [1]  Essentially this involves creating a Python function that 
takes the experiment metadata and the output file and returns a label and a URL.

Let me know if you have any questions.

Thanks,

Marcus

[1] 
https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/custom_output_view_provider/#display-type-link

> On Oct 6, 2020, at 12:49 PM, Luke Peyralans  wrote:
> 
> Is it possible to include a link as an experiment output? I would like to 
> upload some experiment data to a third-party hosted service that creates 
> interactive visualizations, and then provide a *unique link* that would let a 
> user access these visualizations upon conclusion of the experiment. 



smime.p7s
Description: S/MIME cryptographic signature

Re: [External] Regarding deployment issue in deploying Django project on Apache server

[Christie, Marcus Aaron]

Hi Bhakti,

No I don't have any other ideas. I'll follow up with you separately to see if I 
can get access to the server.

> On Oct 1, 2020, at 10:20 AM, Narvekar, Bhakti Sunil  wrote:
> 
> Hi Marcus,
>  
> I tried both the ways. I’m still facing the same issue. Is there anything 
> else that we can try?
>  
> From: Christie, Marcus Aaron <mailto:machr...@iu.edu>
> Sent: Tuesday, September 29, 2020 10:52 AM
> To: Narvekar, Bhakti Sunil <mailto:bhn...@iu.edu>; dev@airavata.apache.org 
> <mailto:dev@airavata.apache.org>
> Cc: Ranawaka, Isuru Janith <mailto:isjar...@iu.edu>
> Subject: Re: [External] Regarding deployment issue in deploying Django 
> project on Apache server
>  
> Hi Bhakti,
> 
> Make sure that the PATH, LD_LIBRARY_PATH and LD_RUN_PATH environment 
> variables are also set for whatever user is used to run Apache and/or the 
> Django mod_wsgi server.
> 
> I'm not sure what is the best way to set these for Apache/mod_wsgi. You can 
> try in /etc/sysconfig/httpd (https://serverfault.com/a/233207 
> <https://serverfault.com/a/233207>). If that doesn't work, you may need to 
> set them directly in the wsgi.py file, see 
> https://gist.github.com/GrahamDumpleton/b380652b768e81a7f60c 
> <https://gist.github.com/GrahamDumpleton/b380652b768e81a7f60c> for an example.
> 
> From: Narvekar, Bhakti Sunil 
> Sent: Monday, September 28, 2020 6:31 PM
> To: dev@airavata.apache.org 
> Cc: Ranawaka, Isuru Janith ; Christie, Marcus Aaron 
> 
> Subject: RE: [External] Regarding deployment issue in deploying Django 
> project on Apache server
>  
> Hi Aarushi,
>  
> Thanks for the prompt reply.
>  
> In order to make the migrations I used to the following commands:
> python manage.py makemigrations
> python manage.py migrate
>  
> But the python version is same if use ‘python’ or ‘python3’
>  
> 
>  
>  
> From: Bisht, Aarushi <mailto:abi...@iu.edu>
> Sent: Monday, September 28, 2020 5:19 PM
> To: dev@airavata.apache.org <mailto:dev@airavata.apache.org>
> Cc: Ranawaka, Isuru Janith <mailto:isjar...@iu.edu>; Christie, Marcus Aaron 
> <mailto:machr...@iu.edu>
> Subject: Re: [External] Regarding deployment issue in deploying Django 
> project on Apache server
>  
> Hi Bhakti,
> 
> It seems python3 is using the upgraded version of SQLite. While migrating the 
> django project are you using python 3? Python might be defaulting to python 2.
> 
> Thanks,
> Aarushi Bisht
> 
> On Sep 28, 2020 1:30 PM, "Narvekar, Bhakti Sunil"  wrote:
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Hello Everyone,
>  
> We are facing an issue in deployment of Django project on Apache server. We 
> are using SQLite as a database and we are getting an improper configuration 
> error for the SQLite version 3.7.17. The screenshot of the error-logs is:
>  
> 
>  
> We followed the following steps to upgrade the SQLite version
>  
> <46D974B68B5E4E22891CF83BE6326073.png>
>  
> The version is update to 3.29.0 but the error in the error-log is same.
>  
> <3DC2E068AA0246289812268A152E02B9.png>
> 
> 
> It would be really helpful if anyone could share their inputs or experience.
>  
> Thank you,
>  
> Sincerely,
> Bhakti Narvekar



smime.p7s
Description: S/MIME cryptographic signature

Re: Recursive entity fetching in registry

[Christie, Marcus Aaron]

Hi Dimuthu,

I noticed a couple things:

- This line in ExperimentOutputValuePK [6] should check if instanceof 
ExperimentOutputValuePK. As is, my guess is equals() here always returns false, 
and that could lead OpenJPA to thinking it hasn't already loaded this one and 
it needs to do so again.
- You probably don't want to include the ExperimentOutputValuePK.value in the 
primary key since it's a BLOB. Support for indexing BLOBs is limited [7] and 
performance won't be good. In this case I would suggest a surrogate key. Or 
maybe there is another column that could be used instead, like an ordering 
column (if the values are ordered, maybe they're not).

Thanks,

Marcus

[6] 
https://github.com/apache/airavata/blob/param-sweep/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputValuePK.java#L61
 
<https://github.com/apache/airavata/blob/param-sweep/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputValuePK.java#L61>
[7] https://mariadb.com/kb/en/blob/#indexing 
<https://mariadb.com/kb/en/blob/#indexing>


> On Sep 24, 2020, at 12:43 PM, DImuthu Upeksha  
> wrote:
> 
> Hi Marcus,
> 
> I'm doing further testing to pinpoint the error however I have traced the sql 
> commands coming into the mysql database from Airavata when invoking this [5] 
> line. ExperimentOutputValueEntity has reference to ExperimentOutputEntity and 
> then to ExperimentEntity. I have attached the total SQL queries that came for 
> this line execution.
> 
> [5] 
> https://github.com/apache/airavata/blob/param-sweep/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/repositories/expcatalog/ExperimentOutputValueRepository.java#L72
>  
> <https://github.com/apache/airavata/blob/param-sweep/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/repositories/expcatalog/ExperimentOutputValueRepository.java#L72>
> 
> Thanks
> Dimuthu
> 
> On Wed, Sep 23, 2020 at 6:09 PM Christie, Marcus Aaron  <mailto:machr...@iu.edu>> wrote:
> Hi Dimuthu,
> 
> Thanks for investigating this. I have a few questions/observations:
> - when you fetch an experiment output, it would be understandable if the 
> experiment was then fetched if experimentOutput.getExperiment() was accessed. 
> This would in turn lead to the experiment's experiment outputs being fetch 
> since we have the fetch type set to EAGER [4]. So the sequence you describe 
> doesn't necessarily sound like unexpected behavior.
> - how many times does the recursive fetching go through fetching the same 
> objects?  I'm assuming at some point it stops otherwise we'd see timeouts or 
> stack overflow. Any idea what causes it to eventually stop?
> - do you have a test case that reproduces this behavior?
> - seems that this performance drop is new. The registry code itself hasn't 
> changed recently on master, but we did recently merge in the upgrade of 
> OpenJPA from 2.4.3 to 3.1.0. Could this be related to the newer version?
> - in general this sort of behavior makes me think that equals/hashCode isn't 
> implemented properly somewhere, since that would make it hard for OpenJPA to 
> figure out if it has already loaded an entity into the session. Looking at 
> ExperimentEntity and ExperimentOutputEntity, I don't see any issues there.
> 
> Thanks,
> 
> Marcus
> 
> [4] 
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L89
>  
> <https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L89>
> 
>> On Sep 22, 2020, at 11:01 AM, DImuthu Upeksha > <mailto:dimuthu.upeks...@gmail.com>> wrote:
>> 
>> Hi Folks,
>> 
>> I just noticed that the entity relations that have bidirectional definitions 
>> (OneToMany and ManyToOne) between 2 tables cause recursive object fetching 
>> and it is causing a significant performance drop in database querying. For 
>> example [1] and [2]. When we fetch an experiment output, I can see a stack 
>> of experiment -> experiment outputs -> experiment 
>> 
>> Same issue was mentioned in [3] but the solution is for spring boot json 
>> parsing. As we don't use spring boot, we might have to figure out some other 
>> way to stop this recursive stack creation.
>> 
>> [1] 
>> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L88
>>  
>> <https://github.com/a

Re: Recursive entity fetching in registry

[Christie, Marcus Aaron]

Hi Dimuthu,

Thanks for investigating this. I have a few questions/observations:
- when you fetch an experiment output, it would be understandable if the 
experiment was then fetched if experimentOutput.getExperiment() was accessed. 
This would in turn lead to the experiment's experiment outputs being fetch 
since we have the fetch type set to EAGER [4]. So the sequence you describe 
doesn't necessarily sound like unexpected behavior.
- how many times does the recursive fetching go through fetching the same 
objects?  I'm assuming at some point it stops otherwise we'd see timeouts or 
stack overflow. Any idea what causes it to eventually stop?
- do you have a test case that reproduces this behavior?
- seems that this performance drop is new. The registry code itself hasn't 
changed recently on master, but we did recently merge in the upgrade of OpenJPA 
from 2.4.3 to 3.1.0. Could this be related to the newer version?
- in general this sort of behavior makes me think that equals/hashCode isn't 
implemented properly somewhere, since that would make it hard for OpenJPA to 
figure out if it has already loaded an entity into the session. Looking at 
ExperimentEntity and ExperimentOutputEntity, I don't see any issues there.

Thanks,

Marcus

[4] 
https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L89

> On Sep 22, 2020, at 11:01 AM, DImuthu Upeksha  
> wrote:
> 
> Hi Folks,
> 
> I just noticed that the entity relations that have bidirectional definitions 
> (OneToMany and ManyToOne) between 2 tables cause recursive object fetching 
> and it is causing a significant performance drop in database querying. For 
> example [1] and [2]. When we fetch an experiment output, I can see a stack of 
> experiment -> experiment outputs -> experiment 
> 
> Same issue was mentioned in [3] but the solution is for spring boot json 
> parsing. As we don't use spring boot, we might have to figure out some other 
> way to stop this recursive stack creation.
> 
> [1] 
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentEntity.java#L88
>  
> 
> [2] 
> https://github.com/apache/airavata/blob/master/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/entities/expcatalog/ExperimentOutputEntity.java#L80
>  
> 
> [3] 
> https://medium.com/@udith.indrakantha/issue-related-with-infinite-recursive-fetching-of-data-from-relationships-between-entity-classes-ffc5fac6c816
>  
> 
> 
> Thanks
> Dimuthu



smime.p7s
Description: S/MIME cryptographic signature

Re: Docker-compose PGA Failure, ide-integration setup

[Christie, Marcus Aaron]

Hi Eugene,

I'm getting the same error, and I'm not sure what the problem is.

I could kind of work around that error by running `docker-compose up` in one 
terminal and then in another I also ran:

docker exec -it pga_pga_1 bash
chmod -R 777 /var/www/portals/default/app/storage/

But I'm running into other errors. I'm guessing it is some file permission 
issues.


> On Aug 6, 2020, at 2:52 PM, Eugene Walker  
> wrote:
> 
> Plenty of disk space is available to Docker (10 gig used of 60 gig available)
> Also the directory being bind mounted as part of docker-compose file **is** 
> allowed under the Docker File Sharing rules.
> 
> Thank you Marcus!
> 
> On Thu, Aug 6, 2020 at 11:26 AM Christie, Marcus Aaron  <mailto:machr...@iu.edu>> wrote:
> Hi Eugene,
> 
> Just a guess, but maybe check if Docker has run out of disk space.  In macOS 
> Docker Desktop, go to Preferences > Resources and check "Disk image size".  
> If so, you can use "docker system prune" to free up space and then try again.
> 
>> On Aug 6, 2020, at 11:18 AM, Eugene Walker > <mailto:eugenesunsetwal...@gmail.com>> wrote:
>> 
>> I'm following the ide-integration development environment instructions, and 
>> trying to start the super admin portal according to these instructions:
>> https://github.com/apache/airavata/tree/master/modules/ide-integration 
>> <https://github.com/apache/airavata/tree/master/modules/ide-integration>
>> 
>> However, the pga container crashes with the following error on MacOS 
>> Catalina 10.15.6
>> 
>> Any ideas why `php artisan optimize` is throwing this error?
>> 
>> ```
>> $> cd src/main/resources/pga
>> $> docker-compose up -d
>> $> docker logs -f pga_pga_1
>> ...
>> Script php artisan optimize handling the post-install-cmd event returned 
>> with error code 1
>> Do not run Composer as root/super user! See https://getcomposer.org/root 
>> <https://getcomposer.org/root> for details
>> Loading composer repositories with package information
>> Installing dependencies (including require-dev) from lock file
>> Nothing to install or update
>> Generating autoload files
>> > php artisan clear-compiled
>> > php artisan optimize
>> Generating optimized class loader
>> Compiling common classes
>> Compiling views
>> 
>> 
>>   [ErrorException]
>>   
>> file_put_contents(/var/www/portals/default/app/storage/views/71093b3cc34a08455a6074e52a522f01):
>>  failed to open stream: No such file or directory
>> ```
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Docker-compose PGA Failure, ide-integration setup

[Christie, Marcus Aaron]

Hi Eugene,

Just a guess, but maybe check if Docker has run out of disk space.  In macOS 
Docker Desktop, go to Preferences > Resources and check "Disk image size".  If 
so, you can use "docker system prune" to free up space and then try again.

> On Aug 6, 2020, at 11:18 AM, Eugene Walker  
> wrote:
> 
> I'm following the ide-integration development environment instructions, and 
> trying to start the super admin portal according to these instructions:
> https://github.com/apache/airavata/tree/master/modules/ide-integration 
> 
> 
> However, the pga container crashes with the following error on MacOS Catalina 
> 10.15.6
> 
> Any ideas why `php artisan optimize` is throwing this error?
> 
> ```
> $> cd src/main/resources/pga
> $> docker-compose up -d
> $> docker logs -f pga_pga_1
> ...
> Script php artisan optimize handling the post-install-cmd event returned with 
> error code 1
> Do not run Composer as root/super user! See https://getcomposer.org/root 
>  for details
> Loading composer repositories with package information
> Installing dependencies (including require-dev) from lock file
> Nothing to install or update
> Generating autoload files
> > php artisan clear-compiled
> > php artisan optimize
> Generating optimized class loader
> Compiling common classes
> Compiling views
> 
> 
>   [ErrorException]
>   
> file_put_contents(/var/www/portals/default/app/storage/views/71093b3cc34a08455a6074e52a522f01):
>  failed to open stream: No such file or directory
> ```



smime.p7s
Description: S/MIME cryptographic signature

Re: Updating the Airavata README.md

[Christie, Marcus Aaron]

Thanks Dimuthu. Yes I agree, that would be a great addition. I'll add it.

> On Aug 5, 2020, at 12:35 PM, DImuthu Upeksha  
> wrote:
> 
> Hi Marcus, 
> 
> This is very nice and clean. I suggested a minor change and what do you think 
> if we point to ide integration module in the readme as a quick start point?
> 
> Thanks
> Dimuthu
> 
> On Wed, Aug 5, 2020 at 11:01 AM Christie, Marcus Aaron  <mailto:machr...@iu.edu>> wrote:
> Hi All,
> 
> I've attempted to bring the Airavata README.md up to date. It contained some 
> stale information, like project structure, which I didn't bother trying to 
> update. I added some additional links and information, based on typical 
> things I've seen on other READMEs (like, for example, how to contribute).
> 
> https://github.com/apache/airavata/pull/258 
> <https://github.com/apache/airavata/pull/258>
> 
> Please review the PR when you have a chance. If there is anything else you 
> think should be in the README just let me know, I'd be happy to add it.
> 
> Thanks,
> 
> Marcus



smime.p7s
Description: S/MIME cryptographic signature

Re: Validating user storage quota

[Christie, Marcus Aaron]

Responses below:

> On Aug 5, 2020, at 3:20 AM, Bandaru, Vivek Shresta  wrote:
> 
> Adding restrictions to output files would mean the user will not be able to 
> access the output files even if he was able to create a new experiment with 
> input files. That would mean, the user waits for the experiment’s output only 
> to find out he doesn’t have enough storage space. Reason why I did not add 
> any validation for output data. Is this a valid use case? Also, is there any 
> way to know how much space an experiment’s output will need before launching 
> the experiment?

I agree, we wouldn't want to restrict storing the output files just because the 
user hit their quota limit. This does raise the question, what should happen?  
Probably there should be a soft and hard quota limit. When the user hits the 
soft limit they can't add new files manually (uploading files in the portal), 
and the user is informed of the situation and advised to free up some space or 
request a larger storage quota. If they hit the hard limit then no further 
additions are allowed, even output files.

> After the previous discussion, I assumed once GroupResourceProfile and 
> UserStoragePreference (apparently it’s about to be deprecated now) are in 
> play, Airavata would be able to choose which StoragePreference should be 
> chosen, reason why I went with mechanism where the API server validates the 
> storage limit. With the current architecture as is, if Airavata tracking the 
> user storage(MFT in the future) is not necessary, then maybe the gateway 
> worrying about the storage quota makes more sense.

I think Airavata validating the storage limit can make sense. But if Airavata 
is managing it, then it could just query the storage resource for the amount of 
storage space used by a user. Airavata already has SSH access to get and put 
files on the storage resources, so it can certainly run the equivalent of 'du'. 
That way we don't need to add an API for the portal to report storage space to 
the API server.

Please do raise the PR, it would be good to see how you've implemented your 
approach.

Thanks,

Marcus

smime.p7s
Description: S/MIME cryptographic signature

Updating the Airavata README.md

[Christie, Marcus Aaron]

Hi All,

I've attempted to bring the Airavata README.md up to date. It contained some 
stale information, like project structure, which I didn't bother trying to 
update. I added some additional links and information, based on typical things 
I've seen on other READMEs (like, for example, how to contribute).

https://github.com/apache/airavata/pull/258

Please review the PR when you have a chance. If there is anything else you 
think should be in the README just let me know, I'd be happy to add it.

Thanks,

Marcus

smime.p7s
Description: S/MIME cryptographic signature

Re: Validating user storage quota

[Christie, Marcus Aaron]

Hi Vivek,

Yes, since the Django Portal is the user's data store, it should enforce the 
policy. If I think about the Airavata API in other contexts, I'm not sure it 
makes sense for the API client to inform the API server of how much space is 
being used. Also, this mechanism is insufficient because it doesn't take into 
account output files that get added to the data store when the experiment is 
executed.

I think it would be fine if the API provided information about the quota 
limits. Then the Django portal can query those limits and apply them.


> On Aug 3, 2020, at 6:32 PM, Bandaru, Vivek Shresta  wrote:
> 
> Hi Marcus,
>  
> Thanks for the reply.
>  
> I think it should be up to the data store to calculate the amount of storage 
> space use and apply the quota. Currently, that's the Django portal. However, 
> in the future it will likely be MFT.  -> Do you mean to say that the 
> validation should be done in the gateway rather than Airavata?  In my current 
> implementation, Django-Portal calculates the amount of Data used by a User 
> and sends this to Airavata when an experiment is being created(through a new 
> Airavata API) where the validation happens. Once the user reaches the storage 
> limit (validated against the new entry, UserStorageQuota in 
> StoragePreferences), Airavata throws an exception resulting in the 
> cancellation of the new experiment. Is this approach fine?
>  
> Regards,
> Vivek.
>  
> From: "Christie, Marcus Aaron" 
> Reply-To: 
> Date: Monday, August 3, 2020 at 1:43 PM
> To: Airavata Dev 
> Cc: "Wannipurage, Dimuthu Upeksha" 
> Subject: Re: Validating user storage quota
>  
> Hi Vivek,
> 
> 
> On Jul 30, 2020, at 9:31 PM, Bandaru, Vivek Shresta  <mailto:vivb...@iu.edu>> wrote:
>  
> Hi All,
>  
> I’ve been working on various approaches to validate the storage quota for a 
> gateway user. Though the storage for experiments is taken care by the 
> gateway, the storage quota validation needs to be done on Airavata. This way, 
> the gateways need not develop their own mechanism to track the quotas for 
> every user.
>  
> The only way as far as I know for a user to add his files to the storage are 
> through Storage page and the Create a New experiment page where they can add 
> input files. Utilizing the existing
>  
> Also, Airavata deposits output files into the experimentDataDir and these 
> will count against the user's quota.
> 
> 
> datastore.size(‘directory_path’) api in the Django-portal, the amount of 
> space used by the user can be tracked, and when the Gateway sends this data 
> to Airavata through a new API, Airavata validates the space utilized with the 
> Storage Quota specified in the Storage Preference.
> Currently, whenever a user tries to create a new Experiment, even before the 
> input files are added to the Experiment’s directory, I’ve added the 
> validation and when the storage limit exceeds, this is how the new experiment 
> page is rendered:
>  
> 
>  
> Questions:
>  
> Since UserStoragePreference feature isn’t being used, I’m not really sure if 
> I need to use the StorageResource mentioned in UserStoragePreference. Once 
> the gateways start using this (not sure if it’s only Django-Portal which 
> isn’t using UserStoragePreference), then the necessary code can be added. 
> I would like to know the teams thoughts on this.
>  
> I wouldn't worry about UserStoragePreference. That is somewhat deprecated, to 
> be replaced with a personal GroupResourceProfile.
> 
> 
>  
> I feel that maybe a better way to approach this problem would be if Airavata 
> tracks the size of each and every file going into the user’s directory 
> instead of the gateway telling the size of the user’s directory. The amount 
> of space used by an individual user can be tracked in the USERS table through 
> a new column. This is where I’m currenty stuck.
> Airavata currently uses SCPFileTransferWrapper.java for uploading the input 
> files onto the compute resources. There is no means of knowing the file size 
> being transferred. 
> Using the current implementation, one possible approach that I could think of 
> is to download the file from the gateway to a temporary directory where 
> Airavata is deployed, and through that URI, the file size can be retrieved. 
> But this would involve downloading every input file on all the gateways of 
> all the users onto a temporary location on Airavata and deleting them.
> I’m currently looking for a better alternative to track the file size given a 
> URI.
>  
> I think it should be up to the data store to calculate the amount of storage 
> space use and apply the quota. Currently, that's the Django portal. However, 
> in the future it 

Re: Validating user storage quota

[Christie, Marcus Aaron]

Hi Vivek,

> On Jul 30, 2020, at 9:31 PM, Bandaru, Vivek Shresta  wrote:
> 
> Hi All,
>  
> I’ve been working on various approaches to validate the storage quota for a 
> gateway user. Though the storage for experiments is taken care by the 
> gateway, the storage quota validation needs to be done on Airavata. This way, 
> the gateways need not develop their own mechanism to track the quotas for 
> every user.
>  
> The only way as far as I know for a user to add his files to the storage are 
> through Storage page and the Create a New experiment page where they can add 
> input files. Utilizing the existing

Also, Airavata deposits output files into the experimentDataDir and these will 
count against the user's quota.

> datastore.size(‘directory_path’) api in the Django-portal, the amount of 
> space used by the user can be tracked, and when the Gateway sends this data 
> to Airavata through a new API, Airavata validates the space utilized with the 
> Storage Quota specified in the Storage Preference.
> Currently, whenever a user tries to create a new Experiment, even before the 
> input files are added to the Experiment’s directory, I’ve added the 
> validation and when the storage limit exceeds, this is how the new experiment 
> page is rendered:
>  
> 
>  
> Questions:
>  
> Since UserStoragePreference feature isn’t being used, I’m not really sure if 
> I need to use the StorageResource mentioned in UserStoragePreference. Once 
> the gateways start using this (not sure if it’s only Django-Portal which 
> isn’t using UserStoragePreference), then the necessary code can be added. 
> I would like to know the teams thoughts on this.

I wouldn't worry about UserStoragePreference. That is somewhat deprecated, to 
be replaced with a personal GroupResourceProfile.

> I feel that maybe a better way to approach this problem would be if Airavata 
> tracks the size of each and every file going into the user’s directory 
> instead of the gateway telling the size of the user’s directory. The amount 
> of space used by an individual user can be tracked in the USERS table through 
> a new column. This is where I’m currenty stuck.
> Airavata currently uses SCPFileTransferWrapper.java for uploading the input 
> files onto the compute resources. There is no means of knowing the file size 
> being transferred. 
> Using the current implementation, one possible approach that I could think of 
> is to download the file from the gateway to a temporary directory where 
> Airavata is deployed, and through that URI, the file size can be retrieved. 
> But this would involve downloading every input file on all the gateways of 
> all the users onto a temporary location on Airavata and deleting them.
> I’m currently looking for a better alternative to track the file size given a 
> URI.

I think it should be up to the data store to calculate the amount of storage 
space use and apply the quota. Currently, that's the Django portal. However, in 
the future it will likely be MFT.

> Once MFT is integrated into Airavata, Airavata will not use the existing file 
> transfer protocols. So, does it make sense to develop this tracking mechanism 
> on MFT and for now, use the above mentioned validation mechanism(Gateway gets 
> the size of the user directory)?

I think definitely understanding how MFT factors in to the future plans will be 
good. Maybe Dimuthu can chime in here.

>  
> Any pointers are appreciated.
> Thanks for reading.
>  
> Regards,
> Vivek.



smime.p7s
Description: S/MIME cryptographic signature

Re: mft-portal search functionality for storage units and resources

[Christie, Marcus Aaron]

Hi Akhi,

Yes that's right, the Airavata Django Portal UI is implemented with Vue 
components.

> On Jul 30, 2020, at 5:35 PM, akhil nagulavancha  wrote:
> 
> Hi all,
> 
> I am adding search functionality for searching storage units and storage 
> resources  for mft-portal . I would like to have inputs on whether I should 
> develop search functionality within the Vue Components or with the Django in 
> the mft portal.
> 
>  As per my understanding of the Airavata Django portal , Most of the UI 
> components are developed within the Vue Components as per the current 
> Airavata standards. But would like to confirm again on whether I should go 
> for developing the search functionality within the Vue components or if it is 
> fine with Django too.
> 
> Thanks and Regards 
> Akhil Nagulavancha



smime.p7s
Description: S/MIME cryptographic signature

Re: Creating experiments on local

[Christie, Marcus Aaron]

Hi Vivek,

Check the output of JobEngineStarter. Sounds like Helix is down if the 
experiment is stuck in Launched state.

> On Jul 24, 2020, at 3:12 AM, Bandaru, Vivek Shresta  wrote:
> 
> Gentle reminder. I’m still looking for inputs.
>  
> From: "Bandaru, Vivek Shresta" mailto:vivb...@iu.edu>>
> Date: Monday, July 20, 2020 at 6:14 PM
> To: "dev@airavata.apache.org " 
> mailto:dev@airavata.apache.org>>
> Subject: Creating experiments on local
>  
> Hi All,
>  
> I’ve been trying to understand how a user experiment’s input/output storage 
> works. Currently the only experiment which I can create in my local is the 
> application ‘Echo’ which doesn’t require any user input file and for some 
> reason is never completing and is always in the Launched state. In the below 
> screenshot, you can see experiments which have been running since 3 days. 
> What can the issue be?
>  
> 
>  
> The below screenshot shows that I’ve chosen the default resources for 
> computing the experiment.
>  
> 
>  
>  
> Executing an experiment which requires an input  file and generated an output 
> file will definitely help me understand the process better. The only 
> experiment available in my local is ‘Echo’ which doesn’t generate any output. 
> How do I launch any other experiments which are only available 
> onhttps://testdrive.airavata.org/ ?
>  
> Regards,
> Vivek.



smime.p7s
Description: S/MIME cryptographic signature

Re: [Discussion] Securing single-page-application using JWT token

[Christie, Marcus Aaron]

It might be worth looking at Keycloak's JS client adapter to see how they 
handle some of these things: 
https://www.keycloak.org/docs/latest/securing_apps/index.html#_javascript_adapter
 



> On Jul 8, 2020, at 4:55 PM, Isuru Ranawaka  wrote:
> 
> We need to finalize methods to store access token, refresh token and Id
> token. As Shivam , anuj suggested  can use local storage,  cookies or  API
> Gateway  approach. Any ideas  on
>   - How to handle silent refresh for invalidated tokens

The Keycloak adapter has an updateToken() function that refreshes only if 
needed and returns a promise in which you're guaranteed a valid token. 
https://www.keycloak.org/docs/latest/securing_apps/index.html#updatetoken-minvalidity
 


>   -  Synchronizing logouts for multiple tabs

See the Session Status iframe: 
https://www.keycloak.org/docs/latest/securing_apps/index.html#session-status-iframe
 


>>> 
> I guess httpOnly cookie is added from the server-side. But that won't be
> feasible with the current approach. Custos provides access tokens in the
> body, and not capable of handling cookies.  I guess we can achieve this via
> intermediate proxy. any thoughts?

Is this a grpc-web limitation? Also, I'm guessing you'll need a proxy or 
something to read the access token out of the cookie on the server-side.

smime.p7s
Description: S/MIME cryptographic signature

Re: Updated docs and tools for database schema migrations

[Christie, Marcus Aaron]

I forgot to also mention that the changes affect the ide-integration module:
- the README.md I linked to in registry-core explains how to copy migrations 
into ide-integration module's database init script
- the ide-integration README.md now documents the command to apply migrations 
to the database container [0]
- the database container image is now a MariaDB image. Migration scripts are 
easier with MariaDB because supports IF EXISTS/IF NOT EXISTS.
- *-derby.sql and *-mysql.sql and persistence.xml files were removed since, at 
least in my testing, there were unnecessary. They also have caused some 
confusion in the past since it's not been clear if these files have to be 
updated or not.

[0] 
https://github.com/apache/airavata/blame/develop/modules/ide-integration/README.md#L63

> On Jun 25, 2020, at 5:32 PM, Christie, Marcus Aaron  wrote:
> 
> Hi All,
> 
> I've recently worked on adding some documentation [0] on how to keep the 
> database schema scripts and migration scripts up to date. In addition I've 
> added some Maven commands that will use OpenJPA's MappingTool to 
> automatically create database schema and/or migration scripts (for Derby and 
> MariaDB/MySQL). I've merged this to the develop branch now and plan to soon 
> merge to the master branch as well.
> 
> My hope is that this makes it clear exactly how and where to make database 
> script changes for the registry databases, and that the entire process is 
> easier. Please see the linked README when you have a chance and let me know 
> if you have any questions.
> 
> Also, the Jira issue for this work is AIRAVATA-3326 [1]
> 
> Thanks,
> 
> Marcus
> 
> [0] 
> https://github.com/apache/airavata/blob/develop/modules/registry/registry-core/README.md
>  
> <https://github.com/apache/airavata/blob/develop/modules/registry/registry-core/README.md>
> [1] https://issues.apache.org/jira/browse/AIRAVATA-3326 
> <https://issues.apache.org/jira/browse/AIRAVATA-3326>
> 



smime.p7s
Description: S/MIME cryptographic signature

Updated docs and tools for database schema migrations

[Christie, Marcus Aaron]

Hi All,

I've recently worked on adding some documentation [0] on how to keep the 
database schema scripts and migration scripts up to date. In addition I've 
added some Maven commands that will use OpenJPA's MappingTool to automatically 
create database schema and/or migration scripts (for Derby and MariaDB/MySQL). 
I've merged this to the develop branch now and plan to soon merge to the master 
branch as well.

My hope is that this makes it clear exactly how and where to make database 
script changes for the registry databases, and that the entire process is 
easier. Please see the linked README when you have a chance and let me know if 
you have any questions.

Also, the Jira issue for this work is AIRAVATA-3326 [1]

Thanks,

Marcus

[0] 
https://github.com/apache/airavata/blob/develop/modules/registry/registry-core/README.md
 

[1] https://issues.apache.org/jira/browse/AIRAVATA-3326 




smime.p7s
Description: S/MIME cryptographic signature

Re: GeoGateway Project Update

[Christie, Marcus Aaron]

Hi Nick, this is good summary. I had a couple of other thoughts:

- I really like the "Thinking in React" article [0]. Even though it is specific 
to React, the principles are pretty general. You only need to do a little 
mental translating to make it work for Vue (for example, parent-child 
communication in React involves passing callbacks to children, but, while you 
can do that in Vue too, typically in Vue children will dispatch events to the 
parent).
- I'm sorry you wrestled with the boilerplate so much. We do have documentation 
on how to setup the boilerplate for core Django apps [1], but it would probably 
be good to have the same for custom Django apps as well. Perhaps better than a 
boilerplate template would be a working minimal example that could be copied 
and modified, just my two cents.


[0] https://reactjs.org/docs/thinking-in-react.html 

[1] 
https://apache-airavata-django-portal.readthedocs.io/en/latest/dev/new_django_app/
 



> On Jun 19, 2020, at 1:50 PM, Mowery, Nicholas Benjamin  
> wrote:
> 
> Greetings from sunny Bloomington,
> An update on the GeoGateway Django App:
> This week I've worked on setting up a reverse-proxy to bypass CORS when 
> sending requests to external sources. To do this I've mapped GeoGateway URLs 
> to request functions in the Django backend which are used by axios in the Vue 
> frontend. This method is flexible but not as efficient as it could be. For 
> the resources that take longer to resolve, I will cache the results 
> periodically to improve performance (as per Marlon's suggestion).
> In a previous email referenced Vuex and how it should be implemented in an 
> application of 10 components or more; I'm learning that this is not 
> necessarily true. Vuex is a very powerful tool for large applications but 
> because of its complexity, can be more work to implement than an event bus. 
> I've removed Vuex in favor of a global event bus and it has simplified things 
> immensely. If anyone has developed with Vue before and has had to make a 
> decision regarding state management and event listeners, I'd love to hear 
> what decision was made and why as I'm sure there is room for improvement in 
> my implementation.
> After wrestling with the Vue+Django+Webpack boilerplate and eventually 
> landing on a solid configuration, I've found development proceeds smoothly 
> and working with Vue and Django separately is a pleasure. There are plenty of 
> articles and repos that help to lay down the boilerplate for a 
> Vue+Django+Webpack project but because this is a part of the Airavata Django 
> Portal, the setup is a bit different. I think this setup could be useful to 
> future Airavata developers and I hope to push a versatile boilerplate for 
> future projects using the same technologies.
> 
> In summary:
> Django CORS reverse-proxy
> Caching results to improve performance
> Jettisoned Vuex in favor of event bus (Please share thoughts on this)
> Vue+Django+Webpack boilerplate for future Airavata developers (would this be 
> helpful?)
> Thanks for reading!
> -Nicholas
> 
> 
> From: Mowery, Nicholas Benjamin mailto:nbmow...@iu.edu>>
> Sent: Friday, June 5, 2020 6:04 PM
> To: dev@airavata.apache.org 
> Subject: [External] Re: Project Update and Questions
>  
> This message was sent from a non-IU address. Please exercise caution when 
> clicking links or opening attachments from external sources.
> 
> Hi all, 
> another update on the GeoGateway Django App. This week I've focused on state 
> management with Vuex and developing a modular and lightweight method for 
> adding KML map layers to the Leaflet map. I've opted to use Vuex to make 
> communication between child components easier. I've also removed Vue as a CDN 
> (in favor of Vue CLI + Webpack) and and fully separated the Vue frontend and 
> Django backend. The boilerplate code is significant but I now have a stable 
> development environment that can be deployed quickly. 
> A question I'm considering:
> What is the best way to implement file upload in an Airavata Django app.  
> Django has a relatively straight-forward method but it might not be the best 
> in our case
> https://simpleisbetterthancomplex.com/tutorial/2016/08/01/how-to-upload-files-with-django.html
>  
> 
> Generally, I'm wondering if this problem should be handled locally by the 
> application rather than communicating with Airavata.
> Some things I've learned: 
> If the application is going to have more than 10 Vue components, implement a 
> Vuex store from the start
> vue2-leaflet is not extensible enough for large GIS applications that require 
> customized map layers
> Webpack configuration can be painful and a boilerplate for future Django + 
> Vue + Webpack applications will be 

Re: GSoC Milestone 1 update

[Christie, Marcus Aaron]

> On Jun 17, 2020, at 10:00 PM, Bandaru, Vivek Shresta  wrote:
> 
> 1. I’ve tried building the docker image with ‘CREATE TABLE’ changes in the 
> appcatalog-mysql.sql and appcatalog-derby.sql inregistry/src/main/resources 
> or/and in database-scripts/ but still the schema never changed. Only changes 
> in 01-databases.sql or the release-migration-scripts changed the schema in my 
> case. Where are these files used? Do I need to remove the changes I’ve made 
> in these files?

Yeah, like I said appcatalog-mysql.sql etc. would only be used when there are 
no tables, but ide-integration loads a database dump from the start, so those 
files never get used in ide-integration. Probably they should be removed from 
the ide-integration module and you can remove your changes from them.

when updating the database schema you need to update:
- -mysql.sql in registry-core
- -derby.sql in registry-core
- MariaDB migration script in 
release-migration-scripts/next/DeltaScripts/_schema_delta.sql
- likewise in ide-integration 0*--migrations.sql

> 2.. I’m assuming the folders in the release-migration-scripts represent the 
> mysql version. Is my understanding right?

Right.

> 3.  Why do we have almost similar scripts at two places in the codebase? 
> appcatalog_derby.sql and appcatalog_mysql.sql in the folders 
> ‘ide-integration/src/main/resources/database-scripts’ and 
> ‘registry/src/main/resources’?

See #1, the duplicates in ide-integration I don't think are needed actually. 
Maybe Dimuthu can chime in.

smime.p7s
Description: S/MIME cryptographic signature

Re: GSoC Milestone 1 update

[Christie, Marcus Aaron]

Hi Vivek,

This is good feedback. I'm working on simplifying the db migrations process, 
hopefully it will be clearer in the future.

In general, the appcatalog-mysql.sql, etc. scripts are executed by Airavata API 
server when it starts up when the database has no tables. However, the 
ide-integration docker-compose starts the MySQL database with a full dump of an 
airavata database (01-databases.sql), so appcatalog-mysql.sql doesn't actually 
get executed in that case.

When a MySQL container is created for the first time by docker-compose, it runs 
all of the scripts in database_scripts/init (because these are mapped to the 
volume /docker-entrypoint-initdb.d). As mentioned above, that's the 
01-databases.sql script which is db dump and then migrations scripts that are 
intended to be applied on top of it.

I hope that helps.

Sincerely,

Marcus

> On Jun 16, 2020, at 6:57 PM, Bandaru, Vivek Shresta  wrote:
> 
> That worked! Thanks Marcus.
> 
> Earlier, I faced issues with my ‘alter table’ syntax during docker-compose, 
> so I did try building mysql db without the alter table query, and just the 
> updated ‘CREATE TABLE STORAGE_PREFERENCE’ query with the new 
> ‘userStorageQuota’ entry. Since I deleted the mysql image and database-data 
> folder, shouldn’t the new database image contain the ‘userStorageQuota’ 
> entry? The exception I posted earlier was when the code had no alter table 
> query and just the updated ‘CREATE TABLE STORAGE_PREFERENCE’ query. Is my 
> understanding right? Any idea what mistake I might be doing?
>  
> Most of my changes to the ‘.sql’ scripts were made through trial and error 
> and workspace searches so I would like to get a better understanding on how 
> the database is built.
> For instance, why do we have almost similar scripts at two places in the 
> codebase? appcatalog_derby.sql and appcatalog_mysql.sql in the folders 
> ‘ide-integration/src/main/resources/database-scripts’ and 
> ‘registry/src/main/resources’? 
> 
> It would be really helpful if you can point me towards some documentation if 
> any.
> 
> Regards,
> Vivek.
>  
> From: "Christie, Marcus Aaron" 
> Reply-To: 
> Date: Tuesday, June 16, 2020 at 5:45 PM
> To: Airavata Dev 
> Subject: Re: GSoC Milestone 1 update
>  
> Hi Vivek,
>  
> https://github.com/apache/airavata/pull/254/files#diff-0d0ad3e35c91564651a24674c1a99411R20
>  
> <https://github.com/apache/airavata/pull/254/files#diff-0d0ad3e35c91564651a24674c1a99411R20>
>  
> Try removing "IF NOT EXISTS". That syntax works for MariaDB but not MySQL, 
> which the ide-integration docker-compose uses.
>  
> (An aside, I am updating ide-integration to start up a MariaDB container 
> instead, but that code is still on a branch.)
> 
> 
> On Jun 14, 2020, at 11:26 PM, Bandaru, Vivek Shresta  <mailto:vivb...@iu.edu>> wrote:
>  
> I’ve made sure to delete the existing mysql image and 
> ide-integration/database-data folder before building a new mysql image with 
> the new changes, but the schema still won’t change. I encountered the below 
> mentioned  exception stacktrace during server startup.



smime.p7s
Description: S/MIME cryptographic signature

Re: GSoC Milestone 1 update

[Christie, Marcus Aaron]

Hi Vivek,

https://github.com/apache/airavata/pull/254/files#diff-0d0ad3e35c91564651a24674c1a99411R20
 


Try removing "IF NOT EXISTS". That syntax works for MariaDB but not MySQL, 
which the ide-integration docker-compose uses.

(An aside, I am updating ide-integration to start up a MariaDB container 
instead, but that code is still on a branch.)

> On Jun 14, 2020, at 11:26 PM, Bandaru, Vivek Shresta  wrote:
> 
> I’ve made sure to delete the existing mysql image and 
> ide-integration/database-data folder before building a new mysql image with 
> the new changes, but the schema still won’t change. I encountered the below 
> mentioned  exception stacktrace during server startup.
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Archiving Airavata PHP Gateway Repository

[Christie, Marcus Aaron]

In principle I'm for it, but we still use PGA for the super-admin portal (i.e., 
scigap.org).

> On May 30, 2020, at 2:51 PM, Suresh Marru  wrote:
> 
> Hi All,
> 
> Since all active development is on Airavata Django Portal, any objections to 
> change the PGA repo [1] to archiving and read-only[2]?
> 
> Cheers,
> Suresh
> 
> [1] - https://github.com/apache/airavata-php-gateway 
> 
> [2] - 
> https://help.github.com/en/github/creating-cloning-and-archiving-repositories/archiving-repositories
>  
> 


smime.p7s
Description: S/MIME cryptographic signature

Re: Project Update and Questions

[Christie, Marcus Aaron]

Hi Nick,


> On May 21, 2020, at 5:51 PM, Mowery, Nicholas Benjamin  
> wrote:
> 
> 
> Hi all,
> This summer I am working on developing a GeoGateway Django app for the 
> Airavata Django Portal. So far I have developed a skeleton of the app and 
> integrated Vue.js for the frontend UI components. At this stage I'm looking 
> for input on some of the choices I've made and if any of them should be 
> adjusted. 
> 
> Leaflet and Vue.js (over Google Maps and React.js)
> It seems at this scale that a modular app rather than Vue.js style SPA makes 
> more sense (although some of the other Airavata GIS applications are mostly 
> SPA)

Regarding SPA or not, I would say go with what you are most comfortable. In 
general, if the UI you build consists almost completely of Vue components (that 
is, as opposed to Django templates) then it should be fairly easy to move from 
a modular approach to SPA later. It's probably a little harder to move from SPA 
to modular, but that would be doable too. So you could for example go with a 
modular approach now and revisit that decision later.

> There are a lot of style choices to be made and many I'm likely still unaware 
> of, so any tips on structuring and organizing a project like this will be 
> helpful.
> I also have a couple questions I've accumulated about Django for anyone 
> willing to help:
> When working with a custom Django app in the Airavata Portal, should I 
> redefine the MEDIA_ROOT directory in the apps.py file? Is there a precedent 
> for changing variables like this defined in the project-wide settings.py file 
> on a per-app basis?
There is some precedent, for example, the WEBPACK_LOADER configuration is 
configured globally and custom Django apps that plug in need to update that 
configuration. Custom Django apps can provide 'settings' in their AppConfig 
class that are merged with the global settings, for example [0].

However, I'm curious, why do you want to redefine MEDIA_ROOT?
> I'm finding that {% include .html %} works perfectly but {% extends 
> html %} does not. I use the block/endblock syntax with unique names but 
> the content doesn't show up. Is there a reason I should use extends over 
> include?

'extend' is used so that your template can inherit the common page structure 
and your template then provides content via 'blocks', see [1].

If you isn't working then perhaps there is some path or lookup issue that is 
causing the base template to fail to be found. If you want we can meet and look 
at it further.


[0] 
https://github.com/InterACTWEL/interactactwel-django-app/blob/dev/interactwel/apps.py
 

[1] https://docs.djangoproject.com/en/3.0/ref/templates/language/#id1 




smime.p7s
Description: S/MIME cryptographic signature

Re: [External] Airavata database connection issue

[Christie, Marcus Aaron]

That's great, what was the resolution?

> On May 20, 2020, at 7:56 PM, Narvekar, Bhakti Sunil  wrote:
> 
> Thank you for the help. The issue was resolved last week.
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: [External] Airavata database connection issue

[Christie, Marcus Aaron]

Hi Bhakti,

I have had issues with the mysql server started by docker-compose and it 
appears that the docker image got corrupted somehow. I was able to fix it by 
deleting and redownloading the image which I did by doing

docker-composer down
docker-compose rm
docker system prune --all

Thanks,

Marcus

> On May 15, 2020, at 7:02 PM, Narvekar, Bhakti Sunil  wrote:
> 
> Hi Professor,
>  
> Thank you for the response. 
> I did search on StackOverFlow and other websites and tried changing the 
> version of MariaDB in pom.xml but doing this did not resolve the issue.
> I will try connecting the database container externally. 
>  
> Thanks.
>  
> Sincerely,
> Bhakti
>  
> From: Suresh Marru 
> Sent: Friday, May 15, 2020 4:32 PM
> To: Airavata Dev 
> Subject: Re: [External] Airavata database connection issue
>  
> Hi Bhakti,
> 
> Looks like your issues are generic port and database. May be you should 
> search for issues on stack overflow and places like that. The error message 
> is clear, the service externally is unable to connect to the database. So you 
> may want to ignore airavata for a while and try to connect to the database 
> container (externally) and verify before you switch back to bring up airavata 
> services.
> 
> Suresh
> 
>> On May 14, 2020, at 11:49 PM, Narvekar, Bhakti Sunil > > wrote:
>> 
>> Hi,
>>  
>> I’m still not able to resolve this issue. It would be great if someone can 
>> help me out.
>>  
>> Thanks.
>>  
>> Sincerely,
>> Bhakti
>>  
>> From: Narvekar, Bhakti Sunil 
>> Sent: Wednesday, May 13, 2020 4:14 PM
>> To: dev@airavata.apache.org 
>> Subject: RE: [External] Re: Airavata database connection issue
>>  
>> Hi Aarushi,
>>  
>> I tried stopping the local MySQL instance and also I tried uninstalling the 
>> instance as well but I’m still facing the same issue. All the containers are 
>> up but now I’m not able to access localhost:13306 as well.
>>  
>> I have attached the StackTrace and logs for MySQL. 
>>  
>> Thanks,
>> Bhakti
>>  
>> From: Bisht, Aarushi 
>> Sent: Wednesday, May 13, 2020 3:45 PM
>> To: dev@airavata.apache.org 
>> Subject: RE: [External] Re: Airavata database connection issue
>>  
>> Hi Bhakti,
>> 
>> Since you have a local mysql instance running that is why you are able to 
>> connect to localhost 13306. Can you try to stop the mysql instance running 
>> locally and only start the docker containers. If the container has 
>> successfully started mysql should be available on localhost:3306. You can 
>> see all the running containers using docker-compose ps.
>> 
>> Thanks,
>> Aarushi Bisht
>> 
>> On May 13, 2020 1:03 PM, "Narvekar, Bhakti Sunil" > > wrote:
>> Hi Vivek,
>>  
>> I feel this is a different issue. I have attached the StackTrace as well.
>> Please let me know if you think it is similar.
>>  
>> Thanks,
>> Bhakti
>>  
>> From: Bandaru, Vivek Shresta 
>> Sent: Wednesday, May 13, 2020 12:04 PM
>> To: dev@airavata.apache.org 
>> Subject: Re: [External] Re: Airavata database connection issue
>>  
>> Hi Bhakti,
>>  
>> This looks very similar to the database connection error a couple of us 
>> faced few weeks ago.
>> Without the full stacktrace I can’t say for sure, but you can try the fix 
>> mentioned in this mail: 
>> https://markmail.org/message/oojdzfmczss2gepr?q=Airavata+database+connection+error+while+starting
>>  
>> 
>>  
>> Regards,
>> Vivek.
>>  
>> From: "Narvekar, Bhakti Sunil" mailto:bhn...@iu.edu>>
>> Reply-To: "dev@airavata.apache.org " 
>> mailto:dev@airavata.apache.org>>
>> Date: Wednesday, May 13, 2020 at 11:52 AM
>> To: "dev@airavata.apache.org " 
>> mailto:dev@airavata.apache.org>>
>> Subject: RE: [External] Re: Airavata database connection issue
>>  
>> Hi Aarushi,
>>  
>> The services are up and running. I have attached the screenshots for the 
>> same. The localhost:13306 is reachable but localhost:3306 is not.
>> I have attached the result of the telnet for both the ports 3306 and 13306.
>> Also, I have MySQL server installed on my local machine, would that be an 
>> issue.
>>  
>> Thanks & Regards,
>> Bhakti Narvekar
>>  
>> From: Bisht, Aarushi 
>> Sent: Wednesday, May 13, 2020 4:15 AM
>> To: dev@airavata.apache.org 
>> Subject: RE: [External] Re: Airavata database connection issue
>>  
>> Hi Bhakti, 
>>  
>> From the logs it seems you are not able to connect to database on localhost: 
>> 13306.
>> Make sure db container is up and running. Try using nc or some other command 
>> to make sure that you are able to connect to this port.
>>  
>> - Aarushi Bisht

Re: Current view about airavata-django-portal. Should it be redesigned

[Christie, Marcus Aaron]

> On Apr 13, 2020, at 2:49 PM, Dinuka Desilva  wrote:
> 
> Hi Marcus,
> 
> 
> 
> On Sat, Apr 11, 2020 at 1:41 AM Christie, Marcus Aaron  <mailto:machr...@iu.edu>> wrote:
> Hi Dinuka,
> 
> My comments below. I'm also CCing the dev list and I suggest we take this 
> discussion there.
> 
>> On Apr 10, 2020, at 11:40 AM, Dinuka Desilva > <mailto:l.dinukadesi...@gmail.com>> wrote:
>> 
>> Dear community,
>> 
>> I started look in and dig in to airavata-django-portal to start work on the 
>> desktop application for airavata. But, my current understanding about the 
>> airavata django portal is, it's a good idea to redesign it. Few point to say 
>> like that is,
>> REST api
>> There's improvements needed on the auth flow.
>> Endpoints are only session based and does not support token based 
>> authentication or authorization.
>> Some of the endpoints are redundant and I would say the design can be better.
> 
> This all sounds good and would be good contributions. Long-long term, we're 
> looking at moving to gRPC on the Airavata API backend so eventually the REST 
> api as a separate bridge may go away if the Django portal can make calls to 
> the Airavata API server directly.
> 
> Is this happening already? If so, any timelines? Otherwise is this something 
> we could start off with?

No timelines, but something to be aware of.

> 
> I believe the front-end work has to wait for it. And what about 
> authentication after moving to gRPC? Are we gonna use google authentication 
> or have our own. I would vote for google auth. :)

I imagine it will still be OAuth2 based. When you say google auth do you mean 
OAuth2 or something else?
>> Front end
>> I feel it's better to have the front end application as a separate 
>> application connected to the rest api. So, that it can be managed separately 
>> plus customisations can be done irrespective of the api.
> 
> Yeah, specifically, I think we might want to move to client-side routing, 
> which is used in the admin app, but not in all of the apps.
>> Considering the components reusability between desktop application and web 
>> application, the architecture can be redesigned.
> 
> Absolutely.
> 
>> Considering the project I would say, it's better to think of the long term 
>> approach rather adding up some more code which means for refactoring. Few 
>> more additional points I would like to add are,
>> Storage management is a common requirement not only for airavata portal but 
>> also in general. For an example, the election project i'm currently working 
>> is also having such requirements and actually we also have built something 
>> on our side. But, if we could generalise it to a common component, it could 
>> be reused.
> 
> What do you mean by storage management?
> What I meant was user file storage available in the portal.
>  
> I'm not familiar with your election project, what did you do regarding 
> storage management there?
> The sri lanka election project 
> <https://github.com/ECLK/results-tabulation-api> is an initiative from Lanka 
> Software Foundation. It has a requirement of keeping file and folder storage. 
> I was just thinking whether both of us could come to a generic component.
>  

Yes, sounds good.
>> 
>> Since this api is meant for the clients to consume directly, it'll be better 
>> to have some kind of more documentation as in like open api spec for an 
>> example. So, that they could play around with it.
> 
> There is a browseable API, if you go to localhost:8000/api. However, to be 
> user friendly it does require some more work. There are plugins to 
> django-rest-framework that can generate an open api spec or interoperate with 
> one. Those might be worth investigating.
> 
>> Actually this is my understanding for few weeks and I would like to have 
>> your thoughts and advices.
> 
> This all sounds good. I encourage to post your specific ideas, as detailed as 
> possible, to get the discussion going.



smime.p7s
Description: S/MIME cryptographic signature

Re: Current view about airavata-django-portal. Should it be redesigned

[Christie, Marcus Aaron]

Hi Dinuka,

My comments below. I'm also CCing the dev list and I suggest we take this 
discussion there.

> On Apr 10, 2020, at 11:40 AM, Dinuka Desilva  
> wrote:
> 
> Dear community,
> 
> I started look in and dig in to airavata-django-portal to start work on the 
> desktop application for airavata. But, my current understanding about the 
> airavata django portal is, it's a good idea to redesign it. Few point to say 
> like that is,
> REST api
> There's improvements needed on the auth flow.
> Endpoints are only session based and does not support token based 
> authentication or authorization.
> Some of the endpoints are redundant and I would say the design can be better.

This all sounds good and would be good contributions. Long-long term, we're 
looking at moving to gRPC on the Airavata API backend so eventually the REST 
api as a separate bridge may go away if the Django portal can make calls to the 
Airavata API server directly.
> Front end
> I feel it's better to have the front end application as a separate 
> application connected to the rest api. So, that it can be managed separately 
> plus customisations can be done irrespective of the api.

Yeah, specifically, I think we might want to move to client-side routing, which 
is used in the admin app, but not in all of the apps.
> Considering the components reusability between desktop application and web 
> application, the architecture can be redesigned.

Absolutely.

> Considering the project I would say, it's better to think of the long term 
> approach rather adding up some more code which means for refactoring. Few 
> more additional points I would like to add are,
> Storage management is a common requirement not only for airavata portal but 
> also in general. For an example, the election project i'm currently working 
> is also having such requirements and actually we also have built something on 
> our side. But, if we could generalise it to a common component, it could be 
> reused.

What do you mean by storage management? I'm not familiar with your election 
project, what did you do regarding storage management there?
> Since this api is meant for the clients to consume directly, it'll be better 
> to have some kind of more documentation as in like open api spec for an 
> example. So, that they could play around with it.

There is a browseable API, if you go to localhost:8000/api. However, to be user 
friendly it does require some more work. There are plugins to 
django-rest-framework that can generate an open api spec or interoperate with 
one. Those might be worth investigating.

> Actually this is my understanding for few weeks and I would like to have your 
> thoughts and advices.

This all sounds good. I encourage to post your specific ideas, as detailed as 
possible, to get the discussion going.



smime.p7s
Description: S/MIME cryptographic signature

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[Christie, Marcus Aaron]

> On Apr 9, 2020, at 9:26 PM, DImuthu Upeksha  
> wrote:
> 
> @Marcus, @Isuru
> 
> I will regenerate them for now but in the long run we need to generate them 
> automatically when the docker compose is executed. Probably via another 
> container instance. However I'm not sure why it's even working at least for 
> 30 minutes. It should be failed from the very first call to the Keycloak. For 
> my setup, it doesn't even fail after 30 minutes. 
> 

Thanks Dimuthu. If it's easier we might put Keycloak behind an HTTPS proxy 
instead of configuring Keycloak to handle HTTPS.  Here are some rough notes I 
have from when I set this up with Apache handling HTTPS proxying to Keycloak:


## Proxy Keycloak behind Apache

```
setsebool -P httpd_can_network_connect on
```

https://serverfault.com/a/799090

Turns out not strictly necessary, but I found this too:

```
...

...
RequestHeader set X-Forwarded-Proto "https"
...
```

https://stackoverflow.com/a/57140468

https://www.keycloak.org/docs/latest/server_installation/index.html#_setting-up-a-load-balancer-or-proxy

Fixed by

1. Adding `proxy-address-forwarding="true"` and `redirect-socket="proxy-https"`
   in standalone.xml:

```xml

```

2. Also changed to

```xml

```

3. Added `ProxyPreserveHost On` to Apache VirtualHost

```ini
RequestHeader set X-Forwarded-Proto "https"
ProxyPass / "http://localhost:8080/;
ProxyPassReverse / "http://localhost:8080/;
ProxyPreserveHost On
```



smime.p7s
Description: S/MIME cryptographic signature

Re: How to generate the structured js:es6 client stub for airavata api

[Christie, Marcus Aaron]

> On Apr 9, 2020, at 11:40 AM, Dinuka Desilva  wrote:
> 
> Oh. I got it. My question is how the token is passed for other endpoint like 
> this https://testdrive.airavata.org/api/applications/list_all/ 
> ?

Well, currently, it's not. The token is stored in the user's Django session. If 
the user is logged in, then that user can can call the API endpoints (Django 
assigns a cookie to identify the session).

We could add a token based authentication though. We would need a new custom 
authenticator [1] and we would need to extend the KeycloakBackend [2] to be 
able to authenticate a user based on an access token.

[1] 
https://www.django-rest-framework.org/api-guide/authentication/#custom-authentication
 

[2] 
https://github.com/apache/airavata-django-portal/blob/master/django_airavata/apps/auth/backends.py#L21
 


Then you would be able to call the Django REST API when the user's access token.

smime.p7s
Description: S/MIME cryptographic signature

Re: How to generate the structured js:es6 client stub for airavata api

[Christie, Marcus Aaron]

> On Apr 9, 2020, at 11:25 AM, Dinuka Desilva  wrote:
> 
> I'm not sure how it could be used for django api requests. As in, whether to 
> pass the "code" as a header etc. An example request call will be helpful.

as a query parameter. See [1], in Django request.GET is a dictionary of query 
parameters. Notice that the response is a JSON response.

[1] 
https://github.com/apache/airavata-django-portal/blob/develop/django_airavata/apps/auth/views.py#L458
 




smime.p7s
Description: S/MIME cryptographic signature

Re: How to generate the structured js:es6 client stub for airavata api

[Christie, Marcus Aaron]

Hi Dinuka,

> On Apr 8, 2020, at 3:50 PM, Dinuka Desilva  wrote:
> 
> I tried this out. And some more questions
> 
> 1) http://localhost:8000/auth/login-desktop 
>  from this upon successful login, 
> it's redirected to 
> https://testdrive.airavata.org/auth/login-desktop-success/? 
>  with status, 
> code, refresh_code, valid_time and username as query parameters. But, a 
> custom redirection url cannot be defined. (I couldn't find a way)
> 

The setting LOGIN_REDIRECT_URL sets the url to redirect to after logging in.

> 2) Once above is executed, I ended up with the issue I have mentioned on 
> another email thread [1]. I tried three times and it happened right after 
> "login-desktop" request.
> 
> 3) I tried this on https://testdrive.airavata.org/ 
>  also it has no issues. But, still I want to 
> get the redirection happen to a custom url that I could specify. Could you 
> help me figure out the right settings for that.
> 
> 4) From the "login-desktop" requests, I get only the status, code, 
> refresh_code, valid_time and username. Using these, how can I obtain the 
> access token. Is there an api for that? I found only this 
> "refreshed-token-desktop". But, still it returns no tokens in the response.

The "code" is the access token.  You can supply refresh_code to 
refresh-token-desktop and the django portal will refresh it and give you an 
access token.



smime.p7s
Description: S/MIME cryptographic signature

Re: How to generate the structured js:es6 client stub for airavata api

[Christie, Marcus Aaron]

Hi Dinuka,

There is a /auth/login-desktop/ view in airavata-django-portal that has the 
user login and then exposes the access token, etc. on the URL as query 
parameters. This is what SEAGrid Desktop uses to login.  You could start with 
that.

That's maybe not the best way to do it, so I'm open to suggestions. I think 
ideally the desktop client would have its own client id and secret and log 
users in directly.

> On Apr 7, 2020, at 11:42 AM, Dinuka Desilva  wrote:
> 
> Hi Isuru,
> 
> The django portal front end is not completely separated from backend and its 
> being managed via django views. So, actually both api and front end are 
> running on the same server. Consequently, any session or cookie variables are 
> accessible across. 
> 
> But, when comes to the electron app, it's UI is separate (unless it uses a 
> deployed url). So, for the authentication a redirection has to happen from 
> the electron app to the django app and upon success, it has to be redirected 
> back to the electron app. But, the thing is no cookies or sessions or any 
> variables are passed back. 
> 
> Following is how I got it redirected. Electron can be run on just file system 
> also. But, since redirecting to file:// is not supported, I put up a server 
> (ports to be changed etc.)
> https://testdrive.airavata.org/auth/login?next=http://localhost:8080/ 
> 
> If I could get the right approach I could go ahead. I'm kind of blocked here.
> 
> Regards,
> Dinuka
> 
> 
> 
> On Tue, Apr 7, 2020 at 8:41 AM Isuru Ranawaka  > wrote:
> Hi Dinuka,
> 
> Adding Marcus to the thread. He may also have good ideas on this.
> 
> On Mon, Apr 6, 2020 at 6:03 PM Dinuka Desilva  > wrote:
> Hi Isuru & Suresh,
> 
> Few concerns I have are,
> Existing login implementation is a form submission and a server rendered 
> html. And it's session based. 
> The endpoints are also session based and goes through CSRF verification.
> So, I'm not quite seeing any clear direction than electron app directly 
> accessing the app by url. Any advice is much appreciated. 
> 
> 
> 
> 
> what I have understood from your concerns, is that you are worrying about 
> session management between the backend server and the frontend. Basically, 
> where to decouple frontend view management logic from the frontend server API 
> management layer. Is that your concern?  Can you explain a bit more about how 
> your electron app design decouples view management components (including 
> HTML, CSS, JS) from the server access API layer?.  Does it have any state 
> management mechanism?
> 
> Anyhow, we need  CSRF verifications at least for authentication requests 
> between frontend and backend. But, there should be a CSRF verification 
> process between browser requests and frontend server. 
> 
> 
> thanks
> Isuru
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 
> Regards,
> Dinuka
> 
> On Mon, Apr 6, 2020 at 6:31 PM Isuru Ranawaka  > wrote:
> Hi Dinuka,
> 
> On Mon, Apr 6, 2020 at 3:11 AM Dinuka Desilva  > wrote:
> 
> 
> On Mon, 6 Apr 2020, 06:36 Suresh Marru,  > wrote:
> Hi Dinuka,
> 
> We have not successfully used Thrift generated JS previously (its possible 
> but do not have that experience within Airavata). Django portal uses the 
> python generated code and exposes them as REST API’s using DRF 
> (https://www.django-rest-framework.org/ 
> ). The Vue.js UI components 
> communicate to these REST API’s. I wonder if you can have electronJS talk to 
> the same API’s instead of directly to Airavata API.
> 
> Yes. Since airavata APIs doesn't have any authentication or authorization 
> layer, I  have to use the Django API. My only worry is then this become only 
> a copy of the same application. Is that the only purpose of this?
> 
> Airavata APIs do not have an authentication layer. But it has an 
> authorization layer. You can refer to AiravataAPIServer class there it 
> engages a security interceptor for authorization.  Anyhow, I guess using same 
> APIs that used by Vue.js will enhanced code reusability otherwise there will 
> be two code bases for the same functionality.
> 
> 
> 
> 
> Also, we would like to move from Thrift to Protobuf and gRPC. I wonder if 
> REST support can be more seamless once the migration is done. 
> 
> Suresh
> 
>> On Apr 5, 2020, at 4:17 PM, Dinuka Desilva > > wrote:
>> 
>> Hi,
>> 
>> I'm trying to generate the es6 client stub for airavata api using the 
>> following script.
>> 
>> thrift -r --gen js:es6 
>> ../../airavata/thrift-interface-descriptions/airavata-apis/airavata_api.thrift
>>  
>> 
>> But, I'm not getting it correctly I guess. I'm getting a list of files in a 
>> folder called gen-js. Instead what I need is a structured code as there in 
>> the 

Re: [airavata-django-portal] Cannot keep the localhost db running for more than half a hour

[Christie, Marcus Aaron]

I've seen this problem too. Are there an errors in the 'docker-compose up' 
output?

> On Apr 5, 2020, at 9:55 PM, Isuru Ranawaka  wrote:
> 
> Hi all,
> 
> 
> 
> On Sun, Apr 5, 2020 at 8:59 PM Suresh Marru  > wrote:
> Hi Isuru,
> 
> I wonder if it is related to renew tokens? Any insights?
> 
> There is a chance. But,  AFAIK  the server caches authorization decisions for 
> at least one hour through auth cache (by default this is enabled). So it is 
> unlikely that the server may try to renew tokens before that time. Breaking 
> in 30 minutes interval is wired.  Anyhow, there is a probability of occurring 
> this if sever unable to access  sharing registry. Hence,   I think better to 
> check whether  DB connections and IAM sever connections are correctly 
> established. 
> 
> Is there any error logs on server-side ?
> 
> thanks
> Isuru
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 
> Suresh
> 
>> On Apr 3, 2020, at 6:48 AM, Dinuka Desilva > > wrote:
>> 
>> Hi,
>> 
>> I got the airava and airavata-django-portal running on localhost by 
>> executing following script on 4 terminals one after the other. 
>> docker-compose down && rm -rf database_data/ && docker-compose up
>> mvn exec:java 
>> -Dexec.mainClass="org.apache.airavata.ide.integration.APIServerStarter"
>> . ./build_js.sh
>> source ./venv/bin/activate && rm -rf ./tmp/ && rm -rf db.sqlite3 && python 
>> manage.py migrate && python manage.py load_default_gateway && python 
>> manage.py runserver
>> 
>> With this I could access the localhost server at http://localhost:8000 
>> . But, only for less than half an hour it worked and 
>> then it's ended up with the following error. It has the logged in user 
>> though an error occurred. 
>> 
>> 
>> 
>> Neither restarting the servers nor clearing cache didn't solve this issue. 
>> And I had to execute above four script again which recreates the database. 
>> For kind of each 20-30 minutes, I had to do this. I'm doubtful whether I'm 
>> doing anything wrong. 
>> 
>> Also, below I have attached the settings_local.py.
>> 
>> Thanka & Regards,
>> Dinuka
>> 
> 
> 
> 
> -- 
> Research Software Engineer
> Indiana University, IN



smime.p7s
Description: S/MIME cryptographic signature

Re: cluster reservations metadata

[Christie, Marcus Aaron]

> On Feb 27, 2020, at 10:53 AM, Pamidighantam, Sudhakar  wrote:
> 
> This is great. If we get reservations for all queues/partitions in a resource 
> does it then require separate entries for each queue on the system or can a 
> flag like “all” indicate it should be applied to all queues defined/available 
> in the resource. 
> 

Hi Sudhakar,

It would be one entry for the reservation. In the user interface you might have 
checkboxes for the queues that the reservation applies to. Also there could be 
an All option that would select all checkboxes, that type of thing.

Thanks,

Marcus

smime.p7s
Description: S/MIME cryptographic signature

cluster reservations metadata

[Christie, Marcus Aaron]

Hi Dev,

I'm working on https://issues.apache.org/jira/browse/AIRAVATA-3126 
 to add reservation 
metadata to the GroupComputeResourcePreference data model.  I want to get some 
feedback on the proposed data model:

* A GroupComputeResourcePreference can have one or more Reservations
  * This changes from the current data model which is that there can only be 
one reservation
* Each Reservation can have one or more queues/partitions. A cluster 
reservation applies to one or more (or all) queues of the 
GroupComputeResourcePreference's compute host.
  * This changes from the current data model in which a reservation is applied 
to all queues of a cluster, which isn't correct when a reservation has been 
created for only a subset of a compute resource's queues.
* Each Reservation has a reservationName and a startTime and endTime.

My hope is that this data model reflects how cluster reservations are used in 
practice. If I've missed anything, please let me know.

Thanks,

Marcus



smime.p7s
Description: S/MIME cryptographic signature

Re: Upgrade Java Version

[Christie, Marcus Aaron]

Thanks for this summary of the work involved. Also kudos to you as well for 
making this possible.

> On Dec 6, 2019, at 5:10 PM, DImuthu Upeksha  
> wrote:
> 
> Hi Folks,
> 
> We have upgraded java version to 11 with OpenJDK in development branch and 
> successfully deployed it in development environment. Following changes were 
> performed in the process
> 
> 1. Upgrades of compiler plugin (3.5.1 -> 3.8.0) , surefire plugin (2.18.1 -> 
> 3.0.0-M1), open jpa (2.4.3 -> 3.1.0). See PR 236 [3]
> 2. Adding javax dependencies explicitly as they were discontinued in default 
> java 11 releases. See PR 234 [4]
> 3. Updating JDK to OpenJDK 11 [5]
> 4. Disabling Dozer mapper setter optimization [6]. This is a temporary fix 
> and waiting for dozer team [7] to properly fix this. 
> 5. Fixing remote debug [8]
> 6. Minor test failure fixes [9]
> 7. Upgrading Confluent/ Kafka version to (4.1.0 -> 5.3.0) [10]
> 
> Travis build and develop environment are now stable :)
> 
> Kudos to Isuru, Marcus and Suresh for pulling this off.  
> 
> [3] https://github.com/apache/airavata/pull/236/files 
> <https://github.com/apache/airavata/pull/236/files>
> [4] https://github.com/apache/airavata/pull/234/files 
> <https://github.com/apache/airavata/pull/234/files>
> [5] 
> https://github.com/apache/airavata/commit/fd56623e303e579d7974b721ecd7f931498ad502
>  
> <https://github.com/apache/airavata/commit/fd56623e303e579d7974b721ecd7f931498ad502>
> [6] 
> https://github.com/DImuthuUpe/dozer/commit/4d33729e866982cb7cee61065f0719c9c1212334
>  
> <https://github.com/DImuthuUpe/dozer/commit/4d33729e866982cb7cee61065f0719c9c1212334>
> [7] 
> https://groups.google.com/forum/?fromgroups=#!topic/dozer-mapper/c1OPee1DwTw 
> <https://groups.google.com/forum/?fromgroups=#!topic/dozer-mapper/c1OPee1DwTw>
> [8] 
> https://github.com/apache/airavata/commit/5fb5629dfaaf503e1875737c58f703e5cdfad255
>  
> <https://github.com/apache/airavata/commit/5fb5629dfaaf503e1875737c58f703e5cdfad255>
> [9] 
> https://github.com/apache/airavata/commit/7b1a2b3525f93a7057325b9df1e7251282e5294f
>  
> <https://github.com/apache/airavata/commit/7b1a2b3525f93a7057325b9df1e7251282e5294f>
> [10] 
> https://github.com/apache/airavata/commit/0859211fd8bfb8e2168e95d711fd27d83133c939
>  
> <https://github.com/apache/airavata/commit/0859211fd8bfb8e2168e95d711fd27d83133c939>
> 
> Thanks
> Dimuthu
> 
> On Tue, Oct 22, 2019 at 1:49 PM Suresh Marru  <mailto:sma...@apache.org>> wrote:
> Hi All,
> 
> I posted a question for Apache Legal - 
> https://issues.apache.org/jira/browse/LEGAL-484 
> <https://issues.apache.org/jira/browse/LEGAL-484>
> 
> + 1 for moving to Java 11 and get the Travis build working again. We probably 
> should try it against both OpenJDK and Oracle JDK latest 11 releases. 
> 
> Suresh
> 
>> On Oct 21, 2019, at 4:34 PM, Christie, Marcus Aaron > <mailto:machr...@iu.edu>> wrote:
>> 
>> Might be good to ask Apache legal, but it's not like we're distributing 
>> Oracle JDK, so not sure what kind of guidance they could give.
>> 
>> In the meantime, it would be great to get the Travis build working again.
>> 
>>> On Oct 21, 2019, at 4:07 PM, DImuthu Upeksha >> <mailto:dimuthu.upeks...@gmail.com>> wrote:
>>> 
>>> Hi Folks,
>>> 
>>> What should we do about this? Should we contact Apache legal support to get 
>>> things clarified about the licensing? We might need to stick to a decision 
>>> before next release.
>>> 
>>> Thanks
>>> Dimuthu
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Upgrade Java Version

[Christie, Marcus Aaron]

Might be good to ask Apache legal, but it's not like we're distributing Oracle 
JDK, so not sure what kind of guidance they could give.

In the meantime, it would be great to get the Travis build working again.

> On Oct 21, 2019, at 4:07 PM, DImuthu Upeksha  
> wrote:
> 
> Hi Folks,
> 
> What should we do about this? Should we contact Apache legal support to get 
> things clarified about the licensing? We might need to stick to a decision 
> before next release.
> 
> Thanks
> Dimuthu



smime.p7s
Description: S/MIME cryptographic signature

Re: 0.18 Release Planning

[Christie, Marcus Aaron]

+1

> On Oct 21, 2019, at 3:34 PM, Suresh Marru  wrote:
> 
> Hi All,
> 
> Can we create a 0.18 branch from master and start release process? I can work 
> on this release in 10 days, but unless there are objections, we can 
> immediately create this 0.18 branch and after that merge staging branch into 
> master. 
> 
> Suresh



smime.p7s
Description: S/MIME cryptographic signature

Re: Upgrade Java Version

[Christie, Marcus Aaron]

+1 for Java 11

> On Sep 12, 2019, at 10:28 AM, Suresh Marru  wrote:
> 
> + 1 for Java 11.
> 
> Any objections to move forward to make Java 11 minimum requirement? 
> 
> Suresh
> 
>> On Sep 10, 2019, at 6:06 PM, DImuthu Upeksha > > wrote:
>> 
>> Hi Suresh,
>> 
>> I would vote for Java 11 as it's relatively mature and a LTS version [1]
>> 
>> [1] https://www.oracle.com/technetwork/java/java-se-support-roadmap.html 
>> 
>> 
>> Thanks
>> Dimuthu
>> 
>> On Thu, Sep 5, 2019 at 8:34 AM Suresh Marru > > wrote:
>> Hi All,
>> 
>> I noticed a travis build fail which complains minimum have required of 9 to 
>> 12 - “Expected feature release number in range of 9 to 14, but got: 8”.
>> 
>> Any thoughts on which version of JDK we should upgrade Airavata to before 
>> next release?
>> 
>> Cheers,
>> Suresh
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Generating thrift stubs for PHP is throwing errors

[Christie, Marcus Aaron]

Hi Shivam,

 

Base is relatively new, see base_api.thrift, so probably those thrift stubs 
hadn’t been generated and committed for the PHP code yet. You can go ahead and 
commit those.

 

The Facades are not generated code. We wrote those to integrate with Laravel’s 
service container.

 

From: "Rastogi, Shivam" 
Reply-To: "dev@airavata.apache.org" 
Date: Monday, July 22, 2019 at 5:51 PM
To: "dev@airavata.apache.org" 
Subject: Generating thrift stubs for PHP is throwing errors

 

Hi Team,

 

I am adding a new method to the RegistryAPIServer. I have updated the following 
files airavata_api.thrift and registry-api.thrift.

 

But when I am executing the shell scripts to generate the stubs for php, the 
Airavata.php files refers to Base/API/BaseAPIIf interface but in the actual pga 
library the Base folder is missing and there is a new folder called Facades.


Could someone please direct me whether I have updated all the correct files or 
not? What is the difference between these two Facades and Base?

 

"

interface AiravataIf extends \Airavata\Base\API\BaseAPIIf {


"

 

 

Thanks and regards,
Shivam Rastogi



smime.p7s
Description: S/MIME cryptographic signature

Re: Error in starting Airavata API server locally

[Christie, Marcus Aaron]

Hi Aarushi,

 

You’ll need to run the db migration scripts against your db: 
https://github.com/apache/airavata/tree/develop/modules/registry/release-migration-scripts/0.18-0.19/DeltaScripts

 

You should be able to run, in the DeltaScripts directory:

 

  cat *.sql | mysql

 

 

From: "Bisht, Aarushi" 
Reply-To: "dev@airavata.apache.org" 
Date: Tuesday, July 9, 2019 at 12:29 PM
To: "dev@airavata.apache.org" 
Subject: Error in starting Airavata API server locally

 

Hi All,

 

Since the past few weeks, I am not able to start the airavata server locally 
using the develop branch. 

 

I am getting the following exception:

 

Exception in thread "main" java.lang.RuntimeException: Failed to 
post-initialize the expcatalog database​

Caused by:  
org.apache.openjpa.persistence.ArgumentException: 
"org.apache.airavata.registry.core.entities.expcatalog.ExperimentOutputEntity.metaData"
 declares column "METADATA", but this column does not exist in table 
"EXPERIMENT_OUTPUT".

 

Up till now I had checked out an older commit from git and was using that 
branch to make local changes. But I will not be able to raise a merge request 
to develop branch using this old branch.

 

Is there anybody who is looking into this issue? 

 

Regards,

Aarushi Bisht



smime.p7s
Description: S/MIME cryptographic signature

Re: [Django portal] Error on Manage Users page in local dev environment

[Christie, Marcus Aaron]

Hi Shivam,

 

It was added to the ‘develop’ branch recently [1].

 

[1] 
https://github.com/apache/airavata/commit/c9e0509c974cf907ba2f77d56568eb142a1a4900

 

From: "Rastogi, Shivam" 
Reply-To: "dev@airavata.apache.org" 
Date: Thursday, June 6, 2019 at 6:14 PM
To: "dev@airavata.apache.org" 
Subject: [Django portal] Error on Manage Users page in local dev environment

 

Hi All,

 

I have set up local development environment for the Django portal and 
everything seems to be working fine but when I navigate to the 'Manage users' 
page on the admin dashboard, I get the following error in the browser console:

 

I checked that and found that the getUsers method is available in the python 
API but could not find the corresponding method in the Airavata API. Any ideas 
what might be the problem?

​

FetchUtils.js?1b5f:168 GET 
http://127.0.0.1:8000/api/managed-user-profiles/?limit=10 500 (Internal Server 
Error)

processFetch @ FetchUtils.js?1b5f:168

get @ FetchUtils.js?1b5f:128

serviceObj. @ ServiceFactory.js?2b58:288

..

(anonymous) @ app.js:1564

(anonymous) @ app.js:1567

Show 38 more frames

 

ErrorReporter.js?9189:5 {

"id": 0,

"message": "Invalid method name: 'getUsers'",

"error": {

"details": {

"url": "/api/managed-user-profiles/?limit=10",

"status": 500,

"response": {

"detail": "Invalid method name: 'getUsers'"

}

}

},

"details": {

"url": "/api/managed-user-profiles/?limit=10",

"status": 500,

"response": {

"detail": "Invalid method name: 'getUsers'"

}

},

"suppressDisplay": false,

"suppressLogging": false,

"createdDate": "2019-06-06T21:34:39.193Z"

}

FetchUtils.js?1b5f:189 Uncaught (in promise) Error: Invalid method name: 
'getUsers'

at eval (FetchUtils.js?1b5f:189)

 

 

Thanks and regards,

Shivam Rastogi

 



smime.p7s
Description: S/MIME cryptographic signature

Re: Login error for new users in Dev SciGap

[Christie, Marcus Aaron]

Hi Shivam,

 

I just realized you wrote dev.scigap.org, not dev.seagrid.org. Yeah, let’s take 
a look at it today.

 

From: "Rastogi, Shivam" 
Reply-To: "dev@airavata.apache.org" 
Date: Tuesday, June 4, 2019 at 10:56 AM
To: "dev@airavata.apache.org" 
Subject: Re: Login error for new users in Dev SciGap 

 

Hi Marcus, 

 

Yes, the Django portal is working fine. I will discuss it with you in office 
about how I can get access to the dev pga portal.

 

Regards,

Shivam Rastogi

From: Christie, Marcus Aaron 
Sent: 04 June 2019 09:17
To: dev@airavata.apache.org
Subject: [External] Re: Login error for new users in Dev SciGap 

 

This message was sent from a non-IU address. Please exercise caution when 
clicking links or opening attachments from external sources.


Hi Shivam, 

 

Thanks for the bug report. I think I changed the api when working on the Django 
portal. Try Django.seagrid.org to log in.

 

 

Sent from my iPhone


On Jun 3, 2019, at 5:10 PM, Rastogi, Shivam  wrote:

Hi All,

 

I created a new account in https://dev.scigap.org/login but I am getting an 
error while logging into it.

I have created the following JIRA to track this request 
https://issues.apache.org/jira/browse/AIRAVATA-3046

 

Thanks  and regards,

Shivam Rastogi



smime.p7s
Description: S/MIME cryptographic signature

Re: Login error for new users in Dev SciGap

[Christie, Marcus Aaron]

Hi Shivam,

Thanks for the bug report. I think I changed the api when working on the Django 
portal. Try Django.seagrid.org to log in.


Sent from my iPhone

On Jun 3, 2019, at 5:10 PM, Rastogi, Shivam 
mailto:sras...@iu.edu>> wrote:


Hi All,


I created a new account in https://dev.scigap.org/login but I am getting an 
error while logging into it.

I have created the following JIRA to track this request 
https://issues.apache.org/jira/browse/AIRAVATA-3046


Thanks  and regards,

Shivam Rastogi

Re: docker-compose up errors

[Christie, Marcus Aaron]

Hi Bibrak,

 

I think the KEYCLOAK_USER and KEYCLOAK_PASSWORD env variables are causing 
Keycloak to try to add this user on start up. Try commenting out these lines 
[1] in docker-compose.yml.

 

[1] 
https://github.com/apache/airavata/blob/develop/modules/ide-integration/src/main/resources/docker-compose.yml#L6-L7

 

From: Bibrak Qamar 
Reply-To: "dev@airavata.apache.org" 
Date: Tuesday, May 21, 2019 at 7:01 PM
To: "dev@airavata.apache.org" 
Subject: docker-compose up errors

 

Hi all,

 

docker-compose up gives me these two errors in its long list of output: 

 

keycloak_1   | 22:51:28,774 WARN  
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (ServerService Thread Pool 
-- 49) SQL Error: 23505, SQLState: 23505
keycloak_1   | 22:51:28,774 ERROR 
[org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (ServerService Thread Pool 
-- 49) Unique index or primary key violation: 
"UK_RU8TT6T700S9V50BU18WS5HA6_INDEX_F ON PUBLIC.USER_ENTITY(REALM_ID, USERNAME) 
VALUES ( /* key:22 */ null, null, null, null, null, null, null, null, 'master', 
'admin', null, null)"; SQL statement:
keycloak_1   | insert into USER_ENTITY (CREATED_TIMESTAMP, EMAIL, 
EMAIL_CONSTRAINT, EMAIL_VERIFIED, ENABLED, FEDERATION_LINK, FIRST_NAME, 
LAST_NAME, REALM_ID, SERVICE_ACCOUNT_CLIENT_LINK, USERNAME, ID) values (?, ?, 
?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [23505-173]
keycloak_1   | 22:51:28,788 INFO  
[org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (ServerService 
Thread Pool -- 49) HHH10: On release of batch it still contained JDBC 
statements
keycloak_1   | 22:51:28,793 ERROR [org.keycloak.services] (ServerService Thread 
Pool -- 49) KC-SERVICES0010: Failed to add user 'admin' to realm 'master': user 
with username exists

 

 

 

Is it normal or they need to be fixed before proceeding?

 

Bibrak Qamar



smime.p7s
Description: S/MIME cryptographic signature

Re: Setup a development environment for Airavata

[Christie, Marcus Aaron]

Hi Lahiru,

 

Looks like that Django issue got fixed in 2.x but not in 1.11 [1]. You can give 
2.0.13 a try with “pip install Django==2.0.13”.

 

However, if it’s only load_default_gateway that fails, you can just leave it 
out unless you need to have the wagtail CMS running.

 

[1] https://github.com/django/django/pull/10733

 

From: Lahiru Jayathilake 
Reply-To: "dev@airavata.apache.org" 
Date: Tuesday, May 14, 2019 at 12:18 PM
To: "dev@airavata.apache.org" 
Subject: Re: Setup a development environment for Airavata

 

Hi Marcus, 

 

Yes "python3 manage.py migrate" ran successfully with no errors. 

Python version - 3.7.3

OS - macOS Mojave version 10.14.4

 

Regards,

Lahiru

 

On Tue, 14 May 2019 at 03:26, Christie, Marcus Aaron  wrote:

Hi Lahiru, 

 

Did “python3 manage.py migrate” finish without errors?

 

Also what version of python and what os are you using?

 

Thanks,

 

Marcus 

Sent from my iPhone


On May 4, 2019, at 2:26 PM, Lahiru Jayathilake  wrote:

Hi Everyone, 

 

I have been setting up the development environment and faced several issues and 
found the following solutions.

 

1. When running org.apache.airavata.ide.integration.APIServerStarter

Error - [ 

error-APIServerStarter.pdf

] 

Solution - Updated the ide-integration/pom.xml to have org.apache.openjpa plugin

 



 

Tried several times running the APIServerStarter and got the same error. After 
fixing the issue with afore-mentioned solution, I tried one more time reverting 
back and that time APIServerStarter ran as expected.

 

2. When running pip install -r requirements.txt in airavata-django-portal

Error - [ 

error-pillow.pdf

] because wagtail 2.0 has requirement of Pillow<5.0,>=2.6.1

Solution - Added 'pillow==2.6.1' in the requirements.txt

 

3. When running python3 manage.py migrate 

Error - [ 

error-django-verison.pdf

]

Solution - Updated the version of Django to 1.11.20 

 

And now I am having the issue when running python3 manage.py 
load_default_gateway 

The error was [ 

error-no-such-table-wagtailcore_page__old.pdf

]  django.db.utils.OperationalError: Problem installing fixtures: no such 
table: wagtailcore_page__old

I found an article related to this [1] according to the article, the problem is 
with sqlite version so I tried the solution which is downgrading sqlite (to 
3.24.0) but still I am getting the same error. Can anyone help me with this?

 

[1] - https://github.com/wagtail/wagtail/issues/4941

 

Cheers!

Lahiru

 

On Fri, 15 Mar 2019 at 23:51, Christie, Marcus Aaron  wrote:

Hi Keerthi,



On Mar 15, 2019, at 2:15 PM, Keerthi N  wrote:

 

It got resolved after npm version update. It was 3.6.2 previously and got 
updated to 6.9.0, everything works fine now.

 

 

Glad to hear it. Can you please submit a PR to update the README [1] regarding 
prerequisites? I would suggest you add something about installing the latest 
LTS release of Node [2].  Thanks.

 

[1] 
https://github.com/apache/airavata/blob/develop/modules/ide-integration/README.md

[2] https://nodejs.org/en/download/

 


 

-- 

Lahiru Jayathilake 

Department of Computer Science and Engineering,

Faculty of Engineering,

University of Moratuwa

 










 

-- 

Lahiru Jayathilake 

Department of Computer Science and Engineering,

Faculty of Engineering,

University of Moratuwa

 




smime.p7s
Description: S/MIME cryptographic signature

Re: Login failed: Could not connect to any of [('::1', 8962, 0, 0), ('127.0.0.1', 8962), ('fe80::1%lo0', 8962, 0, 1)]. Please try again.

[Christie, Marcus Aaron]

Hi Bibrak,

 

This sounds similar to the problem Lahiru ran into [1]. See what Lahiru did, 
looks like Lahiru was able to work around it.

 

[1] https://markmail.org/thread/gh26dlze5evfx3vl

 

 

From: Bibrak Qamar 
Reply-To: "dev@airavata.apache.org" 
Date: Tuesday, May 14, 2019 at 1:55 AM
To: "dev@airavata.apache.org" 
Subject: Login failed: Could not connect to any of [('::1', 8962, 0, 0), 
('127.0.0.1', 8962), ('fe80::1%lo0', 8962, 0, 1)]. Please try again.

 

Hi all,

 

So, I have managed to follow the ide-integration guide provided here: 
https://github.com/apache/airavata/tree/develop/modules/ide-integration

 

When I login to the portal it shows the following error message:

 

Login failed: Could not connect to any of [('::1', 8962, 0, 0), ('127.0.0.1', 
8962), ('fe80::1%lo0', 8962, 0, 1)]. Please try again. 

 

I am also copying the output of my APIServerStarter. It seems it might have 
issue? 

 

What do you think could be the problem?

 

 

 

..:...::/Users/bibrakqamar/.m2/repository/javax/mail/mail/1.4.5/mail-1.4.5.jar
 org.apache.airavata.ide.integration.APIServerStarter

objc[5362]: Class JavaLaunchHelper is implemented in both 
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java 
(0x108f324c0) and 
/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/jre/lib/libinstrument.dylib
 (0x109f114e0). One of the two will be used. Which one is undefined.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/Users/bibrakqamar/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/bibrakqamar/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
01:48:39,495 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could 
NOT find resource [logback-test.xml]
01:48:39,497 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could 
NOT find resource [logback.groovy]
01:48:39,497 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found 
resource [logback.xml] at 
[file:/Users/bibrakqamar/work/GSoC/gitrops/airavata/modules/ide-integration/target/classes/logback.xml]
01:48:39,502 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource 
[logback.xml] occurs multiple times on the classpath.
01:48:39,509 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource 
[logback.xml] occurs at 
[file:/Users/bibrakqamar/work/GSoC/gitrops/airavata/modules/configuration/server/target/classes/logback.xml]
01:48:39,509 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource 
[logback.xml] occurs at 
[file:/Users/bibrakqamar/work/GSoC/gitrops/airavata/modules/airavata-helix/helix-spectator/target/classes/logback.xml]
01:48:39,509 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource 
[logback.xml] occurs at 
[file:/Users/bibrakqamar/work/GSoC/gitrops/airavata/modules/job-monitor/email-monitor/target/classes/logback.xml]
01:48:39,509 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource 
[logback.xml] occurs at 
[file:/Users/bibrakqamar/work/GSoC/gitrops/airavata/modules/ide-integration/target/classes/logback.xml]
01:48:39,986 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction 
- debug attribute not set
01:48:40,042 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About 
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
01:48:40,094 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming 
appender as [CONSOLE]
01:48:40,143 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA 
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] 
for [encoder] property
01:48:40,480 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About 
to instantiate appender of type 
[ch.qos.logback.core.rolling.RollingFileAppender]
01:48:40,493 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming 
appender as [LOGFILE]
01:48:40,498 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA 
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] 
for [encoder] property
01:48:40,522 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@797925218 - 
setting totalSizeCap to 1 GB
01:48:40,543 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@797925218 - No 
compression will be used
01:48:40,553 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@797925218 - 
Will use the pattern ../logs/airavata.log.%d{-MM-dd} for the active file
01:48:40,568 |-INFO in 
c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - The date 
pattern is '-MM-dd' from file name pattern 
'../logs/airavata.log.%d{-MM-dd}'.
01:48:40,568 |-INFO in 
c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Roll-over at 

Re: Cleaning up unused code

[Christie, Marcus Aaron]

+1

Sent from my iPhone

On Apr 29, 2019, at 6:22 PM, DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:

Hi Folks,

I'm working on $subject and reducing the size of distributions with minimal 
dependencies. I will keep adding the classes that I'm going to remove to this 
thread and please let me know whether  those are required.

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/DefaultAiravataSecurityManager.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/oauth/DefaultOAuthClient.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/Main.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/xacml/DefaultPAPClient.java

https://github.com/apache/airavata/blob/master/airavata-services/services-security/src/main/java/org/apache/airavata/service/security/xacml/DefaultXACMLPEP.java

Thanks
Dimuthu

Re: Approach towards AIRAVATA-2975

[Christie, Marcus Aaron]

Hi Aravind,

> On Apr 8, 2019, at 1:19 PM, Aravind Parappil  
> wrote:
> 
> Thanks for your feedback, Marcus!
> 
> Regarding the data-product, I am not fetching all experiment inputs from the 
> array experiment.experimentInputs because when I try to use the 
>  tag on the input.value field, Vue throws a warning 
> "Invalid prop: type check failed for prop "dataProduct". Expected 
> DataProduct, got Object" for inputs of type 'URI'. I guessed this is because 
> experimentInputs array contains normal objects and not DataProducts, so I am 
> fetching DataProducts from the existing inputDataProducts array itself (it 
> doesn't have a name field (only productName), hence I couldn't keep the 
> inputName: value format for the bullets).
> 
> Questions:
> Is my above approach fine? It works properly and gets all the required 
> information apart from the inputName for URI inputs

I would loop over experiment.experimentInputs, then if the type is URI, look 
for the DataProduct in inputDataProducts.
> 
> Am I right to assume that the existing inputDataProducts array is being 
> formed by the API using getDataProduct(), like you mentioned? If so, should 
> the API be modified to add an 'inputName' field too?
Yes, getDataProduct() called to build up the FullExperiment REST API response. 
But no we don't want to add 'inputName' to that.
> If not, how would I go about fetching the DataProduct by passing the data 
> product URI from Vue? I checked existing Vue Services and couldn't find one 
> which is related. I might be looking at the wrong places, however.
> Once again, thank you so much for your patience!
> 
> Thanks & Regards,
> Aravind Parappil
> Master of Science in Computer Science, 2020
> School of Informatics, Computing, and Engineering
> Indiana University Bloomington
> 
> 
> On Mon, Apr 8, 2019 at 12:25 PM Christie, Marcus Aaron  <mailto:machr...@iu.edu>> wrote:
> Hi Aravind,
> 
>> On Apr 7, 2019, at 2:31 PM, Aravind Parappil > <mailto:aravindparap...@gmail.com>> wrote:
>> 
>> Hi team,
>> 
>> I have worked on the bug  
>> <https://issues.apache.org/jira/browse/AIRAVATA-2975>where String inputs to 
>> experiments were not being shown on Experiment Summary page of the Django 
>> portal. 
>> 
>> However, I have a couple of doubts before I decide to proceed with a PR on 
>> the same.
>> I was testing the working of string input using already existing Echo 
>> experiments. However, it takes only 1 input. Is there an application that 
>> takes more than 1 string input so that I can test whether my implementation 
>> works for multiple strings? 
> You can create an Application with more than one input of type string. You 
> can look at the Echo application, go to /admin/applications/ to see a list of 
> applications and click on one to edit the application. Go to the Interface 
> tab and then you can see how the inputs are defined and add additional ones.
> 
>> Am I allowed to create new experiments? If yes, on which resource (Big Red?)
> 
> For now, just save the experiments but don't launch them. 
> 
>> Here is my approach:
>> Created a new computed property (stringInputs) that filters out 
>> experimentInputs of type STRING from experiment object
>> Had to do this (and not take everything from experimentInputs) because 
>> experimentInputs do not contain objects of type data-product
>> @Christie, Marcus Aaron <mailto:machr...@iu.edu> - is it possible to convert 
>> an obj into a data-product?
> I'm not sure what you are asking here. The experiment input 'value' is a data 
> product URI when the type of the experiment input is 'URI'. The data product 
> has to be separately fetched from the API using getDataProduct(), passing the 
> data product URI.
>> Populate a bulleted list of inputName: value, if any exist
>> Kept the earlier logic for displaying data-products (wrapped it in a 
>>  so that it doesn't render on the DOM if no input files exist)
>> Here's how it looks now:
>> 
>> 
>> Above is from experiment Abinit on Aug 1, 2018 3:49 PM. First input is a 
>> string, others are files  
> 
> This looks really good. One suggestion, use the "inputName: value" format for 
> all the inputs and display them in the same order as they are in the 
> "experiment.experimentInputs" array.
> 
>> 
>> Thanks & Regards,
>> Aravind Parappil
>> Master of Science in Computer Science, 2020
>> School of Informatics, Computing, and Engineering
>> Indiana University Bloomington
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Approach towards AIRAVATA-2975

[Christie, Marcus Aaron]

Hi Aravind,

> On Apr 7, 2019, at 2:31 PM, Aravind Parappil  
> wrote:
> 
> Hi team,
> 
> I have worked on the bug  
> <https://issues.apache.org/jira/browse/AIRAVATA-2975>where String inputs to 
> experiments were not being shown on Experiment Summary page of the Django 
> portal. 
> 
> However, I have a couple of doubts before I decide to proceed with a PR on 
> the same.
> I was testing the working of string input using already existing Echo 
> experiments. However, it takes only 1 input. Is there an application that 
> takes more than 1 string input so that I can test whether my implementation 
> works for multiple strings? 
You can create an Application with more than one input of type string. You can 
look at the Echo application, go to /admin/applications/ to see a list of 
applications and click on one to edit the application. Go to the Interface tab 
and then you can see how the inputs are defined and add additional ones.

> Am I allowed to create new experiments? If yes, on which resource (Big Red?)

For now, just save the experiments but don't launch them. 

> Here is my approach:
> Created a new computed property (stringInputs) that filters out 
> experimentInputs of type STRING from experiment object
> Had to do this (and not take everything from experimentInputs) because 
> experimentInputs do not contain objects of type data-product
> @Christie, Marcus Aaron <mailto:machr...@iu.edu> - is it possible to convert 
> an obj into a data-product?
I'm not sure what you are asking here. The experiment input 'value' is a data 
product URI when the type of the experiment input is 'URI'. The data product 
has to be separately fetched from the API using getDataProduct(), passing the 
data product URI.
> Populate a bulleted list of inputName: value, if any exist
> Kept the earlier logic for displaying data-products (wrapped it in a 
>  so that it doesn't render on the DOM if no input files exist)
> Here's how it looks now:
> 
> 
> Above is from experiment Abinit on Aug 1, 2018 3:49 PM. First input is a 
> string, others are files  

This looks really good. One suggestion, use the "inputName: value" format for 
all the inputs and display them in the same order as they are in the 
"experiment.experimentInputs" array.

> 
> Thanks & Regards,
> Aravind Parappil
> Master of Science in Computer Science, 2020
> School of Informatics, Computing, and Engineering
> Indiana University Bloomington



smime.p7s
Description: S/MIME cryptographic signature