[jira] [Commented] (AIRAVATA-2739) Django: GroupResourceProfile selector when creating experiment
[ https://issues.apache.org/jira/browse/AIRAVATA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428915#comment-16428915 ] ASF subversion and git services commented on AIRAVATA-2739: --- Commit 5997dd81d0c20ceba27288d915aee44bf1d3e565 in airavata's branch refs/heads/group-based-auth from [~marcuschristie] [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=5997dd8 ] AIRAVATA-2739 Using new "appModuleId" constant > Django: GroupResourceProfile selector when creating experiment > -- > > Key: AIRAVATA-2739 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2739 > Project: Airavata > Issue Type: Bug >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRAVATA-2741) Ideas for better way to deal with arbitrary output files than ARCHIVE
[ https://issues.apache.org/jira/browse/AIRAVATA-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Christie updated AIRAVATA-2741: -- Description: Just want to capture some details of recent conversations with [~eroma_a] and [~spamidig] on how to improve Airavata capabilities so we can move beyond using ARCHIVE. The ARCHIVE capability is a bit of a hack and causes some issues for us. Just briefly, here are some of the problems: * pulls back absolutely every file but some aren't needed and some intermediate files are very large. For some applications it isn't even practical to use ARCHIVE * pulls back duplicates of Application Output files, further filling gateway data storage * these files are basically opaque to Airavata, so there is a limit on what can be done in a programmatic way for some of these files Here are some potential improvements: * improve wildcard support: allow specifying a wildcard that can match a single or multiple files. For multiple files these can all be registered as a URI_COLLECTION type data output. (Side note: I'm not sure what all is currently supported with the wildcard support, need to investigate) * Show all of the job files in the portal, including ones that aren't defined as Application Outputs and haven't actually been staged back to the portal, and allow the user to request pulling back one of these other files. This would be nice because there are certainly going to be cases where a file is generated that wasn't anticipated (either lack of configuration or just something truly not anticipatable). Would mean needing to register every file in the job directory, not just the Application Outputs (not sure where, replica catalog?). Would also mean we need backend task execution support for fetching these files as needed. was: Just want to capture some details of recent conversations with [~eroma_a] and [~spamidig] on how to improve Airavata capabilities so we can move beyond using ARCHIVE. The ARCHIVE capability is a bit of a hack and causes some issues for us. Just briefly, here are some of the problems: * pulls back absolutely every file but some aren't needed and some intermediate files are very large. For some applications it isn't even practical to use ARCHIVE * pulls back duplicates of Application Output files, further filling gateway data storage * these files are basically opaque to Airavata, so there is a limit on what can be done in a programmatic way for some of these files Here are some potential improvements: * improve wildcard support: allow specifying a wildcard that can match a single or multiple files. For multiple files these can all be registered as a URI_COLLECTION type data output. * Show all of the job files in the portal, including ones that aren't defined as Application Outputs and haven't actually been staged back to the portal, and allow the user to request pulling back one of these other files. This would be nice because there are certainly going to be cases where a file is generated that wasn't anticipated (either lack of configuration or just something truly not anticipatable). Would mean needing to register every file in the job directory, not just the Application Outputs (not sure where, replica catalog?). Would also mean we need backend task execution support for fetching these files as needed. > Ideas for better way to deal with arbitrary output files than ARCHIVE > - > > Key: AIRAVATA-2741 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2741 > Project: Airavata > Issue Type: Improvement >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > Just want to capture some details of recent conversations with [~eroma_a] and > [~spamidig] on how to improve Airavata capabilities so we can move beyond > using ARCHIVE. The ARCHIVE capability is a bit of a hack and causes some > issues for us. Just briefly, here are some of the problems: > * pulls back absolutely every file but some aren't needed and some > intermediate files are very large. For some applications it isn't even > practical to use ARCHIVE > * pulls back duplicates of Application Output files, further filling gateway > data storage > * these files are basically opaque to Airavata, so there is a limit on what > can be done in a programmatic way for some of these files > Here are some potential improvements: > * improve wildcard support: allow specifying a wildcard that can match a > single or multiple files. For multiple files these can all be registered as a > URI_COLLECTION type data output. (Side note: I'm not sure what all is > currently supported with the wildcard support, need to investigate) > * Show all of the job files in the portal, including ones that aren't defined
[jira] [Created] (AIRAVATA-2741) Ideas for better way to deal with arbitrary output files than ARCHIVE
Marcus Christie created AIRAVATA-2741: - Summary: Ideas for better way to deal with arbitrary output files than ARCHIVE Key: AIRAVATA-2741 URL: https://issues.apache.org/jira/browse/AIRAVATA-2741 Project: Airavata Issue Type: Improvement Reporter: Marcus Christie Assignee: Marcus Christie Just want to capture some details of recent conversations with [~eroma_a] and [~spamidig] on how to improve Airavata capabilities so we can move beyond using ARCHIVE. The ARCHIVE capability is a bit of a hack and causes some issues for us. Just briefly, here are some of the problems: * pulls back absolutely every file but some aren't needed and some intermediate files are very large. For some applications it isn't even practical to use ARCHIVE * pulls back duplicates of Application Output files, further filling gateway data storage * these files are basically opaque to Airavata, so there is a limit on what can be done in a programmatic way for some of these files Here are some potential improvements: * improve wildcard support: allow specifying a wildcard that can match a single or multiple files. For multiple files these can all be registered as a URI_COLLECTION type data output. * Show all of the job files in the portal, including ones that aren't defined as Application Outputs and haven't actually been staged back to the portal, and allow the user to request pulling back one of these other files. This would be nice because there are certainly going to be cases where a file is generated that wasn't anticipated (either lack of configuration or just something truly not anticipatable). Would mean needing to register every file in the job directory, not just the Application Outputs (not sure where, replica catalog?). Would also mean we need backend task execution support for fetching these files as needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2740) Non-existing file transfer has failed the experiment
[ https://issues.apache.org/jira/browse/AIRAVATA-2740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428425#comment-16428425 ] Dimuthu Upeksha commented on AIRAVATA-2740: --- Fixed in [https://github.com/apache/airavata/commit/721a55abf5b7dff1260f4fd23395003b5460f5e0] [https://github.com/apache/airavata/commit/e45108a00b3542907fc5494a2a4ef288a1fa9e3b] https://github.com/apache/airavata/commit/7bb426a243135e97cda181850fa6b48f1d5e059d > Non-existing file transfer has failed the experiment > > > Key: AIRAVATA-2740 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2740 > Project: Airavata > Issue Type: Bug > Components: helix implementation >Affects Versions: 0.18 > Environment: http://149.165.168.248:8008/ >Reporter: Eroma >Assignee: Dimuthu Upeksha >Priority: Major > > # Due to an 0 byte input file upload tan expected output file is not > generated in the application execution. > # The airavata tries to transfer the file an error is thrown as the file is > not existing. > # But a 0 byte output file is created in the gateway data storage and user > can view it. > # Exception is thrown [1] > # If a specified output file is not in the working directory, it should be > ignored, not create an empty file in the storage > # If a 0 byte file exists in the working directory it should be transferred > to the storage. > [1] > org.apache.airavata.agents.api.AgentException: java.io.FileNotFoundException: > /tmp/PROCESS_ac1064b1-226e-491c-91e6-303448f05f16/temp_inputs/Gaussian.log > (No such file or directory) at > org.apache.airavata.helix.agent.ssh.SshAgentAdaptor.copyFileTo(SshAgentAdaptor.java:307) > at > org.apache.airavata.helix.agent.storage.StorageResourceAdaptorImpl.uploadFile(StorageResourceAdaptorImpl.java:98) > at > org.apache.airavata.helix.impl.task.staging.DataStagingTask.transferFileToStorage(DataStagingTask.java:158) > at > org.apache.airavata.helix.impl.task.staging.OutputDataStagingTask.onRun(OutputDataStagingTask.java:163) > at > org.apache.airavata.helix.impl.task.AiravataTask.onRun(AiravataTask.java:265) > at org.apache.airavata.helix.core.AbstractTask.run(AbstractTask.java:82) at > org.apache.helix.task.TaskRunner.run(TaskRunner.java:70) at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at > java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) Caused by: > java.io.FileNotFoundException: > /tmp/PROCESS_ac1064b1-226e-491c-91e6-303448f05f16/temp_inputs/Gaussian.log > (No such file or directory) at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) at > java.io.FileInputStream.(FileInputStream.java:138) at > java.io.FileInputStream.(FileInputStream.java:93) at > org.apache.airavata.helix.agent.ssh.SshAgentAdaptor.copyFileTo(SshAgentAdaptor.java:276) > ... 13 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2710) How to assign owner of "everyone" group in Sharing Registry?
[ https://issues.apache.org/jira/browse/AIRAVATA-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428208#comment-16428208 ] Suresh Marru commented on AIRAVATA-2710: [~marcuschristie] I do not have any deep thoughts to add. But i am wondering if we step back and think, shouldn't there be a simpler way of support sharing with PUBLIC and AUTHENTICATED_USERS within the sharing registry? I remember the discussion for everyone groups, but for all reasons you mention, I feel we will over-engineer this and may be fundamentally re-thinking on this now might save us from corner cases tomorrow. Just an idle thought. > How to assign owner of "everyone" group in Sharing Registry? > > > Key: AIRAVATA-2710 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2710 > Project: Airavata > Issue Type: Bug >Reporter: Marcus Christie >Assignee: Marcus Christie >Priority: Major > > in AIRAVATA-2662 the "everyone" group is being added to the Sharing Registry. > A UserGroup in the Sharing Registry must have a owner. This presents a > problem, the "everyone" group cannot be created until there is a user who can > be the owner, but createUser should add each user to the "everyone" group. > For now the implementation of createUser creates the "everyone" group if it > doesn't already exist and makes this user the owner of the group. That's > less than ideal since the first user of a domain ends up the owner of the > "everyone" group. > Here are some possible alternatives: > * create a dummy admin user for the domain that is made the owner of the > everyone group > * allow groups to not have an owner (make the OWNER_ID column nullable on > USER_GROUP) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
