[jira] [Commented] (AIRAVATA-2620) Force post processing functionality
[ https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380833#comment-16380833 ] Eroma commented on AIRAVATA-2620: - h1. Tested in: [https://dev.ultrascan.scigap.org|https://dev.ultrascan.scigap.org/] h1. Test Cases h2. NOTE: # The implementation is mainly tested for Ultrascan gateway. # This service will be added as a new method to the bridge code. # Single experiment to execute at a time. # User should be notified when the file transfer is completed. Currently when the force processing is run if there are no files to transfer it will not throw any errors or messages. h2. Test Cases # User save and launch an experiment. Post processing is run while experiment is in EXECUTING, Job in QUEUED. Results: No files will be transferred as the job is still QUEUED. No errors in the FORCE_PROCESSING. Experiment status will be changed to FORCE_PROCESSING_COMPLETED. - PASS # User launch an experiment. Post processing is run while experiment is in EXECUTING, Job in QUEUED (Email monitoring has stopped and emails have received for both job start and completion). Results: Files are transferred as the job is actually done. No errors in the FORCE_PROCESSING. Experiment status will be changed to FORCE_PROCESSINF_EXECUTING and to FORCE_PROCESSING_COMPLETED. - PASS # User creates an experiment and when its in CREATED runs FORCE_PROCESSING. Results: No files will be transferred as the experiment is in CREATED. Results: No files transferred. No experiment status changed. In api-orch log warning message [1] exists. – NEED IMPROVEMENT # User runs the post processing with incorrect non-existing gateway ID – If experiment in created log has [2] If experiment is failed log has [3] - FAIL # Use runs FORCE_PROCESSING with NULL as experiment ID. Error returned [7] and error in Log [8] = PASS # User runs FORCE_RPOCESSING giving NULL as gateway ID –From PGA point of view nothing happened but exception thrown in the backend api-orch level. Log has the exception [6]. Exception does not indicate the gateway ID is non existing. User should have been informed that the gateway ID is missing. - FAIL # When experiment in COMPLETED runs FORCE_PROCESSING. Results: No files will be transferred as the experiment is in COMPLETED as files should be already transferred. – # User runs POST_PROCESSING for an experiment which is FAILED. Log has [4] – No exceptions should be thrown. – This situation should provide user with a descriptive message. - NEED IMPROVEMENT # User runs post processing when the job is failed due to incorrect input file. # Test post processing for experiments with ARCHIVING enabled before job completion. – Ran FORCE_PROCESSING before the job completion. Existing files along with ARCHIVE got transferred but ARCHIVE folder is empty. When the job and experiment actually got completed after FORCE_PROCESSING archive.tar was not removed. GFAC log has the command to remove it but its not removed. Check exp ID Ultrascan-T2_96439421-467d-4b15-884c-5a8fb42c6d36 on 27^th^ February, 18 – FAIL # Test post processing for experiments with ARCHIVING enabled. Run FORCE_PROCESSING after job completion. ARCHIVE folder is empty. – FAIL # Test an experiment which only have ARCHIVE, no other output files defined to stage in app interface. ARCHIVE folder exists but empty. - FAIL # Run FORCE_PROCESSING when GFAC server is down, but job is all completed. # Run FORCE_PROCESSING when GFAC server is down, but job is all completed. Once the FORCE_PROCESSING is completed start the GFAC server. # Run FORCE_PROCESSING when GFAC server is down, job is also not launched. No errors or exceptions. FORCE_PROCESSING didn’t run - PASS # Run FORCE_PROCESSING when GFAC server is down, and job is not launched. Start FORCE_PROCESSING and then start the GFAC server. – When GFAC server was started the FORCE_PROCESSING completed and created the empty ARCHIVE directory. (After the actual experiment completion was run using email monitoring the archive.tar exists in the folder, still ARCHIVE folder is empty) - FAIL h2. Improvements # When the force processing is done, user should be messaged. # If there were no files to transfer or less number of files than expected should be notified. # List the files that got transferred. # When user runs the force processing for experiments in CRETED, COMPLETED or FAILED where files would not be transferred or when gateway ID is wrong should throw exceptions with meaningful messages to the user. h2. Exceptions (Log messages) [1] 2018-02-21 16:53:02,882 [pool-30-thread-11] WARN o.a.a.o.s.OrchestratorServerHandler gateway_id=ultrascan - Can't run post processing of the CREATED experiment with id Ultrascan-JS1_9e998170-0630-46b8-92b7-7f6e06d9a527 2018-02-21 16:53:24,125 [pool-30-thread-12] INFO o.a.a.m.c.i.ExperimentConsumer - Message Received with message id
[jira] [Commented] (AIRAVATA-2620) Force post processing functionality
[ https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369320#comment-16369320 ] Eroma commented on AIRAVATA-2620: - Resolving the JIRA on behalf of [~dimuthuupe] > Force post processing functionality > > > Key: AIRAVATA-2620 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2620 > Project: Airavata > Issue Type: Improvement >Affects Versions: 0.16 >Reporter: Suresh Marru >Assignee: Dimuthu Upeksha >Priority: Major > Fix For: 0.17 > > > Due to current limitations of only relying on email for job monitoring, the > post-processing sometimes has inherent delays. Ultrascan science gateway > would like to have a capability in airavata to request forcing of post > processing. This will be used when clients have out of band knowledge about > job completion (for example through code instrumented UDP messages) and would > like Airavata to force staging of output files. > This improvement has to be carefully added so existing life cycle of an > experiment is not hampred. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2620) Force post processing functionality
[ https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332517#comment-16332517 ] ASF subversion and git services commented on AIRAVATA-2620: --- Commit e3de5a05731c27b6eb640a7da4b97b177aad48fc in airavata's branch refs/heads/AIRAVATA-2620 from [~smarru] [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=e3de5a0 ] Merge branch 'master' into AIRAVATA-2620 > Force post processing functionality > > > Key: AIRAVATA-2620 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2620 > Project: Airavata > Issue Type: Improvement >Affects Versions: 0.16 >Reporter: Suresh Marru >Assignee: Dimuthu Upeksha >Priority: Major > Fix For: 0.17 > > > Due to current limitations of only relying on email for job monitoring, the > post-processing sometimes has inherent delays. Ultrascan science gateway > would like to have a capability in airavata to request forcing of post > processing. This will be used when clients have out of band knowledge about > job completion (for example through code instrumented UDP messages) and would > like Airavata to force staging of output files. > This improvement has to be carefully added so existing life cycle of an > experiment is not hampred. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRAVATA-2620) Force post processing functionality
[ https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316930#comment-16316930 ] ASF subversion and git services commented on AIRAVATA-2620: --- Commit 1e47fbe2308d9f3ff04a50b286df5328202b4162 in airavata's branch refs/heads/AIRAVATA-2620 from [~smarru] [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=1e47fbe ] Merge branch 'AIRAVATA-2620' of github.com:apache/airavata into AIRAVATA-2620 > Force post processing functionality > > > Key: AIRAVATA-2620 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2620 > Project: Airavata > Issue Type: Improvement >Affects Versions: 0.16 >Reporter: Suresh Marru >Assignee: Dimuthu Upeksha > Fix For: 0.17 > > > Due to current limitations of only relying on email for job monitoring, the > post-processing sometimes has inherent delays. Ultrascan science gateway > would like to have a capability in airavata to request forcing of post > processing. This will be used when clients have out of band knowledge about > job completion (for example through code instrumented UDP messages) and would > like Airavata to force staging of output files. > This improvement has to be carefully added so existing life cycle of an > experiment is not hampred. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRAVATA-2620) Force post processing functionality
[ https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316928#comment-16316928 ] ASF subversion and git services commented on AIRAVATA-2620: --- Commit 1e47fbe2308d9f3ff04a50b286df5328202b4162 in airavata's branch refs/heads/AIRAVATA-2620 from [~smarru] [ https://gitbox.apache.org/repos/asf?p=airavata.git;h=1e47fbe ] Merge branch 'AIRAVATA-2620' of github.com:apache/airavata into AIRAVATA-2620 > Force post processing functionality > > > Key: AIRAVATA-2620 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2620 > Project: Airavata > Issue Type: Improvement >Affects Versions: 0.16 >Reporter: Suresh Marru >Assignee: Dimuthu Upeksha > Fix For: 0.17 > > > Due to current limitations of only relying on email for job monitoring, the > post-processing sometimes has inherent delays. Ultrascan science gateway > would like to have a capability in airavata to request forcing of post > processing. This will be used when clients have out of band knowledge about > job completion (for example through code instrumented UDP messages) and would > like Airavata to force staging of output files. > This improvement has to be carefully added so existing life cycle of an > experiment is not hampred. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRAVATA-2620) Force post processing functionality
[ https://issues.apache.org/jira/browse/AIRAVATA-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303021#comment-16303021 ] Dimuthu Upeksha commented on AIRAVATA-2620: --- Fixed in https://github.com/apache/airavata/commit/10734eeb96faf77f5bb4692833194c5abb8c3e17 > Force post processing functionality > > > Key: AIRAVATA-2620 > URL: https://issues.apache.org/jira/browse/AIRAVATA-2620 > Project: Airavata > Issue Type: Improvement >Affects Versions: 0.16 >Reporter: Suresh Marru >Assignee: Dimuthu Upeksha > Fix For: 0.17 > > > Due to current limitations of only relying on email for job monitoring, the > post-processing sometimes has inherent delays. Ultrascan science gateway > would like to have a capability in airavata to request forcing of post > processing. This will be used when clients have out of band knowledge about > job completion (for example through code instrumented UDP messages) and would > like Airavata to force staging of output files. > This improvement has to be carefully added so existing life cycle of an > experiment is not hampred. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
