Eroma created AIRAVATA-1725:
-------------------------------
Summary: When GFAC is stopped and started while experiments are
launched in PGA getting Bad GFAC version error and exp fails
Key: AIRAVATA-1725
URL: https://issues.apache.org/jira/browse/AIRAVATA-1725
Project: Airavata
Issue Type: Sub-task
Components: GFac
Environment:
http://dev.test-drive.airavata.org/portal/ultrascan-testing/public
Reporter: Eroma
Assignee: Shameera Rathnayaka
Steps
Similar to above parent task
When the GFAC server is started immediately number of experiments failed with
error [1]. Experiment status is FAILED and job status is QUEUED
[1]
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:KeeperErrorCode
= BadVersion for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler/state
org.apache.airavata.gfac.GFacException: Error launching the Job at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:480)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:KeeperErrorCode
= BadVersion for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler/state
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:718)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:467)
... 5 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:KeeperErrorCode = BadVersion for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler/state
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:885)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:661)
... 6 more Caused by:
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.SSHDirectorySetupHandler/state
at org.apache.zookeeper.KeeperException.create(KeeperException.java:115) at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at
org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1228) at
org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:274)
at
org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:270)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at
org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:267)
at
org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:253)
at
org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:41)
at
org.apache.airavata.gfac.core.utils.GFacUtils.createHandlerZnode(GFacUtils.java:332)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:858)
... 7 more org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:KeeperErrorCode = NoNode for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler
org.apache.airavata.gfac.GFacException: Error launching the Job at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:480)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:179)
at
org.apache.airavata.gfac.core.utils.InputHandlerWorker.run(InputHandlerWorker.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
org.apache.airavata.gfac.GFacException: Error Invoking Handlers:KeeperErrorCode
= NoNode for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:718)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.submitJob(BetterGfacImpl.java:467)
... 5 more Caused by: org.apache.airavata.gfac.GFacException: Error Invoking
Handlers:KeeperErrorCode = NoNode for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:885)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.launch(BetterGfacImpl.java:661)
... 6 more Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/gfac-experiments/gfac-node1/SLM1-US-LoneStar-06-09_17-10-24_68c1219a-9652-4a1c-b983-d43cbc7f1f18/org.apache.airavata.gfac.ssh.handler.AdvancedSCPInputHandler
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:696)
at
org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:679)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) at
org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:676)
at
org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453)
at
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443)
at
org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:251)
at
org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:205)
at
org.apache.airavata.gfac.core.utils.GFacUtils.createHandlerZnode(GFacUtils.java:346)
at
org.apache.airavata.gfac.core.utils.GFacUtils.updateHandlerState(GFacUtils.java:376)
at
org.apache.airavata.gfac.core.cpi.BetterGfacImpl.invokeInFlowHandlers(BetterGfacImpl.java:871)
... 7 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
