Sailaja Mada created CLOUDSTACK-4228:
----------------------------------------
Summary: [VMWARE]NPE & operation timed out issues while migrating
volumes across different Zone wide primary storages
Key: CLOUDSTACK-4228
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4228
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller, VMware
Affects Versions: 4.2.0
Reporter: Sailaja Mada
Priority: Critical
Steps:
1. Upgraded from 2.2.16 to 4.2 with VMWARE cluster
2. Configured new zone with VMWARE cluster of 2 hosts with zone wide primary
storage
3. Added one more Zone wide primary storage to this new zone
4. Deploy instance on this zone
5. Add new DATA volume and attach to this instance
6. Tried to migrate ROOT/DATA volume of this instance from Zone wide primary
storage 1 to second Zone wide primary storage which is added to the same zone
Observation:
1. NPE & operation timed out issues are observed while migrating volumes across
different Zone wide primary storage's
2. It is observed 3 times when tried about 6 times.
log & DB dumb is attached . This is a single node management server
2013-08-10 00:02:25,453 DEBUG [agent.manager.AgentAttache]
(StorageManager-Scavenger-4:null) Seq 18-1413808894: Waiting some more time
because this is the current command
2013-08-10 00:02:25,453 INFO [utils.exception.CSExceptionErrorCode]
(StorageManager-Scavenger-4:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for exceptions
2013-08-10 00:02:25,453 WARN [agent.manager.AgentAttache]
(StorageManager-Scavenger-4:null) Seq 18-1413808894: Timed out on Seq
18-1413808894: { Cmd , MgmtId: 90310994128556, via: 18, Ver: v1, Flags:
100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache]
(StorageManager-Scavenger-4:null) Seq 18-1413808894: Cancelling.
2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache]
(StorageManager-Scavenger-4:null) Seq 18-1413808894: No more commands found
2013-08-10 00:02:25,454 DEBUG [cloud.storage.StorageManagerImpl]
(StorageManager-Scavenger-4:null) Unable to send storage pool command to
org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5760aaa2 via 18
com.cloud.exception.OperationTimedoutException: Commands 1413808894 to Host 18
timed out after 3600
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:987)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:438)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:452)
at
com.cloud.template.TemplateManagerImpl.evictTemplateFromStoragePool(TemplateManagerImpl.java:793)
at
com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1039)
at
com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1335)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-08-10 00:02:25,456 DEBUG [agent.transport.Request]
(StorageManager-Scavenger-4:null) Seq 14-647496884: Sending { Cmd , MgmtId:
90310994128556, via: 14, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
2013-08-10 00:02:25,456 DEBUG [agent.transport.Request]
(StorageManager-Scavenger-4:null) Seq 14-647496884: Executing: { Cmd , MgmtId:
90310994128556, via: 14, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
}
2013-08-10 00:02:25,456 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-151:null) Seq 14-647496884: Executing request
2013-08-10 00:02:25,457 WARN [agent.manager.DirectAgentAttache]
(DirectAgent-151:null) Seq 14-647496884: Exception caught
java.lang.NullPointerException
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:195)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-08-10 00:02:26,138 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-13:null) SeqA 10-13769: Processing Seq 10-13769: { Cmd ,
MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:26,145 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-13:null) SeqA 10-13769: Sending Seq 10-13769: { Ans: ,
MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-10 00:02:31,705 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-10:null) SeqA 16-11861: Processing Seq 16-11861: { Cmd ,
MgmtId: -1, via: 16, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:31,712 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-10:null) SeqA 16-11861: Sending Seq 16-11861: { Ans: ,
MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-10 00:02:36,142 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-11:null) SeqA 10-13770: Processing Seq 10-13770: { Cmd ,
MgmtId: -1, via: 10, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:36,151 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-11:null) SeqA 10-13770: Sending Seq 10-13770: { Ans: ,
MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-10 00:02:36,707 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-14:null) SeqA 16-11862: Processing Seq 16-11862: { Cmd ,
MgmtId: -1, via: 16, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-08-10 00:02:36,710 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-14:null) SeqA 16-11862: Sending Seq 16-11862: { Ans: ,
MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
