[ https://issues.apache.org/jira/browse/CLOUDSTACK-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sailaja Mada updated CLOUDSTACK-4228: ------------------------------------- Fix Version/s: 4.2.0 > [VMWARE]NPE & operation timed out issues while migrating volumes across > different Zone wide primary storages > ------------------------------------------------------------------------------------------------------------ > > Key: CLOUDSTACK-4228 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4228 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Storage Controller, VMware > Affects Versions: 4.2.0 > Reporter: Sailaja Mada > Priority: Critical > Fix For: 4.2.0 > > Attachments: apilog.log, db.dmp, management-server.log > > > Steps: > 1. Upgraded from 2.2.16 to 4.2 with VMWARE cluster > 2. Configured new zone with VMWARE cluster of 2 hosts with zone wide primary > storage > 3. Added one more Zone wide primary storage to this new zone > 4. Deploy instance on this zone > 5. Add new DATA volume and attach to this instance > 6. Tried to migrate ROOT/DATA volume of this instance from Zone wide primary > storage 1 to second Zone wide primary storage which is added to the same zone > Observation: > 1. NPE & operation timed out issues are observed while migrating volumes > across different Zone wide primary storage's > 2. It is observed 3 times when tried about 6 times. > log & DB dumb is attached . This is a single node management server > 2013-08-10 00:02:25,453 DEBUG [agent.manager.AgentAttache] > (StorageManager-Scavenger-4:null) Seq 18-1413808894: Waiting some more time > because this is the current command > 2013-08-10 00:02:25,453 INFO [utils.exception.CSExceptionErrorCode] > (StorageManager-Scavenger-4:null) Could not find exception: > com.cloud.exception.OperationTimedoutException in error code list for > exceptions > 2013-08-10 00:02:25,453 WARN [agent.manager.AgentAttache] > (StorageManager-Scavenger-4:null) Seq 18-1413808894: Timed out on Seq > 18-1413808894: { Cmd , MgmtId: 90310994128556, via: 18, Ver: v1, Flags: > 100111, > [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}] > } > 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] > (StorageManager-Scavenger-4:null) Seq 18-1413808894: Cancelling. > 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache] > (StorageManager-Scavenger-4:null) Seq 18-1413808894: No more commands found > 2013-08-10 00:02:25,454 DEBUG [cloud.storage.StorageManagerImpl] > (StorageManager-Scavenger-4:null) Unable to send storage pool command to > org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5760aaa2 via 18 > com.cloud.exception.OperationTimedoutException: Commands 1413808894 to Host > 18 timed out after 3600 > at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486) > at > com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439) > at > com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:987) > at > com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:438) > at > com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:452) > at > com.cloud.template.TemplateManagerImpl.evictTemplateFromStoragePool(TemplateManagerImpl.java:793) > at > com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1039) > at > com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1335) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] > (StorageManager-Scavenger-4:null) Seq 14-647496884: Sending { Cmd , MgmtId: > 90310994128556, via: 14, Ver: v1, Flags: 100111, > [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}] > } > 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request] > (StorageManager-Scavenger-4:null) Seq 14-647496884: Executing: { Cmd , > MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111, > [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}] > } > 2013-08-10 00:02:25,456 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-151:null) Seq 14-647496884: Executing request > 2013-08-10 00:02:25,457 WARN [agent.manager.DirectAgentAttache] > (DirectAgent-151:null) Seq 14-647496884: Exception caught > java.lang.NullPointerException > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:195) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > 2013-08-10 00:02:26,138 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-13:null) SeqA 10-13769: Processing Seq 10-13769: { Cmd > , MgmtId: -1, via: 10, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2013-08-10 00:02:26,145 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-13:null) SeqA 10-13769: Sending Seq 10-13769: { Ans: , > MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 2013-08-10 00:02:31,705 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-10:null) SeqA 16-11861: Processing Seq 16-11861: { Cmd > , MgmtId: -1, via: 16, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2013-08-10 00:02:31,712 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-10:null) SeqA 16-11861: Sending Seq 16-11861: { Ans: , > MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 2013-08-10 00:02:36,142 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-11:null) SeqA 10-13770: Processing Seq 10-13770: { Cmd > , MgmtId: -1, via: 10, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2013-08-10 00:02:36,151 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-11:null) SeqA 10-13770: Sending Seq 10-13770: { Ans: , > MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 2013-08-10 00:02:36,707 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-14:null) SeqA 16-11862: Processing Seq 16-11862: { Cmd > , MgmtId: -1, via: 16, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2013-08-10 00:02:36,710 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-14:null) SeqA 16-11862: Sending Seq 16-11862: { Ans: , > MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010, > [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } > 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira