云平台部署环境：cloudstack4.0.2+xenServer6.0.2基本网络模式
问题描述：
SSVM、CPVM无法启动成功，然后xenServer集群下的所有主机自动重启。
管理节点上报日志如下：
2015-02-03 10:09:45,562 INFO [utils.exception.CSExceptionErrorCode]
(secstorage-1:null) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for exceptions
2015-02-03 10:09:45,562 WARN [agent.manager.AgentAttache] (secstorage-1:null)
Seq 33-160759819: Timed out on Seq 33-160759819: { Cmd , MgmtId: 345052348430,
via: 33, Ver: v1, Flags: 100111,
[{"storage.CreateCommand":{"volId":3325,"pool":{"id":200,"uuid":"1a8e96b6-04a7-3b02-a04a-56ccdf4c54d3","host":"10.6.26.200","path":"/BMDPrimary1","port":2049,"type":"NetworkFilesystem"},"diskCharacteristics":{"size":565240320,"tags":[],"type":"ROOT","name":"ROOT-3251","useLocalStorage":false,"recreatable":true,"diskOfferingId":7,"volumeId":3325,"hyperType":"XenServer"},"templateUrl":"3651f783-6bf2-416b-8a8f-f5406c948ccd","wait":0}}]
}
2015-02-03 10:09:45,562 DEBUG [agent.manager.AgentAttache] (secstorage-1:null)
Seq 33-160759819: Cancelling.
2015-02-03 10:09:45,562 DEBUG [cloud.storage.StorageManagerImpl]
(secstorage-1:null) Unable to send storage pool command to
Pool[200|NetworkFilesystem] via 33
com.cloud.exception.OperationTimedoutException: Commands 160759819 to Host 33
timed out after 1800
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:421)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:505)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:458)
at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:2215)
at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:482)
at
com.cloud.storage.StorageManagerImpl.createVolume(StorageManagerImpl.java:3464)
at com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:3351)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:754)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2015-02-03 10:09:45,565 DEBUG [agent.transport.Request] (secstorage-1:null) Seq
1-1717043215: Sending { Cmd , MgmtId: 345052348430, via: 1, Ver: v1, Flags:
100111,
[{"storage.CreateCommand":{"volId":3325,"pool":{"id":200,"uuid":"1a8e96b6-04a7-3b02-a04a-56ccdf4c54d3","host":"10.6.26.200","path":"/BMDPrimary1","port":2049,"type":"NetworkFilesystem"},"diskCharacteristics":{"size":565240320,"tags":[],"type":"ROOT","name":"ROOT-3251","useLocalStorage":false,"recreatable":true,"diskOfferingId":7,"volumeId":3325,"hyperType":"XenServer"},"templateUrl":"3651f783-6bf2-416b-8a8f-f5406c948ccd","wait":0}}]
}
2015-02-03 10:09:45,565 DEBUG [agent.transport.Request] (secstorage-1:null) Seq
1-1717043215: Executing: { Cmd , MgmtId: 345052348430, via: 1, Ver: v1, Flags:
100111,
[{"storage.CreateCommand":{"volId":3325,"pool":{"id":200,"uuid":"1a8e96b6-04a7-3b02-a04a-56ccdf4c54d3","host":"10.6.26.200","path":"/BMDPrimary1","port":2049,"type":"NetworkFilesystem"},"diskCharacteristics":{"size":565240320,"tags":[],"type":"ROOT","name":"ROOT-3251","useLocalStorage":false,"recreatable":true,"diskOfferingId":7,"volumeId":3325,"hyperType":"XenServer"},"templateUrl":"3651f783-6bf2-416b-8a8f-f5406c948ccd","wait":0}}]
}
2015-02-03 10:09:45,565 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-160:null) Seq 1-1717043215: Executing request
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager vm manager to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager vm manager because it is not in a state to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager upgrade manager to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager StorageManager to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager StorageManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager Alert Manager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager Template Manager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager Template Manager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager Snapshot Manager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager Snapshot Manager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager SnapshotScheduler to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager SecurityGroupManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager SecurityGroupManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager EntityManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager LoadBalancingRulesManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager LoadBalancingRulesManager because it is not in a state to
shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager RulesManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager RulesManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager RemoteAccessVpnManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager Capacity Manager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager VirtualMachineManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager HypervisorGuruManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ResourceManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager ResourceManager because it is not in a state to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager IdentityManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager OCFS2Manager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager FirewallManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager FirewallManager because it is not in a state to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ConsoleProxyManager to shutdown.
2015-02-03 10:09:46,565 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl]
(Thread-5:null) Stop console proxy manager
2015-02-03 10:09:46,566 DEBUG [agent.manager.AgentAttache]
(consoleproxy-1:null) Seq 33-160759818: Interrupted
2015-02-03 10:09:46,566 DEBUG [agent.manager.AgentAttache]
(consoleproxy-1:null) Seq 33-160759818: Waiting some more time because this is
the current command
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager ConsoleProxyManager because it is not in a state to shutdown.
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ProjectManager to shutdown.
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager ProjectManager because it is not in a state to shutdown.
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager SwiftManager to shutdown.
2015-02-03 10:09:47,566 INFO [storage.swift.SwiftManagerImpl] (Thread-5:null)
Stop Swift Manager
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager StorageNetworkManager to shutdown.
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager StorageNetworkManager because it is not in a state to shutdown.
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ExternalLoadBalancerUsageManager to shutdown.
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager HA Manager to shutdown.
2015-02-03 10:09:47,567 INFO [cloud.ha.HighAvailabilityManagerImpl]
(HA-Worker-1:null) Time to go home!
2015-02-03 10:09:47,568 INFO [cloud.ha.HighAvailabilityManagerImpl]
(HA-Worker-4:null) Time to go home!
2015-02-03 10:09:47,568 INFO [cloud.ha.HighAvailabilityManagerImpl]
(HA-Worker-0:null) Time to go home!
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager VPC Manager to shutdown.
2015-02-03 10:09:47,568 INFO [cloud.ha.HighAvailabilityManagerImpl]
(HA-Worker-3:null) Time to go home!
2015-02-03 10:09:47,567 INFO [cloud.ha.HighAvailabilityManagerImpl]
(HA-Worker-2:null) Time to go home!
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager VPC Manager because it is not in a state to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager VpcVirtualRouterManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager VpcVirtualRouterManager because it is not in a state to
shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager NetworkACLManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager NetworkACLManager because it is not in a state to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager TaggedResourcesManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager Site2SiteVpnManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager Site2SiteVpnManager because it is not in a state to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ExternalNetworkManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager BareMetalVmManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Skipping Manager BareMetalVmManager because it is not in a state to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ExternalDhcpManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager PxeServerManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager NetworkUsageManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager OvsTunnelManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager NetappManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager ElasticLoadBalancerManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null)
Asking Manager VmwareManager to shutdown.
2015-02-03 10:09:47,569 INFO [vmware.manager.VmwareManagerImpl] (Thread-5:null)
Cleanup mounted NFS mount points used in current session
2015-02-03 10:09:47,990 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-46:null) Unable to create slave connection to
host(c568f879-25ee-4a2e-9cae-df53f39e5cc6) due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect
timed out
zt
