????????????????????Storage agent or storage VM
disconnected,????SSVM??????????????????????
????SSVM??????????????SSVM????????????????
2014-07-15 10:41:46,679 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(AgentManager-Handler-7:null) create ClusteredAgentAttache for 41
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener:
XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener:
VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener:
ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:46,685 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener: SecurityGroupListener
2014-07-15 10:41:46,685 INFO [network.security.SecurityGroupListener]
(AgentManager-Handler-7:null) Received a host startup notification
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener: StoragePoolMonitor
2014-07-15 10:41:46,687 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener:
SecondaryStorageListener
2014-07-15 10:41:46,687 INFO [storage.secondary.SecondaryStorageListener]
(AgentManager-Handler-7:null) Received a host startup notification
com.cloud.agent.api.StartupSecondaryStorageCommand
2014-07-15 10:41:46,714 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640129: Sending { Cmd , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 100111,
[{"SecStorageSetupCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","certs":{},"wait":0}}]
}
2014-07-15 10:41:46,727 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 41
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request]
(AgentManager-Handler-10:null) Seq 42-1808072705: Processing: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110,
[{"SecStorageSetupAnswer":{"_dir":"a35b9ab9-d061-3ed7-bbe7-82022eec78e4","result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:46,776 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072705: Received: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-10:null) Seq 42-1808072705: No more commands found
2014-07-15 10:41:46,777 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Details from executing class
com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:46,809 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-2:null) Successfully programmed secondary storage
nfs://10.11.18.7/export/d3secondary in secondary storage VM s-909-VM
2014-07-15 10:41:46,831 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 41-766640130: Sending { Cmd , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 100011,
[{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
}
2014-07-15 10:41:46,880 DEBUG [agent.transport.Request]
(AgentManager-Handler-4:null) Seq 41-766640130: Processing: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 10,
[{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,881 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 41-766640130: Received: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,881 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-2:null) Successfully programmed firewall rules into SSVM
s-889-VM
2014-07-15 10:41:46,891 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072706: Sending { Cmd , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 100011,
[{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
}
2014-07-15 10:41:46,901 DEBUG [agent.transport.Request]
(AgentManager-Handler-8:null) Seq 42-1808072706: Processing: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 10,
[{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:46,902 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072706: Received: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:46,902 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-2:null) Successfully programmed firewall rules into
s-909-VM
2014-07-15 10:41:46,929 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072707: Sending { Cmd , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 100111,
[{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
}
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request]
(AgentManager-Handler-11:null) Seq 42-1808072707: Processing: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110,
[{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-11:null) Seq 42-1808072707: No more commands found
2014-07-15 10:41:47,083 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072707: Received: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Details from executing class
com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:47,083 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-2:null) Successfully programmed http auth into s-909-VM
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Connect to listener:
NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:47,083 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Connect to listener:
AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:47,084 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:47,101 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072708: Sending { Cmd , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 100111,
[{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}]
}
2014-07-15 10:41:47,629 DEBUG [agent.transport.Request]
(AgentManager-Handler-3:null) Seq 42-1808072708: Processing: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110,
[{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c":{"templateName":"275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c","installPath":"template/tmpl/2/275//90034c67-a7d5-33cb-be93-b3d0bd02de9d.qcow2","size":292001153024,"physicalSize":17294753792,"id":275,"isPublic":true,"isCorrupted":false},"217-2-0245468c-5e25-3513-92a4-89b4a045751f":{"templateName":"217-2-0245468c-5e25-3513-92a4-89b4a045751f","installPath":"template/tmpl/2/217//3da99fcd-4447-3e84-98c1-6ae7424ff649.qcow2","size":107374182400,"physicalSize":4641128448,"id":217,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//6c37c8df-1225-4bb5-bd0f-bf47eead5d09.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//e970d345-e30b-30c4-879c-88d63200a721.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false},"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1":{"templateName":"218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1","installPath":"template/tmpl/2/218//d3bfbf4b-902c-38db-9d45-2e500ac81dd7.qcow2","size":107374182400,"physicalSize":15100084224,"id":218,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-3:null) Seq 42-1808072708: No more commands found
2014-07-15 10:41:47,630 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072708: Received: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:47,630 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Details from executing class
com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:47,653 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync found routing-3 already in the
template host table
2014-07-15 10:41:47,705 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync found centos55-x86_64 already in
the template host table
2014-07-15 10:41:47,771 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync did not find centos56-x86_64-xen
ready on server 11, will request download to start/resume shortly
2014-07-15 10:41:47,775 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync did not find centos53-x64 ready on
server 11, will request download to start/resume shortly
2014-07-15 10:41:47,785 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync found
217-2-0245468c-5e25-3513-92a4-89b4a045751f already in the template host table
2014-07-15 10:41:47,837 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync found
218-2-00f60a5c-4771-3f10-a3f7-baae34372eb1 already in the template host table
2014-07-15 10:41:47,885 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync found
221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request]
(AgentManager-Handler-6:null) Seq 41-766640129: Processing: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110,
[{"SecStorageSetupAnswer":{"_dir":"a5fd862f-5d3e-3c41-b912-3918446c341a","result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-6:null) Seq 41-766640129: No more commands found
2014-07-15 10:41:47,913 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640129: Received: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110, { SecStorageSetupAnswer } }
2014-07-15 10:41:47,913 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Details from executing class
com.cloud.agent.api.SecStorageSetupCommand: success
2014-07-15 10:41:47,933 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-2:null) Template Sync found
275-2-bb1fbd41-a14c-3f06-ba59-f815a250b45c already in the template host table
2014-07-15 10:41:47,946 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-7:null) Successfully programmed secondary storage
nfs://10.11.18.7/export/d2secondary in secondary storage VM s-889-VM
2014-07-15 10:41:47,968 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 42-1808072709: Sending { Cmd , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 100011,
[{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.19.8","port":"443","intf":"eth2"}],"isAppendAIp":true,"wait":0}}]
}
2014-07-15 10:41:47,976 DEBUG [agent.transport.Request]
(AgentManager-Handler-13:null) Seq 42-1808072709: Processing: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 10,
[{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:47,977 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 42-1808072709: Received: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:47,977 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-7:null) Successfully programmed firewall rules into SSVM
s-909-VM
2014-07-15 10:41:47,985 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640131: Sending { Cmd , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 100011,
[{"SecStorageFirewallCfgCommand":{"portConfigs":[{"add":true,"sourceIp":"10.11.17.1","port":"443","intf":"eth2"}],"isAppendAIp":false,"wait":0}}]
}
2014-07-15 10:41:48,016 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072710: Sending { Cmd , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 100111,
[{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d3secondary","wait":0}}]
}
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request]
(AgentManager-Handler-5:null) Seq 41-766640131: Processing: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 10,
[{"Answer":{"result":true,"wait":0}}] }
2014-07-15 10:41:48,033 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640131: Received: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 10, { Answer } }
2014-07-15 10:41:48,033 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-7:null) Successfully programmed firewall rules into
s-889-VM
2014-07-15 10:41:48,059 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640132: Sending { Cmd , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 100111,
[{"SecStorageVMSetupCommand":{"allowedInternalSites":["10.11.18.0/24","10.11.19.0/24","10.11.16.0/24","10.11.17.0/24","10.11.24.0/24","192.168.1.0/24"],"copyUserName":"cloud","copyPassword":"nD7iafqpzcsxqme","wait":0}}]
}
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request]
(AgentManager-Handler-12:null) Seq 42-1808072710: Processing: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110,
[{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d3secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-12:null) Seq 42-1808072710: No more commands found
2014-07-15 10:41:48,077 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 42-1808072710: Received: { Ans: , MgmtId:
73548179740843, via: 42, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,077 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Details from executing class
com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,085 ERROR [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Monitor DownloadListener says there is an error
in the connect process for 42 due to null
java.lang.NullPointerException
at
com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at
com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at
com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at
com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at
com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at
com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,089 INFO [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Host 42 is disconnecting with event
AgentDisconnected
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) The next status of agent 42is Alert, current
status is Connecting
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Deregistering link for 42 with state Alert
2014-07-15 10:41:48,093 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Remove Agent : 42
2014-07-15 10:41:48,093 DEBUG [agent.manager.ConnectedAgentAttache]
(AgentManager-Handler-2:null) Processing Disconnect.
2014-07-15 10:41:48,094 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,095 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,096 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,101 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Sending Disconnect to listener:
com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,101 DEBUG [cloud.network.NetworkUsageManagerImpl]
(AgentManager-Handler-2:null) Disconnected called on 42 with status Alert
2014-07-15 10:41:48,101 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null)
Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id
= 42, name = s-909-VM]
2014-07-15 10:41:48,163 DEBUG [cloud.host.Status] (AgentManager-Handler-2:null)
Agent status update: [id = 42; name = s-909-VM; old status = Connecting; event
= AgentDisconnected; new status = Alert; old update count = 209; new update
count = 210]
2014-07-15 10:41:48,163 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(AgentManager-Handler-2:null) Notifying other nodes of to disconnect
2014-07-15 10:41:48,178 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Can not send command
com.cloud.agent.api.ReadyCommand due to Host 42 is not up
2014-07-15 10:41:48,178 WARN [agent.manager.AgentManagerImpl]
(AgentManager-Handler-2:null) Unable to create attache for agent: Seq 42-208:
{ Cmd , MgmtId: -1, via: 42, Ver: v1, Flags: 101,
[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"2","pod":"2","guid":"s-909-VM-NfsSecondaryStorageResource","name":"s-909-VM","id":42,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.17.1","publicNetmask":"255.255.255.0","publicMacAddress":"06:ed:f0:00:00:29","privateIpAddress":"10.11.16.70","privateMacAddress":"06:5c:62:00:00:14","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.16.70","storageNetmask":"255.255.255.0","storageMacAddress":"06:5c:62:00:00:14","resourceName":"NfsSecondaryStorageResource","wait":0}}]
}
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request]
(AgentManager-Handler-9:null) Seq 41-766640132: Processing: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110,
[{"Answer":{"result":true,"details":"","wait":0}}] }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-9:null) Seq 41-766640132: No more commands found
2014-07-15 10:41:48,208 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640132: Received: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110, { Answer } }
2014-07-15 10:41:48,208 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Details from executing class
com.cloud.agent.api.SecStorageVMSetupCommand:
2014-07-15 10:41:48,208 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(AgentManager-Handler-7:null) Successfully programmed http auth into s-889-VM
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener:
NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener:
AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,209 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Connect to listener: DownloadListener
2014-07-15 10:41:48,227 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640133: Sending { Cmd , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 100111,
[{"storage.ListTemplateCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}]
}
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request]
(AgentManager-Handler-1:null) Seq 41-766640133: Processing: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110,
[{"storage.ListTemplateAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{"221-2-3706787c-d305-39e0-9680-9f84359df07a":{"templateName":"221-2-3706787c-d305-39e0-9680-9f84359df07a","installPath":"template/tmpl/2/221//221-2-3706787c-d305-39e0-9680-9f84359df07a.iso","size":45762560,"physicalSize":45762560,"id":221,"isPublic":true,"isCorrupted":false},"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80":{"templateName":"206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80","installPath":"template/tmpl/2/206//206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80.iso","size":3548971164,"physicalSize":3548971164,"id":206,"isPublic":true,"isCorrupted":false},"a1e5cdd4-b892-45a0-aeb4-a337cc96b442":{"templateName":"a1e5cdd4-b892-45a0-aeb4-a337cc96b442","installPath":"template/tmpl/2/209//a1e5cdd4-b892-45a0-aeb4-a337cc96b442.qcow2","size":107374182400,"physicalSize":15100084224,"id":1,"isPublic":true,"isCorrupted":false},"routing-3":{"templateName":"routing-3","installPath":"template/tmpl/1/3//3e38f707-5006-4d73-92a5-5be21b6f178f.qcow2","size":725811200,"physicalSize":725811200,"id":3,"isPublic":true,"isCorrupted":false},"5deebbfd-e603-4d09-912d-de5848d7118e":{"templateName":"5deebbfd-e603-4d09-912d-de5848d7118e","installPath":"template/tmpl/2/208//5deebbfd-e603-4d09-912d-de5848d7118e.qcow2","size":107374182400,"physicalSize":4641128448,"id":1,"isPublic":true,"isCorrupted":false},"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c":{"templateName":"205-2-5ffdab64-da2e-3afc-b166-061397e0de7c","installPath":"template/tmpl/2/205//205-2-5ffdab64-da2e-3afc-b166-061397e0de7c.iso","size":4239566848,"physicalSize":4239566848,"id":205,"isPublic":true,"isCorrupted":false},"centos55-x86_64":{"templateName":"centos55-x86_64","installPath":"template/tmpl/1/4//20fb26b3-a01a-39ab-a39b-3b459dc862ed.qcow2","size":8589934592,"physicalSize":1769537536,"id":4,"isPublic":true,"isCorrupted":false}},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-1:null) Seq 41-766640133: No more commands found
2014-07-15 10:41:48,498 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640133: Received: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110, { ListTemplateAnswer } }
2014-07-15 10:41:48,498 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Details from executing class
com.cloud.agent.api.storage.ListTemplateCommand: success
2014-07-15 10:41:48,521 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found routing-3 already in the
template host table
2014-07-15 10:41:48,567 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found centos55-x86_64 already in
the template host table
2014-07-15 10:41:48,609 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync did not find centos56-x86_64-xen
ready on server 2, will request download to start/resume shortly
2014-07-15 10:41:48,613 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync did not find centos53-x64 ready on
server 2, will request download to start/resume shortly
2014-07-15 10:41:48,625 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found
205-2-5ffdab64-da2e-3afc-b166-061397e0de7c already in the template host table
2014-07-15 10:41:48,687 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found
206-2-33e4a593-fa36-3fb6-b68e-a587a8b1af80 already in the template host table
2014-07-15 10:41:48,747 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found
5deebbfd-e603-4d09-912d-de5848d7118e already in the template host table
2014-07-15 10:41:48,813 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found
a1e5cdd4-b892-45a0-aeb4-a337cc96b442 already in the template host table
2014-07-15 10:41:48,860 INFO [storage.download.DownloadMonitorImpl]
(AgentManager-Handler-7:null) Template Sync found
221-2-3706787c-d305-39e0-9680-9f84359df07a already in the template host table
2014-07-15 10:41:48,922 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640134: Sending { Cmd , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 100111,
[{"storage.ListVolumeCommand":{"secUrl":"nfs://10.11.18.7/export/d2secondary","wait":0}}]
}
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request]
(AgentManager-Handler-15:null) Seq 41-766640134: Processing: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110,
[{"storage.ListVolumeAnswer":{"secUrl":"nfs://10.11.18.7/export/d2secondary","templateInfos":{},"result":true,"details":"success","wait":0}}]
}
2014-07-15 10:41:48,939 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-15:null) Seq 41-766640134: No more commands found
2014-07-15 10:41:48,939 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 41-766640134: Received: { Ans: , MgmtId:
73548179740843, via: 41, Ver: v1, Flags: 110, { ListVolumeAnswer } }
2014-07-15 10:41:48,940 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Details from executing class
com.cloud.agent.api.storage.ListVolumeCommand: success
2014-07-15 10:41:48,948 ERROR [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Monitor DownloadListener says there is an error
in the connect process for 41 due to null
java.lang.NullPointerException
at
com.cloud.storage.download.DownloadMonitorImpl.handleVolumeSync(DownloadMonitorImpl.java:704)
at
com.cloud.storage.download.DownloadMonitorImpl.handleSync(DownloadMonitorImpl.java:630)
at
com.cloud.storage.download.DownloadListener.processConnect(DownloadListener.java:385)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:605)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1157)
at
com.cloud.agent.manager.AgentManagerImpl.access$100(AgentManagerImpl.java:142)
at
com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1235)
at
com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1374)
at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:618)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-07-15 10:41:48,951 INFO [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Host 41 is disconnecting with event
AgentDisconnected
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) The next status of agent 41is Alert, current
status is Connecting
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Deregistering link for 41 with state Alert
2014-07-15 10:41:48,956 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Remove Agent : 41
2014-07-15 10:41:48,956 DEBUG [agent.manager.ConnectedAgentAttache]
(AgentManager-Handler-7:null) Processing Disconnect.
2014-07-15 10:41:48,957 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.hypervisor.vmware.manager.VmwareManagerImpl$$EnhancerByCGLIB$$4ad0e44c
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$15bc7206
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.network.security.SecurityGroupListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.storage.listener.StoragePoolMonitor
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$bf7c5122
2014-07-15 10:41:48,958 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$301cd382
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.storage.download.DownloadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.storage.upload.UploadListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.storage.LocalStoragePoolListener$$EnhancerByCGLIB$$c00af9b6
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.capacity.StorageCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.capacity.ComputeCapacityListener
2014-07-15 10:41:48,959 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.consoleproxy.ConsoleProxyListener
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2014-07-15 10:41:48,963 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl$$EnhancerByCGLIB$$c27578aa
2014-07-15 10:41:48,964 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Sending Disconnect to listener:
com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-07-15 10:41:48,964 DEBUG [cloud.network.NetworkUsageManagerImpl]
(AgentManager-Handler-7:null) Disconnected called on 41 with status Alert
2014-07-15 10:41:48,964 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null)
Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id
= 41, name = s-889-VM]
2014-07-15 10:41:49,006 DEBUG [cloud.host.Status] (AgentManager-Handler-7:null)
Agent status update: [id = 41; name = s-889-VM; old status = Connecting; event
= AgentDisconnected; new status = Alert; old update count = 391; new update
count = 392]
2014-07-15 10:41:49,006 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(AgentManager-Handler-7:null) Notifying other nodes of to disconnect
2014-07-15 10:41:49,014 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Can not send command
com.cloud.agent.api.ReadyCommand due to Host 41 is not up
2014-07-15 10:41:49,015 WARN [agent.manager.AgentManagerImpl]
(AgentManager-Handler-7:null) Unable to create attache for agent: Seq 41-390:
{ Cmd , MgmtId: -1, via: 41, Ver: v1, Flags: 101,
[{"StartupSecondaryStorageCommand":{"type":"SecondaryStorage","dataCenter":"1","pod":"1","guid":"s-889-VM-NfsSecondaryStorageResource","name":"s-889-VM","id":41,"version":"2.2.20130913151423","iqn":"NoIqn","publicIpAddress":"10.11.19.8","publicNetmask":"255.255.255.0","publicMacAddress":"06:fb:ee:00:00:1c","privateIpAddress":"10.11.18.67","privateMacAddress":"06:89:b0:00:00:11","privateNetmask":"255.255.255.0","storageIpAddress":"10.11.18.67","storageNetmask":"255.255.255.0","storageMacAddress":"06:89:b0:00:00:11","resourceName":"NfsSecondaryStorageResource","wait":0}}]
}
2014-07-15 10:41:50,555 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2014-07-15 10:41:50,609 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Zone 2 is ready to launch secondary storage VM
2014-07-15 10:41:50,851 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is ready to launch console proxy
2014-07-15 10:41:50,868 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 2 is ready to launch console proxy
2014-07-15 10:41:51,848 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) StorageCollector is running...
2014-07-15 10:41:51,864 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) There is no secondary storage VM for secondary storage
host nfs://10.11.18.7/export/d2secondary
2014-07-15 10:41:51,869 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) There is no secondary storage VM for secondary storage
host nfs://10.11.18.7/export/d3secondary
2014-07-15 10:41:51,928 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers.
2014-07-15 10:41:51,991 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 7-513215195: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1,
Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,059 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 9-1410599594: Received: { Ans: , MgmtId: 73548179740843, via: 9, Ver: v1,
Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,128 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 14-1320747056: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver:
v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,150 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 16-940116823: Received: { Ans: , MgmtId: 73548179740843, via: 16, Ver: v1,
Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,231 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 14-1320747057: Received: { Ans: , MgmtId: 73548179740843, via: 14, Ver:
v1, Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:52,312 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 7-513215196: Received: { Ans: , MgmtId: 73548179740843, via: 7, Ver: v1,
Flags: 10, { GetStorageStatsAnswer } }
2014-07-15 10:41:53,351 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-6:null) Ping from 23
2014-07-15 10:41:54,315 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-13:null) Ping from 8
2014-07-15 10:41:54,532 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) Ping from 40?6?9
