您好：
CS4.6 system vms is struck in starting state . But KVM display running !
Managerment Server/NFS
Host KVM
csm
cs1
192.168.1.10
192.168.1.11
RedHat6.4_X64
RedHat6.4_X64
CS4.6
CS4.6
sysvm模板为systemvm64template-4.6.0-kvm.qcow2.bz2，已校验md5。
在esxi上虚拟出来的csm和cs1搭建此环境，csm和cs1各有3块网卡
网卡名称 vlan id IP地址 lable 用途
csm vmnetwork none 172.28.201.191 仅仅管理使用
vnetmgrt 10 192.168.1.10 cloudbr0 管理与guest
vnetpublic 11 172.16.1.0/24 cloudbr1 public
cs1 vmnetwork none 172.28.201.192 仅仅管理使用
vnetmgrt 10 192.168.1.11 cloudbr0 管理与guest
vnetpublic 11 172.16.1.0/24 cloudbr1 public
已修改全局参数secstorage.allowed.intern为192.168.1.0/24,192.168.1.10
创建完高级zone后system vms一直显示starting状态，但是在cs1上使用virsh list
--all显示为running，重启managerment，agent，libvirtd后在页面显示为running。
[root@cs1 ~]# virsh list --all
Id Name State
----------------------------------------------------
3 s-8-VM running
4 v-7-VM running
agent有如下日志
2016-09-28 10:55:49,939 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
s-8-VM -p
%template=domP%type=secstorage%host=192.168.1.10%port=8250%name=s-8-VM%zone=1%pod=1%guid=s-8-VM%workers=5%resource=com.cloud.storage.resource.PremiumSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=172.16.1.32%eth2mask=255.255.255.0%gateway=172.16.1.254%public.network.device=eth2%eth0ip=169.254.2.237%eth0mask=255.255.0.0%eth1ip=192.168.1.72%eth1mask=255.255.255.0%mgmtcidr=192.168.1.0/24%localgw=192.168.1.254%private.network.device=eth1%internaldns1=192.168.1.254%dns1=8.8.8.8
. Output is:
2016-09-28 10:55:55,135 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Trying to fetch storage pool
fd7d94b4-1672-3337-89f2-7dbc82e716f2 from libvirt
2016-09-28 10:55:55,150 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Asking libvirt to refresh storage pool
fd7d94b4-1672-3337-89f2-7dbc82e716f2
2016-09-28 10:56:10,945 WARN [kvm.resource.LibvirtComputingResource]
(Script-5:null) Interrupting script.
2016-09-28 10:56:10,946 WARN [kvm.resource.LibvirtComputingResource]
(Script-5:null) Interrupting script.
2016-09-28 10:56:10,946 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-1:null) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
v-7-VM -p
%template=domP%type=consoleproxy%host=192.168.1.10%port=8250%name=v-7-VM%zone=1%pod=1%guid=Proxy.7%proxy_vm=7%disable_rp_filter=true%eth2ip=172.16.1.31%eth2mask=255.255.255.0%gateway=172.16.1.254%eth0ip=169.254.1.225%eth0mask=255.255.0.0%eth1ip=192.168.1.87%eth1mask=255.255.255.0%mgmtcidr=192.168.1.0/24%localgw=192.168.1.254%internaldns1=192.168.1.254%dns1=8.8.8.8
. Output is:
2016-09-28 10:56:10,946 WARN [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-2:null) Timed out:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
s-8-VM -p
%template=domP%type=secstorage%host=192.168.1.10%port=8250%name=s-8-VM%zone=1%pod=1%guid=s-8-VM%workers=5%resource=com.cloud.storage.resource.PremiumSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=172.16.1.32%eth2mask=255.255.255.0%gateway=172.16.1.254%public.network.device=eth2%eth0ip=169.254.2.237%eth0mask=255.255.0.0%eth1ip=192.168.1.72%eth1mask=255.255.255.0%mgmtcidr=192.168.1.0/24%localgw=192.168.1.254%private.network.device=eth1%internaldns1=192.168.1.254%dns1=8.8.8.8
. Output is:
managerment后台日志中有如下错误信息，请问是网络哪块出问题了？
2016-09-28 11:02:53,235 DEBUG [c.c.s.StatsCollector]
(StatsCollector-1:ctx-f47d9415) AutoScaling Monitor is running...
2016-09-28 11:02:53,281 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-a7efaf1d) VmStatsCollector is running...
2016-09-28 11:02:53,695 DEBUG [c.c.s.StatsCollector]
(StatsCollector-4:ctx-5034a236) HostStatsCollector is running...
2016-09-28 11:02:53,718 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-5034a236)
Seq 1-808959083066425394: Received: { Ans: , MgmtId: 345048851725, via:
1(cs1), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2016-09-28 11:02:54,883 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-2f74a5a7) StorageCollector is running...
2016-09-28 11:02:54,888 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-2f74a5a7) There is no secondary storage VM for secondary
storage host SStorage
2016-09-28 11:02:54,956 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-2f74a5a7)
Seq 1-808959083066425395: Received: { Ans: , MgmtId: 345048851725, via:
1(cs1), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2016-09-28 11:02:58,070 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-215db595) Begin cleanup expired async-jobs
2016-09-28 11:02:58,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-215db595) End cleanup expired async-jobs
2016-09-28 11:03:01,602 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) Ping from 1
2016-09-28 11:03:01,603 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) Process host VM state report from ping process.
host: 1
2016-09-28 11:03:01,610 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) Process VM state report. host: 1, number of
records in report: 2
2016-09-28 11:03:01,610 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) VM state report. host: 1, vm id: 7, power state:
PowerOn
2016-09-28 11:03:01,628 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) VM state report is updated. host: 1, vm id: 7,
power state: PowerOn
2016-09-28 11:03:01,630 INFO [c.c.v.VirtualMachineManagerImpl]
(AgentManager-Handler-11:null) There is pending job or HA tasks working on the
VM. vm id: 7, postpone power-change report by resetting power-change counters
2016-09-28 11:03:01,653 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) VM state report. host: 1, vm id: 8, power state:
PowerOn
2016-09-28 11:03:01,672 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) VM state report is updated. host: 1, vm id: 8,
power state: PowerOn
2016-09-28 11:03:01,674 INFO [c.c.v.VirtualMachineManagerImpl]
(AgentManager-Handler-11:null) There is pending job or HA tasks working on the
VM. vm id: 8, postpone power-change report by resetting power-change counters
2016-09-28 11:03:01,712 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-11:null) Done with process of VM state report. host: 1
2016-09-28 11:03:08,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-476e9849) Begin cleanup expired async-jobs
2016-09-28 11:03:08,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-476e9849) End cleanup expired async-jobs
2016-09-28 11:03:08,202 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-9696b0ad) Found 0 routers to update status.
2016-09-28 11:03:08,204 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-9696b0ad) Found 0 VPC networks to update Redundant
State.
2016-09-28 11:03:08,205 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-9696b0ad) Found 0 networks to update RvR status.
2016-09-28 11:03:08,242 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-76fc0b61) Found 0 routers to update status.
2016-09-28 11:03:08,243 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-76fc0b61) Found 0 VPC networks to update Redundant
State.
2016-09-28 11:03:08,244 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-76fc0b61) Found 0 networks to update RvR status.
2016-09-28 11:03:18,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-fbfabe80) Begin cleanup expired async-jobs
2016-09-28 11:03:18,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-fbfabe80) End cleanup expired async-jobs
2016-09-28 11:03:28,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-5601a321) Begin cleanup expired async-jobs
2016-09-28 11:03:28,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-5601a321) End cleanup expired async-jobs
2016-09-28 11:03:29,252 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-0be61564) Task (job-66) has been pending for 1019 seconds
2016-09-28 11:03:29,253 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-0be61564) Task (job-67) has been pending for 1019 seconds
2016-09-28 11:03:38,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-153039ea) Begin cleanup expired async-jobs
2016-09-28 11:03:38,077 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-153039ea) End cleanup expired async-jobs
2016-09-28 11:03:38,202 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-b5675452) Found 0 routers to update status.
2016-09-28 11:03:38,204 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-b5675452) Found 0 VPC networks to update Redundant
State.
2016-09-28 11:03:38,205 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-b5675452) Found 0 networks to update RvR status.
2016-09-28 11:03:38,241 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-18c9b227) Found 0 routers to update status.
2016-09-28 11:03:38,243 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-18c9b227) Found 0 VPC networks to update Redundant
State.
2016-09-28 11:03:38,244 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-18c9b227) Found 0 networks to update RvR status.
2016-09-28 11:03:43,163 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager
Timer:ctx-080a7551) Resetting hosts suitable for reconnect
2016-09-28 11:03:43,165 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager
Timer:ctx-080a7551) Completed resetting hosts suitable for reconnect
2016-09-28 11:03:43,165 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager
Timer:ctx-080a7551) Acquiring hosts for clusters already owned by this
management server
2016-09-28 11:03:43,166 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager
Timer:ctx-080a7551) Completed acquiring hosts for clusters already owned by
this management server
2016-09-28 11:03:43,166 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager
Timer:ctx-080a7551) Acquiring hosts for clusters not owned by any management
server
2016-09-28 11:03:43,167 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager
Timer:ctx-080a7551) Completed acquiring hosts for clusters not owned by any
management server
2016-09-28 11:03:48,070 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-871ba457) Begin cleanup expired async-jobs
2016-09-28 11:03:48,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-871ba457) End cleanup expired async-jobs
2016-09-28 11:03:53,236 DEBUG [c.c.s.StatsCollector]
(StatsCollector-1:ctx-f9198145) AutoScaling Monitor is running...
2016-09-28 11:03:53,285 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-ece47c17) VmStatsCollector is running...
2016-09-28 11:03:53,719 DEBUG [c.c.s.StatsCollector]
(StatsCollector-4:ctx-67496858) HostStatsCollector is running...
2016-09-28 11:03:53,739 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-67496858)
Seq 1-808959083066425396: Received: { Ans: , MgmtId: 345048851725, via:
1(cs1), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2016-09-28 11:03:54,957 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-53e6999b) StorageCollector is running...
2016-09-28 11:03:54,961 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-53e6999b) There is no secondary storage VM for secondary
storage host SStorage
2016-09-28 11:03:55,037 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-53e6999b)
Seq 1-808959083066425397: Received: { Ans: , MgmtId: 345048851725, via:
1(cs1), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2016-09-28 11:03:58,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-70730610) Begin cleanup expired async-jobs
2016-09-28 11:03:58,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-70730610) End cleanup expired async-jobs
2016-09-28 11:04:01,590 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) Ping from 1
2016-09-28 11:04:01,591 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) Process host VM state report from ping process.
host: 1
2016-09-28 11:04:01,600 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) Process VM state report. host: 1, number of
records in report: 2
2016-09-28 11:04:01,600 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) VM state report. host: 1, vm id: 7, power state:
PowerOn
2016-09-28 11:04:01,625 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) VM state report is updated. host: 1, vm id: 7,
power state: PowerOn
2016-09-28 11:04:01,627 INFO [c.c.v.VirtualMachineManagerImpl]
(AgentManager-Handler-14:null) There is pending job or HA tasks working on the
VM. vm id: 7, postpone power-change report by resetting power-change counters
2016-09-28 11:04:01,664 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) VM state report. host: 1, vm id: 8, power state:
PowerOn
2016-09-28 11:04:01,693 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) VM state report is updated. host: 1, vm id: 8,
power state: PowerOn
2016-09-28 11:04:01,695 INFO [c.c.v.VirtualMachineManagerImpl]
(AgentManager-Handler-14:null) There is pending job or HA tasks working on the
VM. vm id: 8, postpone power-change report by resetting power-change counters
2016-09-28 11:04:01,726 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(AgentManager-Handler-14:null) Done with process of VM state report. host: 1
2016-09-28 11:04:08,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-cc6322c3) Begin cleanup expired async-jobs
2016-09-28 11:04:08,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-cc6322c3) End cleanup expired async-jobs
2016-09-28 11:04:08,202 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-23ebb3e7) Found 0 routers to update status.
2016-09-28 11:04:08,203 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-23ebb3e7) Found 0 VPC networks to update Redundant
State.
2016-09-28 11:04:08,205 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-23ebb3e7) Found 0 networks to update RvR status.
2016-09-28 11:04:08,242 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-199cd318) Found 0 routers to update status.
2016-09-28 11:04:08,243 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-199cd318) Found 0 VPC networks to update Redundant
State.
2016-09-28 11:04:08,245 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-199cd318) Found 0 networks to update RvR status.
2016-09-28 11:04:18,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-764c45e1) Begin cleanup expired async-jobs
2016-09-28 11:04:18,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-764c45e1) End cleanup expired async-jobs
2016-09-28 11:04:28,069 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-e81105c6) Begin cleanup expired async-jobs
2016-09-28 11:04:28,075 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-e81105c6) End cleanup expired async-jobs
2016-09-28 11:04:29,252 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-c76d7a66) Task (job-66) has been pending for 1079 seconds
2016-09-28 11:04:29,253 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-c76d7a66) Task (job-67) has been pending for 1079 seconds
roarain...@126.com
