[
https://issues.apache.org/jira/browse/CLOUDSTACK-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656573#comment-15656573
]
Wei Zhou commented on CLOUDSTACK-9590:
--------------------------------------
key logs
{code}
2016-11-10 13:23:06,568 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null)
(logid:) Seq -1-4: Scheduling the first command { Cmd , MgmtId: -1, via: -1,
Ver: v1, Flags: 1,
[{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":2,"cpus":8,"speed":2261,"memory":100092813312,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-327.36.3.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.2.1511","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"3","pod":"3","cluster":"9","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","iqn":"iqn.1994-05.com.redhat:eedee56bd952","privateIpAddress":"192.168.85.14","privateMacAddress":"52:80:f7:fc:af:42","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.85.14","storageNetmask":"255.255.255.0","storageMacAddress":"52:80:f7:fc:af:42","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.85.254","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"b3b9ef96-18b1-4136-8a69-5b316c6dc123","host":"192.168.85.14","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":30866534400,"availableBytes":29248552960},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","resourceName":"LibvirtComputingResource","wait":0}}]
}
2016-11-10 13:23:06,570 INFO [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-13:null) (logid:) Connection from /192.168.85.14 closed
but no cleanup was done.
2016-11-10 13:23:06,582 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-15:null) (logid:) Failed to send startupanswer:
java.nio.channels.ClosedChannelException
2016-11-10 13:23:06,605 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Seq -1-4: Processing
the first command { Cmd , MgmtId: -1, via: -1, Ver: v1, Flags: 1,
[{"com.cloud.agent.api.StartupRoutingCommand":{"cpuSockets":2,"cpus":8,"speed":2261,"memory":100092813312,"dom0MinMemory":1073741824,"poolSync":false,"supportsClonedVolumes":false,"caps":"hvm,snapshot","pool":"/root","hypervisorType":"KVM","hostDetails":{"Host.OS.Kernel.Version":"3.10.0-327.36.3.el7.x86_64","com.cloud.network.Networks.RouterPrivateIpStrategy":"HostLocal","Host.OS.Version":"7.2.1511","Host.OS":"CentOS"},"hostTags":[],"groupDetails":{},"type":"Routing","dataCenter":"3","pod":"3","cluster":"9","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","iqn":"iqn.1994-05.com.redhat:eedee56bd952","privateIpAddress":"192.168.85.14","privateMacAddress":"52:80:f7:fc:af:42","privateNetmask":"255.255.255.0","storageIpAddress":"192.168.85.14","storageNetmask":"255.255.255.0","storageMacAddress":"52:80:f7:fc:af:42","resourceName":"LibvirtComputingResource","gatewayIpAddress":"192.168.85.254","wait":0}},{"com.cloud.agent.api.StartupStorageCommand":{"totalSize":0,"poolInfo":{"uuid":"b3b9ef96-18b1-4136-8a69-5b316c6dc123","host":"192.168.85.14","localPath":"/var/lib/libvirt/images","hostPath":"/var/lib/libvirt/images","poolType":"Filesystem","capacityBytes":30866534400,"availableBytes":29248552960},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","guid":"71773d35-5679-34fe-9b15-7e2736d0dd28-LibvirtComputingResource","name":"kvm02.oscloud.local","version":"4.9.0","resourceName":"LibvirtComputingResource","wait":0}}]
}
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to BareMetalDiscoverer
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to NetscalerElement
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to HypervServerDiscoverer
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to BaremetalPxeManagerImpl
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to XcpServerDiscoverer
2016-11-10 13:23:06,626 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to NiciraNvp
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to BrocadeVcsElement
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to Ovm3Discoverer
2016-11-10 13:23:06,627 DEBUG [c.c.h.o.r.Ovm3Discoverer]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb)
createHostVOForConnectedAgent: Host[-51-Routing]
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to LxcServerDiscoverer
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to NetworkUsageManagerImpl
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to PremiumSecondaryStorageManagerImpl
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to Ovs
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to OvmDiscoverer
2016-11-10 13:23:06,627 DEBUG [c.c.r.ResourceManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Dispatching resource
state event CREATE_HOST_VO_FOR_CONNECTED to KvmServerDiscoverer
2016-11-10 13:23:06,687 DEBUG [c.c.r.ResourceState]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Resource state update:
[id = 51; name = kvm02.oscloud.local; old state = Enabled; event =
InternalCreated; new state = Enabled]
2016-11-10 13:23:06,688 DEBUG [c.c.h.Status]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Transition:[Resource
state = Enabled, Agent event = AgentConnected, Host id = 51, name =
kvm02.oscloud.local]
2016-11-10 13:23:06,705 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) create
ClusteredAgentAttache for 51
2016-11-10 13:23:06,709 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Sending Connect to
listener: XcpServerDiscoverer
2016-11-10 13:23:06,709 DEBUG [c.c.h.x.d.XcpServerDiscoverer]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Not XenServer so moving
on.
2016-11-10 13:23:06,709 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Sending Connect to
listener: HypervServerDiscoverer
2016-11-10 13:23:06,709 DEBUG [c.c.h.h.d.HypervServerDiscoverer]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Not Hyper-V hypervisor,
so moving on.
2016-11-10 13:23:06,709 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Sending Connect to
listener: SecurityGroupListener
2016-11-10 13:23:06,709 INFO [c.c.n.s.SecurityGroupListener]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Received a host startup
notification
2016-11-10 13:23:06,714 DEBUG [c.c.a.t.Request]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Seq
51-746753113213370369: Sending { Cmd , MgmtId: 3232257305, via:
51(kvm02.oscloud.local), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CleanupNetworkRulesCmd":{"interval":2382,"wait":0}}] }
2016-11-10 13:23:06,714 INFO [c.c.a.m.AgentAttache]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Seq
51-746753113213370369: Unable to send due to Resource [Host:51] is unreachable:
Host 51: Channel is closed
2016-11-10 13:23:06,714 DEBUG [c.c.a.m.AgentAttache]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Seq
51-746753113213370369: Cancelling.
2016-11-10 13:23:06,714 DEBUG [c.c.n.s.SecurityGroupListener]
(AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Unable to schedule
network rules cleanup for host 51
com.cloud.exception.AgentUnavailableException: Resource [Host:51] is
unreachable: Host 51: Channel is closed
at
com.cloud.agent.manager.ConnectedAgentAttache.send(ConnectedAgentAttache.java:46)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:373)
at
com.cloud.agent.manager.ClusteredAgentAttache.send(ClusteredAgentAttache.java:141)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:507)
at
com.cloud.network.security.SecurityGroupListener.processConnect(SecurityGroupListener.java:169)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:564)
at
com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1087)
at
com.cloud.agent.manager.AgentManagerImpl.access$000(AgentManagerImpl.java:120)
at
com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1171)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> KVM + CentOS 7.2 + Agent in Alert State for long time
> -----------------------------------------------------
>
> Key: CLOUDSTACK-9590
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9590
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: cloudstack-agent
> Affects Versions: 4.9.0
> Environment: entOS Linux release 7.2.1511 (Core)
> cloudstack-agent-4.9.0-1.el7.centos.x86_64
> Reporter: Sven Vogel
> Attachments: agent.log, cloudstack-startup.log, management-server.zip
>
>
> Hi,
> When i add a new host to cloudstack management server it take some time to
> get host out from alert state.
> 1. i add the host and host add not possible
> 2. values are correct set to agent.properties, restart cloustack agent
> 3. agent says connected to server
> 4. management server says "alert"
> management-server.log
> 2016-11-10 13:23:06,783 DEBUG [c.c.h.Status]
> (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Transition:[Resource
> state = Enabled, Agent event = AgentDisconnected, Host
> id = 51, name = kvm02.oscloud.local]
> 2016-11-10 13:23:06,798 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Notifying other nodes
> of to disconnect
> 2016-11-10 13:23:06,806 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentConnectTaskPool-49:ctx-c3b72839) (logid:5a86e1fb) Failed to handle host
> connection: com.cloud.exception.Connection
> Exception: Unable to get an answer to the CheckNetworkCommand from agent: 51
> is there any way to speed up the alert state? is it normal that it take so
> long?
> thanks
> Sven
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
