[ https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210367#comment-15210367 ]
ASF subversion and git services commented on CLOUDSTACK-9285: ------------------------------------------------------------- Commit c69a2428977aed7df497a479a7e802e9d8d0e922 in cloudstack's branch refs/heads/4.7 from [~sweller] [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=c69a242 ] CLOUDSTACK-9285 - Address original on start exception(s) and newline cleanup > Cloudstack 4.8 can't connect to XEN and KVM hosts > ------------------------------------------------- > > Key: CLOUDSTACK-9285 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Xen > Affects Versions: 4.8.0 > Environment: CentOS 7 > Reporter: Jens Fettig > Priority: Blocker > Attachments: agentfailure.pcap > > > We used Cloudstack 4.7 in our testing environment. For future uses we updated > to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 > cloudstack has some problems to connect to the hosts again. Here are the > error logs. > {code:title=XEN Server from the managment-server.log:|borderStyle=solid} > 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly > connected host 13(xen5) > 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] > (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is > running... > 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] > (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is > running... > 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] > (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is > running... > 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] > (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is > running... > 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] > (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary > storage VM for secondary storage host cdsdev-secondary > 2016-02-15 11:22:11,018 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired > async-jobs > 2016-02-15 11:22:11,024 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired > async-jobs > 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] > (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create > master connection to host(192.168.0.97) , due to The credentials given by the > user are incorrect, so access has been denied, and you have not been issued a > session handle. > 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager > Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, > Agent event = AgentDisconnected, Host id = 13, name = xen5] > 2016-02-15 11:22:11,101 WARN [c.c.a.m.ClusteredAgentManagerImpl] > (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) can not load > directly connected host 13(xen5) due to > com.cloud.utils.exception.CloudRuntimeException: Unable to create master > connection to host(192.168.0.97) , due to The credentials given by the user > are incorrect, so access has been denied, and you have not been issued a > session handle. > at > com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827) > at > com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158) > at > com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697) > at > com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217) > at > com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182) > at > com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96) > at > com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233) > at > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > Caused by: The credentials given by the user are incorrect, so access has > been denied, and you have not been issued a session handle. > at com.xensource.xenapi.Types.checkResponse(Types.java:322) > at com.xensource.xenapi.Connection.dispatch(Connection.java:395) > at com.xensource.xenapi.Session.loginWithPassword(Session.java:587) > ... 19 more > {code} > This seems to be the problem: > Unable to create master connection to host(192.168.0.97) , due to The > credentials given by the user are incorrect, so access has been denied, and > you have not been issued a session handle. > I updated the vhd-utils like in the upgrade gourde from 4.6: > http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.7.0/upgrade/upgrade-4.6.html > But maybe a need a new vhd-utils file for 4.8, because this one was for 4.7? > {code:title=KVM from the agent.log:|borderStyle=solid} > 2016-02-15 10:46:20,971 INFO [cloud.agent.AgentShell] (main:null) (logid:) > Agent started > 2016-02-15 10:46:20,973 INFO [cloud.agent.AgentShell] (main:null) (logid:) > Implementation Version is 4.8.0 > 2016-02-15 10:46:20,974 INFO [cloud.agent.AgentShell] (main:null) (logid:) > agent.properties found at /etc/cloudstack/agent/agent.properties > 2016-02-15 10:46:20,977 INFO [cloud.agent.AgentShell] (main:null) (logid:) > Defaulting to using properties file for storage > 2016-02-15 10:46:20,978 INFO [cloud.agent.AgentShell] (main:null) (logid:) > Defaulting to the constant time backoff algorithm > 2016-02-15 10:46:20,988 INFO [cloud.utils.LogUtils] (main:null) (logid:) > log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml > 2016-02-15 10:46:20,999 INFO [cloud.agent.AgentShell] (main:null) (logid:) > Preferring IPv4 address family for agent connection > 2016-02-15 10:46:21,095 INFO [cloud.agent.Agent] (main:null) (logid:) id is > 17 > 2016-02-15 10:46:21,116 INFO [kvm.resource.LibvirtConnection] (main:null) > (logid:) No existing libvirtd connection found. Opening a new one > 2016-02-15 10:46:21,373 INFO [org.reflections.Reflections] (main:null) > (logid:) Reflections took 66 ms to scan 1 urls, producing 7 keys and 10 values > 2016-02-15 10:46:21,514 INFO [kvm.resource.LibvirtComputingResource] > (main:null) (logid:) No libvirt.vif.driver specified. Defaults to > BridgeVifDriver. > 2016-02-15 10:46:21,543 INFO [cloud.agent.Agent] (main:null) (logid:) Agent > [id = 17 : type = LibvirtComputingResource : zone = 1 : pod = 1 : workers = 5 > : host = 192.168.0.72 : port = 8250 > 2016-02-15 10:46:21,549 INFO [utils.nio.NioClient] (main:null) (logid:) > Connecting to 192.168.0.72:8250 > 2016-02-15 10:46:21,609 ERROR [utils.nio.NioConnection] (main:null) (logid:) > Unable to initialize the threads. > java.io.IOException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.Link.doHandshake(Link.java:513) > at com.cloud.utils.nio.NioClient.init(NioClient.java:80) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > at com.cloud.agent.Agent.start(Agent.java:228) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at com.cloud.agent.AgentShell.start(AgentShell.java:461) > at com.cloud.agent.AgentShell.main(AgentShell.java:496) > 2016-02-15 10:46:21,611 INFO [utils.exception.CSExceptionErrorCode] > (main:null) (logid:) Could not find exception: > com.cloud.utils.exception.NioConnectionException in error code list for > exceptions > 2016-02-15 10:46:21,612 ERROR [cloud.agent.AgentShell] (main:null) (logid:) > Unable to start agent: > com.cloud.utils.exception.CloudRuntimeException: Unable to start the > connection! > at com.cloud.agent.Agent.start(Agent.java:230) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at com.cloud.agent.AgentShell.start(AgentShell.java:461) > at com.cloud.agent.AgentShell.main(AgentShell.java:496) > Caused by: com.cloud.utils.exception.NioConnectionException: Connection > closed with -1 on reading size. > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:94) > at com.cloud.agent.Agent.start(Agent.java:228) > ... 5 more > Caused by: java.io.IOException: Connection closed with -1 on reading size. > at com.cloud.utils.nio.Link.doHandshake(Link.java:513) > at com.cloud.utils.nio.NioClient.init(NioClient.java:80) > at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > ... 6 more > {code} > cloudstack-agent on the kvm host and the cloudstack-management on the > cloudstack server is running. Everything on version 4.8.0. > Maybe there are some bugs or did I something wrong? > Greetings J. Fettig -- This message was sent by Atlassian JIRA (v6.3.4#6332)