[jira] [Updated] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts

2016-02-15 Thread Jens Fettig (JIRA)

 [ 
https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jens Fettig updated CLOUDSTACK-9285:

Description: 
We used Cloudstack 4.7 in our testing environment. For future uses we updated 
to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 
cloudstack has some problems to connect to the hosts again. Here are the error 
logs.

XEN Server from the managment-server.log:

2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
(ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly 
connected host 13(xen5)
2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is 
running...
2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is running...
2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is 
running...
2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is running...
2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary storage 
VM for secondary storage host cdsdev-secondary
2016-02-15 11:22:11,018 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
(AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired 
async-jobs
2016-02-15 11:22:11,024 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
(AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired 
async-jobs
2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] 
(ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create 
master connection to host(192.168.0.97) , due to The credentials given by the 
user are incorrect, so access has been denied, and you have not been issued a 
session handle.
2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager 
Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, 
Agent event = AgentDisconnected, Host id = 13, name = xen5]
2016-02-15 11:22:11,101 WARN  [c.c.a.m.ClusteredAgentManagerImpl] 
(ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec)  can not load 
directly connected host 13(xen5) due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to create master 
connection to host(192.168.0.97) , due to The credentials given by the user are 
incorrect, so access has been denied, and you have not been issued a session 
handle.
at 
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
at 
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
at 
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
at 
com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
at 
com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: The credentials given by the user are incorrect, so access has been 
denied, and you have not been issued a session handle.
at com.xensource.xenapi.Types.checkResponse(Types.java:322)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at com.xensource.xenapi.Session.loginWithPassword(Session.java:587)
... 19 more 

This seems to

[jira] [Updated] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts

2016-02-15 Thread Jens Fettig (JIRA)

 [ 
https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jens Fettig updated CLOUDSTACK-9285:

Description: 
We used Cloudstack 4.7 in our testing environment. For future uses we updated 
to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 
cloudstack has some problems to connect to the hosts again. Here are the error 
logs.
{code:title=XEN Server from the managment-server.log:|borderStyle=solid}
2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
(ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly 
connected host 13(xen5)
2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is 
running...
2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is running...
2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is 
running...
2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is running...
2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] 
(StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary storage 
VM for secondary storage host cdsdev-secondary
2016-02-15 11:22:11,018 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
(AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired 
async-jobs
2016-02-15 11:22:11,024 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
(AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired 
async-jobs
2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] 
(ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create 
master connection to host(192.168.0.97) , due to The credentials given by the 
user are incorrect, so access has been denied, and you have not been issued a 
session handle.
2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager 
Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, 
Agent event = AgentDisconnected, Host id = 13, name = xen5]
2016-02-15 11:22:11,101 WARN  [c.c.a.m.ClusteredAgentManagerImpl] 
(ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec)  can not load 
directly connected host 13(xen5) due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to create master 
connection to host(192.168.0.97) , due to The credentials given by the user are 
incorrect, so access has been denied, and you have not been issued a session 
handle.
at 
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
at 
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
at 
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
at 
com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
at 
com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: The credentials given by the user are incorrect, so access has been 
denied, and you have not been issued a session handle.
at com.xensource.xenapi.Types.checkResponse(Types.java:322)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at com.xensource.xenapi.Session.loginWithPassword(Session.java:587)
  

[jira] [Updated] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts

2016-03-03 Thread Simon Weller (JIRA)

 [ 
https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Weller updated CLOUDSTACK-9285:
-
Attachment: agentfailure.pcap

Here's a pcap of the issue. haproxy is sending a RST

> Cloudstack 4.8 can't connect to XEN and KVM hosts
> -
>
> Key: CLOUDSTACK-9285
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: KVM, Xen
>Affects Versions: 4.8.0
> Environment: CentOS 7
>Reporter: Jens Fettig
>Priority: Blocker
> Attachments: agentfailure.pcap
>
>
> We used Cloudstack 4.7 in our testing environment. For future uses we updated 
> to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 
> cloudstack has some problems to connect to the hosts again. Here are the 
> error logs.
> {code:title=XEN Server from the managment-server.log:|borderStyle=solid}
> 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly 
> connected host 13(xen5)
> 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is 
> running...
> 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is 
> running...
> 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is 
> running...
> 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is 
> running...
> 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary 
> storage VM for secondary storage host cdsdev-secondary
> 2016-02-15 11:22:11,018 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired 
> async-jobs
> 2016-02-15 11:22:11,024 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired 
> async-jobs
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create 
> master connection to host(192.168.0.97) , due to The credentials given by the 
> user are incorrect, so access has been denied, and you have not been issued a 
> session handle.
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager 
> Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, 
> Agent event = AgentDisconnected, Host id = 13, name = xen5]
> 2016-02-15 11:22:11,101 WARN  [c.c.a.m.ClusteredAgentManagerImpl] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec)  can not load 
> directly connected host 13(xen5) due to 
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master 
> connection to host(192.168.0.97) , due to The credentials given by the user 
> are incorrect, so access has been denied, and you have not been issued a 
> session handle.
>   at 
> com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
>   at 
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
>   at 
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
>   at 
> com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
>   at 
> com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
>   at 
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>   at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>   at 
> org.apache.cloudstack.managed.context

[jira] [Updated] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts

2016-05-16 Thread Koushik Das (JIRA)

 [ 
https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koushik Das updated CLOUDSTACK-9285:

Fix Version/s: 4.7.1
   4.9.0
   4.8.1

> Cloudstack 4.8 can't connect to XEN and KVM hosts
> -
>
> Key: CLOUDSTACK-9285
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: KVM, Xen
>Affects Versions: 4.7.0, 4.8.0
> Environment: CentOS 7
>Reporter: Jens Fettig
>Priority: Blocker
> Fix For: 4.7.1, 4.8.1, 4.9.0
>
> Attachments: agentfailure.pcap
>
>
> We used Cloudstack 4.7 in our testing environment. For future uses we updated 
> to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 
> cloudstack has some problems to connect to the hosts again. Here are the 
> error logs.
> {code:title=XEN Server from the managment-server.log:|borderStyle=solid}
> 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly 
> connected host 13(xen5)
> 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is 
> running...
> 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is 
> running...
> 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is 
> running...
> 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is 
> running...
> 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary 
> storage VM for secondary storage host cdsdev-secondary
> 2016-02-15 11:22:11,018 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired 
> async-jobs
> 2016-02-15 11:22:11,024 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired 
> async-jobs
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create 
> master connection to host(192.168.0.97) , due to The credentials given by the 
> user are incorrect, so access has been denied, and you have not been issued a 
> session handle.
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager 
> Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, 
> Agent event = AgentDisconnected, Host id = 13, name = xen5]
> 2016-02-15 11:22:11,101 WARN  [c.c.a.m.ClusteredAgentManagerImpl] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec)  can not load 
> directly connected host 13(xen5) due to 
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master 
> connection to host(192.168.0.97) , due to The credentials given by the user 
> are incorrect, so access has been denied, and you have not been issued a 
> session handle.
>   at 
> com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
>   at 
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
>   at 
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
>   at 
> com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
>   at 
> com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
>   at 
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>   at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>   at 
> 

[jira] [Updated] (CLOUDSTACK-9285) Cloudstack 4.8 can't connect to XEN and KVM hosts

2016-05-16 Thread Koushik Das (JIRA)

 [ 
https://issues.apache.org/jira/browse/CLOUDSTACK-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koushik Das updated CLOUDSTACK-9285:

Affects Version/s: 4.7.0

> Cloudstack 4.8 can't connect to XEN and KVM hosts
> -
>
> Key: CLOUDSTACK-9285
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9285
> Project: CloudStack
>  Issue Type: Bug
>  Security Level: Public(Anyone can view this level - this is the 
> default.) 
>  Components: KVM, Xen
>Affects Versions: 4.7.0, 4.8.0
> Environment: CentOS 7
>Reporter: Jens Fettig
>Priority: Blocker
> Fix For: 4.7.1, 4.8.1, 4.9.0
>
> Attachments: agentfailure.pcap
>
>
> We used Cloudstack 4.7 in our testing environment. For future uses we updated 
> to Cloudstack 4.8 for some tests. But after the update from 4.7 to 4.8 
> cloudstack has some problems to connect to the hosts again. Here are the 
> error logs.
> {code:title=XEN Server from the managment-server.log:|borderStyle=solid}
> 2016-02-15 11:22:03,476 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Loading directly 
> connected host 13(xen5)
> 2016-02-15 11:22:06,204 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-1:ctx-aaf933f3) (logid:d157e37d) AutoScaling Monitor is 
> running...
> 2016-02-15 11:22:06,206 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-2:ctx-8e2fc084) (logid:379074cb) VmStatsCollector is 
> running...
> 2016-02-15 11:22:06,217 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-4:ctx-cc1cf960) (logid:b6397e2e) HostStatsCollector is 
> running...
> 2016-02-15 11:22:06,298 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) StorageCollector is 
> running...
> 2016-02-15 11:22:06,302 DEBUG [c.c.s.StatsCollector] 
> (StatsCollector-3:ctx-efda9c07) (logid:5731423e) There is no secondary 
> storage VM for secondary storage host cdsdev-secondary
> 2016-02-15 11:22:11,018 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) Begin cleanup expired 
> async-jobs
> 2016-02-15 11:22:11,024 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (AsyncJobMgr-Heartbeat-1:ctx-bc740746) (logid:cc619bad) End cleanup expired 
> async-jobs
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.x.r.XenServerConnectionPool] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec) Unable to create 
> master connection to host(192.168.0.97) , due to The credentials given by the 
> user are incorrect, so access has been denied, and you have not been issued a 
> session handle.
> 2016-02-15 11:22:11,055 DEBUG [c.c.h.Status] (ClusteredAgentManager 
> Timer:ctx-43ef0a0a) (logid:82d7b1ec) Transition:[Resource state = Enabled, 
> Agent event = AgentDisconnected, Host id = 13, name = xen5]
> 2016-02-15 11:22:11,101 WARN  [c.c.a.m.ClusteredAgentManagerImpl] 
> (ClusteredAgentManager Timer:ctx-43ef0a0a) (logid:82d7b1ec)  can not load 
> directly connected host 13(xen5) due to 
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master 
> connection to host(192.168.0.97) , due to The credentials given by the user 
> are incorrect, so access has been denied, and you have not been issued a 
> session handle.
>   at 
> com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:163)
>   at 
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:523)
>   at 
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.configure(CitrixResourceBase.java:827)
>   at 
> com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:158)
>   at 
> com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
>   at 
> com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
>   at 
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>   at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>   at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>   at 
> org.apache.cloudstack.managed.context.impl.Def