After running for about 30-40 minutes, the new vm instance creation (via gui)
fails, with the following error :
2014-07-28 13:37:04,831 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911: {
Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111,
[{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-14:null) Seq 1-988152032: Forwarding Seq 1-988152032: {
Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to
279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911: {
Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111,
[{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-4:null) Seq 1-988152032: Forwarding Seq 1-988152032: {
Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to
279278805451651
2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911: {
Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111,
[{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}]
} to 279278805451651
2014-07-28 13:37:04,833 DEBUG [agent.manager.ClusteredAgentAttache]
(AgentManager-Handler-2:null) Seq 1-988152032: Forwarding Seq 1-988152032: {
Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to
279278805451651
tail -20 /var/log/cloudstack/agent/agent.log
2014-07-28 13:37:03,430 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to XXX.XXX.X.3:8250
2014-07-28 13:37:06,429 ERROR [utils.nio.NioConnection] (Agent-Selector:null)
Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
at java.lang.Thread.run(Thread.java:701)
2014-07-28 13:37:11,430 INFO [utils.nio.NioClient] (Agent-Selector:null)
Connecting to XXX.XXX.X.3:8250
2014-07-28 13:37:14,227 INFO [cloud.agent.Agent] (AgentShutdownThread:null)
Stopping the agent: Reason = sig.kill
2014-07-28 13:37:14,429 ERROR [utils.nio.NioConnection] (Agent-Selector:null)
Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534)
at com.cloud.utils.nio.NioClient.init(NioClient.java:64)
at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108)
at java.lang.Thread.run(Thread.java:701)
tail -10 /var/log/cloudstack/management/access_log.2014-07-28.txt
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET
/client/scripts/system.js?t=1406534804765 HTTP/1.1" 200 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET
/client/scripts/docs.js?t=1406534804765 HTTP/1.1" 200 31975
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET
/client/scripts/zoneWizard.js?t=1406534804765 HTTP/1.1" 200 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET
/client/scripts/globalSettings.js?t=1406534804765 HTTP/1.1" 200 5870
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET
/client/api?command=listCapabilities&response=json&sessionkey=null&_=1406534804986
HTTP/1.1" 401 98
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET
/client/images/logo-login-oss.png HTTP/1.1" 304 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/sprites.png
HTTP/1.1" 304 -
XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/bg-login.png
HTTP/1.1" 304 -
________________________________
From: Rajani Karuturi <[email protected]>
To: Giri Prasad <[email protected]>
Cc: "[email protected]" <[email protected]>
Sent: Monday, 28 July 2014 12:39 PM
Subject: Re: Unable to stop or delete vm's
looks like agent isn’t able to connect to management server.
Did your management server ip change? is the agent state connected in
cloudstack UI for the system vms?
~Rajani
