After running for about 30-40 minutes, the new vm instance creation (via gui) fails, with the following error :
2014-07-28 13:37:04,831 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-988151911: Forwarding Seq 1-988151911: { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651 2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-14:null) Seq 1-988152032: Forwarding Seq 1-988152032: { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to 279278805451651 2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-9:null) Seq 1-988151911: Forwarding Seq 1-988151911: { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651 2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-4:null) Seq 1-988152032: Forwarding Seq 1-988152032: { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to 279278805451651 2014-07-28 13:37:04,832 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-12:null) Seq 1-988151911: Forwarding Seq 1-988151911: { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"routing.DhcpEntryCommand":{"vmMac":"06:7c:3a:00:00:14","vmIpAddress":"XXX.XXX.X.179","vmName":"CentOS-6-5-new","defaultRouter":"XXX.XXX.X.1","defaultDns":"XXX.XXX.X.172","duid":"00:03:00:01:06:7c:3a:00:00:14","isDefault":true,"accessDetails":{"router.guest.ip":"XXX.XXX.X.172","zone.network.type":"Basic","router.ip":"169.254.3.178","router.name":"r-4-VM"},"wait":0}}] } to 279278805451651 2014-07-28 13:37:04,833 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-2:null) Seq 1-988152032: Forwarding Seq 1-988152032: { Cmd , MgmtId: 187796561440775, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"i-2-11-VM","wait":0}}] } to 279278805451651 tail -20 /var/log/cloudstack/agent/agent.log 2014-07-28 13:37:03,430 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250 2014-07-28 13:37:06,429 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads. java.net.NoRouteToHostException: No route to host at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534) at com.cloud.utils.nio.NioClient.init(NioClient.java:64) at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108) at java.lang.Thread.run(Thread.java:701) 2014-07-28 13:37:11,430 INFO [utils.nio.NioClient] (Agent-Selector:null) Connecting to XXX.XXX.X.3:8250 2014-07-28 13:37:14,227 INFO [cloud.agent.Agent] (AgentShutdownThread:null) Stopping the agent: Reason = sig.kill 2014-07-28 13:37:14,429 ERROR [utils.nio.NioConnection] (Agent-Selector:null) Unable to initialize the threads. java.net.NoRouteToHostException: No route to host at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:534) at com.cloud.utils.nio.NioClient.init(NioClient.java:64) at com.cloud.utils.nio.NioConnection.run(NioConnection.java:108) at java.lang.Thread.run(Thread.java:701) tail -10 /var/log/cloudstack/management/access_log.2014-07-28.txt XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/system.js?t=1406534804765 HTTP/1.1" 200 - XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/docs.js?t=1406534804765 HTTP/1.1" 200 31975 XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/zoneWizard.js?t=1406534804765 HTTP/1.1" 200 - XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/scripts/globalSettings.js?t=1406534804765 HTTP/1.1" 200 5870 XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/api?command=listCapabilities&response=json&sessionkey=null&_=1406534804986 HTTP/1.1" 401 98 XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/logo-login-oss.png HTTP/1.1" 304 - XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/sprites.png HTTP/1.1" 304 - XXX.XXX.X.5 - - [28/Jul/2014:13:36:44 +0530] "GET /client/images/bg-login.png HTTP/1.1" 304 - ________________________________ From: Rajani Karuturi <rajani.karut...@citrix.com> To: Giri Prasad <g_p...@yahoo.com> Cc: "users@cloudstack.apache.org" <users@cloudstack.apache.org> Sent: Monday, 28 July 2014 12:39 PM Subject: Re: Unable to stop or delete vm's looks like agent isn’t able to connect to management server. Did your management server ip change? is the agent state connected in cloudstack UI for the system vms? ~Rajani