Hello All
I'm facing the following issue since early morning today. I tried to update the system offering of the virtual router and restart the VM, but the VM starts and then after it reaches the login prompt throws some dnsmasq errors and halts. I've not been able to start the Virtual Router after that. Any help in this issue would be much appreciated. The errors found in the log files: 2014-11-05 15:29:20,327 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-210:ctx-193f97e2) callHostPlugin failed for cmd: createFileInDomr with args filepath: /var/cache/cloud/VR-5ef8abcd-0141-4f3f-9bcd-c5cc8a8221be.cfg, domrip: 169.254.0.178, filecontents: #Apache CloudStack Virtual Router Config File <version> 1.0 </version> <script> /opt/cloud/bin/monitor_service.sh -c [dhcp]:processname=dnsmasq:servicename=dnsmasq:pidfile=/var/run/dnsmasq/dnsmasq.pid:,[ssh]:processname=sshd:servicename=ssh:pidfile=/var/run/sshd.pid:,[webserver]:processname=apache2:servicename=apache2:pidfile=/var/run/apache2.pid:, </script> <script> ... .... .. /opt/cloud/bin/vmdata.py -d eyIxNDYuMTY5LjQ2LjE4OCI6W1sidXNlcmRhdGEiLCJ1c2VyLWRhdGEiLG51bGxdLFsibWV0YWRhdGEiLCJzZXJ2aWNlLW9mZmVyaW5nIiwiTG9jYWwgU3RvcmFnZSJdLFsibWV0YWRhdGEiLCJhdmFpbGFiaWxpdHktem9uZSIsImRvYy00NiJdLFsibWV0YWRhdGEiLCJsb2NhbC1pcHY0IiwiMTQ2LjE2OS40Ni4xODgiXSxbIm1ldGFkYXRhIiwibG9jYWwtaG9zdG5hbWUiLCJ0ZXN0Y3NnMyJdLFsibWV0YWRhdGEiLCJwdWJsaWMtaXB2NCIsIjE0Ni4xNjkuNDYuMTg4Il0sWyJtZXRhZGF0YSIsInB1YmxpYy1ob3N0bmFtZSIsInRlc3Rjc2czIl0sWyJtZXRhZGF0YSIsImluc3RhbmNlLWlkIiwiZWIxMDIxYTEtMTlmOS00YjY5LWFjNGMtODExMTg0NjE5ZGQ4Il0sWyJtZXRhZGF0YSIsInZtLWlkIiwiZWIxMDIxYTEtMTlmOS00YjY5LWFjNGMtODExMTg0NjE5ZGQ4Il0sWyJtZXRhZGF0YSIsInB1YmxpYy1rZXlzIixudWxsXSxbIm1ldGFkYXRhIiwiY2xvdWQtaWRlbnRpZmllciIsIkNsb3VkU3RhY2stezRkNzVjZDBlLTZmOTMtNDdiZC1iOGE0LTlmODNhMzU3NDlmY30iXV19 </script> , due to There was a failure communicating with the plugin. 2014-11-05 15:29:20,337 WARN [c.c.a.m.DirectAgentAttache] (DirectAgent-210:ctx-193f97e2) Seq 11-7965178890958143578: Exception Caught while executing command com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: createFileInDomr with args filepath: /var/cache/cloud/VR-5ef8abcd-0141-4f3f-9bcd-c5cc8a8221be.cfg, domrip: 169.254.0.178, filecontents: #Apache CloudStack Virtual Router Config File <version> 1.0 </version> ..... ..... <script> /opt/cloud/bin/vmdata.py -d eyIxNDYuMTY5LjQ2LjE4OCI6W1sidXNlcmRhdGEiLCJ1c2VyLWRhdGEiLG51bGxdLFsibWV0YWRhdGEiLCJzZXJ2aWNlLW9mZmVyaW5nIiwiTG9jYWwgU3RvcmFnZSJdLFsibWV0YWRhdGEiLCJhdmFpbGFiaWxpdHktem9uZSIsImRvYy00NiJdLFsibWV0YWRhdGEiLCJsb2NhbC1pcHY0IiwiMTQ2LjE2OS40Ni4xODgiXSxbIm1ldGFkYXRhIiwibG9jYWwtaG9zdG5hbWUiLCJ0ZXN0Y3NnMyJdLFsibWV0YWRhdGEiLCJwdWJsaWMtaXB2NCIsIjE0Ni4xNjkuNDYuMTg4Il0sWyJtZXRhZGF0YSIsInB1YmxpYy1ob3N0bmFtZSIsInRlc3Rjc2czIl0sWyJtZXRhZGF0YSIsImluc3RhbmNlLWlkIiwiZWIxMDIxYTEtMTlmOS00YjY5LWFjNGMtODExMTg0NjE5ZGQ4Il0sWyJtZXRhZGF0YSIsInZtLWlkIiwiZWIxMDIxYTEtMTlmOS00YjY5LWFjNGMtODExMTg0NjE5ZGQ4Il0sWyJtZXRhZGF0YSIsInB1YmxpYy1rZXlzIixudWxsXSxbIm1ldGFkYXRhIiwiY2xvdWQtaWRlbnRpZmllciIsIkNsb3VkU3RhY2stezRkNzVjZDBlLTZmOTMtNDdiZC1iOGE0LTlmODNhMzU3NDlmY30iXV19 </script> , due to There was a failure communicating with the plugin. at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3782) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.createFileInVR(CitrixResourceBase.java:588) at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:1161) at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:171) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:444) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ... ... 2014-11-05 15:29:20,344 DEBUG [c.c.a.t.Request] (Work-Job-Executor-6:ctx-e652ea17 job-9535/job-9536 ctx-f3deae88) Seq 11-7965178890958143578: Received: { Ans: , MgmtId: 233845174477512, via: 11, Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer, NetworkUsageAnswer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answe r , Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answ e r, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer, Answer } } 2014-11-05 15:29:20,350 WARN [o.a.c.alerts] (Work-Job-Executor-6:ctx-e652ea17 job-9535/job-9536 ctx-f3deae88) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Command: com.cloud.agent.api.Command failed while starting virtual router 2014-11-05 15:29:20,355 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-6:ctx-e652ea17 job-9535/job-9536 ctx-f3deae88) Command: com.cloud.agent.api.Command failed while starting virtual router 2014-11-05 15:29:20,355 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-6:ctx-e652ea17 job-9535/job-9536 ctx-f3deae88) The guru did not like the answers so stopping VM[DomainRouter|r-673-VM] 2014-11-05 15:29:20,358 DEBUG [c.c.a.t.Request] (Work-Job-Executor-6:ctx-e652ea17 job-9535/job-9536 ctx-f3deae88) Seq 11-7965178890958143581: Sending { Cmd , MgmtId: 233845174477512, via: 11(motmot), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"r-673-VM","wait":0}}] } 2014-11-05 15:29:20,359 DEBUG [c.c.a.t.Request] (Work-Job-Executor-6:ctx-e652ea17 job-9535/job-9536 ctx-f3deae88) Seq 11-7965178890958143581: Executing: { Cmd , MgmtId: 233845174477512, via: 11(motmot), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"r-673-VM","wait":0}}] } 2014-11-05 15:29:20,359 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-354:ctx-9b42d6df) Seq 11-7965178890958143581: Executing request 2014-11-05 15:29:20,382 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-354:ctx-9b42d6df) 9. The VM r-673-VM is in Stopping state 2014-11-05 15:29:20,682 INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-354:ctx-9b42d6df) Removed network rules for vm r-673-VM -- best wishes, Thomas Joseph