[ https://issues.apache.org/jira/browse/CLOUDSTACK-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Animesh Chaturvedi updated CLOUDSTACK-7726: ------------------------------------------- Assignee: Anthony Xu > [Automation][XenServer] Failed to Remove LoadBalancer Rule due to > callHostPlugin failure > ---------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-7726 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7726 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Automation > Affects Versions: 4.5.0 > Reporter: Chandan Purushothama > Assignee: Anthony Xu > Priority: Critical > Fix For: 4.5.0 > > Attachments: management-server.zip > > > ======================================================= > CallHost Plugin Failure while trying to remove the Load Balancing Rule: > ======================================================= > 2014-10-13 19:20:06,408 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-45:ctx-fb2667f4) Seq 1-122723089845846136: Executing request > 2014-10-13 19:20:06,410 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-89:ctx-0a5fbdf2) callHostPlugin failed for cmd: createFileInDomr > with args filepath: /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: > 169.254.2.250, filecontents: global > log 127.0.0.1:3914 local0 warning > maxconn 4096 > maxpipes 1024 > chroot /var/lib/haproxy > user haproxy > group haproxy > daemon > > defaults > log global > mode tcp > option dontlognull > retries 3 > option redispatch > option forwardfor > option forceclose > timeout connect 5000 > timeout client 50000 > timeout server 50000 > listen stats_on_public 10.220.166.19:8081 > mode http > option httpclose > stats enable > stats uri /admin?stats > stats realm Haproxy\ Statistics > stats auth admin1:AdMiN123 > > listen 10_220_166_28-2222 10.220.166.28:2222 > balance roundrobin > > > , due to There was a failure communicating with the plugin. > 2014-10-13 19:20:06,410 WARN [c.c.a.m.DirectAgentAttache] > (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Throwable caught > while executing command > com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for > cmd: createFileInDomr with args filepath: > /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250, > filecontents: global > log 127.0.0.1:3914 local0 warning > maxconn 4096 > maxpipes 1024 > chroot /var/lib/haproxy > user haproxy > group haproxy > daemon > > defaults > log global > mode tcp > option dontlognull > retries 3 > option redispatch > option forwardfor > option forceclose > timeout connect 5000 > timeout client 50000 > timeout server 50000 > listen stats_on_public 10.220.166.19:8081 > mode http > option httpclose > stats enable > stats uri /admin?stats > stats realm Haproxy\ Statistics > stats auth admin1:AdMiN123 > > listen 10_220_166_28-2222 10.220.166.28:2222 > balance roundrobin > > > , due to There was a failure communicating with the plugin. > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:3670) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createFileInVR(CitrixResourceBase.java:575) > at > com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:161) > at > com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:155) > at > com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:179) > at > com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:125) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:430) > at > com.cloud.hypervisor.xenserver.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:64) > at > com.cloud.hypervisor.xenserver.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:87) > at > com.cloud.hypervisor.xenserver.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-10-13 19:20:06,411 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-89:ctx-0a5fbdf2) Seq 1-122723089845846135: Response Received: > 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] (DirectAgent-89:ctx-0a5fbdf2) > Seq 1-122723089845846135: Processing: { Ans: , MgmtId: 59408290029589, via: > 1, Ver: v1, Flags: 0, > [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: > callHostPlugin failed for cmd: createFileInDomr with args filepath: > /etc/haproxy/haproxy.cfg.new.1413228006238, domrip: 169.254.2.250, > filecontents: global\n\tlog 127.0.0.1:3914 local0 warning\n\tmaxconn > 4096\n\tmaxpipes 1024\n\tchroot /var/lib/haproxy\n\tuser haproxy\n\tgroup > haproxy\n\tdaemon\n\t \ndefaults\n\tlog global\n\tmode tcp\n\toption > dontlognull\n\tretries 3\n\toption redispatch\n\toption forwardfor\n\toption > forceclose\n\ttimeout connect 5000\n\ttimeout client 50000\n\ttimeout > server 50000\n\nlisten stats_on_public 10.220.166.19:8081\n\tmode > http\n\toption httpclose\n\tstats enable\n\tstats uri > /admin?stats\n\tstats realm Haproxy\\ Statistics\n\tstats auth > admin1:AdMiN123\n\n\t \nlisten 10_220_166_28-2222 > 10.220.166.28:2222\n\tbalance roundrobin\n\t \n\t \n, due to There was a > failure communicating with the plugin.","wait":0}}] } > 2014-10-13 19:20:06,412 DEBUG [c.c.a.t.Request] > (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Seq > 1-122723089845846135: Received: { Ans: , MgmtId: 59408290029589, via: 1, > Ver: v1, Flags: 0, { Answer } } > 2014-10-13 19:20:06,424 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id: > 11,while removing vmId 4 > 2014-10-13 19:20:06,427 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) LB Rollback rule id: > 11 while removing vm instances > 2014-10-13 19:20:06,427 WARN [c.c.n.l.LoadBalancingRulesManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84 ctx-803aafba) Unable to apply the > load balancer config because resource is unavaliable. > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is > unreachable: Unable to apply loadbalancing rules on router > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:4050) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3901) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy190.applyLoadBalancingRules(Unknown Source) > at > com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:384) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1845) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2433) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1881) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1805) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1377) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy138.removeFromLoadBalancer(Unknown Source) > at > org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-10-13 19:20:06,444 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-23:ctx-238092a2) Created a vif > 825c5404-4bc8-3a35-8961-db3d95bde69d on 1 > 2014-10-13 19:20:06,452 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Unexpected exception while > executing > org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd > com.cloud.utils.exception.CloudRuntimeException: Failed to remove specified > load balancer rule id for vms [4] > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancerInternal(LoadBalancingRulesManagerImpl.java:1413) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.removeFromLoadBalancer(LoadBalancingRulesManagerImpl.java:1163) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy138.removeFromLoadBalancer(Unknown Source) > at > org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd.execute(RemoveFromLoadBalancerRuleCmd.java:162) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2014-10-13 19:20:06,455 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Complete async job-84, jobStatus: > FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > to remove specified load balancer rule id for vms [4]"} > 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Publish async job-84 complete on > message bus > 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs related to job- 84 > 2014-10-13 19:20:06,456 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Update db status for job- 84 > 2014-10-13 19:20:06,458 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Wake up jobs joined with job- 84 > and disjoin all subjobs created from job- 84 > 2014-10-13 19:20:06,463 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-41:ctx-9ee60aa8 job-84) Done executing > org.apache.cloudstack.api.command.user.loadbalancer.RemoveFromLoadBalancerRuleCmd > for job-84 -- This message was sent by Atlassian JIRA (v6.3.4#6332)