[ https://issues.apache.org/jira/browse/CLOUDSTACK-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhinandan Prateek resolved CLOUDSTACK-2127. -------------------------------------------- Resolution: Fixed > Used Master Branch System VM Template for VMWare: Failed to Create Load > Balancing Rule > -------------------------------------------------------------------------------------- > > Key: CLOUDSTACK-2127 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2127 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Reporter: Chandan Purushothama > Assignee: Abhinandan Prateek > Priority: Critical > Fix For: 4.2.0 > > > =============== > Steps to Reproduce: > =============== > 1. Create a Load Balancing Rule where the Virtual Router is deployed using > the new Master System VM Template for VMWare > ============ > Observations: > ============ > ------------------------------------------------ > Create Load Balancing Rule Job: > ------------------------------------------------ > [root@asfmgmt ~]# grep -i "job-24" > /var/log/cloudstack/management/management-server.log > 2013-04-21 17:22:03,866 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-3:null) submit async job-24, details: AsyncJobVO {id:24, > userId: 3, accountId: 3, sessionKey: null, instanceType: null, instanceId: > null, cmd: > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd, > cmdOriginator: null, cmdInfo: > {"response":"json","id":"0d569a49-ee02-4f32-93b5-31f1bc9aaa99","sessionkey":"LmU2BWQx2jc+w2FawY1qyY9foFk\u003d","virtualmachineids":"d68f7c60-d963-4a32-bcd4-1b5f37aaa992,3a5f350d-0740-424a-9172-7fb893effdeb","ctxUserId":"3","_":"1366590152786","ctxAccountId":"3","ctxStartEventId":"115"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 7200344900649, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-04-21 17:22:03,870 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-30:job-24) Executing > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd > for job-24 > 2013-04-21 17:22:03,882 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-30:job-24) Sync job-24 execution on object network.204 > 2013-04-21 17:22:03,898 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-30:job-24) job > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd > for job-24 was queued, processing the queue. > 2013-04-21 17:22:03,904 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-30:job-24) Executing sync queue item: SyncQueueItemVO {id:7, > queueId: 1, contentType: AsyncJob, contentId: 24, lastProcessMsid: > 7200344900649, lastprocessNumber: 7, lastProcessTime: Sun Apr 21 17:22:03 PDT > 2013, created: Sun Apr 21 17:22:03 PDT 2013} > 2013-04-21 17:22:03,905 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-30:job-24) Schedule queued job-24 > 2013-04-21 17:22:03,909 DEBUG [cloud.async.SyncQueueManagerImpl] > (Job-Executor-30:job-24) There is a pending process in sync queue(id: 1) > 2013-04-21 17:22:03,911 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-31:job-24) Executing > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd > for job-24 > 2013-04-21 17:22:03,944 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-31:job-24) Access to Rule[4-LoadBalancing-Add] granted to > Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710 > 2013-04-21 17:22:03,945 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-31:job-24) Access to VM[User|Atoms-VM-2] granted to > Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710 > 2013-04-21 17:22:03,947 DEBUG [network.lb.LoadBalancingRulesManagerImpl] > (Job-Executor-31:job-24) Adding VM[User|Atoms-VM-2] to the load balancer pool > 2013-04-21 17:22:03,954 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-31:job-24) Access to Rule[4-LoadBalancing-Add] granted to > Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710 > 2013-04-21 17:22:03,955 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-31:job-24) Access to VM[User|Atoms-VM-1] granted to > Acct[3-atoms] by DomainChecker_EnhancerByCloudStack_64236710 > 2013-04-21 17:22:03,957 DEBUG [network.lb.LoadBalancingRulesManagerImpl] > (Job-Executor-31:job-24) Adding VM[User|Atoms-VM-1] to the load balancer pool > 2013-04-21 17:22:04,109 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-31:job-24) > Applying ip association in network Ntwk[204|Guest|8] > 2013-04-21 17:22:04,129 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875769: Sending { Cmd , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 100001, > [{"routing.IpAssocCommand":{"ipAddresses":[{"accountId":3,"publicIp":"10.223.139.70","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:c6:4a:00:00:0c","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.74","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:ef:5a:00:00:10","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.75","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:89:90:00:00:11","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.73","sourceNat":false,"add":true,"oneToOneNat":true,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:81:ce:00:00:0f","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] > } > 2013-04-21 17:22:04,130 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875769: Executing: { Cmd , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 100001, > [{"routing.IpAssocCommand":{"ipAddresses":[{"accountId":3,"publicIp":"10.223.139.70","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:c6:4a:00:00:0c","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.74","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:ef:5a:00:00:10","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.75","sourceNat":false,"add":true,"oneToOneNat":false,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:89:90:00:00:11","networkRate":200,"trafficType":"Public"},{"accountId":3,"publicIp":"10.223.139.73","sourceNat":false,"add":true,"oneToOneNat":true,"firstIP":false,"vlanId":"1391","vlanGateway":"10.223.139.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:81:ce:00:00:0f","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] > } > 2013-04-21 17:22:22,917 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875769: Received: { Ans: , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } } > 2013-04-21 17:22:22,925 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-31:job-24) > Applying firewall rules in network Ntwk[204|Guest|8] > 2013-04-21 17:22:22,965 DEBUG [cloud.network.NetworkModelImpl] > (Job-Executor-31:job-24) Service SecurityGroup is not supported in the > network id=204 > 2013-04-21 17:22:23,002 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875771: Sending { Cmd , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 100001, > [{"routing.LoadBalancerConfigCommand":{"loadBalancers":[{"uuid":"0d569a49-ee02-4f32-93b5-31f1bc9aaa99","srcIp":"10.223.139.75","srcPort":22,"protocol":"tcp","algorithm":"roundrobin","revoked":false,"alreadyAdded":false,"inline":false,"destinations":[{"destIp":"10.1.1.63","destPort":22,"revoked":false,"alreadyAdded":false},{"destIp":"10.1.1.146","destPort":22,"revoked":false,"alreadyAdded":false}]}],"lbStatsVisibility":"global","lbStatsPublicIP":"10.223.139.70","lbStatsPrivateIP":"10.223.59.80","lbStatsGuestIP":"10.1.1.1","lbStatsPort":"8081","lbStatsSrcCidrs":"0/0","lbStatsAuth":"admin1:AdMiN123","lbStatsUri":"/admin?stats","nic":{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"6e104794-6262-4516-b886-6130eabc1e5f","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:40:dd:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2586","isolationUri":"vlan://2586","isSecurityGroupEnabled":false},"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] > } > 2013-04-21 17:22:23,002 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875771: Executing: { Cmd , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 100001, > [{"routing.LoadBalancerConfigCommand":{"loadBalancers":[{"uuid":"0d569a49-ee02-4f32-93b5-31f1bc9aaa99","srcIp":"10.223.139.75","srcPort":22,"protocol":"tcp","algorithm":"roundrobin","revoked":false,"alreadyAdded":false,"inline":false,"destinations":[{"destIp":"10.1.1.63","destPort":22,"revoked":false,"alreadyAdded":false},{"destIp":"10.1.1.146","destPort":22,"revoked":false,"alreadyAdded":false}]}],"lbStatsVisibility":"global","lbStatsPublicIP":"10.223.139.70","lbStatsPrivateIP":"10.223.59.80","lbStatsGuestIP":"10.1.1.1","lbStatsPort":"8081","lbStatsSrcCidrs":"0/0","lbStatsAuth":"admin1:AdMiN123","lbStatsUri":"/admin?stats","nic":{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"6e104794-6262-4516-b886-6130eabc1e5f","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:40:dd:00:02","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2586","isolationUri":"vlan://2586","isSecurityGroupEnabled":false},"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"10.223.59.80","router.name":"r-4-SYSVMTMPL"},"wait":0}}] > } > 2013-04-21 17:22:30,957 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875771: Received: { Ans: , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 0, { Answer } } > 2013-04-21 17:22:30,961 WARN [network.lb.LoadBalancingRulesManagerImpl] > (Job-Executor-31:job-24) Unable to apply the load balancer config because > resource is unavaliable. > 2013-04-21 17:22:30,963 INFO [cloud.utils.AnnotationHelper] > (Job-Executor-31:job-24) > 2013-04-21 17:22:30,984 ERROR [cloud.async.AsyncJobManagerImpl] > (Job-Executor-31:job-24) Unexpected exception while executing > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd > 2013-04-21 17:22:30,985 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-31:job-24) Complete async job-24, jobStatus: 2, resultCode: > 530, result: Error Code: 530 Error text: Failed to add specified > loadbalancerruleid for vms [6, 3] > 2013-04-21 17:22:30,996 DEBUG [cloud.async.SyncQueueManagerImpl] > (Job-Executor-31:job-24) Sync queue (1) is currently empty > 2013-04-21 17:22:31,092 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-25:null) Async job-24 completed > ================ > ERROR: > ================ > 2013-04-21 17:22:29,784 ERROR [utils.ssh.SshHelper] > (DirectAgent-126:10.223.59.66) SSH execution of command /root/loadbalancer.sh > -i 10.223.59.80 -f /tmp/10_223_59_80.cfg -a 10.223.139.75:22:, -s > 10.223.139.70:8081:0/0:,, has an error status code in return. result output: > mv: cannot stat `/var/run/haproxy.pid': No such file or directory > cat: /var/run/haproxy.pid.old: No such file or directory > kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or > kill -l [sigspec] > /root/reconfigLB.sh: line 28: haproxy: command not found > cat: /var/run/haproxy.pid.old: No such file or directory > kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or > kill -l [sigspec] > mv: cannot stat `/var/run/haproxy.pid.old': No such file or directory > 2013-04-21 17:22:29,791 ERROR [vmware.resource.VmwareResource] > (DirectAgent-126:10.223.59.66) LoadBalancerConfigCommand on domain router > 10.223.59.80 failed. message: mv: cannot stat `/var/run/haproxy.pid': No such > file or directory > cat: /var/run/haproxy.pid.old: No such file or directory > kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or > kill -l [sigspec] > /root/reconfigLB.sh: line 28: haproxy: command not found > cat: /var/run/haproxy.pid.old: No such file or directory > kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or > kill -l [sigspec] > mv: cannot stat `/var/run/haproxy.pid.old': No such file or directory > 2013-04-21 17:22:30,957 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-126:null) Seq 1-1166875771: Response Received: > 2013-04-21 17:22:30,957 DEBUG [agent.transport.Request] > (DirectAgent-126:null) Seq 1-1166875771: Processing: { Ans: , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 0, > [{"Answer":{"result":false,"details":"LoadBalancerConfigCommand on domain > router 10.223.59.80 failed. message: mv: cannot stat `/var/run/haproxy.pid': > No such file or directory\ncat: /var/run/haproxy.pid.old: No such file or > directory\nkill: usage: kill [-s sigspec | -n signum | -sigspec] pid | > jobspec ... or kill -l [sigspec]\n/root/reconfigLB.sh: line 28: haproxy: > command not found\ncat: /var/run/haproxy.pid.old: No such file or > directory\nkill: usage: kill [-s sigspec | -n signum | -sigspec] pid | > jobspec ... or kill -l [sigspec]\nmv: cannot stat `/var/run/haproxy.pid.old': > No such file or directory\n","wait":0}}] } > 2013-04-21 17:22:30,957 DEBUG [agent.transport.Request] > (Job-Executor-31:job-24) Seq 1-1166875771: Received: { Ans: , MgmtId: > 7200344900649, via: 1, Ver: v1, Flags: 0, { Answer } } > 2013-04-21 17:22:30,961 WARN [network.lb.LoadBalancingRulesManagerImpl] > (Job-Executor-31:job-24) Unable to apply the load balancer config because > resource is unavaliable. > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is > unreachable: Unable to apply firewall rules on router > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3439) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3294) > at > com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:361) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyRules(LoadBalancingRulesManagerImpl.java:1425) > at > com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2428) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1461) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerConfig(LoadBalancingRulesManagerImpl.java:1385) > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.assignToLoadBalancer(LoadBalancingRulesManagerImpl.java:909) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd.execute(AssignToLoadBalancerRuleCmd.java:100) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:164) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-04-21 17:22:30,963 INFO [cloud.utils.AnnotationHelper] > (Job-Executor-31:job-24) > class java.lang.Stringdoes not have a Table annotation > 2013-04-21 17:22:30,984 ERROR [cloud.async.AsyncJobManagerImpl] > (Job-Executor-31:job-24) Unexpected exception while executing > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd > com.cloud.utils.exception.CloudRuntimeException: Failed to add specified > loadbalancerruleid for vms [6, 3] > at > com.cloud.network.lb.LoadBalancingRulesManagerImpl.assignToLoadBalancer(LoadBalancingRulesManagerImpl.java:932) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.loadbalancer.AssignToLoadBalancerRuleCmd.execute(AssignToLoadBalancerRuleCmd.java:100) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:164) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-04-21 17:22:30,985 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-31:job-24) Complete async job-24, jobStatus: 2, resultCode: > 530, result: Error Code: 530 Error text: Failed to add specified > loadbalancerruleid for vms [6, 3] > 2013-04-21 17:22:30,996 DEBUG [cloud.async.SyncQueueManagerImpl] > (Job-Executor-31:job-24) Sync queue (1) is currently empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira