[ https://issues.apache.org/jira/browse/CLOUDSTACK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
venkata swamybabu budumuru updated CLOUDSTACK-2761: --------------------------------------------------- Attachment: logs.tgz > [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC > --------------------------------------------------------- > > Key: CLOUDSTACK-2761 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2761 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Network Controller > Affects Versions: 4.2.0 > Environment: commit # 8d1189c2ae87216bc1c4a1443f75e9a8629abdc2 > Reporter: venkata swamybabu budumuru > Priority: Blocker > Fix For: 4.2.0 > > Attachments: logs.tgz > > > Steps to reproduce: > 1. Have latest CloudStack build with at least 1 advanced zone > 2. Have at least 1 VMware cluster with 1 host. > 3. Create a VPC with at least 1 Tier > 4. deploy a VM > 5. apply an ACL to allow all > 6. acquire at least 1 IP > 7. create PF/staticNAT on the above created IP to the VM created in step (4) > Observations: > (i) It failed to create staticNAT with the following error > 2013-05-30 08:05:20,203 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-38:null) Seq 1-1416495168: Response Received: > 2013-05-30 08:05:20,206 DEBUG [agent.transport.Request] (DirectAgent-38:null) > Seq 1-1416495168: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver: > v1, Flags: 0, [{"routing.IpAssocAnswer":{"results":["10.147.44.63 - > success"],"result":true,"wait":0}}] } > 2013-05-30 08:05:20,207 DEBUG [agent.transport.Request] > (catalina-exec-19:null) Seq 1-1416495168: Received: { Ans: , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } } > 2013-05-30 08:05:20,211 INFO [cloud.network.NetworkManagerImpl] > (catalina-exec-19:null) Let VpcVirtualRouter handle StaticNat in network 204 > 2013-05-30 08:05:20,223 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-19:null) > Applying static nat rules in network Ntwk[204|Guest|11] > 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] > (catalina-exec-19:null) Seq 1-1416495169: Sending { Cmd , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 100001, > [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}] > } > 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] > (catalina-exec-19:null) Seq 1-1416495169: Executing: { Cmd , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 100001, > [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}] > } > 2013-05-30 08:05:20,289 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-16:null) Seq 1-1416495169: Executing request > 2013-05-30 08:05:20,290 INFO [vmware.resource.VmwareResource] > (DirectAgent-16:10.147.40.12) Executing resource SetFirewallRuleCommand: > {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0} > 2013-05-30 08:05:20,290 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-16:10.147.40.12) Use router's private IP for SSH control. IP : > 10.147.40.62 > 2013-05-30 08:05:21,746 ERROR [utils.ssh.SshHelper] > (DirectAgent-16:10.147.40.12) SSH execution of command /root/firewall.sh -A > -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G has an error status code in return. > result output: Bad argument `10.147.44.63' > Try `iptables -h' or 'iptables --help' for more information. > Bad argument `10.147.44.63' > Try `iptables -h' or 'iptables --help' for more information. > iptables v1.4.14: option "--set-mark" requires an argument > Try `iptables -h' or 'iptables --help' for more information. > iptables: No chain/target/match by that name. > Bad argument `10.147.44.63' > Try `iptables -h' or 'iptables --help' for more information. > Bad argument `eth0' > Try `iptables -h' or 'iptables --help' for more information. > Bad argument `10.147.44.63' > Try `iptables -h' or 'iptables --help' for more information. > iptables: No chain/target/match by that name. > iptables: No chain/target/match by that name. > Bad argument `10.147.44.63' > Try `iptables -h' or 'iptables --help' for more information. > 2013-05-30 08:05:21,790 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-16:10.147.40.12) Executing script on domain router 10.147.40.62: > /root/firewall.sh -A -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G > (ii) After changing the firewall.sh as mentioned below, it went fine. > get_dev_list() { > ip link show | grep -e eth[2-9] | awk -F ":" '{print $2}' > ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}' > } > Changed the above with the following to include eth1 device as well. > get_dev_list() { > ip link show | grep -e eth[1-9] | awk -F ":" '{print $2}' > ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}' > } > > 2013-05-30 08:32:52,492 INFO [cloud.network.NetworkManagerImpl] > (catalina-exec-3:null) Let VpcVirtualRouter handle StaticNat in network 204 > 2013-05-30 08:32:52,506 DEBUG > [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-3:null) > Applying static nat rules in network Ntwk[204|Guest|11] > 2013-05-30 08:32:52,523 DEBUG [agent.transport.Request] > (catalina-exec-3:null) Seq 1-1416495239: Sending { Cmd , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 100001, > [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}] > } > 2013-05-30 08:32:52,524 DEBUG [agent.transport.Request] > (catalina-exec-3:null) Seq 1-1416495239: Executing: { Cmd , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 100001, > [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}] > } > 2013-05-30 08:32:52,524 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-12:null) Seq 1-1416495239: Executing request > 2013-05-30 08:32:52,525 INFO [vmware.resource.VmwareResource] > (DirectAgent-12:10.147.40.12) Executing resource SetFirewallRuleCommand: > {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0} > 2013-05-30 08:32:52,529 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-12:10.147.40.12) Use router's private IP for SSH control. IP : > 10.147.40.62 > 2013-05-30 08:32:53,937 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-12:10.147.40.12) Executing script on domain router 10.147.40.62: > /root/firewall.sh -A -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G > 2013-05-30 08:32:53,938 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-12:null) Seq 1-1416495239: Response Received: > 2013-05-30 08:32:53,938 DEBUG [agent.transport.Request] (DirectAgent-12:null) > Seq 1-1416495239: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver: > v1, Flags: 0, > [{"routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}] > } > (iii) Tried to access the staticNat rule but, it failed because, it created a > firewall rule with DROP policy which is not required in case of staticNAt. > Ideally instead of firewall.sh, it should call "vlc_staticnat.sh" script to > configure staticnat. > Here is the snippet of iptable output. > root@r-3-VM:~# iptables -L -nv -t mangle > Chain PREROUTING (policy ACCEPT 3060 packets, 346K bytes) > pkts bytes target prot opt in out source > destination > 1 60 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0 > state NEW CONNMARK set 0x1 > 2360 3256K FIREWALL_10.147.44.63 all -- * * 0.0.0.0/0 > 10.147.44.63 > 1388 77740 CONNMARK all -- eth2 * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED CONNMARK restore > 27 1620 ACL_OUTBOUND_eth2 all -- eth2 * 10.0.1.0/24 > !10.0.1.1 state NEW > 0 0 MARK all -- eth1 * 0.0.0.0/0 > 10.147.44.63 state NEW MARK set 0x1 > 0 0 CONNMARK all -- eth1 * 0.0.0.0/0 > 10.147.44.63 state NEW CONNMARK save > 0 0 MARK all -- eth0 * 10.0.1.188 0.0.0.0/0 > state NEW MARK set 0x1 > 0 0 CONNMARK all -- eth0 * 10.0.1.188 0.0.0.0/0 > state NEW CONNMARK save > Chain INPUT (policy ACCEPT 3048 packets, 345K bytes) > pkts bytes target prot opt in out source > destination > Chain FORWARD (policy ACCEPT 21 packets, 10108 bytes) > pkts bytes target prot opt in out source > destination > 3774 3335K VPN_STATS_eth1 all -- * * 0.0.0.0/0 > 0.0.0.0/0 > Chain OUTPUT (policy ACCEPT 2402 packets, 384K bytes) > pkts bytes target prot opt in out source > destination > 0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 > udp dpt:68 CHECKSUM fill > Chain POSTROUTING (policy ACCEPT 2423 packets, 395K bytes) > pkts bytes target prot opt in out source > destination > Chain ACL_OUTBOUND_eth2 (1 references) > pkts bytes target prot opt in out source > destination > 27 1620 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 > > 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 > > Chain FIREWALL_10.147.44.63 (1 references) > pkts bytes target prot opt in out source > destination > 2359 3256K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 > state RELATED,ESTABLISHED > 1 60 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 > > Chain VPN_STATS_eth1 (1 references) > pkts bytes target prot opt in out source > destination > 0 0 all -- * eth1 0.0.0.0/0 0.0.0.0/0 > mark match 0x525 > 0 0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0 > mark match 0x524 > (iv) Tried to configure PF rule but that as well failed but with a different > error. > here is the snippet from mgmt server log. > 2013-05-30 11:30:35,264 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-495:10.147.40.12) Use router's private IP for SSH control. IP : > 10.147.40.62 > 2013-05-30 11:30:36,548 ERROR [utils.ssh.SshHelper] > (DirectAgent-495:10.147.40.12) SSH execution of command > /opt/cloud/bin/vpc_portforwarding.sh -A -P tcp -l 10.147.44.64 -p 22-22 -r > 10.0.1.188 -d 22-22 has an error status code in return. result output: > 2013-05-30 11:30:36,555 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-495:null) Seq 1-1416495677: Response Received: > 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] > (DirectAgent-495:null) Seq 1-1416495677: Processing: { Ans: , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 0, > [{"routing.SetPortForwardingRulesAnswer":{"results":["Failed"],"result":false,"wait":0}}] > } > 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] > (Job-Executor-26:job-23) Seq 1-1416495677: Received: { Ans: , MgmtId: > 7280707764394, via: 1, Ver: v1, Flags: 0, { SetPortForwardingRulesAnswer } } > 2013-05-30 11:30:36,556 WARN [network.rules.RulesManagerImpl] > (Job-Executor-26:job-23) Failed to apply port forwarding rules for ip due to > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is > unreachable: Unable to apply firewall rules on router > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3739) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3567) > at > com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:787) > at > com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:565) > at > com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2913) > at > com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:509) > at > com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:889) > at > com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:1072) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.firewall.CreatePortForwardingRuleCmd.execute(CreatePortForwardingRuleCmd.java:184) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > (v) Tried the following iptables command manually and found an error with > syntax > root@r-3-VM:~# sudo iptables -t nat -A PREROUTING --proto tcp -d 10.147.44.64 > --destination-port 22-22 -j DNAT --to-destination 10.0.1.188:22-22 > iptables v1.4.14: invalid port/service `22-22' specified > Try `iptables -h' or 'iptables --help' for more information. > Attaching all the mgmt server logs to the bug. > Here is the system vm template version that I used for testing. > root@r-3-VM:~# cat /etc/cloudstack-release > Cloudstack Release 4.2.0 Tue Apr 16 04:09:58 UTC 2013 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira