[
https://issues.apache.org/jira/browse/CLOUDSTACK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayapal Reddy resolved CLOUDSTACK-2761.
---------------------------------------
Resolution: Fixed
> [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
> ---------------------------------------------------------
>
> Key: CLOUDSTACK-2761
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2761
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Network Controller
> Affects Versions: 4.2.0
> Environment: commit # 8d1189c2ae87216bc1c4a1443f75e9a8629abdc2
> Reporter: venkata swamybabu budumuru
> Assignee: Jayapal Reddy
> Priority: Blocker
> Fix For: 4.2.0
>
> Attachments: logs.tgz
>
>
> Steps to reproduce:
> 1. Have latest CloudStack build with at least 1 advanced zone
> 2. Have at least 1 VMware cluster with 1 host.
> 3. Create a VPC with at least 1 Tier
> 4. deploy a VM
> 5. apply an ACL to allow all
> 6. acquire at least 1 IP
> 7. create PF/staticNAT on the above created IP to the VM created in step (4)
> Observations:
> (i) It failed to create staticNAT with the following error
> 2013-05-30 08:05:20,203 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-38:null) Seq 1-1416495168: Response Received:
> 2013-05-30 08:05:20,206 DEBUG [agent.transport.Request] (DirectAgent-38:null)
> Seq 1-1416495168: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver:
> v1, Flags: 0, [{"routing.IpAssocAnswer":{"results":["10.147.44.63 -
> success"],"result":true,"wait":0}}] }
> 2013-05-30 08:05:20,207 DEBUG [agent.transport.Request]
> (catalina-exec-19:null) Seq 1-1416495168: Received: { Ans: , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } }
> 2013-05-30 08:05:20,211 INFO [cloud.network.NetworkManagerImpl]
> (catalina-exec-19:null) Let VpcVirtualRouter handle StaticNat in network 204
> 2013-05-30 08:05:20,223 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-19:null)
> Applying static nat rules in network Ntwk[204|Guest|11]
> 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request]
> (catalina-exec-19:null) Seq 1-1416495169: Sending { Cmd , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 100001,
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
> }
> 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request]
> (catalina-exec-19:null) Seq 1-1416495169: Executing: { Cmd , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 100001,
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
> }
> 2013-05-30 08:05:20,289 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-16:null) Seq 1-1416495169: Executing request
> 2013-05-30 08:05:20,290 INFO [vmware.resource.VmwareResource]
> (DirectAgent-16:10.147.40.12) Executing resource SetFirewallRuleCommand:
> {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
> 2013-05-30 08:05:20,290 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-16:10.147.40.12) Use router's private IP for SSH control. IP :
> 10.147.40.62
> 2013-05-30 08:05:21,746 ERROR [utils.ssh.SshHelper]
> (DirectAgent-16:10.147.40.12) SSH execution of command /root/firewall.sh -A
> -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G has an error status code in return.
> result output: Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.14: option "--set-mark" requires an argument
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: No chain/target/match by that name.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `eth0'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> 2013-05-30 08:05:21,790 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-16:10.147.40.12) Executing script on domain router 10.147.40.62:
> /root/firewall.sh -A -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G
> (ii) After changing the firewall.sh as mentioned below, it went fine.
> get_dev_list() {
> ip link show | grep -e eth[2-9] | awk -F ":" '{print $2}'
> ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
> }
> Changed the above with the following to include eth1 device as well.
> get_dev_list() {
> ip link show | grep -e eth[1-9] | awk -F ":" '{print $2}'
> ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
> }
>
> 2013-05-30 08:32:52,492 INFO [cloud.network.NetworkManagerImpl]
> (catalina-exec-3:null) Let VpcVirtualRouter handle StaticNat in network 204
> 2013-05-30 08:32:52,506 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-3:null)
> Applying static nat rules in network Ntwk[204|Guest|11]
> 2013-05-30 08:32:52,523 DEBUG [agent.transport.Request]
> (catalina-exec-3:null) Seq 1-1416495239: Sending { Cmd , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 100001,
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
> }
> 2013-05-30 08:32:52,524 DEBUG [agent.transport.Request]
> (catalina-exec-3:null) Seq 1-1416495239: Executing: { Cmd , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 100001,
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
> }
> 2013-05-30 08:32:52,524 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-12:null) Seq 1-1416495239: Executing request
> 2013-05-30 08:32:52,525 INFO [vmware.resource.VmwareResource]
> (DirectAgent-12:10.147.40.12) Executing resource SetFirewallRuleCommand:
> {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
> 2013-05-30 08:32:52,529 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-12:10.147.40.12) Use router's private IP for SSH control. IP :
> 10.147.40.62
> 2013-05-30 08:32:53,937 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-12:10.147.40.12) Executing script on domain router 10.147.40.62:
> /root/firewall.sh -A -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G
> 2013-05-30 08:32:53,938 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-12:null) Seq 1-1416495239: Response Received:
> 2013-05-30 08:32:53,938 DEBUG [agent.transport.Request] (DirectAgent-12:null)
> Seq 1-1416495239: Processing: { Ans: , MgmtId: 7280707764394, via: 1, Ver:
> v1, Flags: 0,
> [{"routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}]
> }
> (iii) Tried to access the staticNat rule but, it failed because, it created a
> firewall rule with DROP policy which is not required in case of staticNAt.
> Ideally instead of firewall.sh, it should call "vlc_staticnat.sh" script to
> configure staticnat.
> Here is the snippet of iptable output.
> root@r-3-VM:~# iptables -L -nv -t mangle
> Chain PREROUTING (policy ACCEPT 3060 packets, 346K bytes)
> pkts bytes target prot opt in out source
> destination
> 1 60 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0
> state NEW CONNMARK set 0x1
> 2360 3256K FIREWALL_10.147.44.63 all -- * * 0.0.0.0/0
> 10.147.44.63
> 1388 77740 CONNMARK all -- eth2 * 0.0.0.0/0 0.0.0.0/0
> state RELATED,ESTABLISHED CONNMARK restore
> 27 1620 ACL_OUTBOUND_eth2 all -- eth2 * 10.0.1.0/24
> !10.0.1.1 state NEW
> 0 0 MARK all -- eth1 * 0.0.0.0/0
> 10.147.44.63 state NEW MARK set 0x1
> 0 0 CONNMARK all -- eth1 * 0.0.0.0/0
> 10.147.44.63 state NEW CONNMARK save
> 0 0 MARK all -- eth0 * 10.0.1.188 0.0.0.0/0
> state NEW MARK set 0x1
> 0 0 CONNMARK all -- eth0 * 10.0.1.188 0.0.0.0/0
> state NEW CONNMARK save
> Chain INPUT (policy ACCEPT 3048 packets, 345K bytes)
> pkts bytes target prot opt in out source
> destination
> Chain FORWARD (policy ACCEPT 21 packets, 10108 bytes)
> pkts bytes target prot opt in out source
> destination
> 3774 3335K VPN_STATS_eth1 all -- * * 0.0.0.0/0
> 0.0.0.0/0
> Chain OUTPUT (policy ACCEPT 2402 packets, 384K bytes)
> pkts bytes target prot opt in out source
> destination
> 0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0
> udp dpt:68 CHECKSUM fill
> Chain POSTROUTING (policy ACCEPT 2423 packets, 395K bytes)
> pkts bytes target prot opt in out source
> destination
> Chain ACL_OUTBOUND_eth2 (1 references)
> pkts bytes target prot opt in out source
> destination
> 27 1620 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
>
> 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
>
> Chain FIREWALL_10.147.44.63 (1 references)
> pkts bytes target prot opt in out source
> destination
> 2359 3256K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
> state RELATED,ESTABLISHED
> 1 60 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
>
> Chain VPN_STATS_eth1 (1 references)
> pkts bytes target prot opt in out source
> destination
> 0 0 all -- * eth1 0.0.0.0/0 0.0.0.0/0
> mark match 0x525
> 0 0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0
> mark match 0x524
> (iv) Tried to configure PF rule but that as well failed but with a different
> error.
> here is the snippet from mgmt server log.
> 2013-05-30 11:30:35,264 DEBUG [vmware.resource.VmwareResource]
> (DirectAgent-495:10.147.40.12) Use router's private IP for SSH control. IP :
> 10.147.40.62
> 2013-05-30 11:30:36,548 ERROR [utils.ssh.SshHelper]
> (DirectAgent-495:10.147.40.12) SSH execution of command
> /opt/cloud/bin/vpc_portforwarding.sh -A -P tcp -l 10.147.44.64 -p 22-22 -r
> 10.0.1.188 -d 22-22 has an error status code in return. result output:
> 2013-05-30 11:30:36,555 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-495:null) Seq 1-1416495677: Response Received:
> 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request]
> (DirectAgent-495:null) Seq 1-1416495677: Processing: { Ans: , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 0,
> [{"routing.SetPortForwardingRulesAnswer":{"results":["Failed"],"result":false,"wait":0}}]
> }
> 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request]
> (Job-Executor-26:job-23) Seq 1-1416495677: Received: { Ans: , MgmtId:
> 7280707764394, via: 1, Ver: v1, Flags: 0, { SetPortForwardingRulesAnswer } }
> 2013-05-30 11:30:36,556 WARN [network.rules.RulesManagerImpl]
> (Job-Executor-26:job-23) Failed to apply port forwarding rules for ip due to
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
> unreachable: Unable to apply firewall rules on router
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3739)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3567)
> at
> com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:787)
> at
> com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:565)
> at
> com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2913)
> at
> com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:509)
> at
> com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:889)
> at
> com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:1072)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.firewall.CreatePortForwardingRuleCmd.execute(CreatePortForwardingRuleCmd.java:184)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> (v) Tried the following iptables command manually and found an error with
> syntax
> root@r-3-VM:~# sudo iptables -t nat -A PREROUTING --proto tcp -d 10.147.44.64
> --destination-port 22-22 -j DNAT --to-destination 10.0.1.188:22-22
> iptables v1.4.14: invalid port/service `22-22' specified
> Try `iptables -h' or 'iptables --help' for more information.
> Attaching all the mgmt server logs to the bug.
> Here is the system vm template version that I used for testing.
> root@r-3-VM:~# cat /etc/cloudstack-release
> Cloudstack Release 4.2.0 Tue Apr 16 04:09:58 UTC 2013
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
