[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayapal Reddy resolved CLOUDSTACK-2761.
---------------------------------------

    Resolution: Fixed
    
> [VMware] [VPC] Failed to create PF/StaticNAT rules on VPC
> ---------------------------------------------------------
>
>                 Key: CLOUDSTACK-2761
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2761
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the 
> default.) 
>          Components: Network Controller
>    Affects Versions: 4.2.0
>         Environment: commit # 8d1189c2ae87216bc1c4a1443f75e9a8629abdc2
>            Reporter: venkata swamybabu budumuru
>            Assignee: Jayapal Reddy
>            Priority: Blocker
>             Fix For: 4.2.0
>
>         Attachments: logs.tgz
>
>
> Steps to reproduce:
> 1. Have latest CloudStack build with at least 1 advanced zone
> 2. Have at least 1 VMware cluster with 1 host.
> 3. Create a VPC with at least 1 Tier
> 4. deploy a VM
> 5. apply an ACL to allow all 
> 6. acquire at least 1 IP
> 7. create PF/staticNAT on the above created IP to the VM created in step (4) 
> Observations:
> (i) It failed to create staticNAT with the following error
> 2013-05-30 08:05:20,203 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-38:null) Seq 1-1416495168: Response Received:
> 2013-05-30 08:05:20,206 DEBUG [agent.transport.Request] (DirectAgent-38:null) 
> Seq 1-1416495168: Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: 
> v1, Flags: 0, [{"routing.IpAssocAnswer":{"results":["10.147.44.63 - 
> success"],"result":true,"wait":0}}] }
> 2013-05-30 08:05:20,207 DEBUG [agent.transport.Request] 
> (catalina-exec-19:null) Seq 1-1416495168: Received:  { Ans: , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 0, { IpAssocAnswer } }
> 2013-05-30 08:05:20,211 INFO  [cloud.network.NetworkManagerImpl] 
> (catalina-exec-19:null) Let VpcVirtualRouter handle StaticNat in network 204
> 2013-05-30 08:05:20,223 DEBUG 
> [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-19:null) 
> Applying static nat rules in network Ntwk[204|Guest|11]
> 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] 
> (catalina-exec-19:null) Seq 1-1416495169: Sending  { Cmd , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 100001, 
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
>  }
> 2013-05-30 08:05:20,288 DEBUG [agent.transport.Request] 
> (catalina-exec-19:null) Seq 1-1416495169: Executing:  { Cmd , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 100001, 
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
>  }
> 2013-05-30 08:05:20,289 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-16:null) Seq 1-1416495169: Executing request
> 2013-05-30 08:05:20,290 INFO  [vmware.resource.VmwareResource] 
> (DirectAgent-16:10.147.40.12) Executing resource SetFirewallRuleCommand: 
> {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
> 2013-05-30 08:05:20,290 DEBUG [vmware.resource.VmwareResource] 
> (DirectAgent-16:10.147.40.12) Use router's private IP for SSH control. IP : 
> 10.147.40.62
> 2013-05-30 08:05:21,746 ERROR [utils.ssh.SshHelper] 
> (DirectAgent-16:10.147.40.12) SSH execution of command /root/firewall.sh  -A  
> -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G  has an error status code in return. 
> result output: Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> iptables v1.4.14: option "--set-mark" requires an argument
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: No chain/target/match by that name.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `eth0'
> Try `iptables -h' or 'iptables --help' for more information.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> iptables: No chain/target/match by that name.
> iptables: No chain/target/match by that name.
> Bad argument `10.147.44.63'
> Try `iptables -h' or 'iptables --help' for more information.
> 2013-05-30 08:05:21,790 DEBUG [vmware.resource.VmwareResource] 
> (DirectAgent-16:10.147.40.12) Executing script on domain router 10.147.40.62: 
> /root/firewall.sh  -A  -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G
> (ii) After changing the firewall.sh as mentioned below, it went fine.
> get_dev_list() {
>   ip link show | grep -e eth[2-9] | awk -F ":" '{print $2}'
>   ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
> }
> Changed the above with the following to include eth1 device as well.
> get_dev_list() {
>   ip link show | grep -e eth[1-9] | awk -F ":" '{print $2}'
>   ip link show | grep -e eth1[0-9] | awk -F ":" '{print $2}'
> }
>  
> 2013-05-30 08:32:52,492 INFO  [cloud.network.NetworkManagerImpl] 
> (catalina-exec-3:null) Let VpcVirtualRouter handle StaticNat in network 204
> 2013-05-30 08:32:52,506 DEBUG 
> [network.router.VirtualNetworkApplianceManagerImpl] (catalina-exec-3:null) 
> Applying static nat rules in network Ntwk[204|Guest|11]
> 2013-05-30 08:32:52,523 DEBUG [agent.transport.Request] 
> (catalina-exec-3:null) Seq 1-1416495239: Sending  { Cmd , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 100001, 
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
>  }
> 2013-05-30 08:32:52,524 DEBUG [agent.transport.Request] 
> (catalina-exec-3:null) Seq 1-1416495239: Executing:  { Cmd , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 100001, 
> [{"routing.SetStaticNatRulesCommand":{"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}}]
>  }
> 2013-05-30 08:32:52,524 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-12:null) Seq 1-1416495239: Executing request
> 2013-05-30 08:32:52,525 INFO  [vmware.resource.VmwareResource] 
> (DirectAgent-12:10.147.40.12) Executing resource SetFirewallRuleCommand: 
> {"rules":[{"dstIp":"10.0.1.188","id":0,"srcIp":"10.147.44.63","revoked":false,"alreadyAdded":false,"purpose":"StaticNat","icmpType":0,"icmpCode":0}],"vpcId":1,"accessDetails":{"router.guest.ip":"10.0.1.1","zone.network.type":"Advanced","router.ip":"10.147.40.62","router.name":"r-3-VM"},"wait":0}
> 2013-05-30 08:32:52,529 DEBUG [vmware.resource.VmwareResource] 
> (DirectAgent-12:10.147.40.12) Use router's private IP for SSH control. IP : 
> 10.147.40.62
> 2013-05-30 08:32:53,937 DEBUG [vmware.resource.VmwareResource] 
> (DirectAgent-12:10.147.40.12) Executing script on domain router 10.147.40.62: 
> /root/firewall.sh  -A  -l 10.147.44.63 -r 10.0.1.188 -d 0:0 -G
> 2013-05-30 08:32:53,938 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-12:null) Seq 1-1416495239: Response Received:
> 2013-05-30 08:32:53,938 DEBUG [agent.transport.Request] (DirectAgent-12:null) 
> Seq 1-1416495239: Processing:  { Ans: , MgmtId: 7280707764394, via: 1, Ver: 
> v1, Flags: 0, 
> [{"routing.SetStaticNatRulesAnswer":{"results":[null],"result":true,"wait":0}}]
>  }
> (iii) Tried to access the staticNat rule but, it failed because, it created a 
> firewall rule with DROP policy which is not required in case of staticNAt. 
> Ideally instead of firewall.sh, it should call "vlc_staticnat.sh" script to 
> configure staticnat.
> Here is the snippet of iptable output.
> root@r-3-VM:~# iptables -L -nv -t mangle
> Chain PREROUTING (policy ACCEPT 3060 packets, 346K bytes)
>  pkts bytes target     prot opt in     out     source               
> destination         
>     1    60 CONNMARK   all  --  eth1   *       0.0.0.0/0            0.0.0.0/0 
>            state NEW CONNMARK set 0x1
>  2360 3256K FIREWALL_10.147.44.63  all  --  *      *       0.0.0.0/0          
>   10.147.44.63        
>  1388 77740 CONNMARK   all  --  eth2   *       0.0.0.0/0            0.0.0.0/0 
>            state RELATED,ESTABLISHED CONNMARK restore
>    27  1620 ACL_OUTBOUND_eth2  all  --  eth2   *       10.0.1.0/24         
> !10.0.1.1             state NEW
>     0     0 MARK       all  --  eth1   *       0.0.0.0/0            
> 10.147.44.63         state NEW MARK set 0x1
>     0     0 CONNMARK   all  --  eth1   *       0.0.0.0/0            
> 10.147.44.63         state NEW CONNMARK save
>     0     0 MARK       all  --  eth0   *       10.0.1.188           0.0.0.0/0 
>            state NEW MARK set 0x1
>     0     0 CONNMARK   all  --  eth0   *       10.0.1.188           0.0.0.0/0 
>            state NEW CONNMARK save
> Chain INPUT (policy ACCEPT 3048 packets, 345K bytes)
>  pkts bytes target     prot opt in     out     source               
> destination         
> Chain FORWARD (policy ACCEPT 21 packets, 10108 bytes)
>  pkts bytes target     prot opt in     out     source               
> destination         
>  3774 3335K VPN_STATS_eth1  all  --  *      *       0.0.0.0/0            
> 0.0.0.0/0           
> Chain OUTPUT (policy ACCEPT 2402 packets, 384K bytes)
>  pkts bytes target     prot opt in     out     source               
> destination         
>     0     0 CHECKSUM   udp  --  *      *       0.0.0.0/0            0.0.0.0/0 
>            udp dpt:68 CHECKSUM fill
> Chain POSTROUTING (policy ACCEPT 2423 packets, 395K bytes)
>  pkts bytes target     prot opt in     out     source               
> destination         
> Chain ACL_OUTBOUND_eth2 (1 references)
>  pkts bytes target     prot opt in     out     source               
> destination         
>    27  1620 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0 
>           
>     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0 
>           
> Chain FIREWALL_10.147.44.63 (1 references)
>  pkts bytes target     prot opt in     out     source               
> destination         
>  2359 3256K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0 
>            state RELATED,ESTABLISHED
>     1    60 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0 
>           
> Chain VPN_STATS_eth1 (1 references)
>  pkts bytes target     prot opt in     out     source               
> destination         
>     0     0            all  --  *      eth1    0.0.0.0/0            0.0.0.0/0 
>            mark match 0x525
>     0     0            all  --  eth1   *       0.0.0.0/0            0.0.0.0/0 
>            mark match 0x524
> (iv) Tried to configure PF rule but that as well failed but with a different 
> error.
> here is the snippet from mgmt server log.
> 2013-05-30 11:30:35,264 DEBUG [vmware.resource.VmwareResource] 
> (DirectAgent-495:10.147.40.12) Use router's private IP for SSH control. IP : 
> 10.147.40.62
> 2013-05-30 11:30:36,548 ERROR [utils.ssh.SshHelper] 
> (DirectAgent-495:10.147.40.12) SSH execution of command 
> /opt/cloud/bin/vpc_portforwarding.sh  -A -P tcp -l 10.147.44.64 -p 22-22 -r 
> 10.0.1.188 -d 22-22 has an error status code in return. result output:
> 2013-05-30 11:30:36,555 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-495:null) Seq 1-1416495677: Response Received:
> 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] 
> (DirectAgent-495:null) Seq 1-1416495677: Processing:  { Ans: , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 0, 
> [{"routing.SetPortForwardingRulesAnswer":{"results":["Failed"],"result":false,"wait":0}}]
>  }
> 2013-05-30 11:30:36,556 DEBUG [agent.transport.Request] 
> (Job-Executor-26:job-23) Seq 1-1416495677: Received:  { Ans: , MgmtId: 
> 7280707764394, via: 1, Ver: v1, Flags: 0, { SetPortForwardingRulesAnswer } }
> 2013-05-30 11:30:36,556 WARN  [network.rules.RulesManagerImpl] 
> (Job-Executor-26:job-23) Failed to apply port forwarding rules for ip due to
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is 
> unreachable: Unable to apply firewall rules on router
>         at 
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3739)
>         at 
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRules(VirtualNetworkApplianceManagerImpl.java:3567)
>         at 
> com.cloud.network.element.VirtualRouterElement.applyPFRules(VirtualRouterElement.java:787)
>         at 
> com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:565)
>         at 
> com.cloud.network.NetworkManagerImpl.applyRules(NetworkManagerImpl.java:2913)
>         at 
> com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImpl.java:509)
>         at 
> com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:889)
>         at 
> com.cloud.network.rules.RulesManagerImpl.applyPortForwardingRules(RulesManagerImpl.java:1072)
>         at 
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at 
> org.apache.cloudstack.api.command.user.firewall.CreatePortForwardingRuleCmd.execute(CreatePortForwardingRuleCmd.java:184)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
>         at 
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> (v) Tried the following iptables command manually and found an error with 
> syntax
> root@r-3-VM:~# sudo iptables -t nat -A PREROUTING --proto tcp -d 10.147.44.64 
> --destination-port 22-22 -j DNAT --to-destination 10.0.1.188:22-22
> iptables v1.4.14: invalid port/service `22-22' specified
> Try `iptables -h' or 'iptables --help' for more information.
> Attaching all the mgmt server logs to the bug.
> Here is the system vm template version that I used for testing.
> root@r-3-VM:~# cat /etc/cloudstack-release
> Cloudstack Release 4.2.0 Tue Apr 16 04:09:58 UTC 2013

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to