[
https://issues.apache.org/jira/browse/CLOUDSTACK-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayapal Reddy updated CLOUDSTACK-9940:
--------------------------------------
Labels: PVR (was: )
> Rules ( PF , Firewall )when deleted during the VR stopped state are still
> persistent on the VR iptables.
> --------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-9940
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9940
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Virtual Router
> Affects Versions: 4.10.0.0
> Reporter: DeepthiMachiraju
> Labels: PVR
> Fix For: 4.10.0.0
>
> Attachments: cloud.log
>
>
> - Create a network and configure PF , FW , LB rules on the source Nat IP.
> - Stop the VR and delete the above rules , and add new rules with different
> port numbers.
> - Start the VR and check if the above rules are configured .
> Observation :
> - Rules which are newly added during the VR stop state are configured
> properly.
> - Rules which are deleted are still retained in the respective json files and
> reflecting in the iptable rules.
> - Rules which are deleted are cleaned up from the DB and UI , but still
> persistent in iptables.
> **********************************************
> mysql> select * from port_forwarding_rules;
> +----+-------------+-----------------+-----------------+---------------+
> | id | instance_id | dest_ip_address | dest_port_start | dest_port_end |
> +----+-------------+-----------------+-----------------+---------------+
> | 38 | 4 | 172.16.1.227 | 22 | 22 |
> | 51 | 23 | 10.1.1.18 | 888 | 888 |
> +----+-------------+-----------------+-----------------+---------------+
> < forwardingrules.json >
> - 2000 port is the one which was deleted when router is in stopped state.
> - 888 port is the newly added rule when VR in stopped state .
> root@r-29-VM:/etc/cloudstack# cat forwardingrules.json
> {
> "10.147.52.21": [
> {
> "internal_ip": "10.1.1.18",
> "internal_ports": "2000:2000",
> "protocol": "tcp",
> "public_ip": "10.147.52.21",
> "public_ports": "2000:2000",
> "type": "forward"
> },
> {
> "internal_ip": "10.1.1.18",
> "internal_ports": "888:888",
> "protocol": "tcp",
> "public_ip": "10.147.52.21",
> "public_ports": "888:888",
> "type": "forward"
> }
> ],
> "id": "forwardingrules"
> ******************************************************
> Firewall Rules :
> mysql> select * from firewall_rules where network_id=209 and
> purpose='Firewall';
> +----+--------------------------------------+---------------+------------+----------+--------+----------+----------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
> | id | uuid | ip_address_id | start_port |
> end_port | state | protocol | purpose | account_id | domain_id | network_id
> | xid | created | icmp_code |
> icmp_type | related | type | vpc_id | traffic_type | display |
> +----+--------------------------------------+---------------+------------+----------+--------+----------+----------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
> | 50 | a41a75b3-ba8b-4126-b098-f52fa8151891 | 12 | 8888 |
> 8888 | Active | tcp | Firewall | 2 | 1 | 209 |
> e608b208-6e27-41c4-9163-40f3f3829929 | 2017-06-05 10:29:02 | NULL |
> NULL | NULL | User | NULL | Ingress | 1 |
> +----+--------------------------------------+---------------+------------+----------+--------+----------+----------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
> 1 row in set (0.00 sec)
> < firewallrules.json >
> - 555 port was deleted when VR in stopped state .
> - 8888 port was added when VR in stopped state
> root@r-29-VM:/etc/cloudstack# cat firewallrules.json
> {
> "0": {
> "already_added": false,
> "default_egress_policy": false,
> "id": 0,
> "protocol": "all",
> "purpose": "Firewall",
> "revoked": false,
> "source_cidr_list": [],
> "src_ip": "",
> "traffic_type": "Egress"
> },
> "49": {
> "already_added": false,
> "default_egress_policy": false,
> "id": 49,
> "protocol": "tcp",
> "purpose": "Firewall",
> "revoked": false,
> "source_cidr_list": [
> "1.1.1.1/32"
> ],
> "src_ip": "10.147.52.21",
> "src_port_range": [
> 555,
> 555
> ],
> "traffic_type": "Ingress"
> },
> "50": {
> "already_added": true,
> "default_egress_policy": false,
> "id": 50,
> "protocol": "tcp",
> "purpose": "Firewall",
> "revoked": false,
> "source_cidr_list": [
> "2.2.2.0/24"
> ],
> "src_ip": "10.147.52.21",
> "src_port_range": [
> 8888,
> 8888
> ],
> "traffic_type": "Ingress"
> },
> "id": "firewallrules"
> **************************************************************
> root@r-29-VM:/etc/cloudstack# iptables -t nat -L
> Chain PREROUTING (policy ACCEPT)
> target prot opt source destination
> DNAT tcp -- anywhere 10.147.52.21 tcp
> dpt:cisco-sccp to:10.1.1.18:2000
> DNAT tcp -- anywhere 10.147.52.21 tcp
> dpt:cisco-sccp to:10.1.1.18:2000
> MARK tcp -- anywhere 10.147.52.21 tcp
> dpt:cisco-sccp MARK set 0x2
> CONNMARK tcp -- anywhere 10.147.52.21 tcp
> dpt:cisco-sccp state NEW CONNMARK save
> DNAT tcp -- anywhere 10.147.52.21 tcp dpt:888
> to:10.1.1.18:888
> DNAT tcp -- anywhere 10.147.52.21 tcp dpt:888
> to:10.1.1.18:888
> MARK tcp -- anywhere 10.147.52.21 tcp dpt:888
> MARK set 0x2
> CONNMARK tcp -- anywhere 10.147.52.21 tcp dpt:888
> state NEW CONNMARK save
> Chain INPUT (policy ACCEPT)
> target prot opt source destination
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
> DNAT tcp -- anywhere 10.147.52.21 tcp
> dpt:cisco-sccp to:10.1.1.18:2000
> DNAT tcp -- anywhere 10.147.52.21 tcp dpt:888
> to:10.1.1.18:888
> Chain POSTROUTING (policy ACCEPT)
> target prot opt source destination
> SNAT all -- anywhere anywhere to:10.147.52.21
> SNAT tcp -- 10.1.1.0/24 v22 tcp
> dpt:cisco-sccp to:10.1.1.1
> SNAT tcp -- 10.1.1.0/24 v22 tcp dpt:888
> to:10.1.1.1
> ***********************************************************
> 2017-06-05 06:33:12,633 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-192:ctx-0deed038) (logid:e11c0ab3) VR Config file
> VR-e16a066b-d50e-48d2-9ba2-e582afd185e5.cfg got created in VR, ip
> 169.254.1.97 with content
> #Apache CloudStack Virtual Router Config File
> <version>
> 1.0
> </version>
> <file>
> /var/cache/cloud/ip_associations.json
> {"ip_address":[{"public_ip":"10.147.52.21","source_nat":true,"add":true,"one_to_one_nat":false,"first_i_p":true,"gateway":"10.147.52.1","netmask":"255.255.255.0","vif_mac_address":"06:45:88:00:00:17","nic_dev_id":2,"new_nic":false,"nw_type":"Public"}],"type":"ips"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py ip_associations.json
> </script>
> <file>
> /var/cache/cloud/firewall_rules.json
> {"rules":[{"id":50,"src_ip":"10.147.52.21","protocol":"tcp","src_port_range":[8888,8888],"revoked":false,"already_added":true,"source_cidr_list":["2.2.2.0/24"],"purpose":"Firewall","traffic_type":"Ingress","default_egress_policy":false}],"type":"firewallrules"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py firewall_rules.json
> </script>
> <file>
> /var/cache/cloud/forwarding_rules.json
> {"rules":[{"revoke":false,"protocol":"tcp","source_ip_address":"10.147.52.21","source_port_range":"888:888","destination_ip_address":"10.1.1.18","destination_port_range":"888:888"}],"type":"forwardrules"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py forwarding_rules.json
> </script>
> <file>
> /var/cache/cloud/load_balancer.json
> {"rules":[{"configuration":["global","\tlog 127.0.0.1:3914 local0
> warning","\tmaxconn 4096","\tmaxpipes 1024","\tchroot
> /var/lib/haproxy","\tuser haproxy","\tgroup haproxy","\tdaemon","\t
> ","defaults","\tlog global","\tmode tcp","\toption
> dontlognull","\tretries 3","\toption redispatch","\toption
> forwardfor","\toption forceclose","\ttimeout connect 5000","\ttimeout
> client 50000","\ttimeout server 50000","\nlisten stats_on_public
> 10.147.52.21:8081\n\tmode http\n\toption httpclose\n\tstats enable\n\tstats
> uri /admin?stats\n\tstats realm Haproxy\\ Statistics\n\tstats auth
> admin1:AdMiN123\n","\t ","listen 10_147_52_21-666
> 10.147.52.21:666","\tbalance roundrobin","\tserver 10_147_52_21-666_0
> 10.1.1.18:666 check","\t ","\t
> "],"tmp_cfg_file_path":"/etc/haproxy/","tmp_cfg_file_name":"haproxy.cfg.new.1496658791133","add_rules":["10.147.52.21:666:"],"remove_rules":[],"stat_rules":["10.147.52.21:8081:0/0:,"],"router_ip":"169.254.1.97"}],"type":"loadbalancer"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py load_balancer.json
> </script>
> <file>
> /var/cache/cloud/monitor_service.json
> {"config":"[dhcp]:processname=dnsmasq:servicename=dnsmasq:pidfile=/var/run/dnsmasq/dnsmasq.pid:,[loadbalancing]:processname=haproxy:servicename=haproxy:pidfile=/var/run/haproxy.pid:,[ssh]:processname=sshd:servicename=ssh:pidfile=/var/run/sshd.pid:,[webserver]:processname=apache2:servicename=apache2:pidfile=/var/run/apache2.pid:,","type":"monitorservice"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py monitor_service.json
> </script>
> <file>
> /var/cache/cloud/vm_dhcp_entry.json
> {"host_name":"v11","mac_address":"02:00:2b:1f:00:01","ipv4_adress":"10.1.1.101","ipv6_duid":"00:03:00:01:02:00:2b:1f:00:01","dns_adresses":"10.1.1.1","default_gateway":"10.1.1.1","default_entry":true,"type":"dhcpentry"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py vm_dhcp_entry.json
> </script>
> <file>
> /var/cache/cloud/vm_dhcp_entry.json
> {"host_name":"v22","mac_address":"02:00:5f:59:00:04","ipv4_adress":"10.1.1.18","ipv6_duid":"00:03:00:01:02:00:5f:59:00:04","dns_adresses":"10.1.1.1","default_gateway":"10.1.1.1","default_entry":true,"type":"dhcpentry"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py vm_dhcp_entry.json
> </script>
> <file>
> /var/cache/cloud/vm_metadata.json
> {"vm_ip_address":"10.1.1.101","vm_metadata":[["userdata","user-data",null],["metadata","service-offering","Small
>
> Instance"],["metadata","availability-zone","z1"],["metadata","local-ipv4","10.1.1.101"],["metadata","local-hostname","v11"],["metadata","public-ipv4","10.147.52.21"],["metadata","public-hostname","10.147.52.21"],["metadata","instance-id","d09cf9fe-cddc-4f8a-952f-5dadede3ab91"],["metadata","vm-id","d09cf9fe-cddc-4f8a-952f-5dadede3ab91"],["metadata","public-keys",null],["metadata","cloud-identifier","CloudStack-{5a63cc81-c2c7-46ac-a959-bff4d5b12f19}"]],"type":"vmdata"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py vm_metadata.json
> </script>
> <file>
> /var/cache/cloud/vm_metadata.json
> {"vm_ip_address":"10.1.1.18","vm_metadata":[["userdata","user-data",null],["metadata","service-offering","Small
>
> Instance"],["metadata","availability-zone","z1"],["metadata","local-ipv4","10.1.1.18"],["metadata","local-hostname","v22"],["metadata","public-ipv4","10.147.52.21"],["metadata","public-hostname","10.147.52.21"],["metadata","instance-id","7f524b97-6e2f-4b19-b40e-6da7f59f3f2e"],["metadata","vm-id","7f524b97-6e2f-4b19-b40e-6da7f59f3f2e"],["metadata","public-keys",null],["metadata","cloud-identifier","CloudStack-{5a63cc81-c2c7-46ac-a959-bff4d5b12f19}"]],"type":"vmdata"}
> </file>
> <script>
> /opt/cloud/bin/update_config.py vm_metadata.json
> </script>
> 2017-06-05 06:33:12,634 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-192:ctx-0deed038) (logid:e11c0ab3) Executing command in VR:
> /opt/cloud/bin/router_proxy.sh vr_cfg.sh 169.254.1.97 -c
> /var/cache/cloud/VR-e16a066b-d50e-48d2-9ba2-e582afd185e5.cfg
> 2017-06-05 06:33:13,059 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-2:ctx-0bd038a0) (logid:d85c5105) ===START=== 10.233.89.32 --
> GET
> command=queryAsyncJobResult&jobId=e11c0ab3-55fb-4a30-a0fd-dc550b1b45f5&response=json&_=1496658792996
> 2017-06-05 06:33:13,142 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-2:ctx-0bd038a0 ctx-7be6be07) (logid:d85c5105) ===END===
> 10.233.89.32 -- GET
> command=queryAsyncJobResult&jobId=e11c0ab3-55fb-4a30-a0fd-dc550b1b45f5&response=json&_=1496658792996
> 2017-06-05 06:33:13,225 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-6:null) (logid:) SeqA 2-144522: Processing Seq
> 2-144522: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2017-06-05 06:33:13,231 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-6:null) (logid:) SeqA 2-144522: Sending Seq 2-144522:
> { Ans: , MgmtId: 6760647622781, via: 2, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2017-06-05 06:33:14,554 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-6:ctx-ca3f95b2) (logid:27ed7e15) VmStatsCollector is
> running...
> 2017-06-05 06:33:14,599 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-13:ctx-42ce21d1) (logid:ee6b66c9) Seq 1-2918895508489535017:
> Executing request
> *******************************************************************************
> Attached cloud.log
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
