Public bug reported:
=====
my env
=====
controller +network node(dvr_snat) + 2 compute nodes(dvr)
DVR: enable DVR when using devstack to deploy this env
FWaaS: manually git clone neutron-fwaas and to configure, using iptables as
driver
====
steps
====
1) create net, subnet, boot VM-1 on CN-1, VM-2 on CN-2, create router, and
attach subnet onto router.
2) create external network, set as router gateway net, create 2 floating IPs
and associate to two VMs.
3) confirm DVR FIP works: fip ns created, iptable rules updated in qrouter ns,
two VMs are pingable by floating IP.
floating IP like: 192.168.0.4 and 192.168.0.5
4) create firewall rules, firewall policy and create firewall on router.
firewall rule like:
fw-r1: ICMP, source: 192.168.0.184/29(none), dest: 192.168.0.0/28(none),
allow
fw-r2: ICMP, source: 192.168.0.0/28(none), dest: 192.168.0.184/29(none),
allow
5) confirm firewall rules updated in qrouter ns.
6) on host who has IP like 192.168.0.190, try to ping floating IPs mentioned in
step 3.
expected: floating IPs should be pingable (for IP 192.168.0.190 is in
192.168.0.184/29, and two firewall rules allows)
observed: no response, "100% packet loss" from ping command. floating IP fail
to ping.
========
more details
========
----------------------------
firewall iptable rules:
----------------------------
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-iv4322a9b15
-A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-ov4322a9b15
-A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-fwaas-defau -j DROP
-A neutron-l3-agent-iv4322a9b15 -m state --state INVALID -j DROP
-A neutron-l3-agent-iv4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp
-j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp
-j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -m state --state INVALID -j DROP
-A neutron-l3-agent-ov4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp
-j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp
-j ACCEPT
-----------------------------------
DVR FIP nat iptable rules:
-----------------------------------
1) for 192.168.0.4:
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
-A neutron-l3-agent-POSTROUTING ! -i rfp-4bf3186c-d ! -o rfp-4bf3186c-d -m
conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
--dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination
20.0.1.7
-A neutron-l3-agent-float-snat -s 20.0.1.7/32 -j SNAT --to-source 192.168.0.4
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on
outgoing traffic." -j neutron-l3-agent-snat
2) for 192.168.0.5:
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.0.5/32 -j DNAT --to-destination 20.0.1.6
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
--dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.0.5/32 ! -i qr-+ -j DNAT
--to-destination 20.0.1.6
-A neutron-l3-agent-float-snat -s 20.0.1.6/32 -j SNAT --to-source 192.168.0.5
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on
outgoing traffic." -j neutron-l3-agent-snat
----------------------------------------------------------------------
tcpdump result: (192.168.0.190 ping 192.168.0.4)
----------------------------------------------------------------------
1) on fg in fip ns, ingress traffic caught:
fa:16:3e:b3:3e:8c > fa:16:3e:9d:ea:ed, ethertype IPv4 (0x0800), length 98:
192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 31, length 64
and fg:
40: fg-59c9ce49-3a: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN group default
link/ether fa:16:3e:9d:ea:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.0.133/24 brd 192.168.0.255 scope global fg-59c9ce49-3a
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe9d:eaed/64 scope link
valid_lft forever preferred_lft forever
2) on fpr in fip ns, ingress traffic caught:
a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98:
192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 159, length 64
and fpr:
13: fpr-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
link/ether a2:c4:79:bf:c0:1a brd ff:ff:ff:ff:ff:ff
inet 169.254.31.143/31 scope global fpr-4bf3186c-d
valid_lft forever preferred_lft forever
inet6 fe80::a0c4:79ff:febf:c01a/64 scope link
valid_lft forever preferred_lft forever
and rfp:
10: rfp-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
link/ether aa:ab:39:2f:ac:df brd ff:ff:ff:ff:ff:ff
inet 169.254.31.142/31 scope global rfp-4bf3186c-d
valid_lft forever preferred_lft forever
inet 192.168.0.4/32 brd 192.168.0.4 scope global rfp-4bf3186c-d
valid_lft forever preferred_lft forever
inet6 fe80::a8ab:39ff:fe2f:acdf/64 scope link
valid_lft forever preferred_lft forever
3) on rfp in qrouter ns, ingress traffic caught:
a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98:
192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 372, length 64
4) on qr in qrouter ns, no traffic caught :(
========
idea/discuss
========
in my understanding, we have DNAT rules in PREROUTING chain, like:
-A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination
20.0.1.7
will change ingress traffic destination IP from 192.168.0.4(firewall know and
allow) to 20.0.1.7(firewall dont know and deny by neutron-l3-agent-fwaas-defau).
So before firewall rules in FORWARD related chains like:
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp
-j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp
-j ACCEPT
have chance to accept, the ingress traffic has already been changed, and will
go to neutron-l3-agent-fwaas-defau chain and be dropped.
an idea is try to move firewall rules into fip ns(but this will raise
other potential problems, for firewall rules will no longer be isolated
by namespace), I tested this, it can work.
** Affects: neutron
Importance: Undecided
Status: New
** Tags: l3-dvr-backlog
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1513574
Title:
firewall rules on DVR FIP fails to work for ingress traffic
Status in neutron:
New
Bug description:
=====
my env
=====
controller +network node(dvr_snat) + 2 compute nodes(dvr)
DVR: enable DVR when using devstack to deploy this env
FWaaS: manually git clone neutron-fwaas and to configure, using iptables as
driver
====
steps
====
1) create net, subnet, boot VM-1 on CN-1, VM-2 on CN-2, create router, and
attach subnet onto router.
2) create external network, set as router gateway net, create 2 floating IPs
and associate to two VMs.
3) confirm DVR FIP works: fip ns created, iptable rules updated in qrouter
ns, two VMs are pingable by floating IP.
floating IP like: 192.168.0.4 and 192.168.0.5
4) create firewall rules, firewall policy and create firewall on router.
firewall rule like:
fw-r1: ICMP, source: 192.168.0.184/29(none), dest: 192.168.0.0/28(none),
allow
fw-r2: ICMP, source: 192.168.0.0/28(none), dest: 192.168.0.184/29(none),
allow
5) confirm firewall rules updated in qrouter ns.
6) on host who has IP like 192.168.0.190, try to ping floating IPs mentioned
in step 3.
expected: floating IPs should be pingable (for IP 192.168.0.190 is in
192.168.0.184/29, and two firewall rules allows)
observed: no response, "100% packet loss" from ping command. floating IP fail
to ping.
========
more details
========
----------------------------
firewall iptable rules:
----------------------------
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-iv4322a9b15
-A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-ov4322a9b15
-A neutron-l3-agent-FORWARD -o rfp-+ -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-FORWARD -i rfp-+ -j neutron-l3-agent-fwaas-defau
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-fwaas-defau -j DROP
-A neutron-l3-agent-iv4322a9b15 -m state --state INVALID -j DROP
-A neutron-l3-agent-iv4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp
-j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp
-j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -m state --state INVALID -j DROP
-A neutron-l3-agent-ov4322a9b15 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp
-j ACCEPT
-A neutron-l3-agent-ov4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp
-j ACCEPT
-----------------------------------
DVR FIP nat iptable rules:
-----------------------------------
1) for 192.168.0.4:
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.0.4/32 -j DNAT --to-destination 20.0.1.7
-A neutron-l3-agent-POSTROUTING ! -i rfp-4bf3186c-d ! -o rfp-4bf3186c-d -m
conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
--dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination
20.0.1.7
-A neutron-l3-agent-float-snat -s 20.0.1.7/32 -j SNAT --to-source 192.168.0.4
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on
outgoing traffic." -j neutron-l3-agent-snat
2) for 192.168.0.5:
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 192.168.0.5/32 -j DNAT --to-destination 20.0.1.6
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp
--dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 192.168.0.5/32 ! -i qr-+ -j DNAT
--to-destination 20.0.1.6
-A neutron-l3-agent-float-snat -s 20.0.1.6/32 -j SNAT --to-source 192.168.0.5
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on
outgoing traffic." -j neutron-l3-agent-snat
----------------------------------------------------------------------
tcpdump result: (192.168.0.190 ping 192.168.0.4)
----------------------------------------------------------------------
1) on fg in fip ns, ingress traffic caught:
fa:16:3e:b3:3e:8c > fa:16:3e:9d:ea:ed, ethertype IPv4 (0x0800), length 98:
192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 31, length 64
and fg:
40: fg-59c9ce49-3a: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UNKNOWN group default
link/ether fa:16:3e:9d:ea:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.0.133/24 brd 192.168.0.255 scope global fg-59c9ce49-3a
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe9d:eaed/64 scope link
valid_lft forever preferred_lft forever
2) on fpr in fip ns, ingress traffic caught:
a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98:
192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 159, length 64
and fpr:
13: fpr-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP group default qlen 1000
link/ether a2:c4:79:bf:c0:1a brd ff:ff:ff:ff:ff:ff
inet 169.254.31.143/31 scope global fpr-4bf3186c-d
valid_lft forever preferred_lft forever
inet6 fe80::a0c4:79ff:febf:c01a/64 scope link
valid_lft forever preferred_lft forever
and rfp:
10: rfp-4bf3186c-d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP group default qlen 1000
link/ether aa:ab:39:2f:ac:df brd ff:ff:ff:ff:ff:ff
inet 169.254.31.142/31 scope global rfp-4bf3186c-d
valid_lft forever preferred_lft forever
inet 192.168.0.4/32 brd 192.168.0.4 scope global rfp-4bf3186c-d
valid_lft forever preferred_lft forever
inet6 fe80::a8ab:39ff:fe2f:acdf/64 scope link
valid_lft forever preferred_lft forever
3) on rfp in qrouter ns, ingress traffic caught:
a2:c4:79:bf:c0:1a > aa:ab:39:2f:ac:df, ethertype IPv4 (0x0800), length 98:
192.168.0.190 > 192.168.0.4: ICMP echo request, id 28356, seq 372, length 64
4) on qr in qrouter ns, no traffic caught :(
========
idea/discuss
========
in my understanding, we have DNAT rules in PREROUTING chain, like:
-A neutron-l3-agent-PREROUTING -d 192.168.0.4/32 -j DNAT --to-destination
20.0.1.7
will change ingress traffic destination IP from 192.168.0.4(firewall know and
allow) to 20.0.1.7(firewall dont know and deny by neutron-l3-agent-fwaas-defau).
So before firewall rules in FORWARD related chains like:
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.0/28 -d 192.168.0.184/29 -p icmp
-j ACCEPT
-A neutron-l3-agent-iv4322a9b15 -s 192.168.0.184/29 -d 192.168.0.0/28 -p icmp
-j ACCEPT
have chance to accept, the ingress traffic has already been changed, and will
go to neutron-l3-agent-fwaas-defau chain and be dropped.
an idea is try to move firewall rules into fip ns(but this will raise
other potential problems, for firewall rules will no longer be
isolated by namespace), I tested this, it can work.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1513574/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
