Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Daniel, hope you are good, had peaceful Christmas time, entering yet better NY 2024 hope so... sorry for overlooking this, even wanted to respond early December, then got delayed again.. Now I do so as its still interesting to me! 1) yes, my sole quick method was "ip nei" command to confirm the ARP passthrough 2) no firewall at all, plain Debian installation 3) you will not believe --> but before Xmas and now, it all works and MAC is passed e2e. That's so pitty. Only change I made was my underlay change of vSwitch uplink to another port... because I re-considered my overall lab setup, yet it hardly could improve this as the external MAC made it to external (VLAN) iface of the bridge, before/. Anyhow, possibly I understand the "bridge fbd" only shows learned MACs on given interface (my VLAN199) and is not supposed to attribute it to all others all way up to NS, like I attempted to guess.. Finally, either this of MACVLAN setup (where I found this), I have new finding which I don’t like as it creates a hell of duplicate traffic into network. The problem is, that either VETH or MACVLAN-configured IP host's VM duplicates incoming packets on its receiving port, connected to vSphere vSwitch, which in turn just dully floods it to uplinks, where my Wireshark sniffer sees it. This is how I discovered that. I prepared this diagram for you to see and tell. https://docs.google.com/document/d/1mNkZswDSG_OjLnsgXJvIX2tUGSEebcZf720eS29eFCA/edit?usp=sharing BR, all the best wishes in NY2024! Peter -Original Message- From: Daniel Gröber Sent: nedeľa 3. decembra 2023 11:09 To: GASPAROVIC Peter OBS/MKT Cc: 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Peter, > So now we move to VLAN level? Yeah, but I'm still waiting for the answers to my questions from two emails ago: > I'd be happy to still track this bug down but I need you to > investigate the behaviour in your environment. If you've torn down the > lab already we can also just call it quits. > > If you do want to continue some questions are still pending, see quoted below. > > > On Mon, Oct 30, 2023 at 07:25:38PM +, peter.gasparo...@orange.com wrote: > > > 1) As was reported, foreign external world MAC@ does not pass into > > > network namespace, just external border point "vlan199" > > > > How did you check this? > > > > > 2) now collecting data for you, honestly I don’t see external mac > > > address on "inet-br" object, so my previous statement was incorrect.. > > > {ossibly I might mixed this up with another "labinet-br" (working > > > in its limited > > > scope) which is IP-defined, while "inet-br" in question is not. > > > > > > 3) so question is, if the MACs learnt via vlan199 are supposed to > > > be paired (displayed) with "inet-br" object and all way up into NS > > > > I want to make sure we're on the same page, how do you check if the > > MAC is reaching into the NS? I assume using `ip neigh`? I'd have a > > look at tcpdump this will tell you whether ARP is even reaching the NS or > > not. > > > > Something simple like > > > > $ tcpdump -enli $IFACE 'arp or icmp' > > > > optionally you can filter by MAC (`ether host` in pcap-filter speak): > > > > $ tcpdump -enli $IFACE ('arp or icmp) and ether host > > aa:00:00:00:00:01 > > > > Oh and one last thing: please double and tripple check that a firewall > > isn't interfering. --Daniel Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Peter, > So now we move to VLAN level? Yeah, but I'm still waiting for the answers to my questions from two emails ago: > I'd be happy to still track this bug down but I need you to investigate > the behaviour in your environment. If you've torn down the lab already we > can also just call it quits. > > If you do want to continue some questions are still pending, see quoted below. > > > On Mon, Oct 30, 2023 at 07:25:38PM +, peter.gasparo...@orange.com wrote: > > > 1) As was reported, foreign external world MAC@ does not pass into > > > network namespace, just external border point "vlan199" > > > > How did you check this? > > > > > 2) now collecting data for you, honestly I don’t see external mac > > > address on "inet-br" object, so my previous statement was incorrect.. > > > {ossibly I might mixed this up with another "labinet-br" (working in > > > its limited > > > scope) which is IP-defined, while "inet-br" in question is not. > > > > > > 3) so question is, if the MACs learnt via vlan199 are supposed to be > > > paired (displayed) with "inet-br" object and all way up into NS > > > > I want to make sure we're on the same page, how do you check if the MAC > > is reaching into the NS? I assume using `ip neigh`? I'd have a look at > > tcpdump this will tell you whether ARP is even reaching the NS or not. > > > > Something simple like > > > > $ tcpdump -enli $IFACE 'arp or icmp' > > > > optionally you can filter by MAC (`ether host` in pcap-filter speak): > > > > $ tcpdump -enli $IFACE ('arp or icmp) and ether host > > aa:00:00:00:00:01 > > > > Oh and one last thing: please double and tripple check that a firewall > > isn't interfering. --Daniel signature.asc Description: PGP signature
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Daniel, of course we can steadily move on, no problem. So now we move to VLAN level? BR Peter -Original Message- From: Daniel Gröber Sent: sobota 18. novembra 2023 3:43 To: GASPAROVIC Peter OBS/MKT Cc: 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Peter, On Mon, Nov 13, 2023 at 09:40:46AM +, peter.gasparo...@orange.com wrote: > In the meantime, I was stubborn to find a solution to what I need in > order to progress and MACVLAN tech actually delivered it (private mode > enough), I used to love macvlan too but now I do L3 instead ;P > something newer than VETH tech what I could read about, and it's just > perfect avoiding bridge itself. So no problem to cooperate on this > fix, I will be glad, just it can get lower priority on your side if > you even attributed it some I'd be happy to still track this bug down but I need you to investigate the behaviour in your environment. If you've torn down the lab already we can also just call it quits. If you do want to continue some questions are still pending, see quoted below. > On Mon, Oct 30, 2023 at 07:25:38PM +, peter.gasparo...@orange.com wrote: > > 1) As was reported, foreign external world MAC@ does not pass into > > network namespace, just external border point "vlan199" > > How did you check this? > > > 2) now collecting data for you, honestly I don’t see external mac > > address on "inet-br" object, so my previous statement was incorrect.. > > {ossibly I might mixed this up with another "labinet-br" (working in > > its limited > > scope) which is IP-defined, while "inet-br" in question is not. > > > > 3) so question is, if the MACs learnt via vlan199 are supposed to be > > paired (displayed) with "inet-br" object and all way up into NS > > I want to make sure we're on the same page, how do you check if the MAC is > reaching into the NS? I assume using `ip neigh`? I'd have a look at tcpdump > this will tell you whether ARP is even reaching the NS or not. > > Something simple like > > $ tcpdump -enli $IFACE 'arp or icmp' > > optionally you can filter by MAC (`ether host` in pcap-filter speak): > > $ tcpdump -enli $IFACE ('arp or icmp) and ether host > aa:00:00:00:00:01 > > Oh and one last thing: please double and tripple check that a firewall isn't > interfering. --Daniel Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Peter, On Mon, Nov 13, 2023 at 09:40:46AM +, peter.gasparo...@orange.com wrote: > In the meantime, I was stubborn to find a solution to what I need in > order to progress and MACVLAN tech actually delivered it (private mode > enough), I used to love macvlan too but now I do L3 instead ;P > something newer than VETH tech what I could read about, and it's > just perfect avoiding bridge itself. So no problem to cooperate on this > fix, I will be glad, just it can get lower priority on your side if you > even attributed it some I'd be happy to still track this bug down but I need you to investigate the behaviour in your environment. If you've torn down the lab already we can also just call it quits. If you do want to continue some questions are still pending, see quoted below. > On Mon, Oct 30, 2023 at 07:25:38PM +, peter.gasparo...@orange.com wrote: > > 1) As was reported, foreign external world MAC@ does not pass into > > network namespace, just external border point "vlan199" > > How did you check this? > > > 2) now collecting data for you, honestly I don’t see external mac > > address on "inet-br" object, so my previous statement was incorrect.. > > {ossibly I might mixed this up with another "labinet-br" (working in > > its limited > > scope) which is IP-defined, while "inet-br" in question is not. > > > > 3) so question is, if the MACs learnt via vlan199 are supposed to be > > paired (displayed) with "inet-br" object and all way up into NS > > I want to make sure we're on the same page, how do you check if the MAC is > reaching into the NS? I assume using `ip neigh`? I'd have a look at tcpdump > this will tell you whether ARP is even reaching the NS or not. > > Something simple like > > $ tcpdump -enli $IFACE 'arp or icmp' > > optionally you can filter by MAC (`ether host` in pcap-filter speak): > > $ tcpdump -enli $IFACE ('arp or icmp) and ether host aa:00:00:00:00:01 > > Oh and one last thing: please double and tripple check that a firewall isn't > interfering. --Daniel signature.asc Description: PGP signature
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Daniel, Thank you honestly for you time to look at this and cooperation.. Good decision to supply my directly with scripts like you want to deal with this. So this one was success, it worked: peterg@debian:~/Downloads$ ./repro.sh Cannot remove namespace file "/var/run/netns/ns0": No such file or directory Cannot remove namespace file "/var/run/netns/ns1": No such file or directory Cannot find device "br0" PING 10.0.0.101 (10.0.0.101): 56 data bytes 64 bytes from 10.0.0.101: icmp_seq=0 ttl=64 time=0.057 ms 64 bytes from 10.0.0.101: icmp_seq=1 ttl=64 time=0.072 ms 64 bytes from 10.0.0.101: icmp_seq=2 ttl=64 time=0.037 ms 64 bytes from 10.0.0.101: icmp_seq=3 ttl=64 time=0.055 ms --- 10.0.0.101 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.037/0.055/0.072/0.000 ms In the meantime, I was stubborn to find a solution to what I need in order to progress and MACVLAN tech actually delivered it (private mode enough), something newer than VETH tech what I could read about, and it's just perfect avoiding bridge itself. So no problem to cooperate on this fix, I will be glad, just it can get lower priority on your side if you even attributed it some Thanks, wishing successful new week. Peter -Original Message- From: Daniel Gröber Sent: sobota 11. novembra 2023 1:35 To: GASPAROVIC Peter OBS/MKT ; 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Peter, looking at the ip/bridge dumps I don't see anything obviously broken so I started by building a reproducer using two netns'en and a bridge on the host to simulate your setup, leaving out the vlan stuff for now. I setup two namespaces ns0/ns1 with a veth pair each connected to br0 on the host. I assign MAC addresses statically to make looking at `bridge fdb` easier (grep ^aa:). The script looks like this (trimmed, full version attached): ip netns add ns0 ip link add veth0 type veth \ peer name veth0 address aa:00:00:00:00:00 netns ns0 ip netns add ns1 ip link add veth1 type veth \ peer name veth1 address aa:00:00:00:00:01 netns ns1 ip link add br0 address aa:bb:bb:bb:bb:bb type bridge forward_delay 0 #^ forward_delay=0 to disable STP as this delays interfaces coming up ip link set dev veth0 master br0 ip link set dev veth1 master br0 ip -n ns0 addr add 10.0.0.100/24 dev veth0 ip -n ns1 addr add 10.0.0.101/24 dev veth1 iplink set br0 up iplink set dev veth0 up ip -n ns0 link set dev veth0 up iplink set dev veth1 up ip -n ns1 link set dev veth1 up ip -n ns0 link set dev lo up ip -n ns1 link set dev lo up ip netns exec ns0 ping -c4 10.0.0.101 Seems to work fine. So we can establish the basic functionality does work and we need to go deeper. Peter, can you confirm this script works on your system? If so the next step is introducing vlans. On Mon, Oct 30, 2023 at 07:25:38PM +, peter.gasparo...@orange.com wrote: > 1) As was reported, foreign external world MAC@ does not pass into > network namespace, just external border point "vlan199" How did you check this? > 2) now collecting data for you, honestly I don’t see external mac > address on "inet-br" object, so my previous statement was incorrect.. > {ossibly I might mixed this up with another "labinet-br" (working in > its limited > scope) which is IP-defined, while "inet-br" in question is not. > > 3) so question is, if the MACs learnt via vlan199 are supposed to be > paired (displayed) with "inet-br" object and all way up into NS I want to make sure we're on the same page, how do you check if the MAC is reaching into the NS? I assume using `ip neigh`? I'd have a look at tcpdump this will tell you whether ARP is even reaching the NS or not. Something simple like $ tcpdump -enli $IFACE 'arp or icmp' optionally you can filter by MAC (`ether host` in pcap-filter speak): $ tcpdump -enli $IFACE ('arp or icmp) and ether host aa:00:00:00:00:01 Oh and one last thing: please double and tripple check that a firewall isn't interfering. --Daniel Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or pri
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Peter, looking at the ip/bridge dumps I don't see anything obviously broken so I started by building a reproducer using two netns'en and a bridge on the host to simulate your setup, leaving out the vlan stuff for now. I setup two namespaces ns0/ns1 with a veth pair each connected to br0 on the host. I assign MAC addresses statically to make looking at `bridge fdb` easier (grep ^aa:). The script looks like this (trimmed, full version attached): ip netns add ns0 ip link add veth0 type veth \ peer name veth0 address aa:00:00:00:00:00 netns ns0 ip netns add ns1 ip link add veth1 type veth \ peer name veth1 address aa:00:00:00:00:01 netns ns1 ip link add br0 address aa:bb:bb:bb:bb:bb type bridge forward_delay 0 #^ forward_delay=0 to disable STP as this delays interfaces coming up ip link set dev veth0 master br0 ip link set dev veth1 master br0 ip -n ns0 addr add 10.0.0.100/24 dev veth0 ip -n ns1 addr add 10.0.0.101/24 dev veth1 iplink set br0 up iplink set dev veth0 up ip -n ns0 link set dev veth0 up iplink set dev veth1 up ip -n ns1 link set dev veth1 up ip -n ns0 link set dev lo up ip -n ns1 link set dev lo up ip netns exec ns0 ping -c4 10.0.0.101 Seems to work fine. So we can establish the basic functionality does work and we need to go deeper. Peter, can you confirm this script works on your system? If so the next step is introducing vlans. On Mon, Oct 30, 2023 at 07:25:38PM +, peter.gasparo...@orange.com wrote: > 1) As was reported, foreign external world MAC@ does not pass into > network namespace, just external border point "vlan199" How did you check this? > 2) now collecting data for you, honestly I don’t see external mac address > on "inet-br" object, so my previous statement was incorrect.. {ossibly I > might mixed this up with another "labinet-br" (working in its limited > scope) which is IP-defined, while "inet-br" in question is not. > > 3) so question is, if the MACs learnt via vlan199 are supposed to be > paired (displayed) with "inet-br" object and all way up into NS I want to make sure we're on the same page, how do you check if the MAC is reaching into the NS? I assume using `ip neigh`? I'd have a look at tcpdump this will tell you whether ARP is even reaching the NS or not. Something simple like $ tcpdump -enli $IFACE 'arp or icmp' optionally you can filter by MAC (`ether host` in pcap-filter speak): $ tcpdump -enli $IFACE ('arp or icmp) and ether host aa:00:00:00:00:01 Oh and one last thing: please double and tripple check that a firewall isn't interfering. --Daniel repro.sh Description: Bourne shell script signature.asc Description: PGP signature
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Daniel, Hope you are good. What is the outlook after a week here? Thanks. BR Peter -Original Message- From: GASPAROVIC Peter OBS/MKT Sent: pondelok 30. októbra 2023 20:26 To: Daniel Gröber Cc: 1054...@bugs.debian.org Subject: RE: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Daniel, Definitely I can't do any script at the moment, so manual steps could be enough I hope so. 1) As was reported, foreign external world MAC@ does not pass into network namespace, just external border point "vlan199" 2) now collecting data for you, honestly I don’t see external mac address on "inet-br" object, so my previous statement was incorrect.. {ossibly I might mixed this up with another "labinet-br" (working in its limited scope) which is IP-defined, while "inet-br" in question is not. 3) so question is, if the MACs learnt via vlan199 are supposed to be paired (displayed) with "inet-br" object and all way up into NS 4) I collected all into text file. If this is problem, then I paste it here. Thanks, BR Peter -Original Message- From: Daniel Gröber Sent: pondelok 30. októbra 2023 13:04 To: GASPAROVIC Peter OBS/MKT Cc: 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Peter, On Mon, Oct 30, 2023 at 10:43:39AM +, peter.gasparo...@orange.com wrote: > Would it be possible to join a Webex session setup by me to check this > out quickly? It's all lab environment. I don't think that would help with reproducing your environment in this case, besides I only offer synchronous debugging sessions for paid consulting engagements. > If not I will proceed per your instructions Please do. --Daniel Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Daniel, Definitely I can't do any script at the moment, so manual steps could be enough I hope so. 1) As was reported, foreign external world MAC@ does not pass into network namespace, just external border point "vlan199" 2) now collecting data for you, honestly I don’t see external mac address on "inet-br" object, so my previous statement was incorrect.. {ossibly I might mixed this up with another "labinet-br" (working in its limited scope) which is IP-defined, while "inet-br" in question is not. 3) so question is, if the MACs learnt via vlan199 are supposed to be paired (displayed) with "inet-br" object and all way up into NS 4) I collected all into text file. If this is problem, then I paste it here. Thanks, BR Peter -Original Message- From: Daniel Gröber Sent: pondelok 30. októbra 2023 13:04 To: GASPAROVIC Peter OBS/MKT Cc: 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Peter, On Mon, Oct 30, 2023 at 10:43:39AM +, peter.gasparo...@orange.com wrote: > Would it be possible to join a Webex session setup by me to check this > out quickly? It's all lab environment. I don't think that would help with reproducing your environment in this case, besides I only offer synchronous debugging sessions for paid consulting engagements. > If not I will proceed per your instructions Please do. --Daniel Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. peterg@debian:~$ peterg@debian:~$ peterg@debian:~$ peterg@debian:~$ ip -d addr show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max _segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens161: mtu 9000 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:01:01:04 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9000 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_ma x_segs 65535 inet6 fe80::250:56ff:fe01:104/64 scope link valid_lft forever preferred_lft forever 3: ens192: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:01:01:01 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9000 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_ma x_segs 65535 inet 172.31.254.50/28 scope global ens192 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:fe01:101/64 scope link valid_lft forever preferred_lft forever 4: ens224: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:01:01:02 brd ff:ff:ff:ff:ff:ff promiscuity 2 minmtu 60 maxmtu 9000 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_ma x_segs 65535 inet6 fe80::250:56ff:fe01:102/64 scope link valid_lft forever preferred_lft forever 5: ens256: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:50:56:01:01:03 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 60 maxmtu 9000 numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_ma x_segs 65535 inet6 fe80::250:56ff:fe01:103/64 scope link valid_lft forever preferred_lft forever 6: vlan11@ens224: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:50:56:01:01:02 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535 vlan protocol 802.1Q id 11 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 192.168.255.254/24 brd 192.168.255.255 scope global vlan11 valid_lft forever preferred_lft forever inet6 fe80::250:56ff:fe01:102/64 scope link valid_lft forever preferred_lft forever 20: vlan77@ens224: mtu 1500 qdisc noqueue master labinet-br state UP group default qlen 1000 link/ether 00:50:56:01:01:02 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 0 maxmtu 65535 vlan
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Peter, On Mon, Oct 30, 2023 at 10:43:39AM +, peter.gasparo...@orange.com wrote: > Would it be possible to join a Webex session setup by me to check this > out quickly? It's all lab environment. I don't think that would help with reproducing your environment in this case, besides I only offer synchronous debugging sessions for paid consulting engagements. > If not I will proceed per your instructions Please do. --Daniel
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Daniel, Thank you for this yet hope even my joy to help fix something in this amazing Linux world you dive so deep in in contrast to me. Would it be possible to join a Webex session setup by me to check this out quickly? It's all lab environment. If not I will proceed per your instructions BR Peter -Original Message- From: Daniel Gröber Sent: nedeľa 29. októbra 2023 11:47 To: GASPAROVIC Peter OBS/MKT Cc: Luca Boccassi ; 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Hi Peter, On Fri, Oct 27, 2023 at 09:29:25AM +, peter.gasparo...@orange.com wrote: > No attempt at all? Then it's against your own rules I've read before > submitting this. I think Luca was a bit harsh here, I'd be happy to help debug this. From a first look it seems unlikely this is related to iproute2, smells more like a kernel issue to me, but either way we need a reproducer. So first step to move this forward is to put together a self contained set of instructions to reproduce the problem. Your original report is a bit sparse on context and details. If you don't feel up to compiling the reproducer script yourself you could start by showing us your system state using $ ip -d addr show # on the host and inside the NS if you could $ bridge -d link; bridge fdb snippets from /etc/network/interfaces or similar relevant config would help too. --Daniel Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Hi Peter, On Fri, Oct 27, 2023 at 09:29:25AM +, peter.gasparo...@orange.com wrote: > No attempt at all? Then it's against your own rules I've read before > submitting this. I think Luca was a bit harsh here, I'd be happy to help debug this. From a first look it seems unlikely this is related to iproute2, smells more like a kernel issue to me, but either way we need a reproducer. So first step to move this forward is to put together a self contained set of instructions to reproduce the problem. Your original report is a bit sparse on context and details. If you don't feel up to compiling the reproducer script yourself you could start by showing us your system state using $ ip -d addr show # on the host and inside the NS if you could $ bridge -d link; bridge fdb snippets from /etc/network/interfaces or similar relevant config would help too. --Daniel signature.asc Description: PGP signature
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
No attempt at all? Then it's against your own rules I've read before submitting this. -Original Message- From: Luca Boccassi Sent: piatok 27. októbra 2023 11:17 To: GASPAROVIC Peter OBS/MKT ; 1054...@bugs.debian.org Subject: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Control: tags -1 upstream On Fri, 27 Oct 2023 at 09:18, wrote: > > Package: iproute2 > Version: 4.20.0-2+deb10u1 > Problem: External MAC@ reaches max Linux bridge, but not net namespace > linked via veth pair to it, based on very minimal config > > Hello Debian team, > > I would like to report problem which possibly has to do with IPROUTE2 > package, I’m experiencing it both Debian 10 (this) and 12 (6.1.0-3). I really > did my best reviewing at least 7 stack-exchange (and like) stories and I’m at > my wit’s end, wondering why this is possibly not fixed in 2023 seeing debates > go back into like 2014.. > > So it’s plain simple to want to make multiple namespaces able to communicate > via common host bridge to external network. VETH tech is all time documented > as solution to this. The problem on given path in subject is this: > > NS veth IP@ = .251 , 0e:61:72:97:aa:ff > (Bridge) veth IP@ = .44 , ce:18:16:4b:0c:c2 Bridge IP@ = .254 , > 00:50:56:01:01:02 External IP@ = .other > > 1) When I initially set up plain “veth port --> NS veth port”, with IP@ at > each end, it’s all seamless, ARP and pings.. > >== 2) Once I enslave veth port to bridge, I can’t reach external > >network <=== > 3) Veth also does not work on IP level anymore, all the time with ICMP echo > from NS space it runs ARP first, though both “ip nei” are populated with > mutual MAC records. The following goes in loop.. > peterg@debian:~$ sudo tcpdump -ni vinet-brp > tcpdump: verbose output suppressed, use -v or -vv for full protocol > decode listening on vinet-brp, link-type EN10MB (Ethernet), capture > size 262144 bytes > 11:18:12.966955 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, > seq 0, length 64 > 11:18:12.966984 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length > 28 > 11:18:12.966989 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length > 28 > 11:18:13.967994 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, > seq 1, length > 4) Once I configure bridge iface with some IP address of same subnet > /24 like veth and NS veth (also externals) use → the NS nei can show > changing MAC address for bridge veth iface > 11:30:27.860907 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length > 28 > 11:30:28.848251 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2352, > seq 14, length 64 > 11:30:28.884892 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length > 28 > 11:30:28.884908 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length > 28 > 11:30:28.980890 ARP, Request who-has 70.0.0.44 tell 70.0.0.251, length > 28 > 11:30:28.980909 ARP, Reply 70.0.0.44 is-at 00:50:56:01:01:02, length > 28 <--- > > inet_bash >> ip nei > 70.0.0.1 dev vinet FAILED > 70.0.0.44 dev vinet lladdr ce:18:16:4b:0c:c2 DELAY <--- > > 5) The bridge vs NS veth pinging works > 6) The bridge relays ARP into external network and back (checked on Cisco > switch), learns of external MAC@s > ===> 7) External MAC@ does not make it to NS space by ARP<=== > 8) I don’t aim to deploy IP@ with bridge and bridge veth ifaces → this > is just to check how it works > 9) This blog was quite surprising stating that bridge without IP@ can > affect routing in global namespace, few /sys kernel tweaks → no help > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Funix > .stackexchange.com%2Fquestions%2F655602%2Fwhy-two-bridged-veth-cannot- > ping-each-other%2F674621%23674621=05%7C01%7Cpeter.gasparovic%40or > ange.com%7C5d4b8a4dcdf140886f7608dbd6cd941d%7C90c7a20af34b40bfbc48b925 > 3b6f5d20%7C0%7C0%7C638339950657397266%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi > MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 > C%7C=mgDKlgwu7NBcoSLZwC0GooZDiBEiz6GVRL02UaMF40E%3D=0 > 10) Even tried to stop default MAC learning on bridge veth iface by “ip link > set dev vinet-brp type bridge_slave learning off” ⇒ did not work, neigh > flushed and pinging .251 vs .254 just worked. > > So I believe that bridge veth iface is broken in its essential functionality > using default “learning/flooding on” settings. > > Thanks for your time to look at this and give hope or deny this so I need to > create extra ports in my host to get what I want to. > BR > Peter You need to report this upstream, nobody here is going to look at something like this __
Processed: Re: Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Processing control commands: > tags -1 upstream Bug #1054642 [iproute2] Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port Added tag(s) upstream. -- 1054642: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1054642 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Control: tags -1 upstream On Fri, 27 Oct 2023 at 09:18, wrote: > > Package: iproute2 > Version: 4.20.0-2+deb10u1 > Problem: External MAC@ reaches max Linux bridge, but not net namespace linked > via veth pair to it, based on very minimal config > > Hello Debian team, > > I would like to report problem which possibly has to do with IPROUTE2 > package, I’m experiencing it both Debian 10 (this) and 12 (6.1.0-3). I really > did my best reviewing at least 7 stack-exchange (and like) stories and I’m at > my wit’s end, wondering why this is possibly not fixed in 2023 seeing debates > go back into like 2014.. > > So it’s plain simple to want to make multiple namespaces able to communicate > via common host bridge to external network. VETH tech is all time documented > as solution to this. The problem on given path in subject is this: > > NS veth IP@ = .251 , 0e:61:72:97:aa:ff > (Bridge) veth IP@ = .44 , ce:18:16:4b:0c:c2 > Bridge IP@ = .254 , 00:50:56:01:01:02 > External IP@ = .other > > 1) When I initially set up plain “veth port --> NS veth port”, with IP@ at > each end, it’s all seamless, ARP and pings.. > >== 2) Once I enslave veth port to bridge, I can’t reach external network <=== > 3) Veth also does not work on IP level anymore, all the time with ICMP echo > from NS space it runs ARP first, though both “ip nei” are populated with > mutual MAC records. The following goes in loop.. > peterg@debian:~$ sudo tcpdump -ni vinet-brp > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on vinet-brp, link-type EN10MB (Ethernet), capture size 262144 bytes > 11:18:12.966955 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 0, > length 64 > 11:18:12.966984 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28 > 11:18:12.966989 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28 > 11:18:13.967994 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 1, > length > 4) Once I configure bridge iface with some IP address of same subnet /24 like > veth and NS veth (also externals) use → the NS nei can show changing MAC > address for bridge veth iface > 11:30:27.860907 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28 > 11:30:28.848251 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2352, seq > 14, length 64 > 11:30:28.884892 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28 > 11:30:28.884908 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28 > 11:30:28.980890 ARP, Request who-has 70.0.0.44 tell 70.0.0.251, length 28 > 11:30:28.980909 ARP, Reply 70.0.0.44 is-at 00:50:56:01:01:02, length 28 <--- > > inet_bash >> ip nei > 70.0.0.1 dev vinet FAILED > 70.0.0.44 dev vinet lladdr ce:18:16:4b:0c:c2 DELAY <--- > > 5) The bridge vs NS veth pinging works > 6) The bridge relays ARP into external network and back (checked on Cisco > switch), learns of external MAC@s > ===> 7) External MAC@ does not make it to NS space by ARP<=== > 8) I don’t aim to deploy IP@ with bridge and bridge veth ifaces → this is > just to check how it works > 9) This blog was quite surprising stating that bridge without IP@ can affect > routing in global namespace, few /sys kernel tweaks → no help > https://unix.stackexchange.com/questions/655602/why-two-bridged-veth-cannot-ping-each-other/674621#674621 > 10) Even tried to stop default MAC learning on bridge veth iface by “ip link > set dev vinet-brp type bridge_slave learning off” ⇒ did not work, neigh > flushed and pinging .251 vs .254 just worked. > > So I believe that bridge veth iface is broken in its essential functionality > using default “learning/flooding on” settings. > > Thanks for your time to look at this and give hope or deny this so I need to > create extra ports in my host to get what I want to. > BR > Peter You need to report this upstream, nobody here is going to look at something like this
Bug#1054642: Failing ARP relay from external -> Linux bridge -> veth port --> NS veth port
Package: iproute2 Version: 4.20.0-2+deb10u1 Problem: External MAC@ reaches max Linux bridge, but not net namespace linked via veth pair to it, based on very minimal config Hello Debian team, I would like to report problem which possibly has to do with IPROUTE2 package, I’m experiencing it both Debian 10 (this) and 12 (6.1.0-3). I really did my best reviewing at least 7 stack-exchange (and like) stories and I’m at my wit’s end, wondering why this is possibly not fixed in 2023 seeing debates go back into like 2014.. So it’s plain simple to want to make multiple namespaces able to communicate via common host bridge to external network. VETH tech is all time documented as solution to this. The problem on given path in subject is this: NS veth IP@ = .251 , 0e:61:72:97:aa:ff (Bridge) veth IP@ = .44 , ce:18:16:4b:0c:c2 Bridge IP@ = .254 , 00:50:56:01:01:02 External IP@ = .other 1) When I initially set up plain “veth port --> NS veth port”, with IP@ at each end, it’s all seamless, ARP and pings.. >== 2) Once I enslave veth port to bridge, I can’t reach external network <=== 3) Veth also does not work on IP level anymore, all the time with ICMP echo from NS space it runs ARP first, though both “ip nei” are populated with mutual MAC records. The following goes in loop.. peterg@debian:~$ sudo tcpdump -ni vinet-brp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vinet-brp, link-type EN10MB (Ethernet), capture size 262144 bytes 11:18:12.966955 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 0, length 64 11:18:12.966984 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28 11:18:12.966989 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28 11:18:13.967994 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2333, seq 1, length 4) Once I configure bridge iface with some IP address of same subnet /24 like veth and NS veth (also externals) use → the NS nei can show changing MAC address for bridge veth iface 11:30:27.860907 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28 11:30:28.848251 IP 70.0.0.251 > 70.0.0.44: ICMP echo request, id 2352, seq 14, length 64 11:30:28.884892 ARP, Request who-has 70.0.0.251 tell 70.0.0.44, length 28 11:30:28.884908 ARP, Reply 70.0.0.251 is-at 0e:61:72:97:aa:ff, length 28 11:30:28.980890 ARP, Request who-has 70.0.0.44 tell 70.0.0.251, length 28 11:30:28.980909 ARP, Reply 70.0.0.44 is-at 00:50:56:01:01:02, length 28 <--- inet_bash >> ip nei 70.0.0.1 dev vinet FAILED 70.0.0.44 dev vinet lladdr ce:18:16:4b:0c:c2 DELAY <--- 5) The bridge vs NS veth pinging works 6) The bridge relays ARP into external network and back (checked on Cisco switch), learns of external MAC@s ===> 7) External MAC@ does not make it to NS space by ARP<=== 8) I don’t aim to deploy IP@ with bridge and bridge veth ifaces → this is just to check how it works 9) This blog was quite surprising stating that bridge without IP@ can affect routing in global namespace, few /sys kernel tweaks → no help https://unix.stackexchange.com/questions/655602/why-two-bridged-veth-cannot-ping-each-other/674621#674621 10) Even tried to stop default MAC learning on bridge veth iface by “ip link set dev vinet-brp type bridge_slave learning off” ⇒ did not work, neigh flushed and pinging .251 vs .254 just worked. So I believe that bridge veth iface is broken in its essential functionality using default “learning/flooding on” settings. Thanks for your time to look at this and give hope or deny this so I need to create extra ports in my host to get what I want to. BR Peter Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.