Re: Bridge broken in 6.0?

[Henning Brauer]

* Aaron Riekenberg  [2016-09-05 13:04]:
> Thanks for the explanation.
> 
> I am curious though - is dhclient really the right place to fix this?  I
> might use some other dhcp client (dhcpcd in ports for example) or some
> other application that uses BPF.  Should every userland program using BPF
> have to worry whether or not it is breaking bridging?

no, the key is the dropping in BPF, which is an OpenBSD extension.
[I don't know whether others have similiar schemes or followed our
lead, and that's NOT THE TOPIC HERE. point is, it is nonstandard and
not in widespread use by 3rd party code if at all]

strictly speaking, the bpf filters in the base dhcp programs have been
matching (and thus eating) too much forever since we added them, it
just didn't show up because it was covered by the behaviour (strictly
speaking, I'd say misbehaviour) of the stack with bridge so far.

-- 
Henning Brauer, h...@bsws.de, henn...@openbsd.org
BS Web Services GmbH, http://bsws.de, Full-Service ISP
Secure Hosting, Mail and DNS. Virtual & Dedicated Servers, Root to Fully Managed
Henning Brauer Consulting, http://henningbrauer.com/

Re: Bridge broken in 6.0?

[Theo de Raadt]

>I am curious though - is dhclient really the right place to fix this?  I
>might use some other dhcp client (dhcpcd in ports for example) or some
>other application that uses BPF.  Should every userland program using BPF
>have to worry whether or not it is breaking bridging?

Various solutions are being discussed, so far all have downsides.
For now, the situation must remain because changing it back would
impact ongoing work.

Re: Bridge broken in 6.0?

[Aaron Riekenberg]

Thanks for the explanation.

I am curious though - is dhclient really the right place to fix this?  I
might use some other dhcp client (dhcpcd in ports for example) or some
other application that uses BPF.  Should every userland program using BPF
have to worry whether or not it is breaking bridging?

On Sun, Sep 4, 2016 at 9:12 AM, Ted Unangst  wrote:

> Aaron Riekenberg wrote:
> > Based on Martin's hint I tried an experiment - I configured em0
> statically
> > instead of using dhcp.  This fixes my problem.
> >
> > So to summarize: in 6.0 running dhclient on a bridge-member interface
> > breaks dhcp traffic passing through the bridge.  This worked in 5.9.
> Also
> > the FAQ bridge example uses DHCP on a member interface:
> > http://www.openbsd.org/faq/faq6.html#Bridge
>
> So everybody understands what's happening behind the scenes and why this
> is a
> new problem...
>
> Long ago, a flag was added to BPF filters to drop matching packets.
> dhclient
> sets this flag on the filter it uses to listen to dhcp packets, which has
> the
> (intended) effect of causing dhclient to eat the packet so nobody else sees
> it.
>
> Between 5.9 and 6.0, some code in the network stack was shuffled about and
> this changed the order in which some operations were performed. In
> particular,
> now BPF sees packets before bridge forwards them, instead of after.
> Therefore,
> if dhclient eats the packet, it won't be forwarded.
>
> dhclient was modified to only match packets intended for *this* machine by
> specifying a filter for the local machine's MAC address. However, there are
> buggy dhcp servers that always send packets to the broadcast address. The
> dhclient filter change was reverted and it once again eats all dhcp
> packets.
>
>

Re: Bridge broken in 6.0?

[Ted Unangst]

Aaron Riekenberg wrote:
> Based on Martin's hint I tried an experiment - I configured em0 statically
> instead of using dhcp.  This fixes my problem.
> 
> So to summarize: in 6.0 running dhclient on a bridge-member interface
> breaks dhcp traffic passing through the bridge.  This worked in 5.9.  Also
> the FAQ bridge example uses DHCP on a member interface:
> http://www.openbsd.org/faq/faq6.html#Bridge

So everybody understands what's happening behind the scenes and why this is a
new problem...

Long ago, a flag was added to BPF filters to drop matching packets. dhclient
sets this flag on the filter it uses to listen to dhcp packets, which has the
(intended) effect of causing dhclient to eat the packet so nobody else sees
it.

Between 5.9 and 6.0, some code in the network stack was shuffled about and
this changed the order in which some operations were performed. In particular,
now BPF sees packets before bridge forwards them, instead of after. Therefore,
if dhclient eats the packet, it won't be forwarded.

dhclient was modified to only match packets intended for *this* machine by
specifying a filter for the local machine's MAC address. However, there are
buggy dhcp servers that always send packets to the broadcast address. The
dhclient filter change was reverted and it once again eats all dhcp packets.

Bridge broken in 6.0?

[Aaron Riekenberg]

Based on Martin's hint I tried an experiment - I configured em0 statically
instead of using dhcp.  This fixes my problem.

So to summarize: in 6.0 running dhclient on a bridge-member interface
breaks dhcp traffic passing through the bridge.  This worked in 5.9.  Also
the FAQ bridge example uses DHCP on a member interface:
http://www.openbsd.org/faq/faq6.html#Bridge

My working configuration without dhclient:

$ cat /etc/hostname.em0
inet 192.168.0.100 255.255.255.0

$ cat /etc/hostname.em1
up

$ cat /etc/hostname.bridge0
add em0
add em1
up

$ netstat -rn
Routing tables

Internet:
DestinationGatewayFlags   Refs  Use   Mtu  Prio
Iface
default192.168.0.1UGS4  732 - 8 em0
224/4  127.0.0.1  URS0 1278 32768 8 lo0
127/8  127.0.0.1  UGRS   00 32768 8 lo0
127.0.0.1  127.0.0.1  UHl2   30 32768 1 lo0
192.168.0/24   192.168.0.100  UC 3 1449 - 4 em0
192.168.0.15a:ea:1d:96:c6:a8  UHLc   1  852 - 4 em0
192.168.0.714:99:e2:2b:f8:bc  UHLc   0  572 - 4 em0
192.168.0.15   d8:a2:5e:95:8e:e8  UHLc   1  239 - 4 em0
192.168.0.100  00:07:e9:19:e0:02  UHLl   0  247 - 1 em0
192.168.0.255  192.168.0.100  UHb0   34 - 1 em0

Internet6:
DestinationGatewayFlags
Refs  Use   Mtu  Prio Iface
::/96  ::1UGRS
  00 32768 8 lo0
::/104 ::1UGRS
  00 32768 8 lo0
::1::1UHl
14   14 32768 1 lo0
::127.0.0.0/104::1UGRS
  00 32768 8 lo0
::224.0.0.0/100::1UGRS
  00 32768 8 lo0
::255.0.0.0/104::1UGRS
  00 32768 8 lo0
:::0.0.0.0/96  ::1UGRS
  00 32768 8 lo0
2002::/24  ::1UGRS
  00 32768 8 lo0
2002:7f00::/24 ::1UGRS
  00 32768 8 lo0
2002:e000::/20 ::1UGRS
  00 32768 8 lo0
2002:ff00::/24 ::1UGRS
  00 32768 8 lo0
fe80::/10  ::1UGRS
  00 32768 8 lo0
fec0::/10  ::1UGRS
  00 32768 8 lo0
fe80::1%lo0fe80::1%lo0UHl
 00 32768 1 lo0
ff01::/16  ::1UGRS
  00 32768 8 lo0
ff01::%lo0/32  ::1Um
  01 32768 4 lo0
ff02::/16  ::1UGRS
  00 32768 8 lo0
ff02::%lo0/32  ::1Um
  01 32768 4 lo0

# cat /etc/pf.conf
#   $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf
- hide quoted text -

ext_if = "em0"
int_if = "em1"

set skip on lo
set loginterface $ext_if
set block-policy drop
set limit states 5

pass on $ext_if all
pass on $int_if all

$ dmesg
OpenBSD 6.0 (GENERIC.MP) #1: Sat Sep  3 06:59:42 CDT 2016
root@server.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP
RTC BIOS diagnostic error 80
real mem = 1045643264 (997MB)
avail mem = 1009541120 (962MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe35a0 (23 entries)
bios0: vendor Intel Corp. version "LF94510J.86A.0278.2010.0414.2000" date
04/14/2010
bios0: Intel Corporation D945GCLF2
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S3 S4 S5
acpi0: tables DSDT FACP APIC WDDT MCFG ASF!
acpi0: wakeup devices SLPB(S4) P32_(S4) UAR1(S4) UAR2(S4) PEX0(S4) PEX1(S4)
PEX2(S4) PEX3(S4) PEX4(S4) PEX5(S4) UHC1(S3) UHC2(S3) UHC3(S3) UHC4(S3)
EHCI(S3) AC9M(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Atom(TM) CPU 330 @ 1.60GHz, 1596.27 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 7 var ranges, 88 fixed ranges
cpu0: apic clock running at 133MHz
cpu0: mwait min=64, max=64, C-substates=0.1, IBE
cpu1 at 

Fwd: Bridge broken in 6.0?

[aaron . riekenberg]

Forwarding to tech.

On Saturday, 3 September 2016 08:36:43 UTC-5, aaron.ri...@gmail.com wrote:
>
> On Saturday, 3 September 2016 07:15:27 UTC-5, Martin Pieuchot  wrote: 
> > Yes something changed. That might be the cause for your regression. 
> > Sadly your bug report does not contain enough information for us to 
> > figure out.  What is your whole config: dmesg, routing table, hostname* 
>
> More info below. 
>
> > Do you run dhclient on your bridge or any of its interface? 
>
> Yes.  I run dhclient on em0, which is attached to a modem/router device 
> provided by my ISP.  That modem/router device is a DHCP server that hands 
> out IPs in the 192.168.0.0/24 subnet.  em0 on my box gets 192.168.0.100 
> from the DHCP server.  em1 on my box has no IP address and is attached to 
> my internal network of devices.  The symptom I am seeing is devices on the 
> internal network are nearly always unable to obtain a DHCP lease from the 
> model/router devices when the openbsd bridge is between them.  Occasionally 
> it works, but it is very unreliable. 
>
> This same configuration worked perfectly on 5.9. 
>
> Also note the box has a 3rd interface (re0) but it unused and 
> unconfigured. 
>
> > And please next time send bug reports to bugs@ :) 
>
> Will do. 
>
> More information: 
>
> $ hostname 
> server.domain 
>
> $ cat /etc/hostname.em0 
> dhcp 
>
> $ cat /etc/hostname.em1 
> up 
>
> $ cat /etc/hostname.bridge0 
> add em0 
> add em1 
> up 
>
> $ dmesg 
> OpenBSD 6.0 (GENERIC.MP) #1: Sat Sep  3 06:59:42 CDT 2016 
> root@server.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP 
> RTC BIOS diagnostic error 80 
> real mem = 1045643264 (997MB) 
> avail mem = 1009541120 (962MB) 
> mpath0 at root 
> scsibus0 at mpath0: 256 targets 
> mainbus0 at root 
> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe35a0 (23 entries) 
> bios0: vendor Intel Corp. version "LF94510J.86A.0278.2010.0414.2000" date 
> 04/14/2010 
> bios0: Intel Corporation D945GCLF2 
> acpi0 at bios0: rev 2 
> acpi0: sleep states S0 S1 S3 S4 S5 
> acpi0: tables DSDT FACP APIC WDDT MCFG ASF! 
> acpi0: wakeup devices SLPB(S4) P32_(S4) UAR1(S4) UAR2(S4) PEX0(S4) 
> PEX1(S4) PEX2(S4) PEX3(S4) PEX4(S4) PEX5(S4) UHC1(S3) UHC2(S3) UHC3(S3) 
> UHC4(S3) EHCI(S3) AC9M(S4) [...] 
> acpitimer0 at acpi0: 3579545 Hz, 24 bits 
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat 
> cpu0 at mainbus0: apid 0 (boot processor) 
> cpu0: Intel(R) Atom(TM) CPU 330 @ 1.60GHz, 1596.22 MHz 
> cpu0: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR
>  
>
> cpu0: 512KB 64b/line 16-way L2 cache 
> cpu0: smt 0, core 0, package 0 
> mtrr: Pentium Pro MTRR support, 7 var ranges, 88 fixed ranges 
> cpu0: apic clock running at 133MHz 
> cpu0: mwait min=64, max=64, C-substates=0.1, IBE 
> cpu1 at mainbus0: apid 1 (application processor) 
> cpu1: Intel(R) Atom(TM) CPU 330 @ 1.60GHz, 1596.01 MHz 
> cpu1: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG,LAHF,PERF,SENSOR
>  
>
> cpu1: 512KB 64b/line 16-way L2 cache 
> cpu1: smt 0, core 1, package 0 
> ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 24 pins 
> acpimcfg0 at acpi0 addr 0xf000, bus 0-127 
> acpiprt0 at acpi0: bus 0 (PCI0) 
> acpiprt1 at acpi0: bus 4 (P32_) 
> acpiprt2 at acpi0: bus 1 (PEX0) 
> acpiprt3 at acpi0: bus -1 (PEX1) 
> acpiprt4 at acpi0: bus 2 (PEX2) 
> acpiprt5 at acpi0: bus 3 (PEX3) 
> acpiprt6 at acpi0: bus -1 (PEX4) 
> acpiprt7 at acpi0: bus -1 (PEX5) 
> acpicpu0 at acpi0: C1(@1 halt!) 
> acpicpu1 at acpi0: C1(@1 halt!) 
> acpibtn0 at acpi0: SLPB 
> "PNP0003" at acpi0 not configured 
> pci0 at mainbus0 bus 0 
> pchb0 at pci0 dev 0 function 0 "Intel 82945G Host" rev 0x02 
> inteldrm0 at pci0 dev 2 function 0 "Intel 82945G Video" rev 0x02 
> drm0 at inteldrm0 
> intagp0 at inteldrm0 
> agp0 at intagp0: aperture at 0x4000, size 0x1000 
> inteldrm0: apic 2 int 16 
> inteldrm0: 1920x1080 
> wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation) 
> wsdisplay0: screen 1-5 added (std, vt100 emulation) 
> ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: msi 
> pci1 at ppb0 bus 1 
> 1:0:0: mem address conflict 0xfffe/0x2 
> re0 at pci1 dev 0 function 0 "Realtek 8168" rev 0x02: RTL8168C/8111C 
> (0x3c00), msi, address 00:1c:c0:f0:cf:9c 
> rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 2 
> ppb1 at pci0 dev 28 function 2 "Intel 82801GB PCIE" rev 0x01: msi 
> pci2 at ppb1 bus 2 
> ppb2 at pci0 dev 28 function 3 "Intel 82801GB PCIE" rev 0x01: msi 
> pci3 at ppb2 bus 3 
> uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 
> 23 
> uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 
> 19 
> uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 2 int 
> 

Re: Bridge broken in 6.0?

[Martin Pieuchot]

On 09/03/16 13:55, Aaron Riekenberg wrote:

I have been using a 5.9 amd64 box with 2 em interfaces configured as a
bridge for some time between my ISP's modem/router and my home network.
With this setup on 5.9, things worked perfectly.

On 6.0 I have tried the same very simple configuration of creating a bridge
with em0 and em1 interfaces and passing all traffic (see configuration
below).  With 6.0 this is unstable - clients are often unable to obtain a
dhcp lease from the router through the bridge, which makes the it
unusable.  If I remove the openbsd 6 bridge from my network and replace it
with a simple cable, things work fine again.

Did something change or break with bridge in 6.0?


Yes something changed. That might be the cause for your regression.
Sadly your bug report does not contain enough information for us to
figure out.  What is your whole config: dmesg, routing table, hostname*

Do you run dhclient on your bridge or any of its interface?

And please next time send bug reports to bugs@ :)

Bridge broken in 6.0?

[Aaron Riekenberg]

I have been using a 5.9 amd64 box with 2 em interfaces configured as a
bridge for some time between my ISP's modem/router and my home network.
With this setup on 5.9, things worked perfectly.

On 6.0 I have tried the same very simple configuration of creating a bridge
with em0 and em1 interfaces and passing all traffic (see configuration
below).  With 6.0 this is unstable - clients are often unable to obtain a
dhcp lease from the router through the bridge, which makes the it
unusable.  If I remove the openbsd 6 bridge from my network and replace it
with a simple cable, things work fine again.

Did something change or break with bridge in 6.0?


/etc/hostname.bridge0:
add em0
add em1
up


pf.conf:

ext_if = "em0"
int_if = "em1"

set skip on lo
set loginterface $ext_if
set block-policy drop
set limit states 5

pass on $ext_if all
pass on $int_if all