Hi I am experiencing packet loss and ENOBUFs, I have a program with with two tap(4) interfaces and i am using kqueue(2) to determine when and which tap interface to process the packet. I email bugs@ over a month ago with no reply https://marc.info/?l=openbsd-bugs&m=156229879107900&w=2 so i thought i would try here, paraphrased below.
Back Story: Over the last few years i have ported Solo5, used by MirageOS and others to run on OpenBSD's vmm(4). In Solo5 we have been working towards supporting multiple network interfaces and implemented this using kqueue(2) and tap(4). The Problem is demonstrated as follows. setting up two Tap interfaces, starting up the Unikernel(Solo5) on vmm. In another session flood pinging the first Tap interface, Solo5 handles this with no packets dropped. In another session ping the second Tap interface, then for every ping to the second interface a packet is dropped on the first. If you switch to a flood ping on the second tab interface, you will observe massive packet loss on both interfaces, and ping complaining about No buffer space available (ENOBUFS). see https://github.com/Solo5/solo5/issues/374 for more information. I have been able to reproduce this in a hacked up exampled program, available here https://github.com/adamsteen/test_net_2if. Please note this is hacked, generally butchered program, which demonstrates the problem. (if required i can try and clean up this test case) 01. git clone https://github.com/adamsteen/test_net_2if 02. cd test_net_2if 03. make 04. doas setup.sh (Setup up the Tap interfaces) 05. doas ./test_net_2if 06. in another seesion start a flood ping doas ping -f 10.0.0.2 07. Observe that the flood ping is functioning correctly, with no packets dropped. 08. In another session, start a normal ping ping 10.1.0.2 09. Observe that, for each ping sent to service1, a packet is dropped. 10. Kill the normal ping 11. start a flood ping doas ping -f 10.1.0.2 12. Observe massive packet loss on both interfaces, and ping complaining about No buffer space available (ENOBUFS). Cheers Adam ps Sorry for the poor english. dmesg: OpenBSD 6.5-current (GENERIC.MP) #123: Sat Jun 29 19:39:46 AWST 2019 ast...@x220.adamsteen.com.au:/sys/arch/amd64/compile/GENERIC.MP real mem = 17041059840 (16251MB) avail mem = 16514461696 (15749MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xdae9c000 (64 entries) bios0: vendor LENOVO version "8DET69WW (1.39 )" date 07/18/2013 bios0: LENOVO 4291N58 acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SLIC SSDT SSDT SSDT HPET APIC MCFG ECDT ASF! TCPA SSDT SSDT \ DMAR UEFI UEFI UEFI acpi0: wakeup devices LID_(S3) SLPB(S3) IGBE(S4) EXP4(S4) EXP7(S4) EHC1(S3) EHC2(S3) \ HDEF(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 2492.31 MHz, 06-2a-07 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \ S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2 \ ,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTS \ CP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1.2, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz, 2491.91 MHz, 06-2a-07 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \ S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2 \ ,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTS \ CP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf8000000, bus 0-63 acpiec0 at acpi0 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (PEG_) acpiprt2 at acpi0: bus 2 (EXP1) acpiprt3 at acpi0: bus 3 (EXP2) acpiprt4 at acpi0: bus 5 (EXP4) acpiprt5 at acpi0: bus 13 (EXP5) acpiprt6 at acpi0: bus -1 (EXP7) acpicpu0 at acpi0: C3(350@104 io@0x415), C1(1000@1 halt), PSS acpicpu1 at acpi0: C3(350@104 io@0x415), C1(1000@1 halt), PSS acpipwrres0 at acpi0: PUBS, resource for EHC1, EHC2 acpitz0 at acpi0: critical temperature is 99 degC acpibtn0 at acpi0: LID_ acpibtn1 at acpi0: SLPB acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 acpicmos0 at acpi0 acpibat0 at acpi0: BAT0 model "42T4861" serial 16466 type LION oem "SANYO" acpiac0 at acpi0: AC unit online acpithinkpad0 at acpi0 "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured acpidock0 at acpi0: GDCK not docked (0) acpivideo0 at acpi0: VID_ acpivout at acpivideo0 not configured acpivideo1 at acpi0: VID_ cpu0: using VERW MDS workaround (except on vmm entry) cpu0: Enhanced SpeedStep 2492 MHz: speeds: 2501, 2500, 2200, 2000, 1800, 1600, 1400, \ 1200, 1000, 800 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Core 2G Host" rev 0x09 inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 3000" rev 0x09 drm0 at inteldrm0 inteldrm0: msi "Intel 6 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured em0 at pci0 dev 25 function 0 "Intel 82579LM" rev 0x04: msi, address \ f0:de:f1:77:c2:c8 ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic \ 2 int 16 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 azalia0 at pci0 dev 27 function 0 "Intel 6 Series HD Audio" rev 0x04: msi azalia0: codecs: Conexant CX20590, Intel/0x2805, using Conexant CX20590 audio0 at azalia0 ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi pci1 at ppb0 bus 2 ppb1 at pci0 dev 28 function 1 "Intel 6 Series PCIE" rev 0xb4: msi pci2 at ppb1 bus 3 iwn0 at pci2 dev 0 function 0 "Intel Centrino Advanced-N 6205" rev 0x34: msi, MIMO \ 2T2R, MoW, address a0:88:b4:c0:16:78 ppb2 at pci0 dev 28 function 3 "Intel 6 Series \ PCIE" rev 0xb4: msi pci3 at ppb2 bus 5 ppb3 at pci0 dev 28 function 4 "Intel 6 Series PCIE" rev 0xb4: msi pci4 at ppb3 bus 13 sdhc0 at pci4 dev 0 function 0 "Ricoh 5U822 SD/MMC" rev 0x07: apic 2 int 16 sdhc0: SDHC 3.0, 50 MHz base clock sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 2 int 23 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 pcib0 at pci0 dev 31 function 0 "Intel QM67 LPC" rev 0x04 ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3 ahci0: port 2: 3.0Gb/s scsibus1 at ahci0: 32 targets sd0 at scsibus1 targ 2 lun 0: <ATA, Samsung SSD 860, RVT4> SCSI3 0/direct fixed \ naa.5002538e400d9cf5 sd0: 476940MB, 512 bytes/sector, 976773168 sectors, thin ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 2 int 18 iic0 at ichiic0 spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM PC3-12800 SO-DIMM spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM PC3-12800 SO-DIMM isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 wsmouse1 at pms0 mux 0 pms0: Synaptics clickpad, firmware 8.0, 0x1e2b1 0x940300 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 aps0 at isa0 port 0x1600/31 vmm0 at mainbus0: VMX/EPT uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \ 2.00/0.00 addr 2 uvideo0 at uhub2 port 6 configuration 1 interface 0 "Chicony \ Electronics Co., Ltd. Integrated Camera" rev 2.00/8.54 addr 3 video0 at uvideo0 uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \ 2.00/0.00 addr 2 umass0 at uhub3 port 1 configuration 1 interface 0 "SanDisk Ultra \ Fit" rev 2.10/1.00 addr 3 umass0: using SCSI over Bulk-Only scsibus2 at umass0: 2 targets, initiator 0 sd1 at scsibus2 targ 1 lun 0: <SanDisk, Ultra Fit, 1.00> SCSI4 0/direct removable \ serial.07815583440513123182 sd1: 29327MB, 512 bytes/sector, 60062500 sectors vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets sd2 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed sd2: 476937MB, 512 bytes/sector, 976767473 sectors root on sd2a (2d000de2c9dbc60d.a) swap on sd2b dump on sd2b inteldrm0: 1366x768, 32bpp wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using wskbd0 wsdisplay0: screen 1-5 added (std, vt100 emulation) iwn0: radio is disabled by hardware switch iwn0: radio is disabled by hardware switch
