Hi, I'm benchmarking OpenBSD 4.8 for use as a firewall/router, and I'm getting some unusually slow results when using a re(4) interface. The hardware is a little OpenVox board[1] as sold by Yawarra[2] - it's a 1.6GHz Atom CPU with two rl(4) 10/100 interfaces and one re(4) gigabit interface.
I have a connection from the re(4) interface to a Gigabit port on a switch. This is on the same VLAN as a laptop (A), and a Cisco router. Laptop (B) sits behind the Cisco router. I have assigned an IP alias (see [5] for the config) on the re(4) interface such that laptop A routes via the OpenVox board. Both laptops run Ubuntu 10.04.1. Here's a routing diagram: Laptop A (10.9.0.2/16) ----- re0 (10.9.0.1/16, alias) OpenBSD re0 (10.20.3.1/16) ----- (10.20.0.1/16) Cisco 6509 (10.15.0.1/16) ----- Laptop B (10.15.200.44/16) I have confirmed that I can get > 900Mb/s between the laptops, if they are on the same VLAN. I can also get > 900 Mb/s between the laptops, passing only through the switch and the Cisco 6509. My problem occurs when I test using the configuration in the diagram above. Using [3] from laptop A (TCP, 1400-byte packets) I get around 240Mb/s. Using [4] (UDP, 1400-byte packets), I get around 500 Mb/s. What gets me is that the CPU is not saturated for either test - it sits at about 70% (interrupts) for the UDP test, and 40% (interrupts) for the TCP test. I've eliminated most of the variables (I think, anyway): - The re(4) interface is connected using a PCIe link, so I would expect there to be sufficient bandwidth there. - Cables are at least CAT5e - The Cisco router, the switch and the laptops can both do in excess of 900 Mb/s (above) - CPU on the OpenBSD box is not saturated by interrupts or anything else - sysctls are at defaults, except for IP forwarding, multicast forwarding, and CARP preemption. I have tried the recommendations at https://calomel.org/network_performance.html with no effect. Does anyone have any ideas on why I'm not seeing higher throughput? Once I've solved this issue, I'll post some decent benchmarks. Cheers, Patrick -- http://www.labyrinthdata.net.au - WA Backup, Web and VPS Hosting [1] http://www.openvox.cn/products/show.php?itemid=161&lang=2 [2] http://www.yawarra.com.au/product.php?productCode=HW-NV16-R [3] On laptop A: netperf -H 10.15.200.44 -t UDP_STREAM -- -m 1400 -M 1400 [4] On laptop A: netperf -H 10.15.200.44 -t TCP_STREAM -- -m 1400 -M 1400 [5] Network config: r...@newrouter:~# ifconfig -a lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33200 priority: 0 groups: lo inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 rl0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500 lladdr a0:98:05:01:00:fa priority: 0 media: Ethernet autoselect status: no carrier rl1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500 lladdr a0:98:05:01:00:f9 priority: 0 media: Ethernet autoselect status: no carrier re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr a0:98:05:01:ab:28 priority: 0 groups: egress media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) status: active inet 10.20.3.1 netmask 0xffff0000 broadcast 10.20.255.255 inet6 fe80::a298:5ff:fe01:ab28%re0 prefixlen 64 scopeid 0x3 enc0: flags=0<> priority: 0 groups: enc status: active pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33200 priority: 0 groups: pflog r...@newrouter:~# ifconfig re0 re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 lladdr a0:98:05:01:ab:28 priority: 0 groups: egress media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause) status: active inet 10.20.3.1 netmask 0xffff0000 broadcast 10.20.255.255 inet6 fe80::a298:5ff:fe01:ab28%re0 prefixlen 64 scopeid 0x3 inet 10.9.0.1 netmask 0xffff0000 broadcast 10.9.255.255 r...@newrouter:~# pfctl -s rules pass inet from 10.9.0.0/16 to any flags S/SA keep state pass inet from any to 10.9.0.0/16 flags S/SA keep state r...@newrouter:~# dmesg below. OpenBSD 4.8 (FLASHRD) #2: Fri Oct 1 15:21:03 PDT 2010 ch...@ref.nmedia.net:/usr/src/sys/arch/i386/compile/FLASHRD cpu0: Intel(R) Atom(TM) CPU Z530 @ 1.60GHz ("GenuineIntel" 686-class) 1.60 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,xTPR,PDCM,MOVBE real mem = 2141278208 (2042MB) avail mem = 2093256704 (1996MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 08/12/10, BIOS32 rev. 0 @ 0xf0010, SMBIOS rev. 2.5 @ 0xfbe70 (18 entries) bios0: vendor American Megatrends Inc. version "080015" date 08/12/2010 bios0: OpenVox IPC100 acpi0 at bios0: rev 2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI acpi0: wakeup devices USB0(S4) USB1(S4) USB2(S4) EUSB(S4) POP4(S4) POP5(S4) LID_(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 132MHz ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 1 (POP4) acpiprt2 at acpi0: bus 2 (P2P6) acpiprt3 at acpi0: bus 3 (POP5) acpicpu0 at acpi0 acpibtn0 at acpi0: SLPB acpibtn1 at acpi0: PWRB acpibtn2 at acpi0: LID_ acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD01 acpivout1 at acpivideo0: DD02 acpivout2 at acpivideo0: DD03 acpivout3 at acpivideo0: DD04 bios0: ROM list: 0xc0000/0xe600! cpu0: unknown Enhanced SpeedStep CPU, msr 0x060b0c1d06000c1d cpu0: using only highest and lowest power states cpu0: Enhanced SpeedStep 1597 MHz: speeds: 1600, 800 MHz pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "Intel US15W Host" rev 0x07 vga1 at pci0 dev 2 function 0 "Intel US15W Video" rev 0x07 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) intagp at vga1 not configured ppb0 at pci0 dev 28 function 0 "Intel SCH PCIE" rev 0x07: apic 1 int 16 (irq 10) pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "TI XIO2000A PCIE-PCI" rev 0x03 pci2 at ppb1 bus 2 rl0 at pci2 dev 2 function 0 "Realtek 8139" rev 0x10: apic 1 int 18 (irq 15), address a0:98:05:01:00:fa rlphy0 at rl0 phy 0: RTL internal PHY rl1 at pci2 dev 3 function 0 "Realtek 8139" rev 0x10: apic 1 int 19 (irq 5), address a0:98:05:01:00:f9 rlphy1 at rl1 phy 0: RTL internal PHY ppb2 at pci0 dev 28 function 1 "Intel SCH PCIE" rev 0x07: apic 1 int 17 (irq 11) pci3 at ppb2 bus 3 re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x03: RTL8168D/8111D (0x2800), apic 1 int 17 (irq 5), address a0:98:05:01:ab:28 rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 2 uhci0 at pci0 dev 29 function 0 "Intel SCH USB" rev 0x07: apic 1 int 16 (irq 10) ehci0 at pci0 dev 29 function 7 "Intel SCH USB" rev 0x07: apic 1 int 19 (irq 5) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 pcib0 at pci0 dev 31 function 0 "Intel SCH LPC" rev 0x07 pciide0 at pci0 dev 31 function 1 "Intel SCH IDE" rev 0x07: DMA, channel 0 wired to compatibility wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFH-016G> wd0: 1-sector PIO, LBA48, 15272MB, 31277232 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 4 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 cannot support dma lance devices pcppi0 at isa0 port 0x61 spkr0 at pcppi0 it0 at isa0 port 0x2e/2: IT8712F rev 8, EC port 0xa10 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 mtrr: Pentium Pro MTRR support rd0: fixed, 7480 blocks softraid0 at root root on rd0a swap on rd0b dump on rd0b
