Hi I need to figure if I can improve isakmpd / ipsec performance in my setup on openbsd -current
I have two boxes connected to each other via 1Gbit link and I'm using iperf to test performance with default ipsec.conf between these two servers: # cat ipsec.conf: ike esp from aaa.aaa.aaa.118 to aaa.aaa.aaa.98 # ipsecctl -sa FLOWS: flow esp in from aaa.aaa.aaa.98 to aaa.aaa.aaa.118 peer aaa.aaa.aaa.98 srcid aaa.aaa.aaa.118/32 dstid aaa.aaa.aaa.98/32 type use flow esp out from aaa.aaa.aaa.118 to aaa.aaa.aaa.98 peer aaa.aaa.aaa.98 srcid aaa.aaa.aaa.118/32 dstid aaa.aaa.aaa.98/32 type require SAD: esp tunnel from aaa.aaa.aaa.98 to aaa.aaa.aaa.118 spi 0x0a04680a auth hmac-sha2-256 enc aes esp tunnel from aaa.aaa.aaa.118 to aaa.aaa.aaa.98 spi 0x4b4f91bb auth hmac-sha2-256 enc aes BOX1 dmesg: # dmesg OpenBSD 5.4-beta (GENERIC.MP) #27: Fri Jul 12 10:35:54 MDT 2013 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 34333917184 (32743MB) avail mem = 33412231168 (31864MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0x9a800 (90 entries) bios0: vendor HP version "O33" date 07/06/2011 bios0: HP ProLiant DL160 G6 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR MCFG SPMI SLIC BOOT OEMB HPET SRAT DMAR SSDT EINJ BERT ERST HEST acpi0: wakeup devices NPE1(S4) NPE2(S4) NPE3(S4) NPE4(S4) NPE5(S4) NPE6(S4) NPE7(S4) NPE8(S4) NPE9(S4) NPEA(S4) P0P1(S4) USB0(S4) USB1(S4) USB2(S4) USB5(S4) EUSB(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.45 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 cpu0: apic clock running at 133MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 18 (application processor) cpu2: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 9, package 0 cpu3 at mainbus0: apid 20 (application processor) cpu3: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 10, package 0 cpu4 at mainbus0: apid 32 (application processor) cpu4: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu4: 256KB 64b/line 8-way L2 cache cpu4: smt 0, core 0, package 1 cpu5 at mainbus0: apid 34 (application processor) cpu5: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu5: 256KB 64b/line 8-way L2 cache cpu5: smt 0, core 1, package 1 cpu6 at mainbus0: apid 50 (application processor) cpu6: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu6: 256KB 64b/line 8-way L2 cache cpu6: smt 0, core 9, package 1 cpu7 at mainbus0: apid 52 (application processor) cpu7: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu7: 256KB 64b/line 8-way L2 cache cpu7: smt 0, core 10, package 1 cpu8 at mainbus0: apid 1 (application processor) cpu8: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu8: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu8: 256KB 64b/line 8-way L2 cache cpu8: smt 1, core 0, package 0 cpu9 at mainbus0: apid 3 (application processor) cpu9: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.08 MHz cpu9: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu9: 256KB 64b/line 8-way L2 cache cpu9: smt 1, core 1, package 0 cpu10 at mainbus0: apid 19 (application processor) cpu10: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.08 MHz cpu10: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu10: 256KB 64b/line 8-way L2 cache cpu10: smt 1, core 9, package 0 cpu11 at mainbus0: apid 21 (application processor) cpu11: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.08 MHz cpu11: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu11: 256KB 64b/line 8-way L2 cache cpu11: smt 1, core 10, package 0 cpu12 at mainbus0: apid 33 (application processor) cpu12: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu12: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu12: 256KB 64b/line 8-way L2 cache cpu12: smt 1, core 0, package 1 cpu13 at mainbus0: apid 35 (application processor) cpu13: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.09 MHz cpu13: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu13: 256KB 64b/line 8-way L2 cache cpu13: smt 1, core 1, package 1 cpu14 at mainbus0: apid 51 (application processor) cpu14: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.08 MHz cpu14: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu14: 256KB 64b/line 8-way L2 cache cpu14: smt 1, core 9, package 1 cpu15 at mainbus0: apid 53 (application processor) cpu15: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2400.08 MHz cpu15: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG, LAHF,PERF,ITSC cpu15: 256KB 64b/line 8-way L2 cache cpu15: smt 1, core 10, package 1 ioapic0 at mainbus0: apid 6 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 4, remapped to apid 6 ioapic1 at mainbus0: apid 7 pa 0xfec8a000, version 20, 24 pins ioapic1: misconfigured as apic 6, remapped to apid 7 acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 8 (NPE1) acpiprt2 at acpi0: bus -1 (NPE2) acpiprt3 at acpi0: bus 7 (NPE3) acpiprt4 at acpi0: bus -1 (NPE4) acpiprt5 at acpi0: bus -1 (NPE5) acpiprt6 at acpi0: bus -1 (NPE6) acpiprt7 at acpi0: bus -1 (NPE8) acpiprt8 at acpi0: bus 5 (NPE9) acpiprt9 at acpi0: bus 4 (NPEA) acpiprt10 at acpi0: bus 3 (P0P4) acpiprt11 at acpi0: bus 2 (P0P8) acpicpu0 at acpi0: C3, C3, C1, PSS acpicpu1 at acpi0: C3, C3, C1, PSS acpicpu2 at acpi0: C3, C3, C1, PSS acpicpu3 at acpi0: C3, C3, C1, PSS acpicpu4 at acpi0: C3, C3, C1, PSS acpicpu5 at acpi0: C3, C3, C1, PSS acpicpu6 at acpi0: C3, C3, C1, PSS acpicpu7 at acpi0: C3, C3, C1, PSS acpicpu8 at acpi0: C3, C3, C1, PSS acpicpu9 at acpi0: C3, C3, C1, PSS acpicpu10 at acpi0: C3, C3, C1, PSS acpicpu11 at acpi0: C3, C3, C1, PSS acpicpu12 at acpi0: C3, C3, C1, PSS acpicpu13 at acpi0: C3, C3, C1, PSS acpicpu14 at acpi0: C3, C3, C1, PSS acpicpu15 at acpi0: C3, C3, C1, PSS acpibtn0 at acpi0: SLPB acpibtn1 at acpi0: PWRB ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2400 MHz: speeds: 2400, 2267, 2133, 2000, 1867, 1733, 1600 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 5520 Host" rev 0x13 ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13: msi pci1 at ppb0 bus 8 ppb1 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13: msi pci2 at ppb1 bus 7 "Emulex LPe12000" rev 0x03 at pci2 dev 0 function 0 not configured ppb2 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13: msi pci3 at ppb2 bus 6 ppb3 at pci0 dev 9 function 0 "Intel X58 PCIE" rev 0x13: msi pci4 at ppb3 bus 5 em0 at pci4 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 9c:8e:99:2e:0c:c0 em1 at pci4 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 9c:8e:99:2e:0c:c1 ppb4 at pci0 dev 10 function 0 "Intel X58 PCIE" rev 0x13: msi pci5 at ppb4 bus 4 ciss0 at pci5 dev 0 function 0 "Hewlett-Packard Smart Array" rev 0x01: apic 7 int 9 ciss0: 1 LD, HW rev 2, FW 5.14/5.14, 64bit fifo rro scsibus0 at ciss0: 1 targets sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 5.14> SCSI3 0/direct fixed sd0: 858210MB, 512 bytes/sector, 1757615708 sectors "Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured "Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured "Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured uhci0 at pci0 dev 26 function 0 "Intel 82801JI USB" rev 0x00: apic 6 int 16 ehci0 at pci0 dev 26 function 7 "Intel 82801JI USB" rev 0x00: apic 6 int 18 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb5 at pci0 dev 28 function 0 "Intel 82801JI PCIE" rev 0x00: msi pci6 at ppb5 bus 3 ppb6 at pci0 dev 28 function 4 "Intel 82801JI PCIE" rev 0x00: msi pci7 at ppb6 bus 2 vga1 at pci7 dev 0 function 0 "Matrox MGA G200e" rev 0x02 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) uhci1 at pci0 dev 29 function 0 "Intel 82801JI USB" rev 0x00: apic 6 int 23 uhci2 at pci0 dev 29 function 1 "Intel 82801JI USB" rev 0x00: apic 6 int 19 uhci3 at pci0 dev 29 function 2 "Intel 82801JI USB" rev 0x00: apic 6 int 18 ehci1 at pci0 dev 29 function 7 "Intel 82801JI USB" rev 0x00: apic 6 int 23 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb7 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x90 pci8 at ppb7 bus 1 pcib0 at pci0 dev 31 function 0 "Intel 82801JIR LPC" rev 0x00 ahci0 at pci0 dev 31 function 2 "Intel 82801JI AHCI" rev 0x00: msi, AHCI 1.2 scsibus1 at ahci0: 32 targets usb2 at uhci0: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 mtrr: Pentium Pro MTRR support vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on sd0a (943fddcea853685b.a) swap on sd0b dump on sd0b BOX2 dmesg: # dmesg OpenBSD 5.4-beta (GENERIC.MP) #27: Fri Jul 12 10:35:54 MDT 2013 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4242747392 (4046MB) avail mem = 4122079232 (3931MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xefffc000 (75 entries) bios0: vendor HP version "J01" date 08/10/2012 bios0: HP ProLiant DL120 G7 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP SPCR MCFG HPET FFFF SPMI ERST APIC FFFF BERT HEST DMAR SSDT SSDT SSDT acpi0: wakeup devices PCI0(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimcfg0 at acpi0 addr 0xf0000000, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.98 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 cpu0: apic clock running at 99MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu4: 256KB 64b/line 8-way L2 cache cpu4: smt 1, core 0, package 0 cpu5 at mainbus0: apid 3 (application processor) cpu5: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu5: 256KB 64b/line 8-way L2 cache cpu5: smt 1, core 1, package 0 cpu6 at mainbus0: apid 5 (application processor) cpu6: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu6: 256KB 64b/line 8-way L2 cache cpu6: smt 1, core 2, package 0 cpu7 at mainbus0: apid 7 (application processor) cpu7: Intel(R) Xeon(R) CPU E31240 @ 3.30GHz, 3292.53 MHz cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX ,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,A ES,XSAVE,AVX,NXE,LONG,LAHF,PERF,ITSC cpu7: 256KB 64b/line 8-way L2 cache cpu7: smt 1, core 3, package 0 ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 8 acpiprt0 at acpi0: bus 11 (IPT1) acpiprt1 at acpi0: bus -1 (IPT2) acpiprt2 at acpi0: bus -1 (IPT3) acpiprt3 at acpi0: bus -1 (IPT4) acpiprt4 at acpi0: bus 2 (IPT5) acpiprt5 at acpi0: bus 3 (IPT6) acpiprt6 at acpi0: bus 14 (IPT7) acpiprt7 at acpi0: bus 1 (IPT8) acpiprt8 at acpi0: bus 4 (PT02) acpiprt9 at acpi0: bus 8 (PT05) acpiprt10 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0: C1 acpicpu1 at acpi0: C1 acpicpu2 at acpi0: C1 acpicpu3 at acpi0: C1 acpicpu4 at acpi0: C1 acpicpu5 at acpi0: C1 acpicpu6 at acpi0: C1 acpitz0 at acpi0: critical temperature is 31 degC ipmi at mainbus0 not configured memory map conflict 0xede3e000/0x21c1000 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09 ppb0 at pci0 dev 1 function 0 "Intel Core 2G PCIE" rev 0x09: msi pci1 at ppb0 bus 4 ppb1 at pci1 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e pci2 at ppb1 bus 5 ppb2 at pci2 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e pci3 at ppb2 bus 6 em0 at pci3 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 90:e2:ba:29:95:a0 em1 at pci3 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 90:e2:ba:29:95:a1 ppb3 at pci2 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e pci4 at ppb3 bus 7 em2 at pci4 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 90:e2:ba:29:95:a4 em3 at pci4 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 90:e2:ba:29:95:a5 ppb4 at pci0 dev 6 function 0 "Intel Core 2G PCIE" rev 0x09: msi pci5 at ppb4 bus 8 ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x05: apic 8 int 21 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb5 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb5 pci6 at ppb5 bus 11 ppb6 at pci0 dev 28 function 4 "Intel 6 Series PCIE" rev 0xb5 pci7 at ppb6 bus 2 em4 at pci7 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address 28:92:4a:2d:23:4c ppb7 at pci0 dev 28 function 5 "Intel 6 Series PCIE" rev 0xb5 pci8 at ppb7 bus 3 em5 at pci8 dev 0 function 0 "Intel 82574L" rev 0x00: msi, address 28:92:4a:2d:23:4d ppb8 at pci0 dev 28 function 6 "Intel 6 Series PCIE" rev 0xb5 pci9 at ppb8 bus 14 ppb9 at pci0 dev 28 function 7 "Intel 6 Series PCIE" rev 0xb5 pci10 at ppb9 bus 1 "Hewlett-Packard iLO3 Slave" rev 0x05 at pci10 dev 0 function 0 not configured vga1 at pci10 dev 0 function 1 vendor "Matrox", unknown product 0x0533 rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) "Hewlett-Packard iLO3 Management" rev 0x05 at pci10 dev 0 function 2 not configured uhci0 at pci10 dev 0 function 4 "Hewlett-Packard USB" rev 0x02: apic 8 int 16 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1 ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x05: apic 8 int 20 usb2 at ehci1: USB revision 2.0 uhub2 at usb2 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb10 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5 pci11 at ppb10 bus 20 pcib0 at pci0 dev 31 function 0 "Intel C204 LPC" rev 0x05 pciide0 at pci0 dev 31 function 2 "Intel 6 Series SATA" rev 0x05: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide0: using apic 8 int 17 for native-PCI interrupt wd0 at pciide0 channel 0 drive 0: <VB0250EAVER> wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 pciide1 at pci0 dev 31 function 5 "Intel 6 Series SATA" rev 0x05: DMA, channel 0 wired to native-PCI, channel 1 wired to native-PCI pciide1: using apic 8 int 17 for native-PCI interrupt isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo com1: probed fifo depth: 0 bytes pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) pckbc0: using irq 12 for aux slot wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 mtrr: Pentium Pro MTRR support uhub3 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2 uhidev0 at uhub3 port 3 configuration 1 interface 0 "Logitech USB Keyboard" rev 1.10/64.00 addr 3 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 variable keys, 6 key codes wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 uhidev1 at uhub3 port 3 configuration 1 interface 1 "Logitech USB Keyboard" rev 1.10/64.00 addr 3 uhidev1: iclass 3/0, 3 report ids uhid0 at uhidev1 reportid 1: input=1, output=0, feature=0 uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0 uhid2 at uhidev1 reportid 3: input=3, output=0, feature=0 umodem0 at uhub3 port 4 configuration 1 interface 0 "Cisco Cisco USB Console" rev 2.00/0.00 addr 4 umodem0: data interface 1, has no CM over data, has no break umodem0: status change notification available ucom0 at umodem0 uhub4 at uhub2 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2 uhidev2 at uhub1 port 1 configuration 1 interface 0 "HP Virtual Keyboard" rev 1.10/0.02 addr 2 uhidev2: iclass 3/1 ukbd1 at uhidev2: 8 variable keys, 6 key codes, country code 33 wskbd2 at ukbd1 mux 1 wskbd2: connecting to wsdisplay0 uhidev3 at uhub1 port 1 configuration 1 interface 1 "HP Virtual Keyboard" rev 1.10/0.02 addr 2 uhidev3: iclass 3/1 ums0 at uhidev3: 3 buttons wsmouse1 at ums0 mux 0 vscsi0 at root scsibus0 at vscsi0: 256 targets softraid0 at root scsibus1 at softraid0: 256 targets root on wd0a (09ef9ad438d5cb88.a) swap on wd0b dump on wd0b I'm getting ~ 270 Mbits/sec over encrypted tunnel and ~ 920 Mbits/sec over unencrypted. Any luck to get more? Where are bottlenecks there? During iperf tests I see that both servers CPU usage is low and iperf is only CPU consumer: load averages: 0.91, 0.36, 0.39 gateway 15:55:06 29 processes: 28 idle, 1 on processor All CPUs: 0.0% user, 0.0% nice, 7.8% system, 5.8% interrupt, 86.3% idle Memory: Real: 19M/95M act/tot Free: 31G Cache: 34M Swap: 0K/32G PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND 7243 root 28 0 1696K 1736K idle - 6:45 33.30% iperf 19984 ejectevg 28 0 1132K 2132K onproc/3 - 0:02 1.46% top 23492 root 2 0 1692K 716K idle - 346:47 0.00% isakmpd Thank you in advance! -- -- With regards, Eugene Sudyr