On Nov 10, 2009, at 2:31 PM, Michael wrote:
> Hi,
>
> when using softraid crypto with OpenBSD 4.6-current I never get more
> than ~10-11 MB/s disk writing speed even though the disk (WD Raptor 73
> GB) itself, without crypto, can do way more.
>
> What I see during transfer in top/systat is a high interrupt load,
> however the interrupt load when writing to a not encrypted partition is
> even higher and the speed faster, so that doesn't seem to be the issue.
>
> The crypto system process is shown as mostly bored in top. So, I am
> wondering why softraid crypto never exceeds ~10-11 MB/s for me?
>
> Ideas are welcome. It would also be nice if someone could check the
> speed with their own softraid crypto setups.
>
> To test the speed I downloaded a huge file (~3,5 GB) using FTP over gbit
> ethernet.
>
> Write performance measured:
>
> to mfs partition: ~40 MB/s
> to ffs partition: ~40 MB/s (doesn't matter if using softdep or not)
> to softraid crypto partition: ~11 MB/s (+- 1 MB/s)
>
> Those 40 MB/s are limited due to the other systems read performance.
> However, softraid crypto seems (unreasonably ?) slow to me.
>
>
> Michael
>
long reply follows:
I've been running softraid(4) for while and recently added the crypto
discipline to my test configuration and find the performance acceptable
but wanted numbers so I ran some tests and found that while my
throughput is better than yours, it isn't huge multiples of
better.
And yet I don't think my experience demonstrate huge performance
issues with softraid(4)'s crypto discipline. I say that because in
addition to testing crypto performance I also tested my server's
encryption performance to see whether my max write speeds were
siginificantly worse than the speed the server can encrypt data.
In summary, writing to a file using softdep or asynchronous was about
20 MB/s or approximately twice your reported speed.
The server-encryption tests results ranged from 25 MB/s to 45 MB/s
depending on whether I used /dev/zero, /dev/arandom or a raw device
as the input. Writing to a file using softdep or asynchronous mount
options gave results ranging from a 57% to 68% of theoretical. And note
the crypto device is sitting on a softraid(4) mirror device on a system
running 4.5.
So while less than system max leaving some room for improvement, based on
the system encryption peformance test, and assuming I constructed
encryption peformance test correctly, it looks like my server's softraid
crypto performance is pretty good, especially when taking into account
that softraid(4) and the crypto discpline are very new to OpenBSD.
One curious result is that my read speeds are lower than write. I was
expecting reads to be a lot faster since the crypto devices are
sitting on a softraid(4) mirror.
Assuming my system-encryption performance test is well-constructed,
perhaps you could run it on your system and see whether you're getting
similar results.
I'd appreciate feedback anyone has about the test setup, especially the
system-encryption test. I used openssl (see below) with the same data
inputs as the crypto devices.
Summary and details of the tests and results below.
--Aaron
------------------------------
System Summary:
OpenBSD 4.5
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ 2.01 GHz
1 GB RAM
Tests Summary:
# dd if=/dev/zero bs=1m count=100 | openssl enc -aes-128-cbc \
-salt [1] > /dev/null
# dd if=/dev/[2] of=[3] bs=1m count=100
[1] softraid crypto key
[2] One of zero | arandom | rwd0a
[3] One of /mnt/backup3/speed_test | /dev/rsd7a
Results Summary:
1) Openssl sytem-encryption test:
Source MB/s
/dev/zero 44.47
/dev/arandom 25.05
/dev/rwd0a 34.18
Write Tests:
2) Asynchronous disk write:
Source MB/s % of System Test
/dev/zero 21.43 48
/dev/arandom 14.84 59
/dev/rwd0a 19.44 57
3) Softdep disk write:
Source MB/s % of System Test
/dev/zero 20.85 47
/dev/arandom 14.74 59
/dev/rwd0a 23.2 57
4) Synchronous disk write:
Source MB/s % of System Test
/dev/zero 5.8 13
/dev/arandom 5.07 20
/dev/rwd0a 5.35 16
5) Raw device write:
Source MB/s % of System Test
/dev/zero 19.59 44
/dev/arandom 14.32 57
/dev/rwd0a 16.55 48
6) Read Tests:
Source MB/s
Mounted Read 15.12
Raw Read 15.72
------------------------------
Tests and Configuration:
I tried several configuration for the hardware tests: write to the raw
device and write to a file on a mounted drive using softdep, asynchronous,
synchronous and raw. The write tests used data from /dev/zero,
/dev/arandom the IDE system boot disk. The write tests are probably
overkill but it didn't take much longer to run them and I was curious.
Read test were performed from a file on the mounted drive and raw device.
The drives are internal SATA II with 32MB of cache spinning at 7200
RPM. The chipset on the motherboard supports the full 3Gb/s speed of
SATA II. The server is low to moderately taxed.
The softraid configuration is below. In summary, the crypto device
is sitting on a softraid mirror.
My configuration:
# bioctl -i softraid0
Volume Status Size Device
softraid0 0 Online 1000204812800 sd4 RAID1
0 Online 1000204812800 0:0.0 noencl <wd5a>
1 Online 1000204812800 0:1.0 noencl <wd6a>
0 Online 500101079552 2:0.0 noencl <sd4a>
softraid0 3 Online 500101079552 sd7 CRYPTO
0 Online 500101079552 3:0.0 noencl <sd4d>
The encryption key is 126 bits.
------------------------------
Results Details:
Tests:
1) System encryption -- /dev/zero multiple runs
# dd if=/dev/zero bs=1m count=100 | openssl enc -aes-128-cbc \
-salt [1] > /dev/null
1.1)
Enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
100+0 records in
100+0 records out
104857600 bytes transferred in 2.357 secs (44471399 bytes/sec)
1.2)
System encryption -- disk device /dev/arandom
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
100+0 records in
100+0 records out
104857600 bytes transferred in 4.186 secs (25047553 bytes/sec)
1.3)
System encryption -- disk device /dev/rwd0a
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
100+0 records in
100+0 records out
104857600 bytes transferred in 3.068 secs (34176365 bytes/sec)
[1] softraid crypto key
------------------------------
2) Mounted crypto drive -- asynchronous mount:
# mount
/dev/sd7a on /mnt/backup3 type ffs (asynchronous, local, nodev, nosuid)
2.1)
# dd if=/dev/zero of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 4.891 secs (21434711 Bytes/sec)
2.2)
# dd if=/dev/arandom of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 7.067 secs (14837329 bytes/sec)
2.3)
# dd if=/dev/rwd0a of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 5.394 secs (19437367 bytes/sec)
------------------------------
3) Mounted crypto drive -- softdep mount:
# mount
/dev/sd7a on /mnt/backup3 type ffs (local, nodev, nosuid, softdep)
3.1)
# dd if=/dev/zero of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 5.030 secs (20845712 bytes/sec)
3.2)
# dd if=/dev/arandom of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 7.112 secs (14743141 bytes/sec)
3.4)
# dd if=/dev/rwd0a of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 4.520 secs (23198276 bytes/sec)
------------------------------
4) Mounted crypto drive -- synchronous mount:
# mount
/dev/sd7a on /mnt/backup3 type ffs (local, nodev, nosuid, synchronous)
4.1)
# dd if=/dev/zero of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 18.077 secs (5800322 bytes/sec)
4.2)
# dd if=/dev/arandom of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 20.663 secs (5074583 bytes/sec)
4.3)
# dd if=/dev/rwd0a of=/mnt/backup3/speed_test bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 19.600 secs (5349676 bytes/sec)
------------------------------
5) Raw crypto device:
5.1)
# dd if=/dev/zero of=/dev/rsd7a bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 5.353 secs (19586419 bytes/sec)
5.2)
# dd if=/dev/arandom of=/dev/rsd7a bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 7.324 secs (14315134 bytes/sec)
5.3)
# dd if=/dev/rwd0a of=/dev/rsd7a bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 6.335 secs (16550757 bytes/sec)
------------------------------
6) Read Tests:
6.1) Mounted crypto drive:
# dd if=/mnt/backup3/speed_test of=/dev/null
204800+0 records in
204800+0 records out
104857600 bytes transferred in 6.936 secs (15115871 bytes/sec)
6.2) Raw crypto device:
# dd if=/dev/rsd7a of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 6.671 secs (15716698 bytes/sec)
------------------------------
OpenBSD 4.5-beta (GENERIC) #1678: Wed Feb 11 10:49:54 MST 2009
t...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ ("AuthenticAMD"
686-class, 512KB L2 cache) 2.01 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16
real mem = 1072193536 (1022MB)
avail mem = 1028501504 (980MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 11/29/06, BIOS32 rev. 0 @ 0xf1fc0,
SMBIOS rev. 2.4 @ 0xf0000 (67 entries)
bios0: vendor Phoenix Technologies, LTD version "ASUS M2NPV-VM ACPI BIOS
Revision 0603" date 11/29/2006
bios0: ASUSTek Computer INC. M2NPV-VM
acpi0 at bios0: rev 2
acpi0: tables DSDT FACP HPET MCFG APIC
acpi0: wakeup devices HUB0(S5) XVRA(S5) XVRB(S5) XVRC(S5) USB0(S4) USB2(S4)
AZAD(S5) MMAC(S5) MMCI(S5) UAR1(S5) UAR2(S5) PS2M(S4) PS2K(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 25000000 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (HUB0)
acpicpu0 at acpi0
acpitz0 at acpi0: critical temperature 75 degC
acpibtn0 at acpi0: PWRB
bios0: ROM list: 0xc0000/0x10000 0xd0000/0x4000! 0xd4000/0x4800
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
"NVIDIA C51 Host" rev 0xa2 at pci0 dev 0 function 0 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 1 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 2 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 3 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 4 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 5 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 6 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 7 not configured
ppb0 at pci0 dev 2 function 0 "NVIDIA C51 PCIE" rev 0xa1
pci1 at ppb0 bus 1
ppb1 at pci0 dev 3 function 0 "NVIDIA C51 PCIE" rev 0xa1
pci2 at ppb1 bus 2
ppb2 at pci0 dev 4 function 0 "NVIDIA C51 PCIE" rev 0xa1
pci3 at ppb2 bus 3
vga1 at pci3 dev 0 function 0 "ATI Radeon X300" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 2 int 16 (irq 11)
drm0 at radeondrm0
"ATI Radeon X300 Sec" rev 0x00 at pci3 dev 0 function 1 not configured
"NVIDIA MCP51 Host" rev 0xa2 at pci0 dev 9 function 0 not configured
pcib0 at pci0 dev 10 function 0 "NVIDIA MCP51 ISA" rev 0xa3
nviic0 at pci0 dev 10 function 1 "NVIDIA MCP51 SMBus" rev 0xa3
iic0 at nviic0
spdmem0 at iic0 addr 0x50: 512MB DDR2 SDRAM non-parity PC2-6400CL3
spdmem1 at iic0 addr 0x51: 512MB DDR2 SDRAM non-parity PC2-6400CL3
iic1 at nviic0
"NVIDIA MCP51 Memory" rev 0xa3 at pci0 dev 10 function 2 not configured
ohci0 at pci0 dev 11 function 0 "NVIDIA MCP51 USB" rev 0xa3: apic 2 int 20
(irq 5), version 1.0, legacy support
ehci0 at pci0 dev 11 function 1 "NVIDIA MCP51 USB" rev 0xa3: apic 2 int 20
(irq 10)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 13 function 0 "NVIDIA MCP51 IDE" rev 0xa1: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: <Maxtor 6L250R0>
wd0: 16-sector PIO, LBA48, 239372MB, 490234752 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets, initiator 7
cd0 at scsibus0 targ 0 lun 0: <SONY, DVD RW DRU-510A, 1.0a> ATAPI 5/cdrom
removable
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 14 function 0 "NVIDIA MCP51 SATA" rev 0xa1: DMA
pciide1: using apic 2 int 20 (irq 5) for native-PCI interrupt
wd1 at pciide1 channel 0 drive 0: <WDC WD10EACS-00ZJB0>
wd1: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
wd2 at pciide1 channel 1 drive 0: <SAMSUNG HD103UJ>
wd2: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd2(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5
pciide2 at pci0 dev 15 function 0 "NVIDIA MCP51 SATA" rev 0xa1: DMA
pciide2: using apic 2 int 20 (irq 5) for native-PCI interrupt
wd3 at pciide2 channel 0 drive 0: <ST3500641AS>
wd3: 16-sector PIO, LBA48, 476938MB, 976771055 sectors
wd3(pciide2:0:0): using PIO mode 4, Ultra-DMA mode 5
wd4 at pciide2 channel 1 drive 0: <ST3500641AS>
wd4: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd4(pciide2:1:0): using PIO mode 4, Ultra-DMA mode 5
ppb3 at pci0 dev 16 function 0 "NVIDIA MCP51 PCI-PCI" rev 0xa2
pci4 at ppb3 bus 4
skc0 at pci4 dev 8 function 0 "D-Link Systems DGE-530T B1" rev 0x11, Yukon
Lite (0x9): apic 2 int 16 (irq 7)
sk0 at skc0 port A: address 00:17:9a:84:62:9c
eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 5
pciide3 at pci4 dev 9 function 0 "CMD Technology SiI3512 SATA" rev 0x01: DMA
pciide3: using apic 2 int 17 (irq 5) for native-PCI interrupt
pciide3: port 0: device present, speed: 1.5Gb/s
wd5 at pciide3 channel 0 drive 0: <ST3500641AS>
wd5: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd5(pciide3:0:0): using BIOS timings, Ultra-DMA mode 6
pciide3: port 1: device present, speed: 1.5Gb/s
wd6 at pciide3 channel 1 drive 0: <ST3500641AS>
wd6: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd6(pciide3:1:0): using BIOS timings, Ultra-DMA mode 6
pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev 0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00
kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00: core rev
BH-F2
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
it0 at isa0 port 0x2e/2: IT8716F rev 0, EC port 0x290
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
umass0 at uhub0 port 7 configuration 1 interface 0 "Atech Flash PRO-Gear
XM-4U" rev 2.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <USB2.0, CardReader CF, 0100> SCSI0 0/direct
removable
sd0: drive offline
sd1 at scsibus1 targ 1 lun 1: <USB2.0, CardReader SM XD, 0100> SCSI0 0/direct
removable
sd1: drive offline
sd2 at scsibus1 targ 1 lun 2: <USB2.0, CardReader MS, 0100> SCSI0 0/direct
removable
sd2: drive offline
sd3 at scsibus1 targ 1 lun 3: <USB2.0, CardReader SD, 0100> SCSI0 0/direct
removable
sd3: drive offline
softraid0 at root
softraid0: roaming device wd5a -> wd1a
softraid0: roaming device wd6a -> wd2a
softraid0: roaming device wd1a -> wd3a
softraid0: roaming device wd3a -> wd4a
root on wd0a swap on wd0b dump on wd0b
softraid0: roaming device wd5a -> wd1a
softraid0: roaming device wd6a -> wd2a
scsibus2 at softraid0: 1 targets, initiator 1
sd4 at scsibus2 targ 0 lun 0: <OPENBSD, SR RAID 1, 003> SCSI2 0/direct fixed
sd4: 953869MB, 512 bytes/sec, 1953525026 sec total
softraid0: roaming device wd1a -> wd3a
softraid0: roaming device wd3a -> wd4a
scsibus3 at softraid0: 1 targets, initiator 1
sd5 at scsibus3 targ 0 lun 0: <OPENBSD, SR RAID 1, 003> SCSI2 0/direct fixed
sd5: 476937MB, 512 bytes/sec, 976767923 sec total
softraid0: invalid metadata format
scsibus4 at softraid0: 1 targets, initiator 1
sd6 at scsibus4 targ 0 lun 0: <OPENBSD, SR CRYPTO, 003> SCSI2 0/direct fixed
sd6: 476933MB, 512 bytes/sec, 976759922 sec total
softraid0: volume sd6 is roaming, it used to be sd8, updating metadata
scsibus5 at softraid0: 1 targets, initiator 1
sd7 at scsibus5 targ 0 lun 0: <OPENBSD, SR CRYPTO, 003> SCSI2 0/direct fixed
sd7: 476933MB, 512 bytes/sec, 976759922 sec total
softraid0: volume sd7 is roaming, it used to be sd6, updating metadata
softraid0: invalid metadata format
scsibus6 at softraid0: 1 targets, initiator 1
sd8 at scsibus6 targ 0 lun 0: <OPENBSD, SR CRYPTO, 003> SCSI2 0/direct fixed
sd8: 476937MB, 512 bytes/sec, 976767923 sec total
softraid0: volume sd8 is roaming, it used to be sd7, updating metadata
arp info overwritten for 10.10.15.104 by 00:16:cb:a7:ca:3c on sk0
arp info overwritten for 10.10.15.104 by 00:26:08:e3:05:f4 on sk0
umass1 at uhub0 port 5 configuration 1 interface 0 "JMicron JM20336 SATA, USB
Combo" rev 2.00/1.00 addr 3
umass1: using SCSI over Bulk-Only
scsibus7 at umass1: 2 targets, initiator 0
sd9 at scsibus7 targ 1 lun 0: <ST375064, 3QD0, 3.AA> SCSI2 0/direct fixed
sd9: 715404MB, 512 bytes/sec, 1465149168 sec total
sd10 at scsibus7 targ 1 lun 1: <ST375064, 5QD2, 3.AA> SCSI2 0/direct fixed
sd10: 715404MB, 512 bytes/sec, 1465149168 sec total
