Hi,
Initially comparing I/O speed between FreeBSD/ZFS/GELI and
OpenBSD/FFS/CRYPTO, I noticed that there were a huge difference between
plain and encrypted filesystem using OpenBSD. I ran the test on a 1
vCore/1GB RAM Vultr VPS, running OpenBSD 6.2-beta. I had / configured in
plain FFS and /home encrypted using bioctl(8). Then I ran a few `dd` and
`bonnie++`
According to those tests, writing FFS/CRYPTO is about 10 times slower
than FFS/PLAIN.
For the record, using the same `dd` on FreeBSD, ZFS with GELI is only 2
times slower than plain ZFS.
Furthemore, comparing FreeBSD/ZFS/PLAIN and OpenBSD/FFS/PLAIN, the speed
is about the same.
Finally, it seems reading OpenBSD/FFS/PLAIN and OpenBSD/FFS/CRYPTO is
done at the same speed.
Is this expected to have so much difference between FFS/PLAIN and
FFS/CRYPTO when writing data?
TIA,
Jo
PS: here's my test data.
# sysctl kern.version hw.machine hw.model hw.ncpu hw.physmem
kern.version=OpenBSD 6.2-beta (GENERIC) #91: Wed Sep 13 22:05:17 MDT
2017
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
hw.machine=amd64
hw.model=Virtual CPU a7769a6388d5
hw.ncpu=1
hw.physmem=1056817152
# disklabel sd0
# /dev/rsd0c:
type: SCSI
disk: SCSI disk
label: Block Device
duid: 69939b6a66c3879a
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 3263
total sectors: 52428800
boundstart: 64
boundend: 52420095
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 16739680 35680384 4.2BSD 2048 16384 12958 # /
b: 4208966 64 swap # none
c: 52428800 0 unused
d: 31471335 4209030 RAID
# disklabel sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: SR CRYPTO
duid: 4179a9e67beb3d4e
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 1958
total sectors: 31470807
boundstart: 64
boundend: 31455270
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
c: 31470807 0 unused
e: 273024 64 4.2BSD 2048 16384 2133 # /etc
h: 31182176 273088 4.2BSD 2048 16384 12958 #
/home
# mount
/dev/sd0a on / type ffs (local, wxallowed)
/dev/sd1e on /etc type ffs (local, softdep)
/dev/sd1h on /home type ffs (local, nodev, nosuid)
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 7.9G 915M 6.6G 12% /
/dev/sd1e 131M 4.9M 120M 4% /etc
/dev/sd1h 14.6G 2.0K 13.9G 0% /home
# sync && time dd if=/dev/zero of=/TEST bs=512 count=3000000 && sync
3000000+0 records in
3000000+0 records out
1536000000 bytes transferred in 8.567 secs (179278802 bytes/sec)
0m08.61s real 0m00.29s user 0m07.70s system
# sync && time dd if=/dev/zero of=/home/TEST bs=512 count=3000000 &&
sync
3000000+0 records in
3000000+0 records out
1536000000 bytes transferred in 20.875 secs (73580525 bytes/sec)
0m20.88s real 0m00.42s user 0m05.54s system
# sync && time dd if=/dev/zero of=/TEST bs=4k count=300000 && sync
300000+0 records in
300000+0 records out
1228800000 bytes transferred in 4.151 secs (296024071 bytes/sec)
0m04.19s real 0m00.04s user 0m04.01s system
# sync && time dd if=/dev/zero of=/home/TEST bs=4k count=300000 && sync
300000+0 records in
300000+0 records out
1228800000 bytes transferred in 22.872 secs (53723676 bytes/sec)
0m22.95s real 0m00.06s user 0m01.89s system
# sync && time dd if=/dev/zero of=/TEST bs=8k count=150000 && sync
150000+0 records in
150000+0 records out
1228800000 bytes transferred in 4.088 secs (300571699 bytes/sec)
0m04.12s real 0m00.05s user 0m03.93s system
# sync && time dd if=/dev/zero of=/home/TEST bs=8k count=150000 && sync
150000+0 records in
150000+0 records out
1228800000 bytes transferred in 21.418 secs (57372236 bytes/sec)
0m21.48s real 0m00.05s user 0m01.72s system
# time dd if=/TEST of=/dev/null
2400000+0 records in
2400000+0 records out
1228800000 bytes transferred in 12.327 secs (99677812 bytes/sec)
0m12.33s real 0m00.39s user 0m03.62s system
# time dd if=/home/TEST of=/dev/null
2400000+0 records in
2400000+0 records out
1228800000 bytes transferred in 12.802 secs (95979204 bytes/sec)
0m12.80s real 0m00.29s user 0m02.87s system
# time dd if=/TEST of=/dev/null bs=512
2400000+0 records in
2400000+0 records out
1228800000 bytes transferred in 12.888 secs (95337724 bytes/sec)
0m12.89s real 0m00.29s user 0m03.41s system
# time dd if=/home/TEST of=/dev/null bs=512
2400000+0 records in
2400000+0 records out
1228800000 bytes transferred in 13.951 secs (88076531 bytes/sec)
0m13.95s real 0m00.24s user 0m02.61s system
# time dd if=/TEST of=/dev/null bs=4k
300000+0 records in
300000+0 records out
1228800000 bytes transferred in 11.478 secs (107052456 bytes/sec)
0m11.48s real 0m00.07s user 0m01.91s system
# time dd if=/home/TEST of=/dev/null bs=4k
300000+0 records in
300000+0 records out
1228800000 bytes transferred in 12.554 secs (97877823 bytes/sec)
0m12.55s real 0m00.06s user 0m01.80s system
# time dd if=/TEST of=/dev/null bs=8k
150000+0 records in
150000+0 records out
1228800000 bytes transferred in 11.956 secs (102775667 bytes/sec)
0m11.95s real 0m00.02s user 0m01.87s system
# time dd if=/home/TEST of=/dev/null bs=8k
150000+0 records in
150000+0 records out
1228800000 bytes transferred in 12.721 secs (96590791 bytes/sec)
0m12.72s real 0m00.06s user 0m01.66s system
# bonnie++ -q -u www -d /tmp > /root/bonnie-PLAIN.csv
Version 1.97 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
cherie.tumfatig. 2G 1693 99 179217 93 60657 34 2314 98 92028 27
1390 118
Latency 7876us 4947us 9914us 6272us 5084us
41171us
Version 1.97 ------Sequential Create------ --------Random
Create--------
cherie.tumfatig.net -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1400 10 +++++ +++ 4976 21 1372 11 +++++ +++
6331 34
Latency 7572us 349us 4004us 18817us 86us
2868us
# bonnie++ -q -u www -d /home/www > /root/bonnie-CRYPTO.csv
Version 1.97 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
cherie.tumfatig. 2G 1668 99 52340 13 18797 16 2420 96 85558 20
1037 84
Latency 7189us 91008us 17138us 4562us 6496us
82148us
Version 1.97 ------Sequential Create------ --------Random
Create--------
cherie.tumfatig.net -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 425 7 +++++ +++ 455 4 423 7 +++++ +++
1750 16
Latency 10694us 859us 25427us 12505us 127us
10133us
# dmesg
OpenBSD 6.2-beta (GENERIC) #91: Wed Sep 13 22:05:17 MDT 2017
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056817152 (1007MB)
avail mem = 1017909248 (970MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf68f0 (9 entries)
bios0: vendor SeaBIOS version
"rel-1.10.1-0-g8891697-prebuilt.qemu-project.org" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Virtual CPU a7769a6388d5, 2400.42 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,ABM,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
"PNP0F13" at acpi0 not configured
"PNP0700" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> ATAPI 5/cdrom
removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int
11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0
int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 56:00:00:8a:f3:5a
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct
fixed
sd0: 25600MB, 512 bytes/sector, 52428800 sectors
virtio1: msix shared
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 10
virtio3 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio3
virtio3: apic 0 int 10
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev
1.00/1.00 addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB
Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (69939b6a66c3879a.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown
sd1 at scsibus4 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct
fixed
sd1: 15366MB, 512 bytes/sector, 31470807 sectors
