On Fri, 8 Feb 2013, Scott McEachern wrote:
> I get a rather curious error when shutting down a machine with a RAID 1
> setup that contains a crypto partition and a "normal" partition:
>
> syncing disks... done
> sd3 detached
> softraid0: I/O error 5 on dev 0x433 at block 16
> softraid0: could not write metadata to sd3d
> sd4 detached
> rebooting...
>
> When the machine starts up again, I see:
> softraid0: sd4 was not shutdown properly (twice)
>
> The funny thing is that everything works just fine. No problems
> whatsoever.
>
> That said, the error message is more than a little unsettling and I'm
> worried that one day I'm going to access files on the crypto volume only
> to find the data isn't available.
>
> Any hints on what I did wrong when setting this up?
While stacked softraid volumes generally work, they are not officially
supported (for a variety of reasons). The problem that you mention above is
due to the way that softraid volumes are shutdown - the shutdown order is
approximately the same as the order they are created. In your case this means
that sd3 gets shutdown before sd4, hence sd4 is unable to write metadata to
sd3. For the time being, in order to avoid the issue you should disassemble
the CRYPTO volume (sd4) before the RAID 1 volume (sd3).
> Details:
>
> The RAID volume, sd3, is made up from two 3TB as sd0 and sd1:
>
> # disklabel -pg sd0
> # /dev/rsd0c:
> type: SCSI
> disk: SCSI disk
> label: ST3000DM001-1CH1
> duid: ec7586498efe98b8
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 364801
> total sectors: 5860533168 # total bytes: 2794.5G
> boundstart: 64
> boundend: 4294961685
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> a: 2794.5G 64 RAID
> c: 2794.5G 0 unused
>
> # disklabel -pg sd1
> # /dev/rsd1c:
> type: SCSI
> disk: SCSI disk
> label: ST3000DM001-1CH1
> duid: 0372b77b79b2e168
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 364801
> total sectors: 5860533168 # total bytes: 2794.5G
> boundstart: 64
> boundend: 4294961685
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> a: 2794.5G 64 RAID
> c: 2794.5G 0 unused
>
> Those two drives then become sd3: (sd2 is an SSD boot drive)
>
> # disklabel -pg sd3
> # /dev/rsd3c:
> type: SCSI
> disk: SCSI disk
> label: SR RAID 1
> duid: 6ca0888211d57886
> flags:
> bytes/sector: 512
> sectors/track: 255
> tracks/cylinder: 511
> sectors/cylinder: 130305
> cylinders: 44975
> total sectors: 5860532576 # total bytes: 2794.5G
> boundstart: 0
> boundend: 5860532576
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> a: 1397.3G 0 4.2BSD 8192 65536 1 # /st2
> c: 2794.5G 0 unused
> d: 1397.3G 2930266240 RAID
>
> from which a crypto volume was created, becoming sd4:
>
> # disklabel -pg sd4
> # /dev/rsd4c:
> type: SCSI
> disk: SCSI disk
> label: SR CRYPTO
> duid: 6c6e53ab843ef6c8
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 182400
> total sectors: 2930265808 # total bytes: 1397.3G
> boundstart: 0
> boundend: 2930265808
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> a: 1397.3G 0 4.2BSD 8192 65536 1
> c: 1397.3G 0 unused
>
>
> I'm running a snapshot from Jan 21. Full dmesg:
>
> OpenBSD 5.2-current (GENERIC.MP) #20: Mon Jan 21 17:23:23 MST 2013
> t...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 12613910528 (12029MB)
> avail mem = 12255641600 (11687MB)
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x9f400 (68 entries)
> bios0: vendor American Megatrends Inc. version "2105" date 07/23/2010
> bios0: ASUSTeK Computer INC. M4A785TD-V EVO
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S1 S3 S4 S5
> acpi0: tables DSDT FACP APIC MCFG OEMB SRAT HPET SSDT
> acpi0: wakeup devices PCE2(S4) PCE3(S4) PCE4(S4) PCE5(S4) PCE6(S4)
> PCE7(S4) PCE9(S4) PCEA(S4) PCEB(S4) PCEC(S4) SBAZ(S4) PS2M(S4) PS2K(S4)
> UAR1(S4) P0PC(S4) UHC1(S4) UHC2(S4) UHC3(S4) USB4(S4) UHC5(S4) UHC6(S4)
> UHC7(S4)
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD Phenom(tm) II X6 1100T Processor, 3300.74 MHz
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFL
>USH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,
>3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,I
>TSC cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
> cpu0: ITLB 32 4KB entries fully associative, 16 4MB entries fully
> associative
> cpu0: DTLB 48 4KB entries fully associative, 48 4MB entries fully
> associative
> cpu0: AMD erratum 721 detected and fixed
> cpu0: apic clock running at 200MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: AMD Phenom(tm) II X6 1100T Processor, 3300.25 MHz
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFL
>USH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,
>3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,I
>TSC cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
> cpu1: ITLB 32 4KB entries fully associative, 16 4MB entries fully
> associative
> cpu1: DTLB 48 4KB entries fully associative, 48 4MB entries fully
> associative
> cpu1: AMD erratum 721 detected and fixed
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: AMD Phenom(tm) II X6 1100T Processor, 3300.25 MHz
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFL
>USH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,
>3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,I
>TSC cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
> cpu2: ITLB 32 4KB entries fully associative, 16 4MB entries fully
> associative
> cpu2: DTLB 48 4KB entries fully associative, 48 4MB entries fully
> associative
> cpu2: AMD erratum 721 detected and fixed
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: AMD Phenom(tm) II X6 1100T Processor, 3300.25 MHz
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFL
>USH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,
>3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,I
>TSC cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
> cpu3: ITLB 32 4KB entries fully associative, 16 4MB entries fully
> associative
> cpu3: DTLB 48 4KB entries fully associative, 48 4MB entries fully
> associative
> cpu3: AMD erratum 721 detected and fixed
> cpu4 at mainbus0: apid 4 (application processor)
> cpu4: AMD Phenom(tm) II X6 1100T Processor, 3300.25 MHz
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFL
>USH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,
>3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,I
>TSC cpu4: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
> cpu4: ITLB 32 4KB entries fully associative, 16 4MB entries fully
> associative
> cpu4: DTLB 48 4KB entries fully associative, 48 4MB entries fully
> associative
> cpu4: AMD erratum 721 detected and fixed
> cpu5 at mainbus0: apid 5 (application processor)
> cpu5: AMD Phenom(tm) II X6 1100T Processor, 3300.25 MHz
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFL
>USH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,
>3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,I
>TSC cpu5: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB
> 64b/line 16-way L2 cache, 6MB 64b/line 48-way L3 cache
> cpu5: ITLB 32 4KB entries fully associative, 16 4MB entries fully
> associative
> cpu5: DTLB 48 4KB entries fully associative, 48 4MB entries fully
> associative
> cpu5: AMD erratum 721 detected and fixed
> ioapic0 at mainbus0: apid 6 pa 0xfec00000, version 21, 24 pins
> acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
> acpihpet0 at acpi0: 14318180 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (P0P1)
> acpiprt2 at acpi0: bus -1 (PCE2)
> acpiprt3 at acpi0: bus -1 (PCE3)
> acpiprt4 at acpi0: bus -1 (PCE4)
> acpiprt5 at acpi0: bus -1 (PCE5)
> acpiprt6 at acpi0: bus -1 (PCE6)
> acpiprt7 at acpi0: bus -1 (PCE7)
> acpiprt8 at acpi0: bus -1 (PCE9)
> acpiprt9 at acpi0: bus 2 (PCEA)
> acpiprt10 at acpi0: bus 3 (P0PC)
> acpicpu0 at acpi0: C2, PSS
> acpicpu1 at acpi0: C2, PSS
> acpicpu2 at acpi0: C2, PSS
> acpicpu3 at acpi0: C2, PSS
> acpicpu4 at acpi0: C2, PSS
> acpicpu5 at acpi0: C2, PSS
> aibs0 at acpi0: RTMP RVLT RFAN GGRP GITM SITM
> acpibtn0 at acpi0: PWRB
> cpu0: 3300 MHz: speeds: 3300 2500 1700 800 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "AMD RS880 Host" rev 0x00
> ppb0 at pci0 dev 1 function 0 vendor "Asustek", unknown product 0x9602
> rev 0x00
> pci1 at ppb0 bus 1
> vga1 at pci1 dev 5 function 0 "ATI Radeon HD 4200" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> radeondrm0 at vga1: apic 6 int 18
> drm0 at radeondrm0
> azalia0 at pci1 dev 5 function 1 "ATI Radeon HD 4200 HD Audio" rev 0x00:
> msi azalia0: no supported codecs
> ppb1 at pci0 dev 10 function 0 "AMD RS780 PCIE" rev 0x00: msi
> pci2 at ppb1 bus 2
> re0 at pci2 dev 0 function 0 "Realtek 8168" rev 0x03: RTL8168D/8111D
> (0x2800), apic 6 int 18, address 20:cf:30:b0:06:02
> rgephy0 at re0 phy 7: RTL8169S/8110S PHY, rev. 2
> ahci0 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x00: apic 6 int
> 22, AHCI 1.1
> scsibus0 at ahci0: 32 targets
> sd0 at scsibus0 targ 0 lun 0: <ATA, ST3000DM001-1CH1, CC43> SCSI3
> 0/direct fixed naa.5000c5004e455146
> sd0: 2861588MB, 512 bytes/sector, 5860533168 sectors
> sd1 at scsibus0 targ 1 lun 0: <ATA, ST3000DM001-1CH1, CC43> SCSI3
> 0/direct fixed naa.5000c5004e4a8141
> sd1: 2861588MB, 512 bytes/sector, 5860533168 sectors
> sd2 at scsibus0 targ 2 lun 0: <ATA, OCZ-VERTEX4, 1.5> SCSI3 0/direct
> fixed naa.5e83a97810e6145a
> sd2: 122104MB, 512 bytes/sector, 250069680 sectors, thin
> ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 6 int 16,
> version 1.0, legacy support
> ohci1 at pci0 dev 18 function 1 "ATI SB700 USB" rev 0x00: apic 6 int 16,
> version 1.0, legacy support
> ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 6 int 17
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "ATI EHCI root hub" rev 2.00/1.00 addr 1
> ohci2 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 6 int 18,
> version 1.0, legacy support
> ohci3 at pci0 dev 19 function 1 "ATI SB700 USB" rev 0x00: apic 6 int 18,
> version 1.0, legacy support
> ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 6 int 19
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 "ATI EHCI root hub" rev 2.00/1.00 addr 1
> piixpm0 at pci0 dev 20 function 0 "ATI SBx00 SMBus" rev 0x3c: SMI
> iic0 at piixpm0
> iic0: addr 0x20 01=19 02=24 03=2e 04=00 05=00 06=00 07=00 08=00 09=00
> 0a=10 0b=10 0c=10 0d=10 0e=22 0f=92 10=3d 11=00 12=00 13=00 14=0a 15=0a
> 16=2c 17=a0 18=e0 1a=ae 1b=a4 1c=b3 1d=00 1e=0c 1f=01 20=09 21=09 22=09
> 23=09 24=bb 3e=03 words 00=ff19 01=1924 02=242e 03=2e00 04=0000 05=0000
> 06=0000 07=0000
> spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600
> spdmem1 at iic0 addr 0x51: 4GB DDR3 SDRAM PC3-10600
> spdmem2 at iic0 addr 0x52: 2GB DDR3 SDRAM PC3-10600
> spdmem3 at iic0 addr 0x53: 2GB DDR3 SDRAM PC3-10600
> pciide0 at pci0 dev 20 function 1 "ATI SB700 IDE" rev 0x00: DMA, channel
> 0 configured to compatibility, channel 1 configured to compatibility
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, DVDRAM GSA-4163B, AX13> ATAPI
> 5/cdrom removable
> cd0(pciide0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 4
> azalia1 at pci0 dev 20 function 2 "ATI SBx00 HD Audio" rev 0x00: apic 6
> int 16
> azalia1: codecs: VIA/0x0397
> audio0 at azalia1
> pcib0 at pci0 dev 20 function 3 "ATI SB700 ISA" rev 0x00
> ppb2 at pci0 dev 20 function 4 "ATI SB600 PCI" rev 0x00
> pci3 at ppb2 bus 3
> re1 at pci3 dev 5 function 0 "D-Link DGE-530T C1" rev 0x10:
> RTL8169/8110SB (0x1000), apic 6 int 20, address 5c:d9:98:ae:3c:7b
> rgephy1 at re1 phy 7: RTL8169S/8110S PHY, rev. 3
> "VIA VT6306 FireWire" rev 0xc0 at pci3 dev 8 function 0 not configured
> ohci4 at pci0 dev 20 function 5 "ATI SB700 USB" rev 0x00: apic 6 int 18,
> version 1.0, legacy support
> pchb1 at pci0 dev 24 function 0 "AMD AMD64 10h HyperTransport" rev 0x00
> pchb2 at pci0 dev 24 function 1 "AMD AMD64 10h Address Map" rev 0x00
> pchb3 at pci0 dev 24 function 2 "AMD AMD64 10h DRAM Cfg" rev 0x00
> km0 at pci0 dev 24 function 3 "AMD AMD64 10h Misc Cfg" rev 0x00
> pchb4 at pci0 dev 24 function 4 "AMD AMD64 10h Link Cfg" rev 0x00
> usb2 at ohci0: USB revision 1.0
> uhub2 at usb2 "ATI OHCI root hub" rev 1.00/1.00 addr 1
> usb3 at ohci1: USB revision 1.0
> uhub3 at usb3 "ATI OHCI root hub" rev 1.00/1.00 addr 1
> usb4 at ohci2: USB revision 1.0
> uhub4 at usb4 "ATI OHCI root hub" rev 1.00/1.00 addr 1
> usb5 at ohci3: USB revision 1.0
> uhub5 at usb5 "ATI OHCI root hub" rev 1.00/1.00 addr 1
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> it0 at isa0 port 0x2e/2: IT8712F rev 8, EC port 0x290
> usb6 at ohci4: USB revision 1.0
> uhub6 at usb6 "ATI OHCI root hub" rev 1.00/1.00 addr 1
> mtrr: Pentium Pro MTRR support
> uhub7 at uhub0 port 4 "HP\M^? f2105 2PORT USB 2.0 HUB" rev 2.00/7.02 addr 2
> ugen0 at uhub2 port 3 "APC Back-UPS ES 550G FW:843.K4 .D USB FW:K4" rev
> 1.10/1.06 addr 2
> uhidev0 at uhub4 port 3 configuration 1 interface 0 "Logitech USB
> Optical Mouse" rev 2.00/43.01 addr 2
> uhidev0: iclass 3/1
> ums0 at uhidev0: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> sd3 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 005> SCSI2 0/direct
> fixed sd3: 2861588MB, 512 bytes/sector, 5860532576 sectors
> root on sd2a (27a551cc8502d62c.a) swap on sd2b dump on sd2b
> softraid0: sd4 was not shutdown properly
> softraid0: sd4 was not shutdown properly
> sd4 at scsibus3 targ 2 lun 0: <OPENBSD, SR CRYPTO, 005> SCSI2 0/direct
> fixed sd4: 1430793MB, 512 bytes/sector, 2930265808 sectors
--
"Reason is not automatic. Those who deny it cannot be conquered by it.
Do not count on them. Leave them alone." -- Ayn Rand
