Hi everyone, long time lurker, first time poster. I've been around since
the 5.* days, so I would consider myself fairly seasoned in the ways of
OpenBSD. I've obviously done the RTFM dance, done it once, done it
twice, been doing it all week long now-- this problem really has me
banging my head against a wall.
I'm cross posting this to both the misc and sparc64 mailing lists in the
hopes that I can entice both the softraid and sparc connoisseurs.
I've recently gotten my hands on a couple shiny new SPARC T4-1 and T3-1
servers and I was looking to install OpenBSD with a softraid mirror on
them for production use. The problem is, is that I end up with this upon
following the install instructions and rebooting:
Boot device: /pci@400/pci@1/pci@0/pci@4/scsi@0/disk@p0 File and
args: sr0a:/bsd OpenBSD IEEE 1275 Bootblock 1.4
OpenBSD BOOT 1.9 Error: /iscsi-hba: no
iscsi-network-bootpath property Unknown device: sr0 Cannot
boot from softraid: Unknown error: code 19 Program terminated
The only other information I can find online involving a softraid error
code 19 was the following:
http://openbsd-archive.7691.n7.nabble.com/5-8-sparc64-boot-from-softraid-4-fails-td284700.html
Considering he was running 5.8 and I am on 6.2 and the fix for his issue
has already been committed, I’m left scratching my head as to what the
problem here could be.
The install procedure I followed on the T4 was:
1) Boot install kernel and drop to shell and provision RAID partitions
on both disks using the letter “a” via disklabel(8)
2) Assemble RAID volume with # bioctl -c 1 -l /dev/sd0a,/dev/sd1a softraid0
3) I zeroed the first 10MB of the RAID volume with # dd if=/dev/urandom
of=/dev/rsd2c bs=1m
4) I finished off the install as usual by typing ”install” into command
line and proceeded normally
5) I then rebooted and set the boot parameters at the ok prompt as per
boot_sparc(8)
Aside from the sparc64 specific stuff, these same steps (plus x86
specific fdisk stuff) get me a working, bootable softraid mirror on my
i386/amd64 test boxes.
It appears the issue I linked to in the OpenBSD archive, was that the
bootloader couldn't probe beyond 4 levels deep in the device tree.
According to Stefan Sperling he committed a patch to allow it to probe
to (an arbitrarily set) 10 levels deep in the device tree. According to
my devalias output, my disk0 appears to be 6 levels deep in the tree if
I am reading it correctly.
I did manage to get the T4 booting from softraid using this guide:
http://brycv.com/blog/2012/openbsd-sparc64-and-root-on-softraid/
That guide however does seem to contravene the guidelines set forth in
softraid(4): “On sparc64, bootable chunks must be RAID partitions using
the letter ‘a’ .”
With the above guide from brycv.com we end up with 3 separate partitions
on our disks. This doesn't seem right as on amd64/i386 you would only
have a single raid partition per disk, not a separate root and swap
partition in addition to the RAID partitions. It would also seem that a
rebuild would involve significantly more manual intervention to maintain
drive bootability in the event of one of the drives in the mirror going
down ie: One wouldn't just be able to run bioctl -R /dev/rsd1a sd2 and
have the array be golden again, they would instead also need to mount
and copy over the contents of the other non RAID partitions on the disk.
In short, on sparc64, is softraid boot like that of i386/amd64 where
everything including root is stored within the RAID volume, or does
sparc64 require me to have a small partition at the beginning of my
physical disks containing the kernel and ofwboot? One fellow on reddit
said he had a T1000 booting softraid crypto no problem so I am wondering
if this is a T4 specific issue.
Sorry for the wall of text, I really appreciate any help you guys can
offer. Let me know if you need any other information and I’ll be happy
to provide it. Does anyone else here have any experience with sparc64
softraid booting or with OpenBSD on a T3 or T4? Please let me know, and
Merry Christmas to all!
Obligatory dmesg:
T4 -1 Output
# dmesg
console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2017 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 6.2 (RAMDISK) #282: Tue Oct 3 23:21:19 MDT 2017
dera...@sparc64.openbsd.org:/usr/src/sys/arch/sparc64/compile/RAMDISK
real mem = 136902082560 (130560MB)
avail mem = 134512222208 (128280MB)
mainbus0 at root: SPARC T4-1
cpu0 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"SPARC-T4" at mainbus0 not configured
"reboot-memory" at mainbus0 not configured
vbus0 at mainbus0
"flashprom" at vbus0 not configured
"tpm" at vbus0 not configured
cbus0 at vbus0
"virtual-channel" at cbus0 not configured
"virtual-channel-client" at cbus0 not configured
"virtual-channel-client" at cbus0 not configured
"virtual-channel" at cbus0 not configured
"virtual-domain-service" at cbus0 not configured
vrng0 at vbus0
vcons0 at vbus0: ivec 0x111, console
vrtc0 at vbus0
vpci0 at mainbus0: bus 0 to 18, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 1 function 0 "Sun SPARC PCIE" rev 0x01
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 vendor "IDT", unknown product 0x807a rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 vendor "IDT", unknown product 0x807a rev
0x02: msi
pci3 at ppb2 bus 3
ppb3 at pci2 dev 4 function 0 vendor "IDT", unknown product 0x807a rev 0x02
pci4 at ppb3 bus 4
mpii0 at pci4 dev 0 function 0 "Symbios Logic SAS2008" rev 0x03: msi
mpii0: RF On-Board, firmware 9.5.0.0 IR, MPI 2.0
scsibus0 at mpii0: 834 targets
ppb4 at pci2 dev 6 function 0 vendor "IDT", unknown product 0x807a rev
0x02: msi
pci5 at ppb4 bus 5
ppb5 at pci2 dev 8 function 0 vendor "IDT", unknown product 0x807a rev
0x02: msi
pci6 at ppb5 bus 6
ppb6 at pci0 dev 2 function 0 "Sun SPARC PCIE" rev 0x01
pci7 at ppb6 bus 7
ppb7 at pci7 dev 0 function 0 vendor "IDT", unknown product 0x8077 rev 0x02
pci8 at ppb7 bus 8
ppb8 at pci8 dev 0 function 0 vendor "IDT", unknown product 0x8077 rev 0x02
pci9 at ppb8 bus 9
ppb9 at pci9 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x02
pci10 at ppb9 bus 10
"ASPEED Technology AST2000" rev 0x10 at pci10 dev 0 function 0 not
configured
ppb10 at pci8 dev 4 function 0 vendor "IDT", unknown product 0x8077 rev 0x02
pci11 at ppb10 bus 11
mpii1 at pci11 dev 0 function 0 "Symbios Logic SAS2008" rev 0x03: msi
mpii1: RF On-Board, firmware 9.5.0.0 IR, MPI 2.0
scsibus1 at mpii1: 834 targets
ppb11 at pci8 dev 6 function 0 vendor "IDT", unknown product 0x8077 rev 0x02
pci12 at ppb11 bus 12
em0 at pci12 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
00:10:e0:3e:a1:de
em1 at pci12 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
00:10:e0:3e:a1:df
ppb12 at pci8 dev 7 function 0 vendor "IDT", unknown product 0x8077 rev 0x02
pci13 at ppb12 bus 13
em2 at pci13 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
00:10:e0:3e:a1:e0
em3 at pci13 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
00:10:e0:3e:a1:e1
ppb13 at pci8 dev 8 function 0 vendor "IDT", unknown product 0x8077 rev
0x02: msi
pci14 at ppb13 bus 14
ppb14 at pci8 dev 10 function 0 vendor "IDT", unknown product 0x8077 rev
0x02: msi
pci15 at ppb14 bus 15
ppb15 at pci8 dev 12 function 0 vendor "IDT", unknown product 0x8077 rev
0x02: msi
pci16 at ppb15 bus 16
ppb16 at pci8 dev 15 function 0 vendor "IDT", unknown product 0x8077 rev
0x02
pci17 at ppb16 bus 17
ppb17 at pci17 dev 0 function 0 "PLX PEX 8112" rev 0xaa
pci18 at ppb17 bus 18
ohci0 at pci18 dev 0 function 0 "NEC USB" rev 0x43: ivec 0x402, version 1.0
ohci1 at pci18 dev 0 function 1 "NEC USB" rev 0x43: ivec 0x403, version 1.0
ehci0 at pci18 dev 0 function 2 "NEC USB" rev 0x04: ivec 0x404
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "NEC EHCI root hub" rev
2.00/1.00 addr 1
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "NEC OHCI root hub" rev
1.00/1.00 addr 1
usb2 at ohci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "NEC OHCI root hub" rev
1.00/1.00 addr 1
"niu" at mainbus0 not configured
"pci-performance-counters" at mainbus0 not configured
cd0 at scsibus1 targ 8 lun 0: <TEAC, DV-W28SS-W, 1.0A> ATAPI 5/cdrom
removable
sd0 at scsibus0 targ 2 lun 0: <HP, EG0300FAWHV, HPDD> SCSI3 0/direct
fixed naa.5000c5001b1e4cd3
sd0: 286102MB, 512 bytes/sector, 585937500 sectors
sd1 at scsibus0 targ 5 lun 0: <ATA, SanDisk SDSSDA24, 30RL> SCSI3
0/direct fixed naa.5001b448b4351630
sd1: 228936MB, 512 bytes/sector, 468862128 sectors
sd2 at scsibus0 targ 3 lun 0: <HP, EG0300FAWHV, HPDD> SCSI3 0/direct
fixed naa.5000c50029674d0f
sd2: 286102MB, 512 bytes/sector, 585937500 sectors
uhub3 at uhub0 port 2 configuration 1 interface 0 "vendor 0x04b4 product
0x6560" rev 2.00/0.0b addr 2
uhub4 at uhub3 port 3 configuration 1 interface 0 "American Megatrends
Inc. Generic Hub" rev 2.00/1.00 addr 3
umass0 at uhub4 port 2 configuration 1 interface 0 "American Megatrends
Inc. Virtual Cdrom Device" rev 2.00/1.00 addr 4
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
cd1 at scsibus2 targ 1 lun 0: <AMI, Virtual CDROM, 1.00> SCSI0 5/cdrom
removable
cdce0 at uhub4 port 3 configuration 2 interface 0 "SunMicro Virtual Eth
Device" rev 2.00/1.00 addr 5
cdce0: address 02:21:28:57:47:17
uhidev0 at uhub3 port 4 configuration 1 interface 0 "American Megatrends
Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 6
uhidev0: iclass 3/1
ukbd0 at uhidev0
wskbd0 at ukbd0
uhidev1 at uhub3 port 4 configuration 1 interface 1 "American Megatrends
Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 6
uhidev1: iclass 3/1
uhid at uhidev1 not configured
uhub5 at uhub0 port 4 configuration 1 interface 0 "vendor 0x04b4 product
0x6560" rev 2.00/0.0b addr 7
softraid0 at root
scsibus3 at softraid0: 256 targets
sd3 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
sd3: 286101MB, 512 bytes/sector, 585934896 sectors
bootpath:
/pci@400,0/pci@1,0/pci@0,0/pci@4,0/scsi@0,0/disk@5001b448b4351630,0:c
root on rd0a swap on rd0b dump on rd0b
umass1 at uhub3 port 1 configuration 1 interface 0 "LEXR PLUG DRIVE LEXR
PLUG DRIVE" rev 1.10/0.01 addr 8
umass1: using SCSI over Bulk-Only
scsibus4 at umass1: 2 targets, initiator 0
sd4 at scsibus4 targ 1 lun 0: <LEXAR, JUMPDRIVE, 1.02> SCSI1 0/direct
removable serial.05dc0080000000000000
sd4: 30MB, 512 bytes/sector, 62464 sectors
sd4 detached
scsibus4 detached
umass1 detached
#
