On February 10, 2008 04:19:52 pm Marco Peereboom wrote:
> This error has all the makings of a firmware issue. I never saw this
> when I tested the 1.0 firmware but have seen reports of this in the
> newer versions. Please do upgrade and share your results. If it still
> fails I'll consider trying to reboot the firmware and see what that
> does.
>
Thanks very much.
We had this error on two Dell 2950's recently. The error was observed roughly
12 hours apart. Both are mail servers running OpenBSD 4.2 and had been up for
close to 60 days when we saw the error (sd0 not queued error 5). Also, while
the error was happening we were getting a lot of NDR's because one user had
set up forwarding to the wrong address before taking off on his vacation. I
don't know whether that contributed to this error in some way.
Going through all the messages over the past two years on this topic, to me it
looks like the problem occurs only on heavily loaded servers that have been
up for a few months or on systems that have a lot of writes and the disks are
mirrored. Is this a valid observation? Hopefully I am not jumping to
conclusions here.
Here is the dmesg after we did a power reset. We have not seen any problems
for the past 15 days.
Since we have not had any other problems, we were going to leave it running
for a month or so, update the firmware on the backup server and see if the
problem reappears. I will update the list if we have the problem again.
If there is any additional information that would be useful or if there is
anything else I can do on the backup server, please let me know.
SERVER 1 (main server)
OpenBSD 4.2 (GENERIC) #375: Tue Aug 28 10:38:44 MDT 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz ("GenuineIntel" 686-class) 1.60 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-
CPL,VMX,TM2,CX16,xTPR
real mem = 1072955392 (1023MB)
avail mem = 1029857280 (982MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 08/10/07, BIOS32 rev. 0 @ 0xffe90,
SMBIOS rev. 2.4 @ 0x3ffbc000 (62 entries)
bios0: vendor Dell Inc. version "1.5.1" date 08/10/2007
bios0: Dell Inc. PowerEdge 2950
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfaa30/384 (22 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 6321ESB LPC" rev 0x00)
pcibios0: PCI bus #16 is the last bus
bios0: ROM list: 0xc0000/0x9000! 0xc9000/0x5200 0xce800/0x1000!
0xec000/0x4000!
acpi0 at mainbus0: rev 2
acpi0: tables DSDT FACP APIC SPCR HPET MCFG
acpitimer at acpi0 not configured
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 6 (PEX2)
acpiprt2 at acpi0: bus 7 (UPST)
acpiprt3 at acpi0: bus 8 (DWN1)
acpiprt4 at acpi0: bus 10 (DWN2)
acpiprt5 at acpi0: bus -1 (PE2X)
acpiprt6 at acpi0: bus 1 (PEX3)
acpiprt7 at acpi0: bus 2 (PE2P)
acpiprt8 at acpi0: bus 12 (PEX4)
acpiprt9 at acpi0: bus -1 (PE2P)
acpiprt10 at acpi0: bus -1 (PEX5)
acpiprt11 at acpi0: bus 0 (PE2P)
acpiprt12 at acpi0: bus 14 (PEX6)
acpiprt13 at acpi0: bus -1 (PXHA)
acpiprt14 at acpi0: bus -1 (PXHB)
acpiprt15 at acpi0: bus 4 (SBEX)
acpiprt16 at acpi0: bus 16 (COMP)
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
acpicpu at acpi0 not configured
ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x12
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x12
pci1 at ppb0 bus 6
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 7
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 8
ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci4 at ppb3 bus 9
bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 10
ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
pci5 at ppb4 bus 10
ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci6 at ppb5 bus 11
ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x12
pci7 at ppb6 bus 1
ppb7 at pci7 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
pci8 at ppb7 bus 2
mfi0 at pci8 dev 14 function 0 "Dell PERC 5" rev 0x00: irq 11
mfi0: logical drives 1, version 5.1.1-0040, 256MB RAM
scsibus0 at mfi0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <DELL, PERC 5/i, 1.03> SCSI3 0/direct fixed
sd0: 139392MB, 17769 cyl, 255 head, 63 sec, 512 bytes/sec, 285474816 sec total
ppb8 at pci7 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
pci9 at ppb8 bus 3
ppb9 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x12
pci10 at ppb9 bus 12
ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x12
pci11 at ppb10 bus 13
ppb11 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x12
pci12 at ppb11 bus 14
ppb12 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x12
pci13 at ppb12 bus 15
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x12
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x12
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x12
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x12
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x12
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x12
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x12
ppb13 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09
pci14 at ppb13 bus 4
ppb14 at pci14 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci15 at ppb14 bus 5
bnx1 at pci15 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 10
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: irq 15
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: irq 14
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: irq 15
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: irq 15
usb0 at ehci0: USB revision 2.0
uhub0 at usb0: Intel EHCI root hub, rev 2.00/1.00, addr 1
ppb15 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xd9
pci16 at ppb15 bus 16
vga1 at pci16 dev 13 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09: PM disabled
usb1 at uhci0: USB revision 1.0
uhub1 at usb1: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3: Intel UHCI root hub, rev 1.00/1.00, addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
biomask fbe5 netmask ffe5 ttymask ffe7
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
uhub4 at uhub0 port 1: Dell product 0xa001, rev 2.00/0.00, addr 2
uhidev0 at uhub4 port 1 configuration 1 interface 0
uhidev0: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub4 port 1 configuration 1 interface 1
uhidev1: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
ums0 at uhidev1
ums0: X report 0x0002 not supported
umass0 at uhub4 port 2 configuration 1 interface 0
umass0: DELL INC. DRAC5 VIRTUAL MEDIA, rev 2.00/0.00, addr 4
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets
cd0 at scsibus1 targ 1 lun 0: <Dell, Virtual CDROM, 123> SCSI0 5/cdrom
removable
umass1 at uhub4 port 2 configuration 1 interface 1
umass1: DELL INC. DRAC5 VIRTUAL MEDIA, rev 2.00/0.00, addr 4
umass1: using SCSI over Bulk-Only
scsibus2 at umass1: 2 targets
sd1 at scsibus2 targ 1 lun 0: <Dell, Virtual Floppy, 123> SCSI0 0/direct
removable
sd1: drive offline
uhub5 at uhub0 port 5: Cypress Semiconductor USB2 Hub, rev 2.00/0.0b, addr 5
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
.
.
.
# bioctl -hiv sd0
Volume Status Size Device
mfi0 0 Online 136G sd0 RAID1
0 Online 137G 1:0.0 noencl <FUJITSU MAX3147RC
D207>
'unknown serial'
1 Online 137G 1:1.0 noencl <FUJITSU MAX3147RC
D207>
'unknown serial'
SERVER 2: (backup server)
OpenBSD 4.2 (GENERIC) #375: Tue Aug 28 10:38:44 MDT 2007
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz ("GenuineIntel" 686-class) 1.60 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,TM2,CX16,xTPR
real mem = 1072955392 (1023MB)
avail mem = 1029857280 (982MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 08/10/07, BIOS32 rev. 0 @ 0xffe90,
SMBIOS rev. 2.4 @ 0x3ffbc000 (62 entries)
bios0: vendor Dell Inc. version "1.5.1" date 08/10/2007
bios0: Dell Inc. PowerEdge 2950
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfaa30/384 (22 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 6321ESB LPC" rev 0x00)
pcibios0: PCI bus #16 is the last bus
bios0: ROM list: 0xc0000/0x9000! 0xc9000/0x5200 0xec000/0x4000!
acpi at mainbus0 not configured
ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x12
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x12
pci1 at ppb0 bus 6
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 7
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 8
ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci4 at ppb3 bus 9
bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 11
ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
pci5 at ppb4 bus 10
ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci6 at ppb5 bus 11
ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x12
pci7 at ppb6 bus 1
ppb7 at pci7 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
pci8 at ppb7 bus 2
mfi0 at pci8 dev 14 function 0 "Dell PERC 5" rev 0x00: irq 5
mfi0: logical drives 1, version 5.1.1-0040, 256MB RAM
scsibus0 at mfi0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <DELL, PERC 5/i, 1.03> SCSI3 0/direct fixed
sd0: 139392MB, 17769 cyl, 255 head, 63 sec, 512 bytes/sec, 285474816 sec total
ppb8 at pci7 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
pci9 at ppb8 bus 3
ppb9 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x12
pci10 at ppb9 bus 12
ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x12
pci11 at ppb10 bus 13
ppb11 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x12
pci12 at ppb11 bus 14
ppb12 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x12
pci13 at ppb12 bus 15
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x12
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x12
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x12
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x12
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x12
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x12
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x12
ppb13 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09
pci14 at ppb13 bus 4
ppb14 at pci14 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci15 at ppb14 bus 5
bnx1 at pci15 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 11
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: irq 11
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: irq 10
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: irq 11
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: irq 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0: Intel EHCI root hub, rev 2.00/1.00, addr 1
ppb15 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xd9
pci16 at ppb15 bus 16
vga1 at pci16 dev 13 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09: PM disabled
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, CD-ROM GCR-8240N, 1.10> SCSI0 5/cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
usb1 at uhci0: USB revision 1.0
uhub1 at usb1: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2: Intel UHCI root hub, rev 1.00/1.00, addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3: Intel UHCI root hub, rev 1.00/1.00, addr 1
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
biomask ffe5 netmask ffe5 ttymask ffe7
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
uhub4 at uhub0 port 1: Dell product 0xa001, rev 2.00/0.00, addr 2
uhidev0 at uhub4 port 1 configuration 1 interface 0
uhidev0: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub4 port 1 configuration 1 interface 1
uhidev1: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
ums0 at uhidev1
ums0: X report 0x0002 not supported
uhub5 at uhub0 port 5: Cypress Semiconductor USB2 Hub, rev 2.00/0.0b, addr 4
uhidev2 at uhub5 port 2 configuration 1 interface 0
uhidev2: Logitech Logitech USB Keyboard, rev 1.10/23.00, addr 5, iclass 3/1
ukbd1 at uhidev2: 8 modifier keys, 6 key codes
wskbd2 at ukbd1 mux 1
wskbd2: connecting to wsdisplay0
uhidev3 at uhub5 port 2 configuration 1 interface 1
uhidev3: Logitech Logitech USB Keyboard, rev 1.10/23.00, addr 5, iclass 3/0
uhidev3: 2 report ids
uhid0 at uhidev3 reportid 1: input=2, output=0, feature=0
uhid1 at uhidev3 reportid 2: input=1, output=0, feature=0
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
.
.
.
# bioctl -hiv sd0
Volume Status Size Device
mfi0 0 Online 136G sd0 RAID1
0 Online 137G 1:0.0 noencl <FUJITSU MAX3147RC
D207>
'unknown serial'
1 Online 137G 1:1.0 noencl <FUJITSU MAX3147RC
D207>
'unknown serial'
--
Vijay Sankar, M.Eng., P.Eng.
President & CEO
ForeTell Technologies Limited
59 Flamingo Avenue, Winnipeg, MB Canada R3J 0X6
Phone: +1 204 885 9535, E-Mail: [EMAIL PROTECTED]
