I think it is some sort of race and we end up hanging the firmware.
Like I said I have never seen it myself but that is how it feels like.
Also tracing the code and that particular error there is not much else
that could be wrong. Let me think about this for a few days.
On Sun, Feb 10, 2008 at 05:37:21PM -0600, Vijay Sankar wrote:
> On February 10, 2008 04:19:52 pm Marco Peereboom wrote:
> > This error has all the makings of a firmware issue. I never saw this
> > when I tested the 1.0 firmware but have seen reports of this in the
> > newer versions. Please do upgrade and share your results. If it still
> > fails I'll consider trying to reboot the firmware and see what that
> > does.
> >
>
> Thanks very much.
>
> We had this error on two Dell 2950's recently. The error was observed roughly
> 12 hours apart. Both are mail servers running OpenBSD 4.2 and had been up for
> close to 60 days when we saw the error (sd0 not queued error 5). Also, while
> the error was happening we were getting a lot of NDR's because one user had
> set up forwarding to the wrong address before taking off on his vacation. I
> don't know whether that contributed to this error in some way.
>
> Going through all the messages over the past two years on this topic, to me
> it
> looks like the problem occurs only on heavily loaded servers that have been
> up for a few months or on systems that have a lot of writes and the disks are
> mirrored. Is this a valid observation? Hopefully I am not jumping to
> conclusions here.
>
> Here is the dmesg after we did a power reset. We have not seen any problems
> for the past 15 days.
>
> Since we have not had any other problems, we were going to leave it running
> for a month or so, update the firmware on the backup server and see if the
> problem reappears. I will update the list if we have the problem again.
>
> If there is any additional information that would be useful or if there is
> anything else I can do on the backup server, please let me know.
>
> SERVER 1 (main server)
>
> OpenBSD 4.2 (GENERIC) #375: Tue Aug 28 10:38:44 MDT 2007
> [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
> cpu0: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz ("GenuineIntel" 686-class) 1.60 GHz
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-
> CPL,VMX,TM2,CX16,xTPR
> real mem = 1072955392 (1023MB)
> avail mem = 1029857280 (982MB)
> mainbus0 at root
> bios0 at mainbus0: AT/286+ BIOS, date 08/10/07, BIOS32 rev. 0 @ 0xffe90,
> SMBIOS rev. 2.4 @ 0x3ffbc000 (62 entries)
> bios0: vendor Dell Inc. version "1.5.1" date 08/10/2007
> bios0: Dell Inc. PowerEdge 2950
> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
> pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfaa30/384 (22 entries)
> pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 6321ESB LPC" rev 0x00)
> pcibios0: PCI bus #16 is the last bus
> bios0: ROM list: 0xc0000/0x9000! 0xc9000/0x5200 0xce800/0x1000!
> 0xec000/0x4000!
> acpi0 at mainbus0: rev 2
> acpi0: tables DSDT FACP APIC SPCR HPET MCFG
> acpitimer at acpi0 not configured
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 6 (PEX2)
> acpiprt2 at acpi0: bus 7 (UPST)
> acpiprt3 at acpi0: bus 8 (DWN1)
> acpiprt4 at acpi0: bus 10 (DWN2)
> acpiprt5 at acpi0: bus -1 (PE2X)
> acpiprt6 at acpi0: bus 1 (PEX3)
> acpiprt7 at acpi0: bus 2 (PE2P)
> acpiprt8 at acpi0: bus 12 (PEX4)
> acpiprt9 at acpi0: bus -1 (PE2P)
> acpiprt10 at acpi0: bus -1 (PEX5)
> acpiprt11 at acpi0: bus 0 (PE2P)
> acpiprt12 at acpi0: bus 14 (PEX6)
> acpiprt13 at acpi0: bus -1 (PXHA)
> acpiprt14 at acpi0: bus -1 (PXHB)
> acpiprt15 at acpi0: bus 4 (SBEX)
> acpiprt16 at acpi0: bus 16 (COMP)
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> acpicpu at acpi0 not configured
> ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
> cpu0 at mainbus0
> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x12
> ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x12
> pci1 at ppb0 bus 6
> ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci2 at ppb1 bus 7
> ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci3 at ppb2 bus 8
> ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
> pci4 at ppb3 bus 9
> bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 10
> ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci5 at ppb4 bus 10
> ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
> pci6 at ppb5 bus 11
> ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x12
> pci7 at ppb6 bus 1
> ppb7 at pci7 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci8 at ppb7 bus 2
> mfi0 at pci8 dev 14 function 0 "Dell PERC 5" rev 0x00: irq 11
> mfi0: logical drives 1, version 5.1.1-0040, 256MB RAM
> scsibus0 at mfi0: 1 targets
> sd0 at scsibus0 targ 0 lun 0: <DELL, PERC 5/i, 1.03> SCSI3 0/direct fixed
> sd0: 139392MB, 17769 cyl, 255 head, 63 sec, 512 bytes/sec, 285474816 sec total
> ppb8 at pci7 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci9 at ppb8 bus 3
> ppb9 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x12
> pci10 at ppb9 bus 12
> ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x12
> pci11 at ppb10 bus 13
> ppb11 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x12
> pci12 at ppb11 bus 14
> ppb12 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x12
> pci13 at ppb12 bus 15
> pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x12
> pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x12
> pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x12
> pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x12
> pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x12
> pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x12
> pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x12
> ppb13 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09
> pci14 at ppb13 bus 4
> ppb14 at pci14 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
> pci15 at ppb14 bus 5
> bnx1 at pci15 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 10
> uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: irq 15
> uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: irq 14
> uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: irq 15
> ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: irq 15
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0: Intel EHCI root hub, rev 2.00/1.00, addr 1
> ppb15 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xd9
> pci16 at ppb15 bus 16
> vga1 at pci16 dev 13 function 0 "ATI ES1000" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ichpcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09: PM disabled
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1: Intel UHCI root hub, rev 1.00/1.00, addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2: Intel UHCI root hub, rev 1.00/1.00, addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3: Intel UHCI root hub, rev 1.00/1.00, addr 1
> isa0 at ichpcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
> pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> biomask fbe5 netmask ffe5 ttymask ffe7
> pctr: 686-class user-level performance counters enabled
> mtrr: Pentium Pro MTRR support
> uhub4 at uhub0 port 1: Dell product 0xa001, rev 2.00/0.00, addr 2
> uhidev0 at uhub4 port 1 configuration 1 interface 0
> uhidev0: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
> ukbd0 at uhidev0: 8 modifier keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> uhidev1 at uhub4 port 1 configuration 1 interface 1
> uhidev1: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
> ums0 at uhidev1
> ums0: X report 0x0002 not supported
> umass0 at uhub4 port 2 configuration 1 interface 0
> umass0: DELL INC. DRAC5 VIRTUAL MEDIA, rev 2.00/0.00, addr 4
> umass0: using SCSI over Bulk-Only
> scsibus1 at umass0: 2 targets
> cd0 at scsibus1 targ 1 lun 0: <Dell, Virtual CDROM, 123> SCSI0 5/cdrom
> removable
> umass1 at uhub4 port 2 configuration 1 interface 1
> umass1: DELL INC. DRAC5 VIRTUAL MEDIA, rev 2.00/0.00, addr 4
> umass1: using SCSI over Bulk-Only
> scsibus2 at umass1: 2 targets
> sd1 at scsibus2 targ 1 lun 0: <Dell, Virtual Floppy, 123> SCSI0 0/direct
> removable
> sd1: drive offline
> uhub5 at uhub0 port 5: Cypress Semiconductor USB2 Hub, rev 2.00/0.0b, addr 5
> dkcsum: sd0 matches BIOS drive 0x80
> root on sd0a swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> .
> .
> .
> # bioctl -hiv sd0
> Volume Status Size Device
> mfi0 0 Online 136G sd0 RAID1
> 0 Online 137G 1:0.0 noencl <FUJITSU MAX3147RC
> D207>
> 'unknown serial'
> 1 Online 137G 1:1.0 noencl <FUJITSU MAX3147RC
> D207>
> 'unknown serial'
>
>
>
> SERVER 2: (backup server)
>
> OpenBSD 4.2 (GENERIC) #375: Tue Aug 28 10:38:44 MDT 2007
> [EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
> cpu0: Intel(R) Xeon(R) CPU 5110 @ 1.60GHz ("GenuineIntel" 686-class) 1.60 GHz
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,TM2,CX16,xTPR
> real mem = 1072955392 (1023MB)
> avail mem = 1029857280 (982MB)
> mainbus0 at root
> bios0 at mainbus0: AT/286+ BIOS, date 08/10/07, BIOS32 rev. 0 @ 0xffe90,
> SMBIOS rev. 2.4 @ 0x3ffbc000 (62 entries)
> bios0: vendor Dell Inc. version "1.5.1" date 08/10/2007
> bios0: Dell Inc. PowerEdge 2950
> pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
> pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfaa30/384 (22 entries)
> pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 6321ESB LPC" rev 0x00)
> pcibios0: PCI bus #16 is the last bus
> bios0: ROM list: 0xc0000/0x9000! 0xc9000/0x5200 0xec000/0x4000!
> acpi at mainbus0 not configured
> ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca8/8 spacing 4
> cpu0 at mainbus0
> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x12
> ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x12
> pci1 at ppb0 bus 6
> ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci2 at ppb1 bus 7
> ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci3 at ppb2 bus 8
> ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
> pci4 at ppb3 bus 9
> bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 11
> ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci5 at ppb4 bus 10
> ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
> pci6 at ppb5 bus 11
> ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x12
> pci7 at ppb6 bus 1
> ppb7 at pci7 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci8 at ppb7 bus 2
> mfi0 at pci8 dev 14 function 0 "Dell PERC 5" rev 0x00: irq 5
> mfi0: logical drives 1, version 5.1.1-0040, 256MB RAM
> scsibus0 at mfi0: 1 targets
> sd0 at scsibus0 targ 0 lun 0: <DELL, PERC 5/i, 1.03> SCSI3 0/direct fixed
> sd0: 139392MB, 17769 cyl, 255 head, 63 sec, 512 bytes/sec, 285474816 sec total
> ppb8 at pci7 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci9 at ppb8 bus 3
> ppb9 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x12
> pci10 at ppb9 bus 12
> ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x12
> pci11 at ppb10 bus 13
> ppb11 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x12
> pci12 at ppb11 bus 14
> ppb12 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x12
> pci13 at ppb12 bus 15
> pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x12
> pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x12
> pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x12
> pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x12
> pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x12
> pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x12
> pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x12
> ppb13 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09
> pci14 at ppb13 bus 4
> ppb14 at pci14 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
> pci15 at ppb14 bus 5
> bnx1 at pci15 dev 0 function 0 "Broadcom BCM5708" rev 0x12: irq 11
> uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: irq 11
> uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: irq 10
> uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: irq 11
> ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: irq 11
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0: Intel EHCI root hub, rev 2.00/1.00, addr 1
> ppb15 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xd9
> pci16 at ppb15 bus 16
> vga1 at pci16 dev 13 function 0 "ATI ES1000" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ichpcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09: PM disabled
> pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel
> 0
> configured to compatibility, channel 1 configured to compatibility
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, CD-ROM GCR-8240N, 1.10> SCSI0
> 5/cdrom
> removable
> cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
> pciide0: channel 1 ignored (disabled)
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1: Intel UHCI root hub, rev 1.00/1.00, addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2: Intel UHCI root hub, rev 1.00/1.00, addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3: Intel UHCI root hub, rev 1.00/1.00, addr 1
> isa0 at ichpcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> midi0 at pcppi0: <PC speaker>
> spkr0 at pcppi0
> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
> pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> biomask ffe5 netmask ffe5 ttymask ffe7
> pctr: 686-class user-level performance counters enabled
> mtrr: Pentium Pro MTRR support
> uhub4 at uhub0 port 1: Dell product 0xa001, rev 2.00/0.00, addr 2
> uhidev0 at uhub4 port 1 configuration 1 interface 0
> uhidev0: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
> ukbd0 at uhidev0: 8 modifier keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> uhidev1 at uhub4 port 1 configuration 1 interface 1
> uhidev1: Dell DRAC5, rev 1.10/0.00, addr 3, iclass 3/1
> ums0 at uhidev1
> ums0: X report 0x0002 not supported
> uhub5 at uhub0 port 5: Cypress Semiconductor USB2 Hub, rev 2.00/0.0b, addr 4
> uhidev2 at uhub5 port 2 configuration 1 interface 0
> uhidev2: Logitech Logitech USB Keyboard, rev 1.10/23.00, addr 5, iclass 3/1
> ukbd1 at uhidev2: 8 modifier keys, 6 key codes
> wskbd2 at ukbd1 mux 1
> wskbd2: connecting to wsdisplay0
> uhidev3 at uhub5 port 2 configuration 1 interface 1
> uhidev3: Logitech Logitech USB Keyboard, rev 1.10/23.00, addr 5, iclass 3/0
> uhidev3: 2 report ids
> uhid0 at uhidev3 reportid 1: input=2, output=0, feature=0
> uhid1 at uhidev3 reportid 2: input=1, output=0, feature=0
> dkcsum: sd0 matches BIOS drive 0x80
> root on sd0a swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> .
> .
> .
>
> # bioctl -hiv sd0
> Volume Status Size Device
> mfi0 0 Online 136G sd0 RAID1
> 0 Online 137G 1:0.0 noencl <FUJITSU MAX3147RC
> D207>
> 'unknown serial'
> 1 Online 137G 1:1.0 noencl <FUJITSU MAX3147RC
> D207>
> 'unknown serial'
>
>
>
>
> --
> Vijay Sankar, M.Eng., P.Eng.
> President & CEO
> ForeTell Technologies Limited
> 59 Flamingo Avenue, Winnipeg, MB Canada R3J 0X6
> Phone: +1 204 885 9535, E-Mail: [EMAIL PROTECTED]
