I just had a strange freeze that killed networking and made software RAID fail two of my harddisks.
There are a bunch of messages from the kernel which I extracted from the system log after reboot at the end of this mail. I hit power off in pure paranoia after the box froze, and then started to do disk I/O again just right after I noticed the messages about two of my RAID disks had failed on the console. The network didn't recover when the harddrive suddenly started working again. I managed to connect an USB keyboard and wake up the monitor from sleep so I could see some of the messages printed on the console. I looked through some other threads and found a mention of smartmontools which I too use (5.37-5ubuntu2). Kernel 2.6.22-14-generic (Ubuntu Gutsy Gibbon 7.10) Motherboard: Asus M2N32 WS Professional nForce 590 SLI MCP (MCP55) CPU: Athlon64 X2 Dual-Core 5600+ RAM: 4GB (passed memtest86 just a few minutes ago) The harddrives are four Samsung HD501LJ 500GB drives. sda and sdb have firmware CR100-10 and sdc and sdd have firmware CR100-11. The drives are just a couple of months old, well cooled and so far there's nothing interesting reported by S.M.A.R.T. Software raid is configured like this: sda1,sdc1 -> md0 (raid 1) sdb1,sdd1 -> md1 (raid 1) Both md0 and md1 are then encrypted with dm-crypt and the dm-devices are then used to form md2 (stripe). -- noah # lspci 00:00.0 RAM memory: nVidia Corporation C51 Host Bridge (rev a2) 00:00.1 RAM memory: nVidia Corporation C51 Memory Controller 0 (rev a2) 00:00.2 RAM memory: nVidia Corporation C51 Memory Controller 1 (rev a2) 00:00.3 RAM memory: nVidia Corporation C51 Memory Controller 5 (rev a2) 00:00.4 RAM memory: nVidia Corporation C51 Memory Controller 4 (rev a2) 00:00.5 RAM memory: nVidia Corporation C51 Host Bridge (rev a2) 00:00.6 RAM memory: nVidia Corporation C51 Memory Controller 3 (rev a2) 00:00.7 RAM memory: nVidia Corporation C51 Memory Controller 2 (rev a2) 00:04.0 PCI bridge: nVidia Corporation C51 PCI Express Bridge (rev a1) 00:08.0 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a1) 00:09.0 ISA bridge: nVidia Corporation MCP55 LPC Bridge (rev a2) 00:09.1 SMBus: nVidia Corporation MCP55 SMBus (rev a2) 00:09.2 RAM memory: nVidia Corporation MCP55 Memory Controller (rev a2) 00:0a.0 USB Controller: nVidia Corporation MCP55 USB Controller (rev a1) 00:0a.1 USB Controller: nVidia Corporation MCP55 USB Controller (rev a2) 00:0c.0 IDE interface: nVidia Corporation MCP55 IDE (rev a1) 00:0d.0 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a2) 00:0d.1 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a2) 00:0d.2 IDE interface: nVidia Corporation MCP55 SATA Controller (rev a2) 00:0e.0 PCI bridge: nVidia Corporation MCP55 PCI bridge (rev a2) 00:0e.1 Audio device: nVidia Corporation MCP55 High Definition Audio (rev a2) 00:10.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) 00:11.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2) 00:12.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2) 00:14.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2) 00:15.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2) 00:16.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2) 00:17.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2) 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:00.0 VGA compatible controller: nVidia Corporation GeForce 8400 GS (rev a1) 02:06.0 Communication controller: Tiger Jet Network Inc. Tiger3XX Modem/ISDN interface 03:00.0 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X Bridge (rev 06) 03:00.1 PCI bridge: NEC Corporation uPD720400 PCI Express - PCI/PCI-X Bridge (rev 06) 08:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE6145 SATA II PCI-E controller (rev a1) kernel: [734344.717844] irq 21: nobody cared (try booting with the "irqpoll" option) kernel: [734344.717866] kernel: [734344.717866] Call Trace: kernel: [734344.717868] <IRQ> [__report_bad_irq+30/128] __report_bad_irq+0x1e/0x80 mdadm: Fail event detected on md device /dev/md1, component device /dev/sdd1 kernel: [734344.717888] [note_interrupt+643/704] note_interrupt+0x283/0x2c0 kernel: [734344.717895] [handle_fasteoi_irq+221/272] handle_fasteoi_irq+0xdd/0x110 mdadm: Fail event detected on md device /dev/md0, component device /dev/sdc1 kernel: [734344.717901] [do_IRQ+123/256] do_IRQ+0x7b/0x100 kernel: [734344.717904] [default_idle+0/64] default_idle+0x0/0x40 kernel: [734344.717907] [ret_from_intr+0/10] ret_from_intr+0x0/0xa kernel: [734344.717909] <EOI> [tcp_poll+0/368] tcp_poll+0x0/0x170 kernel: [734344.717918] [default_idle+41/64] default_idle+0x29/0x40 kernel: [734344.717923] [cpu_idle+112/192] cpu_idle+0x70/0xc0 kernel: [734344.717936] kernel: [734344.717937] handlers: kernel: [734344.717950] [_end+131265960/2130332920] (nv_generic_interrupt+0x0/0xe0 [sata_nv]) kernel: [734344.717973] [_end+131152728/2130332920] (nv_nic_irq_optimized+0x0/0x2b0 [forcedeth]) kernel: [734344.718003] Disabling IRQ #21 kernel: [734356.827155] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen kernel: [734356.827169] ata5.00: cmd 25/00:90:ff:3c:82/00:01:02:00:00/e0 tag 0 cdb 0x0 data 204800 in kernel: [734356.827170] res 40/00:b4:cc:88:7f/40:00:02:00:00/e0 Emask 0x4 (timeout) kernel: [734355.620185] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen kernel: [734355.620199] ata6.00: cmd c8/00:08:ef:52:a9/00:00:00:00:00/e0 tag 0 cdb 0x0 data 4096 in kernel: [734355.620200] res 40/00:00:02:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) kernel: [734355.731595] ata6: soft resetting port kernel: [734355.787308] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) kernel: [734358.738407] ata5: port is slow to respond, please be patient (Status 0xd8) kernel: [734360.418260] ata5: device not ready (errno=-16), forcing hardreset kernel: [734360.418262] ata5: hard resetting port kernel: [734360.643958] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) kernel: [734366.509226] ata6.00: qc timeout (cmd 0x27) kernel: [734366.509231] ata6.00: ata_hpa_resize 1: hpa sectors (0) is smaller than sectors (976773168) kernel: [734366.509241] ata6.00: failed to set xfermode (err_mask=0x40) kernel: [734366.509250] ata6: failed to recover some devices, retrying in 5 secs kernel: [734368.296234] ata6: hard resetting port kernel: [734368.521935] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300) kernel: [734371.365916] ata5.00: qc timeout (cmd 0x27) kernel: [734371.365921] ata5.00: ata_hpa_resize 1: hpa sectors (0) is smaller than sectors (976773168) kernel: [734371.365928] ata5.00: failed to set xfermode (err_mask=0x40) kernel: [734371.365936] ata5: failed to recover some devices, retrying in 5 secs kernel: [734373.152942] ata5: hard resetting port kernel: [734373.378644] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) kernel: [734379.244112] ata6.00: qc timeout (cmd 0x27) kernel: [734379.244118] ata6.00: ata_hpa_resize 1: hpa sectors (0) is smaller than sectors (976773168) kernel: [734379.244126] ata6.00: failed to set xfermode (err_mask=0x40) kernel: [734379.244135] ata6: limiting SATA link speed to 1.5 Gbps kernel: [734379.244138] ata6.00: limiting speed to UDMA/133:PIO3 kernel: [734379.244140] ata6: failed to recover some devices, retrying in 5 secs kernel: [734381.031138] ata6: hard resetting port kernel: [734381.256840] ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 310) kernel: [734384.095108] ata5.00: qc timeout (cmd 0x27) kernel: [734384.095113] ata5.00: ata_hpa_resize 1: hpa sectors (0) is smaller than sectors (976773168) kernel: [734384.095120] ata5.00: failed to set xfermode (err_mask=0x40) kernel: [734384.095129] ata5: limiting SATA link speed to 1.5 Gbps kernel: [734384.095131] ata5.00: limiting speed to UDMA/133:PIO3 kernel: [734384.095133] ata5: failed to recover some devices, retrying in 5 secs kernel: [734385.882133] ata5: hard resetting port kernel: [734386.107836] ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310) kernel: [734391.979018] ata6.00: qc timeout (cmd 0x27) kernel: [734391.979024] ata6.00: ata_hpa_resize 1: hpa sectors (0) is smaller than sectors (976773168) kernel: [734391.979032] ata6.00: failed to set xfermode (err_mask=0x40) kernel: [734391.979041] ata6.00: disabled kernel: [734392.159007] ata6: EH complete kernel: [734392.159018] sd 5:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK kernel: [734392.159022] end_request: I/O error, dev sdd, sector 11096815 kernel: [734392.159026] raid1: sdd1: rescheduling sector 11096752 kernel: [734394.574692] sd 5:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK kernel: [734394.574697] end_request: I/O error, dev sdd, sector 976767935 kernel: [734394.574703] md: super_written gets error=-5, uptodate=0 kernel: [734394.574706] raid1: Disk failure on sdd1, disabling device. kernel: [734394.574707] ^IOperation continuing on 1 devices kernel: [734394.574960] sd 5:0:0:0: [sdd] READ CAPACITY failed kernel: [734394.574961] sd 5:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK kernel: [734394.574964] sd 5:0:0:0: [sdd] Sense not available. kernel: [734394.575233] sd 5:0:0:0: [sdd] Write Protect is off kernel: [734394.575235] sd 5:0:0:0: [sdd] Mode Sense: 00 00 00 00 kernel: [734394.575275] sd 5:0:0:0: [sdd] Asking for cache data failed kernel: [734394.575282] sd 5:0:0:0: [sdd] Assuming drive cache: write through kernel: [734394.577622] RAID1 conf printout: kernel: [734394.577624] --- wd:1 rd:2 kernel: [734394.577626] disk 0, wo:1, o:0, dev:sdd1 kernel: [734394.577627] disk 1, wo:0, o:1, dev:sdb1 kernel: [734394.583008] RAID1 conf printout: kernel: [734394.583010] --- wd:1 rd:2 kernel: [734394.583011] disk 1, wo:0, o:1, dev:sdb1 kernel: [734394.593189] raid1: sdb1: redirecting sector 11096752 to another mirror kernel: [734399.802574] ata5.00: qc timeout (cmd 0x27) kernel: [734399.802580] ata5.00: ata_hpa_resize 1: hpa sectors (0) is smaller than sectors (976773168) kernel: [734399.802588] ata5.00: failed to set xfermode (err_mask=0x40) kernel: [734399.802606] ata5.00: disabled kernel: [734400.306547] ata5: EH complete kernel: [734400.306601] sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK kernel: [734400.306605] end_request: I/O error, dev sdc, sector 42089727 kernel: [734400.306608] raid1: sdc1: rescheduling sector 42089664 kernel: [734400.306626] raid1: sdc1: rescheduling sector 42089672 kernel: [734400.306642] raid1: sdc1: rescheduling sector 42089680 kernel: [734400.306658] raid1: sdc1: rescheduling sector 42089688 kernel: [734400.306674] raid1: sdc1: rescheduling sector 42089696 kernel: [734400.306690] raid1: sdc1: rescheduling sector 42089704 kernel: [734400.306706] raid1: sdc1: rescheduling sector 42089712 kernel: [734400.306722] raid1: sdc1: rescheduling sector 42089720 kernel: [734400.306738] raid1: sdc1: rescheduling sector 42089728 kernel: [734400.307304] sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK kernel: [734400.307307] end_request: I/O error, dev sdc, sector 976767935 kernel: [734400.307313] md: super_written gets error=-5, uptodate=0 kernel: [734400.307315] raid1: Disk failure on sdc1, disabling device. kernel: [734400.307316] ^IOperation continuing on 1 devices kernel: [734400.307643] sd 4:0:0:0: [sdc] READ CAPACITY failed kernel: [734400.307645] sd 4:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK kernel: [734400.307647] sd 4:0:0:0: [sdc] Sense not available. kernel: [734400.307767] sd 4:0:0:0: [sdc] Write Protect is off kernel: [734400.307768] sd 4:0:0:0: [sdc] Mode Sense: 00 00 00 00 kernel: [734400.307947] sd 4:0:0:0: [sdc] Asking for cache data failed kernel: [734400.307963] sd 4:0:0:0: [sdc] Assuming drive cache: write through kernel: [734400.319399] RAID1 conf printout: kernel: [734400.319401] --- wd:1 rd:2 kernel: [734400.319402] disk 0, wo:1, o:0, dev:sdc1 kernel: [734400.319404] disk 1, wo:0, o:1, dev:sda1 kernel: [734400.330537] RAID1 conf printout: kernel: [734400.330539] --- wd:1 rd:2 kernel: [734400.330540] disk 1, wo:0, o:1, dev:sda1 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
