I ran speedtest on our raid-arrya (script included below). After
couple of hours bonnie get stuck in D-state. Also other
processes that access raid got stuck in D-state. I've seen this
before with 0.90-series raid.
Any idea how to debug this problem?
------------------------------------------------------------------------
2 ? DW 44:28 (kflushd)
6145 ? DW< 0:55 (raid5syncd)
5708 a1 SWN 0:00 | \_ (tst_raid5)
6149 a1 S N 0:00 | | \_ (tst_raid5)
6220 a1 D N 6:59 | | \_ bonnie -s 500
6221 a1 SWN 0:00 | | \_ (tee)
6222 a1 D N 7:00 | | \_ bonnie -s 500
6223 a1 SWN 0:00 | | \_ (tee)
6226 a1 D N 5:04 | | \_ (bonnie)
6227 a1 SWN 0:00 | | \_ (tee)
6228 a1 D N 7:02 | | \_ bonnie -s 500
6229 a1 SWN 0:00 | | \_ (tee)
6322 a0 D 0:00 ls -ltr /mnt/raid/
------------------------------------------------------------------------
# cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdh1[4] hdg1[3] hdf1[2] hde1[1] hdc1[0] 52828160 blocks level 5,
256k chunk, algorithm 2 [5/5] [UUUUU] resync=3% finish=25095.5min
unused devices: <none>
------------------------------------------------------------------------
Linux version 2.2.6 (root@localhost) (gcc version 2.7.2.3) #1 Fri Apr 23 20:31:13 EEST
1999
Detected 300691181 Hz processor.
ide_setup: ide0=dma
ide_setup: ide1=dma
ide_setup: ide2=dma
ide_setup: ide3=dma
Console: colour VGA+ 80x25
Calibrating delay loop... 599.65 BogoMIPS
Memory: 128168k/131072k available (840k kernel code, 412k reserved, 1616k data, 36k
init)
VFS: Diskquotas version dquot_6.4.0 initialized
CPU: AMD AMD-K6(tm) 3D processor stepping 00
Checking 386/387 coupling... OK, FPU using exception 16 error reporting.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
PCI: PCI BIOS revision 2.10 entry at 0xfb210
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: 00:38 [1106/0586]: Work around ISA DMA hangs (00)
Activating ISA DMA hang workarounds.
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
Initializing RT netlink socket
Starting kswapd v 1.5
Serial driver version 4.27 with no serial options enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
Real Time Clock Driver v1.09
VP_IDE: IDE controller on PCI bus 00 dev 39
VP_IDE: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xc000-0xc007, BIOS settings: hda:DMA, hdb:DMA
ide0: VIA Bus-Master (U)DMA Timing Config Success
ide1: BM-DMA at 0xc008-0xc00f, BIOS settings: hdc:DMA, hdd:DMA
ide1: VIA Bus-Master (U)DMA Timing Config Success
PDC20246: IDE controller on PCI bus 00 dev 48
PDC20246: not 100% native mode: will probe irqs later
PDC20246: ROM enabled at 0xe9000000
ide2: BM-DMA at 0xd800-0xd807, BIOS settings: hde:DMA, hdf:DMA
ide3: BM-DMA at 0xd808-0xd80f, BIOS settings: hdg:DMA, hdh:DMA
hda: FUJITSU MPC3032AT, ATA DISK drive
hdc: IBM-DTTA-351350, ATA DISK drive
hde: IBM-DTTA-351350, ATA DISK drive
hdf: IBM-DTTA-351350, ATA DISK drive
hdg: IBM-DTTA-351350, ATA DISK drive
hdh: IBM-DTTA-351350, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
ide2 at 0xc800-0xc807,0xcc06 on irq 10
ide3 at 0xd000-0xd007,0xd406 on irq 10
hda: FUJITSU MPC3032AT, 3093MB w/0kB Cache, CHS=785/128/63, UDMA
hdc: IBM-DTTA-351350, 12897MB w/464kB Cache, CHS=26205/16/63, UDMA
hde: IBM-DTTA-351350, 12897MB w/464kB Cache, CHS=26205/16/63, UDMA
hdf: IBM-DTTA-351350, 12897MB w/464kB Cache, CHS=26205/16/63, UDMA
hdg: IBM-DTTA-351350, 12897MB w/464kB Cache, CHS=26205/16/63, UDMA
hdh: IBM-DTTA-351350, 12897MB w/464kB Cache, CHS=26205/16/63, UDMA
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
md driver 0.90.0 MAX_MD_DEVS=256, MAX_REAL=12
raid5 personality registered
raid5: measuring checksumming speed
raid5: using high-speed MMX checksum routine
pII_mmx : 650.748 MB/sec
p5_mmx : 574.548 MB/sec
8regs : 456.438 MB/sec
32regs : 309.372 MB/sec
using fastest function: pII_mmx (650.748 MB/sec)
md.c: sizeof(mdp_super_t) = 4096
Partition check:
hda: hda1 hda2 < hda5 hda6 hda7 hda8 >
hdc: hdc1
hde: hde1
hdf: hdf1
hdg: hdg1
hdh: hdh1
autodetecting RAID arrays
(read) hdc1's sb offset: 13207168 [events: 0000000c]
(read) hde1's sb offset: 13207168 [events: 0000000c]
(read) hdf1's sb offset: 13207168 [events: 0000000c]
(read) hdg1's sb offset: 13207168 [events: 0000000c]
(read) hdh1's sb offset: 13207168 [events: 0000000c]
autorun ...
considering hdh1 ...
adding hdh1 ...
adding hdg1 ...
adding hdf1 ...
adding hde1 ...
adding hdc1 ...
created md0
bind<hdc1,1>
bind<hde1,2>
bind<hdf1,3>
bind<hdg1,4>
bind<hdh1,5>
running: <hdh1><hdg1><hdf1><hde1><hdc1>
now!
hdh1's event counter: 0000000c
hdg1's event counter: 0000000c
hdf1's event counter: 0000000c
hde1's event counter: 0000000c
hdc1's event counter: 0000000c
md0: max total readahead window set to 512k
md0: 4 data-disks, max readahead per data-disk: 128k
raid5: device hdh1 operational as raid disk 4
raid5: device hdg1 operational as raid disk 3
raid5: device hdf1 operational as raid disk 2
raid5: device hde1 operational as raid disk 1
raid5: device hdc1 operational as raid disk 0
raid5: allocated 5299kB for md0
raid5: raid level 5 set md0 active with 5 out of 5 devices, algorithm 2
RAID5 conf printout:
--- rd:5 wd:5 fd:0
disk 0, s:0, o:1, n:0 rd:0 us:1 dev:hdc1
disk 1, s:0, o:1, n:1 rd:1 us:1 dev:hde1
disk 2, s:0, o:1, n:2 rd:2 us:1 dev:hdf1
disk 3, s:0, o:1, n:3 rd:3 us:1 dev:hdg1
disk 4, s:0, o:1, n:4 rd:4 us:1 dev:hdh1
disk 5, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 6, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 7, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 8, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 9, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 10, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 11, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
RAID5 conf printout:
--- rd:5 wd:5 fd:0
disk 0, s:0, o:1, n:0 rd:0 us:1 dev:hdc1
disk 1, s:0, o:1, n:1 rd:1 us:1 dev:hde1
disk 2, s:0, o:1, n:2 rd:2 us:1 dev:hdf1
disk 3, s:0, o:1, n:3 rd:3 us:1 dev:hdg1
disk 4, s:0, o:1, n:4 rd:4 us:1 dev:hdh1
disk 5, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 6, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 7, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 8, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 9, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 10, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
disk 11, s:0, o:0, n:0 rd:0 us:0 dev:[dev 00:00]
md: updating md0 RAID superblock on device
hdh1 [events: 0000000d](write) hdh1's sb offset: 13207168
hdg1 [events: 0000000d](write) hdg1's sb offset: 13207168
hdf1 [events: 0000000d](write) hdf1's sb offset: 13207168
hde1 [events: 0000000d](write) hde1's sb offset: 13207168
hdc1 [events: 0000000d](write) hdc1's sb offset: 13207168
.
... autorun DONE.
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 36k freed
Adding Swap: 130748k swap-space (priority -1)
------------------------------------------------------------------------------
#!/bin/sh
# Test raid
# Markus Linnala <[EMAIL PROTECTED]> February 8 1999 - April 26 1999
#set -x
PATH=$PATH:/sbin:/usr/sbin
DEV=/dev/md0
MNT=/mnt/raid
LOGPREFIX=/tmp/BT_5_2_raid_0_90
export inode block chunk stride DEV MNT LOGPREFIX MKFS_PARAMS
/etc/init.d/knfs stop
function do_bonnie () {
(
cd $MNT
for a in 200 350 500
do
sync
for b in 1 2 3 4 5
do
bonnie -s $a | tee -a $LOGPREFIX.$inode.$block.$chunk.$b &
done
wait
done)
}
function do_start () {
echo "Start"
mkraid --really-force $DEV
raidstart $DEV
mke2fs $MKFS_PARAMS $DEV
mount $DEV $MNT
}
function do_stop () {
echo "Stop"
umount $DEV
raidstop $DEV
}
function do_raidtab () {
echo "Raidtab"
perl -pe 's/\@CS\@/$ENV{chunk}/' < /etc/raidtab.in > /etc/raidtab
}
function do_main () {
cd /
do_stop
for inode in 8192; do
for block in 1024 4096 16384; do
for chunk in 64 128 256 512 1024;do
stride=`expr $chunk \* 1024 \/ $block`
MKFS_PARAMS="-i $inode -b $block -R stride=$stride"
do_raidtab
do_start
do_bonnie
do_stop
done
done
done
}
do_main
------------------------------------------------------------------------------
--
//Markus
