Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
On Sat, 13 Jan 2007, Al Boldi wrote:
> Justin Piszcz wrote:
> > On Sat, 13 Jan 2007, Al Boldi wrote:
> > > Justin Piszcz wrote:
> > > > Btw, max sectors did improve my performance a little bit but
> > > > stripe_cache+read_ahead were the main optimizations that made
> > > > everything go faster by about ~1.5x. I have individual bonnie++
> > > > benchmarks of [only] the max_sector_kb tests as well, it improved the
> > > > times from 8min/bonnie run -> 7min 11 seconds or so, see below and
> > > > then after that is what you requested.
> > >
> > > Can you repeat with /dev/sda only?
> >
> > For sda-- (is a 74GB raptor only)-- but ok.
>
> Do you get the same results for the 150GB-raptor on sd{e,g,i,k}?
>
> > # uptime
> > 16:25:38 up 1 min, 3 users, load average: 0.23, 0.14, 0.05
> > # cat /sys/block/sda/queue/max_sectors_kb
> > 512
> > # echo 3 > /proc/sys/vm/drop_caches
> > # dd if=/dev/sda of=/dev/null bs=1M count=10240
> > 10240+0 records in
> > 10240+0 records out
> > 10737418240 bytes (11 GB) copied, 150.891 seconds, 71.2 MB/s
> > # echo 192 > /sys/block/sda/queue/max_sectors_kb
> > # echo 3 > /proc/sys/vm/drop_caches
> > # dd if=/dev/sda of=/dev/null bs=1M count=10240
> > 10240+0 records in
> > 10240+0 records out
> > 10737418240 bytes (11 GB) copied, 150.192 seconds, 71.5 MB/s
> > # echo 128 > /sys/block/sda/queue/max_sectors_kb
> > # echo 3 > /proc/sys/vm/drop_caches
> > # dd if=/dev/sda of=/dev/null bs=1M count=10240
> > 10240+0 records in
> > 10240+0 records out
> > 10737418240 bytes (11 GB) copied, 150.15 seconds, 71.5 MB/s
> >
> >
> > Does this show anything useful?
>
> Probably a latency issue. md is highly latency sensitive.
>
> What CPU type/speed do you have? Bootlog/dmesg?
>
>
> Thanks!
>
> --
> Al
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to [EMAIL PROTECTED]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> What CPU type/speed do you have? Bootlog/dmesg?
Core Duo E6300
The speed is great since I have tweaked the various settings..
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
On Sat, 13 Jan 2007, Al Boldi wrote:
Justin Piszcz wrote:
On Sat, 13 Jan 2007, Al Boldi wrote:
Justin Piszcz wrote:
Btw, max sectors did improve my performance a little bit but
stripe_cache+read_ahead were the main optimizations that made
everything go faster by about ~1.5x. I have individual bonnie++
benchmarks of [only] the max_sector_kb tests as well, it improved the
times from 8min/bonnie run - 7min 11 seconds or so, see below and
then after that is what you requested.
Can you repeat with /dev/sda only?
For sda-- (is a 74GB raptor only)-- but ok.
Do you get the same results for the 150GB-raptor on sd{e,g,i,k}?
# uptime
16:25:38 up 1 min, 3 users, load average: 0.23, 0.14, 0.05
# cat /sys/block/sda/queue/max_sectors_kb
512
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/sda of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 150.891 seconds, 71.2 MB/s
# echo 192 /sys/block/sda/queue/max_sectors_kb
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/sda of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 150.192 seconds, 71.5 MB/s
# echo 128 /sys/block/sda/queue/max_sectors_kb
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/sda of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 150.15 seconds, 71.5 MB/s
Does this show anything useful?
Probably a latency issue. md is highly latency sensitive.
What CPU type/speed do you have? Bootlog/dmesg?
Thanks!
--
Al
-
To unsubscribe from this list: send the line unsubscribe linux-raid in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
What CPU type/speed do you have? Bootlog/dmesg?
Core Duo E6300
The speed is great since I have tweaked the various settings..
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Justin Piszcz wrote:
> On Sat, 13 Jan 2007, Al Boldi wrote:
> > Justin Piszcz wrote:
> > > Btw, max sectors did improve my performance a little bit but
> > > stripe_cache+read_ahead were the main optimizations that made
> > > everything go faster by about ~1.5x. I have individual bonnie++
> > > benchmarks of [only] the max_sector_kb tests as well, it improved the
> > > times from 8min/bonnie run -> 7min 11 seconds or so, see below and
> > > then after that is what you requested.
> >
> > Can you repeat with /dev/sda only?
>
> For sda-- (is a 74GB raptor only)-- but ok.
Do you get the same results for the 150GB-raptor on sd{e,g,i,k}?
> # uptime
> 16:25:38 up 1 min, 3 users, load average: 0.23, 0.14, 0.05
> # cat /sys/block/sda/queue/max_sectors_kb
> 512
> # echo 3 > /proc/sys/vm/drop_caches
> # dd if=/dev/sda of=/dev/null bs=1M count=10240
> 10240+0 records in
> 10240+0 records out
> 10737418240 bytes (11 GB) copied, 150.891 seconds, 71.2 MB/s
> # echo 192 > /sys/block/sda/queue/max_sectors_kb
> # echo 3 > /proc/sys/vm/drop_caches
> # dd if=/dev/sda of=/dev/null bs=1M count=10240
> 10240+0 records in
> 10240+0 records out
> 10737418240 bytes (11 GB) copied, 150.192 seconds, 71.5 MB/s
> # echo 128 > /sys/block/sda/queue/max_sectors_kb
> # echo 3 > /proc/sys/vm/drop_caches
> # dd if=/dev/sda of=/dev/null bs=1M count=10240
> 10240+0 records in
> 10240+0 records out
> 10737418240 bytes (11 GB) copied, 150.15 seconds, 71.5 MB/s
>
>
> Does this show anything useful?
Probably a latency issue. md is highly latency sensitive.
What CPU type/speed do you have? Bootlog/dmesg?
Thanks!
--
Al
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
On Sat, 13 Jan 2007, Al Boldi wrote: > Justin Piszcz wrote: > > Btw, max sectors did improve my performance a little bit but > > stripe_cache+read_ahead were the main optimizations that made everything > > go faster by about ~1.5x. I have individual bonnie++ benchmarks of > > [only] the max_sector_kb tests as well, it improved the times from > > 8min/bonnie run -> 7min 11 seconds or so, see below and then after that is > > what you requested. > > > > # echo 3 > /proc/sys/vm/drop_caches > > # dd if=/dev/md3 of=/dev/null bs=1M count=10240 > > 10240+0 records in > > 10240+0 records out > > 10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s > > # for i in sde sdg sdi sdk; do echo 192 > > > /sys/block/"$i"/queue/max_sectors_kb; echo "Set > > /sys/block/"$i"/queue/max_sectors_kb to 192kb"; done > > Set /sys/block/sde/queue/max_sectors_kb to 192kb > > Set /sys/block/sdg/queue/max_sectors_kb to 192kb > > Set /sys/block/sdi/queue/max_sectors_kb to 192kb > > Set /sys/block/sdk/queue/max_sectors_kb to 192kb > > # echo 3 > /proc/sys/vm/drop_caches > > # dd if=/dev/md3 of=/dev/null bs=1M count=10240 > > 10240+0 records in > > 10240+0 records out > > 10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s > > > > Awful performance with your numbers/drop_caches settings.. ! > > Can you repeat with /dev/sda only? > > With fresh reboot to shell, then: > $ cat /sys/block/sda/queue/max_sectors_kb > $ echo 3 > /proc/sys/vm/drop_caches > $ dd if=/dev/sda of=/dev/null bs=1M count=10240 > > $ echo 192 > /sys/block/sda/queue/max_sectors_kb > $ echo 3 > /proc/sys/vm/drop_caches > $ dd if=/dev/sda of=/dev/null bs=1M count=10240 > > $ echo 128 > /sys/block/sda/queue/max_sectors_kb > $ echo 3 > /proc/sys/vm/drop_caches > $ dd if=/dev/sda of=/dev/null bs=1M count=10240 > > > What were your tests designed to show? > > A problem with the block-io. > > > Thanks! > > -- > Al > > - > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Here you go: For sda-- (is a 74GB raptor only)-- but ok. # uptime 16:25:38 up 1 min, 3 users, load average: 0.23, 0.14, 0.05 # cat /sys/block/sda/queue/max_sectors_kb 512 # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 150.891 seconds, 71.2 MB/s # # # # echo 192 > /sys/block/sda/queue/max_sectors_kb # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 150.192 seconds, 71.5 MB/s # echo 128 > /sys/block/sda/queue/max_sectors_kb # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 150.15 seconds, 71.5 MB/s Does this show anything useful? Justin. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Justin Piszcz wrote: > Btw, max sectors did improve my performance a little bit but > stripe_cache+read_ahead were the main optimizations that made everything > go faster by about ~1.5x. I have individual bonnie++ benchmarks of > [only] the max_sector_kb tests as well, it improved the times from > 8min/bonnie run -> 7min 11 seconds or so, see below and then after that is > what you requested. > > # echo 3 > /proc/sys/vm/drop_caches > # dd if=/dev/md3 of=/dev/null bs=1M count=10240 > 10240+0 records in > 10240+0 records out > 10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s > # for i in sde sdg sdi sdk; do echo 192 > > /sys/block/"$i"/queue/max_sectors_kb; echo "Set > /sys/block/"$i"/queue/max_sectors_kb to 192kb"; done > Set /sys/block/sde/queue/max_sectors_kb to 192kb > Set /sys/block/sdg/queue/max_sectors_kb to 192kb > Set /sys/block/sdi/queue/max_sectors_kb to 192kb > Set /sys/block/sdk/queue/max_sectors_kb to 192kb > # echo 3 > /proc/sys/vm/drop_caches > # dd if=/dev/md3 of=/dev/null bs=1M count=10240 > 10240+0 records in > 10240+0 records out > 10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s > > Awful performance with your numbers/drop_caches settings.. ! Can you repeat with /dev/sda only? With fresh reboot to shell, then: $ cat /sys/block/sda/queue/max_sectors_kb $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 $ echo 192 > /sys/block/sda/queue/max_sectors_kb $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 $ echo 128 > /sys/block/sda/queue/max_sectors_kb $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 > What were your tests designed to show? A problem with the block-io. Thanks! -- Al - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Justin Piszcz wrote: # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s # for i in sde sdg sdi sdk; do echo 192 > /sys/block/"$i"/queue/max_sectors_kb; echo "Set /sys/block/"$i"/queue/max_sectors_kb to 192kb"; done Set /sys/block/sde/queue/max_sectors_kb to 192kb Set /sys/block/sdg/queue/max_sectors_kb to 192kb Set /sys/block/sdi/queue/max_sectors_kb to 192kb Set /sys/block/sdk/queue/max_sectors_kb to 192kb # echo 3 > /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s Awful performance with your numbers/drop_caches settings.. ! What were your tests designed to show? To start, I expect then to show change in write, not read... and IIRC (I didn't look it up) drop_caches just flushes the caches so you start with known memory contents, none. Justin. On Fri, 12 Jan 2007, Justin Piszcz wrote: On Fri, 12 Jan 2007, Al Boldi wrote: Justin Piszcz wrote: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU This should be 1:14 not 1:06(was with a similarly sized file but not the same) the 1:14 is the same file as used with the other benchmarks. and to get that I used 256mb read-ahead and 16384 stripe size ++ 128 max_sectors_kb (same size as my sw raid5 chunk size) max_sectors_kb is probably your key. On my system I get twice the read performance by just reducing max_sectors_kb from default 512 to 192. Can you do a fresh reboot to shell and then: $ cat /sys/block/hda/queue/* $ cat /proc/meminfo $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 $ echo 192 > /sys/block/hda/queue/max_sectors_kb $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 -- bill davidsen <[EMAIL PROTECTED]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Btw, max sectors did improve my performance a little bit but
stripe_cache+read_ahead were the main optimizations that made everything
go faster by about ~1.5x. I have individual bonnie++ benchmarks of
[only] the max_sector_kb tests as well, it improved the times from 8min/bonnie
run -> 7min 11 seconds or so, see below and then after that is what you
requested.
# Options used:
# blockdev --setra 1536 /dev/md3 (back to default)
# cat /sys/block/sd{e,g,i,k}/queue/max_sectors_kb
# value: 512
# value: 512
# value: 512
# value: 512
# Test with, chunksize of raid array (128)
# echo 128 > /sys/block/sde/queue/max_sectors_kb
# echo 128 > /sys/block/sdg/queue/max_sectors_kb
# echo 128 > /sys/block/sdi/queue/max_sectors_kb
# echo 128 > /sys/block/sdk/queue/max_sectors_kb
max_sectors_kb128_run1:max_sectors_kb128_run1,4000M,46522,98,109829,19,42776,12,46527,97,86206,14,647.7,1,16:10:16/64,874,9,29123,97,2778,16,852,9,25399,86,1396,10
max_sectors_kb128_run2:max_sectors_kb128_run2,4000M,44037,99,107971,19,42420,12,46385,97,85773,14,628.8,1,16:10:16/64,981,10,23006,77,3185,19,848,9,27891,94,1737,13
max_sectors_kb128_run3:max_sectors_kb128_run3,4000M,46501,98,108313,19,42558,12,46314,97,87697,15,617.0,1,16:10:16/64,864,9,29795,99,2744,16,897,9,29021,98,1439,10
max_sectors_kb128_run4:max_sectors_kb128_run4,4000M,40750,98,108959,19,42519,12,45027,97,86484,14,637.0,1,16:10:16/64,929,10,29641,98,2476,14,883,9,29529,99,1867,13
max_sectors_kb128_run5:max_sectors_kb128_run5,4000M,46664,98,108387,19,42801,12,46423,97,87379,14,642.5,0,16:10:16/64,925,10,29756,99,2759,16,915,10,28694,97,1215,8
162.54user 43.96system 7:12.02elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (5major+1104minor)pagefaults 0swaps
168.75user 43.51system 7:14.49elapsed 48%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1092minor)pagefaults 0swaps
162.76user 44.18system 7:12.26elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1096minor)pagefaults 0swaps
178.91user 43.39system 7:24.39elapsed 50%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1094minor)pagefaults 0swaps
162.45user 43.86system 7:11.26elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1092minor)pagefaults 0swaps
---
# cat /sys/block/sd[abcdefghijk]/queue/*
cat: /sys/block/sda/queue/iosched: Is a directory
32767
512
128
128
noop [anticipatory]
cat: /sys/block/sdb/queue/iosched: Is a directory
32767
512
128
128
noop [anticipatory]
cat: /sys/block/sdc/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdd/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sde/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdf/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdg/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdh/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdi/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdj/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdk/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
#
(note I am only using four of these (which are raptors, in raid5 for md3))
# cat /proc/meminfo
MemTotal: 2048904 kB
MemFree: 1299980 kB
Buffers: 1408 kB
Cached: 58032 kB
SwapCached: 0 kB
Active: 65012 kB
Inactive:33796 kB
HighTotal: 1153312 kB
HighFree: 1061792 kB
LowTotal: 895592 kB
LowFree:238188 kB
SwapTotal: 2200760 kB
SwapFree: 2200760 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 39332 kB
Mapped: 20248 kB
Slab:37116 kB
SReclaimable:10580 kB
SUnreclaim: 26536 kB
PageTables: 1284 kB
NFS_Unstable:0 kB
Bounce: 0 kB
CommitLimit: 3225212 kB
Committed_AS: 111056 kB
VmallocTotal: 114680 kB
VmallocUsed: 3828 kB
VmallocChunk: 110644 kB
#
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/md3 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s
# for i in sde sdg sdi sdk; do echo 192 >
/sys/block/"$i"/queue/max_sectors_kb; echo "Set
/sys/block/"$i"/queue/max_sectors_kb to 192kb"; done
Set /sys/block/sde/queue/max_sectors_kb to 192kb
Set /sys/block/sdg/queue/max_sectors_kb to 192kb
Set /sys/block/sdi/queue/max_sectors_kb to 192kb
Set /sys/block/sdk/queue/max_sectors_kb to 192kb
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/md3 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s
Awful performance with your numbers/drop_caches settings.. !
What were your tests designed to show?
Justin.
On Fri,
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
On Fri, 12 Jan 2007, Al Boldi wrote: > Justin Piszcz wrote: > > RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU > > > > This should be 1:14 not 1:06(was with a similarly sized file but not the > > same) the 1:14 is the same file as used with the other benchmarks. and to > > get that I used 256mb read-ahead and 16384 stripe size ++ 128 > > max_sectors_kb (same size as my sw raid5 chunk size) > > max_sectors_kb is probably your key. On my system I get twice the read > performance by just reducing max_sectors_kb from default 512 to 192. > > Can you do a fresh reboot to shell and then: > $ cat /sys/block/hda/queue/* > $ cat /proc/meminfo > $ echo 3 > /proc/sys/vm/drop_caches > $ dd if=/dev/hda of=/dev/null bs=1M count=10240 > $ echo 192 > /sys/block/hda/queue/max_sectors_kb > $ echo 3 > /proc/sys/vm/drop_caches > $ dd if=/dev/hda of=/dev/null bs=1M count=10240 > > > Thanks! > > -- > Al > > - > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Ok. sec - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Justin Piszcz wrote: > RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU > > This should be 1:14 not 1:06(was with a similarly sized file but not the > same) the 1:14 is the same file as used with the other benchmarks. and to > get that I used 256mb read-ahead and 16384 stripe size ++ 128 > max_sectors_kb (same size as my sw raid5 chunk size) max_sectors_kb is probably your key. On my system I get twice the read performance by just reducing max_sectors_kb from default 512 to 192. Can you do a fresh reboot to shell and then: $ cat /sys/block/hda/queue/* $ cat /proc/meminfo $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 $ echo 192 > /sys/block/hda/queue/max_sectors_kb $ echo 3 > /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 Thanks! -- Al - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU This should be 1:14 not 1:06(was with a similarly sized file but not the same) the 1:14 is the same file as used with the other benchmarks. and to get that I used 256mb read-ahead and 16384 stripe size ++ 128 max_sectors_kb (same size as my sw raid5 chunk size) On Fri, 12 Jan 2007, Justin Piszcz wrote: > > > On Fri, 12 Jan 2007, Michael Tokarev wrote: > > > Justin Piszcz wrote: > > > Using 4 raptor 150s: > > > > > > Without the tweaks, I get 111MB/s write and 87MB/s read. > > > With the tweaks, 195MB/s write and 211MB/s read. > > > > > > Using kernel 2.6.19.1. > > > > > > Without the tweaks and with the tweaks: > > > > > > # Stripe tests: > > > echo 8192 > /sys/block/md3/md/stripe_cache_size > > > > > > # DD TESTS [WRITE] > > > > > > DEFAULT: (512K) > > > $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 > > > 10240+0 records in > > > 10240+0 records out > > > 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s > > [] > > > 8192K READ AHEAD > > > $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M > > > 10240+0 records in > > > 10240+0 records out > > > 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s > > > > What exactly are you measuring? Linear read/write, like copying one > > device to another (or to /dev/null), in large chunks? > Check bonnie benchmarks below. > > > > I don't think it's an interesting test. Hint: how many times a day > > you plan to perform such a copy? > It is a measurement of raw performance. > > > > (By the way, for a copy of one block device to another, try using > > O_DIRECT, with two dd processes doing the copy - one reading, and > > another writing - this way, you'll get best results without huge > > affect on other things running on the system. Like this: > > > > dd if=/dev/onedev bs=1M iflag=direct | > > dd of=/dev/twodev bs=1M oflag=direct > > ) > Interesting, I will take this into consideration-- however, an untar test > shows a 2:1 improvement, see below. > > > > /mjt > > > > Decompress/unrar a DVD-sized file: > > On the following RAID volumes with the same set of [4] 150GB raptors: > > RAID 0] 1:13.16 elapsed @ 49% CPU > RAID 4] 2:05.85 elapsed @ 30% CPU > RAID 5] 2:01.94 elapsed @ 32% CPU > RAID 6] 2:39.34 elapsed @ 24% CPU > RAID 10] 1:52.37 elapsed @ 32% CPU > > RAID 5 Tweaked (8192 stripe_cache & 16384 setra/blockdev):: > > RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU > > I did not tweak raid 0, but seeing how RAID5 tweaked is faster than RAID0 > is good enough for me :) > > RAID0 did 278MB/s read and 317MB/s write (by the way) > > Here are the bonnie results, the times alone speak for themselves, from 8 > minutes to min and 48-59 seconds. > > # No optimizations: > # Run Benchmarks > Default Bonnie: > [nr_requests=128,max_sectors_kb=512,stripe_cache_size=256,read_ahead=1536] > default_run1,4000M,42879,98,105436,19,41081,11,46277,96,87845,15,639.2,1,16:10:16/64,380,4,29642,99,2990,18,469,5,11784,40,1712,12 > default_run2,4000M,47145,99,108664,19,40931,11,46466,97,94158,16,634.8,0,16:10:16/64,377,4,16990,56,2850,17,431,4,21066,71,1800,13 > default_run3,4000M,43653,98,109063,19,40898,11,46447,97,97141,16,645.8,1,16:10:16/64,373,4,22302,75,2793,16,420,4,16708,56,1794,13 > default_run4,4000M,46485,98,110664,20,41102,11,46443,97,93616,16,631.3,1,16:10:16/64,363,3,14484,49,2802,17,388,4,25532,86,1604,12 > default_run5,4000M,43813,98,109800,19,41214,11,46457,97,92563,15,635.1,1,16:10:16/64,376,4,28990,95,2827,17,388,4,22874,76,1817,13 > > 169.88user 44.01system 8:02.98elapsed 44%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (6major+1102minor)pagefaults 0swaps > 161.60user 44.33system 7:53.14elapsed 43%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (13major+1095minor)pagefaults 0swaps > 166.64user 45.24system 8:00.07elapsed 44%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (13major+1096minor)pagefaults 0swaps > 161.90user 44.66system 8:00.85elapsed 42%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (13major+1094minor)pagefaults 0swaps > 167.61user 44.12system 8:03.26elapsed 43%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (13major+1092minor)pagefaults 0swaps > > > All optimizations [bonnie++] > > 168.08user 46.05system 5:55.13elapsed 60%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (16major+1092minor)pagefaults 0swaps > 162.65user 46.21system 5:48.47elapsed 59%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (7major+1101minor)pagefaults 0swaps > 168.06user 45.74system 5:59.84elapsed 59%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (7major+1102minor)pagefaults 0swaps > 168.00user 46.18system 5:58.77elapsed 59%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (13major+1095minor)pagefaults 0swaps > 167.98user 45.53system 5:56.49elapsed 59%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (5major+1101minor)pagefaults
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
On Fri, 12 Jan 2007, Michael Tokarev wrote: > Justin Piszcz wrote: > > Using 4 raptor 150s: > > > > Without the tweaks, I get 111MB/s write and 87MB/s read. > > With the tweaks, 195MB/s write and 211MB/s read. > > > > Using kernel 2.6.19.1. > > > > Without the tweaks and with the tweaks: > > > > # Stripe tests: > > echo 8192 > /sys/block/md3/md/stripe_cache_size > > > > # DD TESTS [WRITE] > > > > DEFAULT: (512K) > > $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 > > 10240+0 records in > > 10240+0 records out > > 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s > [] > > 8192K READ AHEAD > > $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M > > 10240+0 records in > > 10240+0 records out > > 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s > > What exactly are you measuring? Linear read/write, like copying one > device to another (or to /dev/null), in large chunks? Check bonnie benchmarks below. > > I don't think it's an interesting test. Hint: how many times a day > you plan to perform such a copy? It is a measurement of raw performance. > > (By the way, for a copy of one block device to another, try using > O_DIRECT, with two dd processes doing the copy - one reading, and > another writing - this way, you'll get best results without huge > affect on other things running on the system. Like this: > > dd if=/dev/onedev bs=1M iflag=direct | > dd of=/dev/twodev bs=1M oflag=direct > ) Interesting, I will take this into consideration-- however, an untar test shows a 2:1 improvement, see below. > > /mjt > Decompress/unrar a DVD-sized file: On the following RAID volumes with the same set of [4] 150GB raptors: RAID 0] 1:13.16 elapsed @ 49% CPU RAID 4] 2:05.85 elapsed @ 30% CPU RAID 5] 2:01.94 elapsed @ 32% CPU RAID 6] 2:39.34 elapsed @ 24% CPU RAID 10] 1:52.37 elapsed @ 32% CPU RAID 5 Tweaked (8192 stripe_cache & 16384 setra/blockdev):: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU I did not tweak raid 0, but seeing how RAID5 tweaked is faster than RAID0 is good enough for me :) RAID0 did 278MB/s read and 317MB/s write (by the way) Here are the bonnie results, the times alone speak for themselves, from 8 minutes to min and 48-59 seconds. # No optimizations: # Run Benchmarks Default Bonnie: [nr_requests=128,max_sectors_kb=512,stripe_cache_size=256,read_ahead=1536] default_run1,4000M,42879,98,105436,19,41081,11,46277,96,87845,15,639.2,1,16:10:16/64,380,4,29642,99,2990,18,469,5,11784,40,1712,12 default_run2,4000M,47145,99,108664,19,40931,11,46466,97,94158,16,634.8,0,16:10:16/64,377,4,16990,56,2850,17,431,4,21066,71,1800,13 default_run3,4000M,43653,98,109063,19,40898,11,46447,97,97141,16,645.8,1,16:10:16/64,373,4,22302,75,2793,16,420,4,16708,56,1794,13 default_run4,4000M,46485,98,110664,20,41102,11,46443,97,93616,16,631.3,1,16:10:16/64,363,3,14484,49,2802,17,388,4,25532,86,1604,12 default_run5,4000M,43813,98,109800,19,41214,11,46457,97,92563,15,635.1,1,16:10:16/64,376,4,28990,95,2827,17,388,4,22874,76,1817,13 169.88user 44.01system 8:02.98elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (6major+1102minor)pagefaults 0swaps 161.60user 44.33system 7:53.14elapsed 43%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1095minor)pagefaults 0swaps 166.64user 45.24system 8:00.07elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1096minor)pagefaults 0swaps 161.90user 44.66system 8:00.85elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1094minor)pagefaults 0swaps 167.61user 44.12system 8:03.26elapsed 43%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1092minor)pagefaults 0swaps All optimizations [bonnie++] 168.08user 46.05system 5:55.13elapsed 60%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (16major+1092minor)pagefaults 0swaps 162.65user 46.21system 5:48.47elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (7major+1101minor)pagefaults 0swaps 168.06user 45.74system 5:59.84elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (7major+1102minor)pagefaults 0swaps 168.00user 46.18system 5:58.77elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1095minor)pagefaults 0swaps 167.98user 45.53system 5:56.49elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (5major+1101minor)pagefaults 0swaps c6300-optimized:4000M,43976,99,167209,29,73109,22,43471,91,208572,40,511.4,1,16:10:16/64,1109,12,26948,89,2469,14,1051,11,29037,97,2167,16 c6300-optimized:4000M,47455,99,190212,35,70402,21,43167,92,206290,40,503.3,1,16:10:16/64,1071,11,29893,99,2804,16,1059,12,24887,84,2090,16 c6300-optimized:4000M,43979,99,172543,29,71811,21,41760,87,201870,39,498.9,1,16:10:16/64,1042,11,30276,99,2800,16,1063,12,29491,99,2257,17 c6300-optimized:4000M,43824,98,164585,29,73470,22,43098,90,207003,40,489.1,1,16:10:16/64,1045,11,30288,98,2512,15,1018,11,27365,92,2097,16
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Justin Piszcz wrote: > Using 4 raptor 150s: > > Without the tweaks, I get 111MB/s write and 87MB/s read. > With the tweaks, 195MB/s write and 211MB/s read. > > Using kernel 2.6.19.1. > > Without the tweaks and with the tweaks: > > # Stripe tests: > echo 8192 > /sys/block/md3/md/stripe_cache_size > > # DD TESTS [WRITE] > > DEFAULT: (512K) > $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 > 10240+0 records in > 10240+0 records out > 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s [] > 8192K READ AHEAD > $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M > 10240+0 records in > 10240+0 records out > 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s What exactly are you measuring? Linear read/write, like copying one device to another (or to /dev/null), in large chunks? I don't think it's an interesting test. Hint: how many times a day you plan to perform such a copy? (By the way, for a copy of one block device to another, try using O_DIRECT, with two dd processes doing the copy - one reading, and another writing - this way, you'll get best results without huge affect on other things running on the system. Like this: dd if=/dev/onedev bs=1M iflag=direct | dd of=/dev/twodev bs=1M oflag=direct ) /mjt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Justin Piszcz wrote: Using 4 raptor 150s: Without the tweaks, I get 111MB/s write and 87MB/s read. With the tweaks, 195MB/s write and 211MB/s read. Using kernel 2.6.19.1. Without the tweaks and with the tweaks: # Stripe tests: echo 8192 /sys/block/md3/md/stripe_cache_size # DD TESTS [WRITE] DEFAULT: (512K) $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s [] 8192K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s What exactly are you measuring? Linear read/write, like copying one device to another (or to /dev/null), in large chunks? I don't think it's an interesting test. Hint: how many times a day you plan to perform such a copy? (By the way, for a copy of one block device to another, try using O_DIRECT, with two dd processes doing the copy - one reading, and another writing - this way, you'll get best results without huge affect on other things running on the system. Like this: dd if=/dev/onedev bs=1M iflag=direct | dd of=/dev/twodev bs=1M oflag=direct ) /mjt - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
On Fri, 12 Jan 2007, Michael Tokarev wrote: Justin Piszcz wrote: Using 4 raptor 150s: Without the tweaks, I get 111MB/s write and 87MB/s read. With the tweaks, 195MB/s write and 211MB/s read. Using kernel 2.6.19.1. Without the tweaks and with the tweaks: # Stripe tests: echo 8192 /sys/block/md3/md/stripe_cache_size # DD TESTS [WRITE] DEFAULT: (512K) $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s [] 8192K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s What exactly are you measuring? Linear read/write, like copying one device to another (or to /dev/null), in large chunks? Check bonnie benchmarks below. I don't think it's an interesting test. Hint: how many times a day you plan to perform such a copy? It is a measurement of raw performance. (By the way, for a copy of one block device to another, try using O_DIRECT, with two dd processes doing the copy - one reading, and another writing - this way, you'll get best results without huge affect on other things running on the system. Like this: dd if=/dev/onedev bs=1M iflag=direct | dd of=/dev/twodev bs=1M oflag=direct ) Interesting, I will take this into consideration-- however, an untar test shows a 2:1 improvement, see below. /mjt Decompress/unrar a DVD-sized file: On the following RAID volumes with the same set of [4] 150GB raptors: RAID 0] 1:13.16 elapsed @ 49% CPU RAID 4] 2:05.85 elapsed @ 30% CPU RAID 5] 2:01.94 elapsed @ 32% CPU RAID 6] 2:39.34 elapsed @ 24% CPU RAID 10] 1:52.37 elapsed @ 32% CPU RAID 5 Tweaked (8192 stripe_cache 16384 setra/blockdev):: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU I did not tweak raid 0, but seeing how RAID5 tweaked is faster than RAID0 is good enough for me :) RAID0 did 278MB/s read and 317MB/s write (by the way) Here are the bonnie results, the times alone speak for themselves, from 8 minutes to min and 48-59 seconds. # No optimizations: # Run Benchmarks Default Bonnie: [nr_requests=128,max_sectors_kb=512,stripe_cache_size=256,read_ahead=1536] default_run1,4000M,42879,98,105436,19,41081,11,46277,96,87845,15,639.2,1,16:10:16/64,380,4,29642,99,2990,18,469,5,11784,40,1712,12 default_run2,4000M,47145,99,108664,19,40931,11,46466,97,94158,16,634.8,0,16:10:16/64,377,4,16990,56,2850,17,431,4,21066,71,1800,13 default_run3,4000M,43653,98,109063,19,40898,11,46447,97,97141,16,645.8,1,16:10:16/64,373,4,22302,75,2793,16,420,4,16708,56,1794,13 default_run4,4000M,46485,98,110664,20,41102,11,46443,97,93616,16,631.3,1,16:10:16/64,363,3,14484,49,2802,17,388,4,25532,86,1604,12 default_run5,4000M,43813,98,109800,19,41214,11,46457,97,92563,15,635.1,1,16:10:16/64,376,4,28990,95,2827,17,388,4,22874,76,1817,13 169.88user 44.01system 8:02.98elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (6major+1102minor)pagefaults 0swaps 161.60user 44.33system 7:53.14elapsed 43%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1095minor)pagefaults 0swaps 166.64user 45.24system 8:00.07elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1096minor)pagefaults 0swaps 161.90user 44.66system 8:00.85elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1094minor)pagefaults 0swaps 167.61user 44.12system 8:03.26elapsed 43%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1092minor)pagefaults 0swaps All optimizations [bonnie++] 168.08user 46.05system 5:55.13elapsed 60%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (16major+1092minor)pagefaults 0swaps 162.65user 46.21system 5:48.47elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (7major+1101minor)pagefaults 0swaps 168.06user 45.74system 5:59.84elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (7major+1102minor)pagefaults 0swaps 168.00user 46.18system 5:58.77elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1095minor)pagefaults 0swaps 167.98user 45.53system 5:56.49elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (5major+1101minor)pagefaults 0swaps c6300-optimized:4000M,43976,99,167209,29,73109,22,43471,91,208572,40,511.4,1,16:10:16/64,1109,12,26948,89,2469,14,1051,11,29037,97,2167,16 c6300-optimized:4000M,47455,99,190212,35,70402,21,43167,92,206290,40,503.3,1,16:10:16/64,1071,11,29893,99,2804,16,1059,12,24887,84,2090,16 c6300-optimized:4000M,43979,99,172543,29,71811,21,41760,87,201870,39,498.9,1,16:10:16/64,1042,11,30276,99,2800,16,1063,12,29491,99,2257,17 c6300-optimized:4000M,43824,98,164585,29,73470,22,43098,90,207003,40,489.1,1,16:10:16/64,1045,11,30288,98,2512,15,1018,11,27365,92,2097,16
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU This should be 1:14 not 1:06(was with a similarly sized file but not the same) the 1:14 is the same file as used with the other benchmarks. and to get that I used 256mb read-ahead and 16384 stripe size ++ 128 max_sectors_kb (same size as my sw raid5 chunk size) On Fri, 12 Jan 2007, Justin Piszcz wrote: On Fri, 12 Jan 2007, Michael Tokarev wrote: Justin Piszcz wrote: Using 4 raptor 150s: Without the tweaks, I get 111MB/s write and 87MB/s read. With the tweaks, 195MB/s write and 211MB/s read. Using kernel 2.6.19.1. Without the tweaks and with the tweaks: # Stripe tests: echo 8192 /sys/block/md3/md/stripe_cache_size # DD TESTS [WRITE] DEFAULT: (512K) $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s [] 8192K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s What exactly are you measuring? Linear read/write, like copying one device to another (or to /dev/null), in large chunks? Check bonnie benchmarks below. I don't think it's an interesting test. Hint: how many times a day you plan to perform such a copy? It is a measurement of raw performance. (By the way, for a copy of one block device to another, try using O_DIRECT, with two dd processes doing the copy - one reading, and another writing - this way, you'll get best results without huge affect on other things running on the system. Like this: dd if=/dev/onedev bs=1M iflag=direct | dd of=/dev/twodev bs=1M oflag=direct ) Interesting, I will take this into consideration-- however, an untar test shows a 2:1 improvement, see below. /mjt Decompress/unrar a DVD-sized file: On the following RAID volumes with the same set of [4] 150GB raptors: RAID 0] 1:13.16 elapsed @ 49% CPU RAID 4] 2:05.85 elapsed @ 30% CPU RAID 5] 2:01.94 elapsed @ 32% CPU RAID 6] 2:39.34 elapsed @ 24% CPU RAID 10] 1:52.37 elapsed @ 32% CPU RAID 5 Tweaked (8192 stripe_cache 16384 setra/blockdev):: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU I did not tweak raid 0, but seeing how RAID5 tweaked is faster than RAID0 is good enough for me :) RAID0 did 278MB/s read and 317MB/s write (by the way) Here are the bonnie results, the times alone speak for themselves, from 8 minutes to min and 48-59 seconds. # No optimizations: # Run Benchmarks Default Bonnie: [nr_requests=128,max_sectors_kb=512,stripe_cache_size=256,read_ahead=1536] default_run1,4000M,42879,98,105436,19,41081,11,46277,96,87845,15,639.2,1,16:10:16/64,380,4,29642,99,2990,18,469,5,11784,40,1712,12 default_run2,4000M,47145,99,108664,19,40931,11,46466,97,94158,16,634.8,0,16:10:16/64,377,4,16990,56,2850,17,431,4,21066,71,1800,13 default_run3,4000M,43653,98,109063,19,40898,11,46447,97,97141,16,645.8,1,16:10:16/64,373,4,22302,75,2793,16,420,4,16708,56,1794,13 default_run4,4000M,46485,98,110664,20,41102,11,46443,97,93616,16,631.3,1,16:10:16/64,363,3,14484,49,2802,17,388,4,25532,86,1604,12 default_run5,4000M,43813,98,109800,19,41214,11,46457,97,92563,15,635.1,1,16:10:16/64,376,4,28990,95,2827,17,388,4,22874,76,1817,13 169.88user 44.01system 8:02.98elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (6major+1102minor)pagefaults 0swaps 161.60user 44.33system 7:53.14elapsed 43%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1095minor)pagefaults 0swaps 166.64user 45.24system 8:00.07elapsed 44%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1096minor)pagefaults 0swaps 161.90user 44.66system 8:00.85elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1094minor)pagefaults 0swaps 167.61user 44.12system 8:03.26elapsed 43%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1092minor)pagefaults 0swaps All optimizations [bonnie++] 168.08user 46.05system 5:55.13elapsed 60%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (16major+1092minor)pagefaults 0swaps 162.65user 46.21system 5:48.47elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (7major+1101minor)pagefaults 0swaps 168.06user 45.74system 5:59.84elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (7major+1102minor)pagefaults 0swaps 168.00user 46.18system 5:58.77elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (13major+1095minor)pagefaults 0swaps 167.98user 45.53system 5:56.49elapsed 59%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (5major+1101minor)pagefaults 0swaps c6300-optimized:4000M,43976,99,167209,29,73109,22,43471,91,208572,40,511.4,1,16:10:16/64,1109,12,26948,89,2469,14,1051,11,29037,97,2167,16
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Justin Piszcz wrote: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU This should be 1:14 not 1:06(was with a similarly sized file but not the same) the 1:14 is the same file as used with the other benchmarks. and to get that I used 256mb read-ahead and 16384 stripe size ++ 128 max_sectors_kb (same size as my sw raid5 chunk size) max_sectors_kb is probably your key. On my system I get twice the read performance by just reducing max_sectors_kb from default 512 to 192. Can you do a fresh reboot to shell and then: $ cat /sys/block/hda/queue/* $ cat /proc/meminfo $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 $ echo 192 /sys/block/hda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 Thanks! -- Al - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
On Fri, 12 Jan 2007, Al Boldi wrote: Justin Piszcz wrote: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU This should be 1:14 not 1:06(was with a similarly sized file but not the same) the 1:14 is the same file as used with the other benchmarks. and to get that I used 256mb read-ahead and 16384 stripe size ++ 128 max_sectors_kb (same size as my sw raid5 chunk size) max_sectors_kb is probably your key. On my system I get twice the read performance by just reducing max_sectors_kb from default 512 to 192. Can you do a fresh reboot to shell and then: $ cat /sys/block/hda/queue/* $ cat /proc/meminfo $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 $ echo 192 /sys/block/hda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 Thanks! -- Al - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Ok. sec - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Btw, max sectors did improve my performance a little bit but
stripe_cache+read_ahead were the main optimizations that made everything
go faster by about ~1.5x. I have individual bonnie++ benchmarks of
[only] the max_sector_kb tests as well, it improved the times from 8min/bonnie
run - 7min 11 seconds or so, see below and then after that is what you
requested.
# Options used:
# blockdev --setra 1536 /dev/md3 (back to default)
# cat /sys/block/sd{e,g,i,k}/queue/max_sectors_kb
# value: 512
# value: 512
# value: 512
# value: 512
# Test with, chunksize of raid array (128)
# echo 128 /sys/block/sde/queue/max_sectors_kb
# echo 128 /sys/block/sdg/queue/max_sectors_kb
# echo 128 /sys/block/sdi/queue/max_sectors_kb
# echo 128 /sys/block/sdk/queue/max_sectors_kb
max_sectors_kb128_run1:max_sectors_kb128_run1,4000M,46522,98,109829,19,42776,12,46527,97,86206,14,647.7,1,16:10:16/64,874,9,29123,97,2778,16,852,9,25399,86,1396,10
max_sectors_kb128_run2:max_sectors_kb128_run2,4000M,44037,99,107971,19,42420,12,46385,97,85773,14,628.8,1,16:10:16/64,981,10,23006,77,3185,19,848,9,27891,94,1737,13
max_sectors_kb128_run3:max_sectors_kb128_run3,4000M,46501,98,108313,19,42558,12,46314,97,87697,15,617.0,1,16:10:16/64,864,9,29795,99,2744,16,897,9,29021,98,1439,10
max_sectors_kb128_run4:max_sectors_kb128_run4,4000M,40750,98,108959,19,42519,12,45027,97,86484,14,637.0,1,16:10:16/64,929,10,29641,98,2476,14,883,9,29529,99,1867,13
max_sectors_kb128_run5:max_sectors_kb128_run5,4000M,46664,98,108387,19,42801,12,46423,97,87379,14,642.5,0,16:10:16/64,925,10,29756,99,2759,16,915,10,28694,97,1215,8
162.54user 43.96system 7:12.02elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (5major+1104minor)pagefaults 0swaps
168.75user 43.51system 7:14.49elapsed 48%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1092minor)pagefaults 0swaps
162.76user 44.18system 7:12.26elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1096minor)pagefaults 0swaps
178.91user 43.39system 7:24.39elapsed 50%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1094minor)pagefaults 0swaps
162.45user 43.86system 7:11.26elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1092minor)pagefaults 0swaps
---
# cat /sys/block/sd[abcdefghijk]/queue/*
cat: /sys/block/sda/queue/iosched: Is a directory
32767
512
128
128
noop [anticipatory]
cat: /sys/block/sdb/queue/iosched: Is a directory
32767
512
128
128
noop [anticipatory]
cat: /sys/block/sdc/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdd/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sde/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdf/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdg/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdh/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdi/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdj/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdk/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
#
(note I am only using four of these (which are raptors, in raid5 for md3))
# cat /proc/meminfo
MemTotal: 2048904 kB
MemFree: 1299980 kB
Buffers: 1408 kB
Cached: 58032 kB
SwapCached: 0 kB
Active: 65012 kB
Inactive:33796 kB
HighTotal: 1153312 kB
HighFree: 1061792 kB
LowTotal: 895592 kB
LowFree:238188 kB
SwapTotal: 2200760 kB
SwapFree: 2200760 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 39332 kB
Mapped: 20248 kB
Slab:37116 kB
SReclaimable:10580 kB
SUnreclaim: 26536 kB
PageTables: 1284 kB
NFS_Unstable:0 kB
Bounce: 0 kB
CommitLimit: 3225212 kB
Committed_AS: 111056 kB
VmallocTotal: 114680 kB
VmallocUsed: 3828 kB
VmallocChunk: 110644 kB
#
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/md3 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s
# for i in sde sdg sdi sdk; do echo 192
/sys/block/$i/queue/max_sectors_kb; echo Set
/sys/block/$i/queue/max_sectors_kb to 192kb; done
Set /sys/block/sde/queue/max_sectors_kb to 192kb
Set /sys/block/sdg/queue/max_sectors_kb to 192kb
Set /sys/block/sdi/queue/max_sectors_kb to 192kb
Set /sys/block/sdk/queue/max_sectors_kb to 192kb
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/md3 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s
Awful performance with your numbers/drop_caches settings.. !
What were your tests designed to show?
Justin.
On Fri, 12 Jan 2007,
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Justin Piszcz wrote: # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s # for i in sde sdg sdi sdk; do echo 192 /sys/block/$i/queue/max_sectors_kb; echo Set /sys/block/$i/queue/max_sectors_kb to 192kb; done Set /sys/block/sde/queue/max_sectors_kb to 192kb Set /sys/block/sdg/queue/max_sectors_kb to 192kb Set /sys/block/sdi/queue/max_sectors_kb to 192kb Set /sys/block/sdk/queue/max_sectors_kb to 192kb # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s Awful performance with your numbers/drop_caches settings.. ! What were your tests designed to show? To start, I expect then to show change in write, not read... and IIRC (I didn't look it up) drop_caches just flushes the caches so you start with known memory contents, none. Justin. On Fri, 12 Jan 2007, Justin Piszcz wrote: On Fri, 12 Jan 2007, Al Boldi wrote: Justin Piszcz wrote: RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU This should be 1:14 not 1:06(was with a similarly sized file but not the same) the 1:14 is the same file as used with the other benchmarks. and to get that I used 256mb read-ahead and 16384 stripe size ++ 128 max_sectors_kb (same size as my sw raid5 chunk size) max_sectors_kb is probably your key. On my system I get twice the read performance by just reducing max_sectors_kb from default 512 to 192. Can you do a fresh reboot to shell and then: $ cat /sys/block/hda/queue/* $ cat /proc/meminfo $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 $ echo 192 /sys/block/hda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/hda of=/dev/null bs=1M count=10240 -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Justin Piszcz wrote: Btw, max sectors did improve my performance a little bit but stripe_cache+read_ahead were the main optimizations that made everything go faster by about ~1.5x. I have individual bonnie++ benchmarks of [only] the max_sector_kb tests as well, it improved the times from 8min/bonnie run - 7min 11 seconds or so, see below and then after that is what you requested. # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s # for i in sde sdg sdi sdk; do echo 192 /sys/block/$i/queue/max_sectors_kb; echo Set /sys/block/$i/queue/max_sectors_kb to 192kb; done Set /sys/block/sde/queue/max_sectors_kb to 192kb Set /sys/block/sdg/queue/max_sectors_kb to 192kb Set /sys/block/sdi/queue/max_sectors_kb to 192kb Set /sys/block/sdk/queue/max_sectors_kb to 192kb # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s Awful performance with your numbers/drop_caches settings.. ! Can you repeat with /dev/sda only? With fresh reboot to shell, then: $ cat /sys/block/sda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 $ echo 192 /sys/block/sda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 $ echo 128 /sys/block/sda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 What were your tests designed to show? A problem with the block-io. Thanks! -- Al - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
On Sat, 13 Jan 2007, Al Boldi wrote: Justin Piszcz wrote: Btw, max sectors did improve my performance a little bit but stripe_cache+read_ahead were the main optimizations that made everything go faster by about ~1.5x. I have individual bonnie++ benchmarks of [only] the max_sector_kb tests as well, it improved the times from 8min/bonnie run - 7min 11 seconds or so, see below and then after that is what you requested. # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s # for i in sde sdg sdi sdk; do echo 192 /sys/block/$i/queue/max_sectors_kb; echo Set /sys/block/$i/queue/max_sectors_kb to 192kb; done Set /sys/block/sde/queue/max_sectors_kb to 192kb Set /sys/block/sdg/queue/max_sectors_kb to 192kb Set /sys/block/sdi/queue/max_sectors_kb to 192kb Set /sys/block/sdk/queue/max_sectors_kb to 192kb # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/md3 of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s Awful performance with your numbers/drop_caches settings.. ! Can you repeat with /dev/sda only? With fresh reboot to shell, then: $ cat /sys/block/sda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 $ echo 192 /sys/block/sda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 $ echo 128 /sys/block/sda/queue/max_sectors_kb $ echo 3 /proc/sys/vm/drop_caches $ dd if=/dev/sda of=/dev/null bs=1M count=10240 What were your tests designed to show? A problem with the block-io. Thanks! -- Al - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Here you go: For sda-- (is a 74GB raptor only)-- but ok. # uptime 16:25:38 up 1 min, 3 users, load average: 0.23, 0.14, 0.05 # cat /sys/block/sda/queue/max_sectors_kb 512 # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 150.891 seconds, 71.2 MB/s # # # # echo 192 /sys/block/sda/queue/max_sectors_kb # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 150.192 seconds, 71.5 MB/s # echo 128 /sys/block/sda/queue/max_sectors_kb # echo 3 /proc/sys/vm/drop_caches # dd if=/dev/sda of=/dev/null bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 150.15 seconds, 71.5 MB/s Does this show anything useful? Justin. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Justin Piszcz wrote:
On Sat, 13 Jan 2007, Al Boldi wrote:
Justin Piszcz wrote:
Btw, max sectors did improve my performance a little bit but
stripe_cache+read_ahead were the main optimizations that made
everything go faster by about ~1.5x. I have individual bonnie++
benchmarks of [only] the max_sector_kb tests as well, it improved the
times from 8min/bonnie run - 7min 11 seconds or so, see below and
then after that is what you requested.
Can you repeat with /dev/sda only?
For sda-- (is a 74GB raptor only)-- but ok.
Do you get the same results for the 150GB-raptor on sd{e,g,i,k}?
# uptime
16:25:38 up 1 min, 3 users, load average: 0.23, 0.14, 0.05
# cat /sys/block/sda/queue/max_sectors_kb
512
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/sda of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 150.891 seconds, 71.2 MB/s
# echo 192 /sys/block/sda/queue/max_sectors_kb
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/sda of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 150.192 seconds, 71.5 MB/s
# echo 128 /sys/block/sda/queue/max_sectors_kb
# echo 3 /proc/sys/vm/drop_caches
# dd if=/dev/sda of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 150.15 seconds, 71.5 MB/s
Does this show anything useful?
Probably a latency issue. md is highly latency sensitive.
What CPU type/speed do you have? Bootlog/dmesg?
Thanks!
--
Al
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read & 195MB/s write)
Using 4 raptor 150s: Without the tweaks, I get 111MB/s write and 87MB/s read. With the tweaks, 195MB/s write and 211MB/s read. Using kernel 2.6.19.1. Without the tweaks and with the tweaks: # Stripe tests: echo 8192 > /sys/block/md3/md/stripe_cache_size # DD TESTS [WRITE] DEFAULT: (512K) $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s 8192 STRIPE CACHE $ dd if=/dev/zero of=10gb.8192k.stripe.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 55.0628 seconds, 195 MB/s (and again...) 10737418240 bytes (11 GB) copied, 61.9902 seconds, 173 MB/s (and again...) 10737418240 bytes (11 GB) copied, 61.3053 seconds, 175 MB/s ** maybe 16384 is better, need to do more testing. 16384 STRIPE CACHE $ dd if=/dev/zero of=10gb.16384k.stripe.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 56.2793 seconds, 191 MB/s 32768 STRIPE CACHE $ dd if=/dev/zero of=10gb.32768.stripe.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 55.8382 seconds, 192 MB/s # Set readahead. blockdev --setra 16384 /dev/md3 # DD TESTS [READ] DEFAULT: (1536K READ AHEAD) $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 298+0 records in 297+0 records out 311427072 bytes (311 MB) copied, 3.5453 seconds, 87.8 MB/s 2048K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 85.4632 seconds, 126 MB/s 8192K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s 16384K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 59.3119 seconds, 181 MB/s 32768 READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 56.6329 seconds, 190 MB/s 65536 READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 54.9768 seconds, 195 MB/s 131072 READ AHEAD 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 52.0896 seconds, 206 MB/s 262144 READ AHEAD** $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 50.8496 seconds, 211 MB/s (and again..) 10737418240 bytes (11 GB) copied, 51.2064 seconds, 210 MB/s*** 524288 READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 59.6508 seconds, 180 MB/s Output (vmstat) during a write test: procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 2 1172 730536 12 95274000 0 357720 1836 107450 0 80 6 15 1 1172 485016 12 119444800 0 171760 1604 42853 0 38 16 46 1 0172 243960 12 143214000 0 223088 1598 63118 0 44 25 31 0 0172 77428 12 159624000 0 199736 1559 56939 0 36 28 36 2 0172 50328 12 16227960016 87496 1726 31251 0 27 73 0 2 1172 50600 12 162205200 0 313432 1739 88026 0 53 16 32 1 1172 51012 12 162121600 0 200656 1586 56349 0 38 9 53 0 3172 50084 12 162240800 0 204320 1588 67085 0 40 24 36 1 1172 51716 12 162076000 0 245672 1608 81564 0 61 13 26 0 2172 51168 12 162143200 0 212740 1622 67203 0 44 22 34 0 2172 51940 12 162051600 0 203704 1614 59396 0 42 24 35 0 0172 51188 12 162134800 0 171744 1582 56664 0 38 28 34 1 0172 52264 12 162081200 0 143792 1724 43543 0 39 59 2 0 1172 48292 12 16239840016 248784 1610 73980 0 40 19 41 0 2172 51868 12 162059600 0 209184 1571 60611 0 40 20 40 1 1172 51168 12 162134000 0 205020 1620 70048 0 38 27 34 2 0172 51076 12 162150800 0 236400 1658 81582 0 59 13 29 0 0172 51284 12 162106400 0 138739 1611 40220 0 30 34 36 1 0172 52020 12 162037600 4 170200 1752 52315 0 38 58 5 Output (vmstat) during a read test: procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 0172 53484 12 176939600 0 0 1005 54 0 0 100 0 0 0172 53148 12 174038000 221752 0 1562 11779 0 22 70 9 0 0172 53868 12 170904800 23176416 1708 14658 0 37 54 9 2 0172 53384
Linux Software RAID 5 Performance Optimizations: 2.6.19.1: (211MB/s read 195MB/s write)
Using 4 raptor 150s: Without the tweaks, I get 111MB/s write and 87MB/s read. With the tweaks, 195MB/s write and 211MB/s read. Using kernel 2.6.19.1. Without the tweaks and with the tweaks: # Stripe tests: echo 8192 /sys/block/md3/md/stripe_cache_size # DD TESTS [WRITE] DEFAULT: (512K) $ dd if=/dev/zero of=10gb.no.optimizations.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 96.6988 seconds, 111 MB/s 8192 STRIPE CACHE $ dd if=/dev/zero of=10gb.8192k.stripe.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 55.0628 seconds, 195 MB/s (and again...) 10737418240 bytes (11 GB) copied, 61.9902 seconds, 173 MB/s (and again...) 10737418240 bytes (11 GB) copied, 61.3053 seconds, 175 MB/s ** maybe 16384 is better, need to do more testing. 16384 STRIPE CACHE $ dd if=/dev/zero of=10gb.16384k.stripe.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 56.2793 seconds, 191 MB/s 32768 STRIPE CACHE $ dd if=/dev/zero of=10gb.32768.stripe.out bs=1M count=10240 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 55.8382 seconds, 192 MB/s # Set readahead. blockdev --setra 16384 /dev/md3 # DD TESTS [READ] DEFAULT: (1536K READ AHEAD) $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 298+0 records in 297+0 records out 311427072 bytes (311 MB) copied, 3.5453 seconds, 87.8 MB/s 2048K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 85.4632 seconds, 126 MB/s 8192K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 64.9454 seconds, 165 MB/s 16384K READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 59.3119 seconds, 181 MB/s 32768 READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 56.6329 seconds, 190 MB/s 65536 READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 54.9768 seconds, 195 MB/s 131072 READ AHEAD 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 52.0896 seconds, 206 MB/s 262144 READ AHEAD** $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 50.8496 seconds, 211 MB/s (and again..) 10737418240 bytes (11 GB) copied, 51.2064 seconds, 210 MB/s*** 524288 READ AHEAD $ dd if=10gb.16384k.stripe.out of=/dev/null bs=1M 10240+0 records in 10240+0 records out 10737418240 bytes (11 GB) copied, 59.6508 seconds, 180 MB/s Output (vmstat) during a write test: procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 2 1172 730536 12 95274000 0 357720 1836 107450 0 80 6 15 1 1172 485016 12 119444800 0 171760 1604 42853 0 38 16 46 1 0172 243960 12 143214000 0 223088 1598 63118 0 44 25 31 0 0172 77428 12 159624000 0 199736 1559 56939 0 36 28 36 2 0172 50328 12 16227960016 87496 1726 31251 0 27 73 0 2 1172 50600 12 162205200 0 313432 1739 88026 0 53 16 32 1 1172 51012 12 162121600 0 200656 1586 56349 0 38 9 53 0 3172 50084 12 162240800 0 204320 1588 67085 0 40 24 36 1 1172 51716 12 162076000 0 245672 1608 81564 0 61 13 26 0 2172 51168 12 162143200 0 212740 1622 67203 0 44 22 34 0 2172 51940 12 162051600 0 203704 1614 59396 0 42 24 35 0 0172 51188 12 162134800 0 171744 1582 56664 0 38 28 34 1 0172 52264 12 162081200 0 143792 1724 43543 0 39 59 2 0 1172 48292 12 16239840016 248784 1610 73980 0 40 19 41 0 2172 51868 12 162059600 0 209184 1571 60611 0 40 20 40 1 1172 51168 12 162134000 0 205020 1620 70048 0 38 27 34 2 0172 51076 12 162150800 0 236400 1658 81582 0 59 13 29 0 0172 51284 12 162106400 0 138739 1611 40220 0 30 34 36 1 0172 52020 12 162037600 4 170200 1752 52315 0 38 58 5 Output (vmstat) during a read test: procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 0172 53484 12 176939600 0 0 1005 54 0 0 100 0 0 0172 53148 12 174038000 221752 0 1562 11779 0 22 70 9 0 0172 53868 12 170904800 23176416 1708 14658 0 37 54 9 2 0172 53384
