Bug#600031: updating kernel causes slow write speed to raid
On 15 October 2010 02:33, Ben Hutchings wrote: > On Thu, 2010-10-14 at 15:16 +0100, Dave Clarke wrote: >> Ben >> That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s >> read with the experimental kernel. I'm happy with these speeds on this >> hardware. It is still a little slower then with windows but I can live >> with that. > > Good, but I'm concerned as to why 2.6.32 is so much worse. Since that > will be the kernel version in Debian 6.0 I want to make sure it does > perform well. I'd be happy to test some fixes to 2.6.32 to find the source of the problem if there is anything you want me to try. >> May be able to get better by tuning read-ahead buffer and >> vm.dirty_ratio. Do you think there is any merit in adjusting these? > > I don't know; this isn't my area of expertise. > > Ben. > > -- > Ben Hutchings > Once a job is fouled up, anything done to improve it makes it worse. > Regards Dave -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktimajjbx8kqlauem98tb1pzqqh0gt5zmuffvm...@mail.gmail.com
Bug#600031: updating kernel causes slow write speed to raid
On Thu, 2010-10-14 at 15:16 +0100, Dave Clarke wrote: > Ben > That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s > read with the experimental kernel. I'm happy with these speeds on this > hardware. It is still a little slower then with windows but I can live > with that. Good, but I'm concerned as to why 2.6.32 is so much worse. Since that will be the kernel version in Debian 6.0 I want to make sure it does perform well. > May be able to get better by tuning read-ahead buffer and > vm.dirty_ratio. Do you think there is any merit in adjusting these? I don't know; this isn't my area of expertise. Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Bug#600031: updating kernel causes slow write speed to raid
Ben That has solved it. Now getting around 27MB/s write on ext4 and 97MB/s read with the experimental kernel. I'm happy with these speeds on this hardware. It is still a little slower then with windows but I can live with that. May be able to get better by tuning read-ahead buffer and vm.dirty_ratio. Do you think there is any merit in adjusting these? Thanks Dave On 14 October 2010 03:46, Ben Hutchings wrote: > On Thu, 2010-10-14 at 00:31 +0100, Dave Clarke wrote: >> Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4. >> using raid5. Also to note that when booted with windows server 2003 >> using Everest disk benchmark it does an average of 34MB/s (peaking to >> 50MB/s). >> >> Attached are bonnie++ output for the two kernel versions and ext3 and >> ext4 (ext4 won't mount with 2.6.26 kernel). >> And meminfo for both kernels taken when the system is loaded and unloaded. > > OK, it looks like it's not so short of memory. And this is really quite > poor write performance. > > Could you also test Linux 2.6.36-rc6, as packaged in experimental? > > Ben. > > -- > Ben Hutchings > Once a job is fouled up, anything done to improve it makes it worse. > == Loaded (bonnie++) == MemTotal: 514348 kB MemFree:5432 kB Buffers: 512 kB Cached: 464020 kB SwapCached: 148 kB Active: 250484 kB Inactive: 239976 kB Active(anon): 13300 kB Inactive(anon):13588 kB Active(file): 237184 kB Inactive(file): 226388 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 514348 kB LowFree:5432 kB SwapTotal: 1508348 kB SwapFree:1506756 kB Dirty:28 kB Writeback: 0 kB AnonPages: 25824 kB Mapped: 3584 kB Shmem: 916 kB Slab: 11876 kB SReclaimable: 5924 kB SUnreclaim: 5952 kB KernelStack: 840 kB PageTables: 1076 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit: 1765520 kB Committed_AS: 166828 kB VmallocTotal: 508152 kB VmallocUsed:7124 kB VmallocChunk: 486968 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 4096 kB DirectMap4k: 16128 kB DirectMap4M: 507904 kB == Unloaded == MemTotal: 514348 kB MemFree: 487468 kB Buffers: 948 kB Cached: 6584 kB SwapCached: 1952 kB Active: 3992 kB Inactive: 7680 kB Active(anon): 2084 kB Inactive(anon): 2444 kB Active(file): 1908 kB Inactive(file): 5236 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 514348 kB LowFree: 487468 kB SwapTotal: 1508348 kB SwapFree:1481796 kB Dirty:40 kB Writeback: 0 kB AnonPages: 2620 kB Mapped: 2636 kB Shmem: 380 kB Slab: 8312 kB SReclaimable: 2220 kB SUnreclaim: 6092 kB KernelStack: 848 kB PageTables: 1088 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit: 1765520 kB Committed_AS: 168020 kB VmallocTotal: 508152 kB VmallocUsed:7124 kB VmallocChunk: 486968 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 4096 kB DirectMap4k: 16128 kB DirectMap4M: 507904 kB Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP heterogeneous1G 384 92 29431 20 8050 5 1327 97 82875 19 514.1 29 Latency 296ms1322ms1178ms 52896us 58501us1105ms Version 1.96 --Sequential Create-- Random Create heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 3980 17 + +++ 7072 14 8390 34 + +++ 11415 23 Latency 55696us1290us1610us 479us 77us 273us 1.96,1.96,heterogeneous,1,1287064582,1G,,384,92,29431,20,8050,5,1327,97,82875,19,514.1,29,16,3980,17,+,+++,7072,14,8390,34,+,+++,11415,23,296ms,1322ms,1178ms,52896us,58501us,1105ms,55696us,1290us,1610us,479us,77us,273us Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP heterogeneou
Bug#600031: updating kernel causes slow write speed to raid
On Thu, 2010-10-14 at 00:31 +0100, Dave Clarke wrote: > Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4. > using raid5. Also to note that when booted with windows server 2003 > using Everest disk benchmark it does an average of 34MB/s (peaking to > 50MB/s). > > Attached are bonnie++ output for the two kernel versions and ext3 and > ext4 (ext4 won't mount with 2.6.26 kernel). > And meminfo for both kernels taken when the system is loaded and unloaded. OK, it looks like it's not so short of memory. And this is really quite poor write performance. Could you also test Linux 2.6.36-rc6, as packaged in experimental? Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Bug#600031: updating kernel causes slow write speed to raid
Running bonnie++ locally. Have formatted / as ext3 and /home/ as ext4. using raid5. Also to note that when booted with windows server 2003 using Everest disk benchmark it does an average of 34MB/s (peaking to 50MB/s). Attached are bonnie++ output for the two kernel versions and ext3 and ext4 (ext4 won't mount with 2.6.26 kernel). And meminfo for both kernels taken when the system is loaded and unloaded. Dave On 13 October 2010 00:15, Ben Hutchings wrote: > On Tue, 2010-10-12 at 23:07 +0100, Dave Clarke wrote: >> Package: linux-image >> Version: 2.6.32-5-686 >> >> After dist-upgrade from Lenny to Squeeze write speed is halved. Shown >> with bonnie++. Before upgrade write speed was 21MB/s, after upgrade >> 10MB/s. > > Let's see the output from bonnie++. Also are you running this locally > or over NFS or other network filesystem? > >> Hardware: Dell Powervault 745n with hardware raid Adaptec 2610SA. >> Celeron(R) CPU 2.40GHz, 512MiB RAM. >> >> Confirmed link to kernel with: >> >> otherserver$ scp 4gbfile raid_server: >> 100% 4000MB 6.6MB/s 10:06 >> >> reboot server with kernel 2.6.26-2-686 leftover from Lenny. >> otherserver$ scp 4gbfile raid_server: >> 100% 4000MB 11.3MB/s 05:55 >> >> Also of note that when copying a large file the system becomes heavily >> loaded. >> iowait: 80.1%wa >> load average: 3.58, 1.48, 0.66 >> Mem: 514428k total, 508276k used, 6152k free, 10084k buffers > > That seems like too little memory for a file server. Can you send the > contents of /proc/meminfo? > > Also what filesystem is used on the RAID? > > Ben. > > -- > Ben Hutchings > Once a job is fouled up, anything done to improve it makes it worse. > # Linux heterogeneous 2.6.26-2-686 #1 SMP Thu Sep 16 19:35:51 UTC 2010 i686 GNU/Linux # 10 reps, for consistancy. Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP heterogeneous1G 421 96 17752 10 7734 4 1515 95 61396 17 516.5 29 Latency 53894us1387ms4065ms 58119us 28090us1526ms Version 1.96 --Sequential Create-- Random Create heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7452 27 + +++ 11835 24 9917 34 + +++ 12541 27 Latency 29878us1355us1420us1142us 25us 107us 1.96,1.96,heterogeneous,1,1286995872,1G,,421,96,17752,10,7734,4,1515,95,61396,17,516.5,29,16,7452,27,+,+++,11835,24,9917,34,+,+++,12541,27,53894us,1387ms,4065ms,58119us,28090us,1526ms,29878us,1355us,1420us,1142us,25us,107us Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP heterogeneous1G 340 95 19407 12 7697 4 989 98 60791 17 512.0 36 Latency 56156us1279ms5718ms 48099us 59930us 99728us Version 1.96 --Sequential Create-- Random Create heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8668 30 + +++ 6363 12 9615 31 + +++ 12282 25 Latency 62301us1336us1362us1096us 154us 104us 1.96,1.96,heterogeneous,1,1286993983,1G,,340,95,19407,12,7697,4,989,98,60791,17,512.0,36,16,8668,30,+,+++,6363,12,9615,31,+,+++,12282,25,56156us,1279ms,5718ms,48099us,59930us,99728us,62301us,1336us,1362us,1096us,154us,104us Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP heterogeneous1G 317 97 17955 11 7684 4 1424 97 58693 18 523.3 35 Latency 29157us2438ms4469ms 34983us 110ms 46833us Version 1.96 --Sequential Create-- Random Create heterogeneous -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 7668 28 + +++ 12340 25 10329 36 + +++ 10508 24 Latency 13663us1364us1382us1110us 68us 82us 1.96,1.96,heterogeneous,1,1286994431,1G,,317,97,17955,11,7684,4,1424,97,58693,18,523.3,35,16,7668,28,+,+++,12340,25,10329,36,+,+++,10508,24,29157us,2438ms,4469ms,34983us,110ms,46833us,13663us,1364us,1382us,1110us,68us,82us Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency
Bug#600031: updating kernel causes slow write speed to raid
On Tue, 2010-10-12 at 23:07 +0100, Dave Clarke wrote: > Package: linux-image > Version: 2.6.32-5-686 > > After dist-upgrade from Lenny to Squeeze write speed is halved. Shown > with bonnie++. Before upgrade write speed was 21MB/s, after upgrade > 10MB/s. Let's see the output from bonnie++. Also are you running this locally or over NFS or other network filesystem? > Hardware: Dell Powervault 745n with hardware raid Adaptec 2610SA. > Celeron(R) CPU 2.40GHz, 512MiB RAM. > > Confirmed link to kernel with: > > otherserver$ scp 4gbfile raid_server: >100% 4000MB 6.6MB/s 10:06 > > reboot server with kernel 2.6.26-2-686 leftover from Lenny. > otherserver$ scp 4gbfile raid_server: >100% 4000MB 11.3MB/s 05:55 > > Also of note that when copying a large file the system becomes heavily loaded. >iowait: 80.1%wa >load average: 3.58, 1.48, 0.66 >Mem:514428k total, 508276k used, 6152k free,10084k buffers That seems like too little memory for a file server. Can you send the contents of /proc/meminfo? Also what filesystem is used on the RAID? Ben. -- Ben Hutchings Once a job is fouled up, anything done to improve it makes it worse. signature.asc Description: This is a digitally signed message part
Processed: Re: Bug#600031: updating kernel causes slow write speed to raid
Processing commands for cont...@bugs.debian.org: > reassign 600031 linux-2.6 Bug #600031 [linux-image] updating kernel causes slow write speed to raid Warning: Unknown package 'linux-image' Bug reassigned from package 'linux-image' to 'linux-2.6'. Bug No longer marked as found in versions 2.6.32-5-686. > -- Stopping processing here. Please contact me if you need assistance. -- 600031: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=600031 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems -- To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/handler.s.c.12869214604222.transcr...@bugs.debian.org
