Re: Fwd: Inconsistency in bonnie++ results for repeated runs
On Tue, 10 Feb 2004 23:58, [EMAIL PROTECTED] (Chris Wagner) wrote: > without extensive trial and error testing. Russell you might want to make > a super debug version of Bonnie that gathers statistics from each step in > the pipeline from the application to the platter. I would look very The experimental version of Bonnie++ (1.9x) displays the worst-case latencies of all operations. It might shed some light on these issues. If your suspicion is correct then a large part of the performance loss would be on a few operations (maybe even a single operation), in which case the latency reported by 1.9x would be extreme. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
Hmm, that's a sticky widget. Have you tried any other HD benchmarks and gotten similar results? I think we need that to narrow it down to either a Bonnie or hardware issue. It could be that some of ur disks are preparing to die. I have seen that before, a disk that's getting flaky will do strange things. If you can get your ear near the disk or put your finger on it you should be able to tell if it starts thrashing. If it's thrashing when you know it shouldn't, I'ld pop that sucker ASAP. If the unnatural thrashing coincides with the throughput drop then I think you have ur culprit. (do this with all the relevent disks of course) Now if the hardware's fine then there's almost no telling where the problem lies without extensive trial and error testing. Russell you might want to make a super debug version of Bonnie that gathers statistics from each step in the pipeline from the application to the platter. I would look very closely at the RAID controller driver. I'm in the middle of a fight right now with Adaptec over file corruption and I eventually narrowed it down to the driver. They want to blame everything except themselves. Incidentally, if ur thinking of upgrading ur storage system check this mugga out: http://www20.tomshardware.com/storage/20030425/index.html Good luck, let me know if you discover anything. -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede males" 0100
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
On Tue, 10 Feb 2004 23:58, [EMAIL PROTECTED] (Chris Wagner) wrote: > without extensive trial and error testing. Russell you might want to make > a super debug version of Bonnie that gathers statistics from each step in > the pipeline from the application to the platter. I would look very The experimental version of Bonnie++ (1.9x) displays the worst-case latencies of all operations. It might shed some light on these issues. If your suspicion is correct then a large part of the performance loss would be on a few operations (maybe even a single operation), in which case the latency reported by 1.9x would be extreme. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
Hmm, that's a sticky widget. Have you tried any other HD benchmarks and gotten similar results? I think we need that to narrow it down to either a Bonnie or hardware issue. It could be that some of ur disks are preparing to die. I have seen that before, a disk that's getting flaky will do strange things. If you can get your ear near the disk or put your finger on it you should be able to tell if it starts thrashing. If it's thrashing when you know it shouldn't, I'ld pop that sucker ASAP. If the unnatural thrashing coincides with the throughput drop then I think you have ur culprit. (do this with all the relevent disks of course) Now if the hardware's fine then there's almost no telling where the problem lies without extensive trial and error testing. Russell you might want to make a super debug version of Bonnie that gathers statistics from each step in the pipeline from the application to the platter. I would look very closely at the RAID controller driver. I'm in the middle of a fight right now with Adaptec over file corruption and I eventually narrowed it down to the driver. They want to blame everything except themselves. Incidentally, if ur thinking of upgrading ur storage system check this mugga out: http://www20.tomshardware.com/storage/20030425/index.html Good luck, let me know if you discover anything. -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede males" 0100 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
On Thu, 5 Feb 2004, Chris Wagner wrote: > Can you tell some more about the hard drive/ controller/ driver setup? My > first guess is a driver or cacheing issue. 1CPU Host --- Motherboard: SuperMicro CPU:1GHz Pentium III w/ 256k L2 cache Disk Controller: Onboard IDE Disks: 20GB IDE UDMA(100) x2 2CPU Host -- Motherboard: HP LH3000r CPU:850MHz Pentium III w/ 256k L2 cache x2 (Coppermine) RAID Controller:AMI MegaRAID SCSI Disks: 10 x 18GB > What is the commonality between the 1-way and 2-way systems? There is not much common: the 2-ways use SCSI MegaRAID/RAID0/5disks(on the relevant volume. But going through again, while there is variation, it is not to the same degree on the 2CPU hosts(only about 25%): the worst offenders are the 1CPU Hosts. > Do you have a host that u've *not* seen this on. > I forgot to mention the kernel for this test is linux-2.4.25-pre6, But on linux 2.6.1, I see similar problem on the Sequential Block Read: http://khack.osdl.org/stp/285814/logs/run-log.txt Version @version@ --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP stp1-000 1G 8080 99 44855 29 15970 10 6272 76 38319 13 151.1 0 stp1-000 1G 8049 99 12655 8 11521 7 5810 71 26823 7 184.7 0 stp1-000 1G 8081 99 44719 29 15934 9 6515 79 40076 10 193.8 0 > > > > > -- > REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- > "...ne cede males" > > 0100 >
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
On Thu, 5 Feb 2004, Chris Wagner wrote: > Can you tell some more about the hard drive/ controller/ driver setup? My > first guess is a driver or cacheing issue. 1CPU Host --- Motherboard: SuperMicro CPU:1GHz Pentium III w/ 256k L2 cache Disk Controller: Onboard IDE Disks: 20GB IDE UDMA(100) x2 2CPU Host -- Motherboard: HP LH3000r CPU:850MHz Pentium III w/ 256k L2 cache x2 (Coppermine) RAID Controller:AMI MegaRAID SCSI Disks: 10 x 18GB > What is the commonality between the 1-way and 2-way systems? There is not much common: the 2-ways use SCSI MegaRAID/RAID0/5disks(on the relevant volume. But going through again, while there is variation, it is not to the same degree on the 2CPU hosts(only about 25%): the worst offenders are the 1CPU Hosts. > Do you have a host that u've *not* seen this on. > I forgot to mention the kernel for this test is linux-2.4.25-pre6, But on linux 2.6.1, I see similar problem on the Sequential Block Read: http://khack.osdl.org/stp/285814/logs/run-log.txt Version @version@ --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP stp1-000 1G 8080 99 44855 29 15970 10 6272 76 38319 13 151.1 0 stp1-000 1G 8049 99 12655 8 11521 7 5810 71 26823 7 184.7 0 stp1-000 1G 8081 99 44719 29 15934 9 6515 79 40076 10 193.8 0 > > > > > -- > REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- > "...ne cede males" > > 0100 > -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
Can you tell some more about the hard drive/ controller/ driver setup? My first guess is a driver or cacheing issue. What is the commonality between the 1-way and 2-way systems? Do you have a host that u've *not* seen this on. -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede males" 0100
Re: Fwd: Inconsistency in bonnie++ results for repeated runs
Can you tell some more about the hard drive/ controller/ driver setup? My first guess is a driver or cacheing issue. What is the commonality between the 1-way and 2-way systems? Do you have a host that u've *not* seen this on. -- REMEMBER THE WORLD TRADE CENTER ---=< WTC 911 >=-- "...ne cede males" 0100 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
