Re: howto boost write(2) performance?
Michael Stiller wrote: Hi list, i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. People regularly report speeds higher than that on the RAID list, and I can get that order of magnitude speed using dd with 1MB buffers to a software RAID-0 array or cheap SATA drives. Are you using a decent controller? Many "RAID" controllers have bandwidth limitations, buffer size issues, etc. I had some chea "SCSI" arrays which were just SCSI controllers in from of a bunch of cheap, slow, non-SCSI drives. I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs). I would limit the write size to a MB and see if that helps, regardless of the buffer size. A circular queue of smaller buffers, like ptbuf, may perform better. The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. Maybe mmaping the disk would work? I don't think it would help, I'd really try limiting the size of the write() calls first, assuming your hardware is adequate. -- Bill Davidsen <[EMAIL PROTECTED]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
Michael Stiller wrote: Hi list, i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. People regularly report speeds higher than that on the RAID list, and I can get that order of magnitude speed using dd with 1MB buffers to a software RAID-0 array or cheap SATA drives. Are you using a decent controller? Many RAID controllers have bandwidth limitations, buffer size issues, etc. I had some chea SCSI arrays which were just SCSI controllers in from of a bunch of cheap, slow, non-SCSI drives. I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs). I would limit the write size to a MB and see if that helps, regardless of the buffer size. A circular queue of smaller buffers, like ptbuf, may perform better. The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. Maybe mmaping the disk would work? I don't think it would help, I'd really try limiting the size of the write() calls first, assuming your hardware is adequate. -- Bill Davidsen [EMAIL PROTECTED] We have more to fear from the bungling of the incompetent than from the machinations of the wicked. - from Slashdot - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
On Tue, 2007-10-09 at 16:56 +0200, Andi Kleen wrote: > Michael Stiller <[EMAIL PROTECTED]> writes: > > > > The write(2) performance is not good enough, the writer threads take to > > much time, and i ask you for ideas, howto to boost the write > > performance. > > You could use an O_DIRECT write if the data is suitably aligned > and your IO sizes are big enough (O_DIRECT is usually a loss > on small IOs). It will also be synchronous, but if you do it > from a separate thread anyways that should be fine. Thanks to all who answered and especially to Andi. Using O_DIRECT did the trick. Cheers, Michael - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
On Tue, 2007-10-09 at 16:56 +0200, Andi Kleen wrote: Michael Stiller [EMAIL PROTECTED] writes: The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine. Thanks to all who answered and especially to Andi. Using O_DIRECT did the trick. Cheers, Michael - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
> > The kernel really cannot sustain 125MB/s? I assume the disk > array is capable? Yes, the array should be capable, LSI controller U320, SATA disks inside the array. > Where is the bottleneck? Does it keep all disks busy, or are > the CPUs overloaded? I'm not sure where the bottleneck is. CPU Load goes up with many pdflush processes in D state. Thanks for the suggestions with sg and O_DIRECT so far, will try this. -Michael - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
The kernel really cannot sustain 125MB/s? I assume the disk array is capable? Yes, the array should be capable, LSI controller U320, SATA disks inside the array. Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded? I'm not sure where the bottleneck is. CPU Load goes up with many pdflush processes in D state. Thanks for the suggestions with sg and O_DIRECT so far, will try this. -Michael - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
On Tuesday 09 October 2007 23:50, Michael Stiller wrote: > Hi list, > > i'm developing an application (in C) which needs to write about > 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. > It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. > > I buffer the data in (currently 4) 400Mb buffers and use write(2) in a > dedicated thread to write them to the raw disk (no fs). > > The write(2) performance is not good enough, the writer threads take to > much time, and i ask you for ideas, howto to boost the write > performance. The kernel really cannot sustain 125MB/s? I assume the disk array is capable? Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
Boaz Harrosh wrote: [] > If your target is a SCSI target you can gain up to 15% by using > sg. Search on the net for the "sg utils" package. The source code > of sg_dd and others are a grate example of how to do it. > > Other wise O_DIRECT is your friend. Also look for asynchronous > I/O so you have a few pending IO buffers to keep the pipes full. What's the advantage of sg_io over O_DIRECT to the block device? I think sg devices are being (slowly?) obsoleted since most stuff which has been in sg now works over normal block device, no? At least, I haven't seen any significant difference between sg_dd and dd with oflag=direct. Thanks. /mjt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
On Tue, Oct 09 2007 at 16:56 +0200, Andi Kleen <[EMAIL PROTECTED]> wrote: > Michael Stiller <[EMAIL PROTECTED]> writes: >> The write(2) performance is not good enough, the writer threads take to >> much time, and i ask you for ideas, howto to boost the write >> performance. > > You could use an O_DIRECT write if the data is suitably aligned > and your IO sizes are big enough (O_DIRECT is usually a loss > on small IOs). It will also be synchronous, but if you do it > from a separate thread anyways that should be fine. > > -Andi If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the "sg utils" package. The source code of sg_dd and others are a grate example of how to do it. Other wise O_DIRECT is your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full. Boaz - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
Michael Stiller <[EMAIL PROTECTED]> writes: > > The write(2) performance is not good enough, the writer threads take to > much time, and i ask you for ideas, howto to boost the write > performance. You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
El Tue, 09 Oct 2007 15:50:17 +0200 Michael Stiller <[EMAIL PROTECTED]> escribió: > Hi list, > > i'm developing an application (in C) which needs to write about > 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. > It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. > > I buffer the data in (currently 4) 400Mb buffers and use write(2) in a > dedicated thread to write them to the raw disk (no fs). > > The write(2) performance is not good enough, the writer threads take > to much time, and i ask you for ideas, howto to boost the write > performance. > > Maybe mmaping the disk would work? > > Cheers, > > -Michael > > PS. I would like to be cc'd as i usually don't read the list due to > high traffic. > > - > To unsubscribe from this list: send the line "unsubscribe > linux-kernel" in the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ Create a 1GB ram disk, write data there, and then backup into a hard disk -- Gustavo Chaín Dumit - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
howto boost write(2) performance?
Hi list, i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs). The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. Maybe mmaping the disk would work? Cheers, -Michael PS. I would like to be cc'd as i usually don't read the list due to high traffic. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
howto boost write(2) performance?
Hi list, i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs). The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. Maybe mmaping the disk would work? Cheers, -Michael PS. I would like to be cc'd as i usually don't read the list due to high traffic. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
El Tue, 09 Oct 2007 15:50:17 +0200 Michael Stiller [EMAIL PROTECTED] escribió: Hi list, i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs). The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. Maybe mmaping the disk would work? Cheers, -Michael PS. I would like to be cc'd as i usually don't read the list due to high traffic. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ Create a 1GB ram disk, write data there, and then backup into a hard disk -- Gustavo Chaín Dumit - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
Michael Stiller [EMAIL PROTECTED] writes: The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine. -Andi - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
On Tue, Oct 09 2007 at 16:56 +0200, Andi Kleen [EMAIL PROTECTED] wrote: Michael Stiller [EMAIL PROTECTED] writes: The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. You could use an O_DIRECT write if the data is suitably aligned and your IO sizes are big enough (O_DIRECT is usually a loss on small IOs). It will also be synchronous, but if you do it from a separate thread anyways that should be fine. -Andi If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the sg utils package. The source code of sg_dd and others are a grate example of how to do it. Other wise O_DIRECT is your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full. Boaz - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
Boaz Harrosh wrote: [] If your target is a SCSI target you can gain up to 15% by using sg. Search on the net for the sg utils package. The source code of sg_dd and others are a grate example of how to do it. Other wise O_DIRECT is your friend. Also look for asynchronous I/O so you have a few pending IO buffers to keep the pipes full. What's the advantage of sg_io over O_DIRECT to the block device? I think sg devices are being (slowly?) obsoleted since most stuff which has been in sg now works over normal block device, no? At least, I haven't seen any significant difference between sg_dd and dd with oflag=direct. Thanks. /mjt - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: howto boost write(2) performance?
On Tuesday 09 October 2007 23:50, Michael Stiller wrote: Hi list, i'm developing an application (in C) which needs to write about 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI. It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7. I buffer the data in (currently 4) 400Mb buffers and use write(2) in a dedicated thread to write them to the raw disk (no fs). The write(2) performance is not good enough, the writer threads take to much time, and i ask you for ideas, howto to boost the write performance. The kernel really cannot sustain 125MB/s? I assume the disk array is capable? Where is the bottleneck? Does it keep all disks busy, or are the CPUs overloaded? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
