Re: [PERFORM] concurrent IO in postgres?
On Thu, 2010-12-23 at 11:24 -0700, Scott Marlowe wrote: > On Thu, Dec 23, 2010 at 10:37 AM, Przemek Wozniak wrote: > > When testing the IO performance of ioSAN storage device from FusionIO > > (650GB MLC version) one of the things I tried is a set of IO intensive > > operations in Postgres: bulk data loads, updates, and queries calling > > for random IO. So far I cannot make Postgres take advantage of this > > So, were you running a lot of these at once? Or just single threaded? > I get very good io concurrency with lots of parallel postgresql > connections on a 34 disk SAS setup with a battery backed controller. In one test I was running between 1 and 32 clients simultaneously writing lots of data using copy binary. The problem is that with a large RAM buffer it all goes there, and then the background writer, a single postgres process, will issue write requests one at a time I suspect. So the actual IO is effectively serialized by the backend. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
[PERFORM] concurrent IO in postgres?
When testing the IO performance of ioSAN storage device from FusionIO (650GB MLC version) one of the things I tried is a set of IO intensive operations in Postgres: bulk data loads, updates, and queries calling for random IO. So far I cannot make Postgres take advantage of this tremendous IO capacity. I can squeeze a factor of a few here and there when caching cannot be utilized, but this hardware can do a lot more. Low level testing with fio shows on average x10 speedups over disk for sequential IO and x500-800 for random IO. With enough threads I can get IOPS in the 100-200K range and 1-1.5GB/s bandwidth, basically what's advertised. But not with Postgres. Is this because the Postgres backend is essentially single threaded and in general does not perform asynchronous IO, or I'm missing something? I found out that the effective_io_concurrency parameter only takes effect for bitmap index scans. Also, is there any work going on to allow concurrent IO in the backend and adapt Postgres to the capabilities of Flash? Will appreciate any comments, experiences, etc. Przemek Wozniak -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
