Re: [HACKERS] wal_buffers, redux

2012-03-14 Thread Jeff Janes
On Mon, Mar 12, 2012 at 7:16 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Mar 11, 2012 at 11:51 PM, Fujii Masao masao.fu...@gmail.com wrote: On Sun, Mar 11, 2012 at 12:55 PM, Robert Haas robertmh...@gmail.com wrote: I've finally been able to run some more tests of the effect of

Re: [HACKERS] wal_buffers, redux

2012-03-14 Thread Robert Haas
On Tue, Mar 13, 2012 at 11:18 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 13, 2012 at 6:44 PM, Josh Berkus j...@agliodbs.com wrote: That's a speedup of nearly a factor of two, so clearly fsync-related stalls are a big problem here, even with wal_buffers cranked up through the

Re: [HACKERS] wal_buffers, redux

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 3:29 PM, Jeff Janes jeff.ja...@gmail.com wrote: I think my analysis is pretty much a re-wording of yours, but I'd emphasize that getting the WALWriteLock is bad not just because they fight over the lock, but because someone else (probably background wal writer) is

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Andres Freund
On Tuesday, March 13, 2012 03:26:34 AM Robert Haas wrote: Meanwhile, here are some TPS graphs at 16MB and 32MB on the IBM POWER7 machine. 32 clients, 1800 seconds, scale factor 300, synchronous commit off. That graph makes me cringe because its pretty representative of what I have seen in

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes jeff.ja...@gmail.com wrote: Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both machines) with fsync=off (as well as synchronous_commit=off still) might help clarify things. If it increases the TPS of Nate@16MB, but doesn't change the

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes jeff.ja...@gmail.com wrote: Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both machines) with fsync=off (as well as synchronous_commit=off still) might help clarify things. I reran the 32-client benchmark on the IBM machine with

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 3:48 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes jeff.ja...@gmail.com wrote: Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both machines) with fsync=off (as well as synchronous_commit=off still) might help

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 4:55 AM, Andres Freund and...@anarazel.de wrote: On Tuesday, March 13, 2012 03:26:34 AM Robert Haas wrote: Meanwhile, here are some TPS graphs at 16MB and 32MB on the IBM POWER7 machine.  32 clients, 1800 seconds, scale factor 300, synchronous commit off. That graph

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Josh Berkus
That's a speedup of nearly a factor of two, so clearly fsync-related stalls are a big problem here, even with wal_buffers cranked up through the ceiling. H. Do you have any ability to test on XFS? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Fujii Masao
On Wed, Mar 14, 2012 at 7:20 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 13, 2012 at 3:48 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes jeff.ja...@gmail.com wrote: Rerunning all 4 benchmarks (both 16MB and 32MB wal_buffers on both

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 10:02 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Mar 14, 2012 at 7:20 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 13, 2012 at 3:48 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes jeff.ja...@gmail.com wrote:

Re: [HACKERS] wal_buffers, redux

2012-03-13 Thread Robert Haas
On Tue, Mar 13, 2012 at 6:44 PM, Josh Berkus j...@agliodbs.com wrote: That's a speedup of nearly a factor of two, so clearly fsync-related stalls are a big problem here, even with wal_buffers cranked up through the ceiling. H.   Do you have any ability to test on XFS? It seems I do.

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Robert Haas
On Sun, Mar 11, 2012 at 11:51 PM, Fujii Masao masao.fu...@gmail.com wrote: On Sun, Mar 11, 2012 at 12:55 PM, Robert Haas robertmh...@gmail.com wrote: I've finally been able to run some more tests of the effect of adjusting wal_buffers to values higher than 16MB.  I ran the test on the 16 core

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Jeff Janes
On Sat, Mar 10, 2012 at 7:55 PM, Robert Haas robertmh...@gmail.com wrote: I've finally been able to run some more tests of the effect of adjusting wal_buffers to values higher than 16MB.  I ran the test on the 16 core (x 4 hw threads/core) IBM POWER7 machine, with my usual configuration

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Robert Haas
On Mon, Mar 12, 2012 at 12:32 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Nate Boley's machine, the difference was ~100% increase rather than ~10%. Oh, right. I had forgotten how dramatic the changes were in those test runs. I guess I should be happy that the absolute numbers on this

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Jeff Janes
On Mon, Mar 12, 2012 at 10:55 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Mar 12, 2012 at 12:32 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Nate Boley's machine, the difference was ~100% increase rather than ~10%. Oh, right.  I had forgotten how dramatic the changes were in those

Re: [HACKERS] wal_buffers, redux

2012-03-12 Thread Robert Haas
On Mon, Mar 12, 2012 at 4:45 PM, Jeff Janes jeff.ja...@gmail.com wrote: Do you think the difference is in the CPU architecture, or the IO subsystem? That is an excellent question.  I tried looking at vmstat output, but a funny thing kept happening: periodically, the iowait column would show

Re: [HACKERS] wal_buffers, redux

2012-03-11 Thread Fujii Masao
On Sun, Mar 11, 2012 at 12:55 PM, Robert Haas robertmh...@gmail.com wrote: I've finally been able to run some more tests of the effect of adjusting wal_buffers to values higher than 16MB.  I ran the test on the 16 core (x 4 hw threads/core) IBM POWER7 machine, with my usual configuration

[HACKERS] wal_buffers, redux

2012-03-10 Thread Robert Haas
I've finally been able to run some more tests of the effect of adjusting wal_buffers to values higher than 16MB. I ran the test on the 16 core (x 4 hw threads/core) IBM POWER7 machine, with my usual configuration settings: shared_buffers = 8GB maintenance_work_mem = 1GB synchronous_commit = off