Re: [HACKERS] mosbench revisited

2011-08-15 Thread Greg Stark
On Wed, Aug 3, 2011 at 7:21 PM, Robert Haas robertmh...@gmail.com wrote:  I'm kind of interested by the result, actually, as I had feared that the spinlock protecting ProcArrayLock was going to be a bigger problem sooner. I think this depends on how many connections you have. If you try to

Re: [HACKERS] mosbench revisited

2011-08-15 Thread Robert Haas
On Mon, Aug 15, 2011 at 6:22 PM, Greg Stark st...@mit.edu wrote: On Wed, Aug 3, 2011 at 7:21 PM, Robert Haas robertmh...@gmail.com wrote:  I'm kind of interested by the result, actually, as I had feared that the spinlock protecting ProcArrayLock was going to be a bigger problem sooner. I

Re: [HACKERS] mosbench revisited

2011-08-10 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: However, it doesn't really do anything to solve this problem. The problem here is not that the size of the relation is changing too frequently - indeed, it's not changing at all in this test case. The problem is rather that testing whether or not the

Re: [HACKERS] mosbench revisited

2011-08-10 Thread Robert Haas
2011/8/10 Dimitri Fontaine dfonta...@hi-media.com: Robert Haas robertmh...@gmail.com writes: However, it doesn't really do anything to solve this problem. The problem here is not that the size of the relation is changing too frequently - indeed, it's not changing at all in this test case.

Re: [HACKERS] mosbench revisited

2011-08-10 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mié ago 10 21:27:08 -0400 2011: 2011/8/10 Dimitri Fontaine dfonta...@hi-media.com: Robert Haas robertmh...@gmail.com writes: However, it doesn't really do anything to solve this problem. The problem here is not that the size of the relation is

Re: [HACKERS] mosbench revisited

2011-08-10 Thread Robert Haas
On Wed, Aug 10, 2011 at 9:46 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mié ago 10 21:27:08 -0400 2011: 2011/8/10 Dimitri Fontaine dfonta...@hi-media.com: Robert Haas robertmh...@gmail.com writes: However, it doesn't really do anything to

Re: [HACKERS] mosbench revisited

2011-08-08 Thread Robert Haas
On Sat, Aug 6, 2011 at 1:43 PM, Jeff Janes jeff.ja...@gmail.com wrote: The approach is to move the important things from a LWLock to a spinlock, and to not do any locking for increments to clock-hand increment and numBufferAllocs. That means that some buffers might occasionally get inspected

Re: [HACKERS] mosbench revisited

2011-08-08 Thread Robert Haas
On Sat, Aug 6, 2011 at 2:16 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will.  For those who may be curious, the problem is in generic_file_llseek()

Re: [HACKERS] mosbench revisited

2011-08-08 Thread Robert Haas
On Sat, Aug 6, 2011 at 3:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: My experiments have shown that the freelist proper is not substantially faster than the freelist clocksweep--and that is even under the assumption that putting things back into the

Re: [HACKERS] mosbench revisited

2011-08-08 Thread Jesper Krogh
On 2011-08-08 15:29, Robert Haas wrote: On Sat, Aug 6, 2011 at 2:16 PM, Dimitri Fontainedimi...@2ndquadrant.fr wrote: Robert Haasrobertmh...@gmail.com writes: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will. For those who may be curious,

Re: [HACKERS] mosbench revisited

2011-08-06 Thread Jeff Janes
On Wed, Aug 3, 2011 at 11:21 AM, Robert Haas robertmh...@gmail.com wrote: About nine months ago, we had a discussion of some benchmarking that was done by the mosbench folks at MIT: http://archives.postgresql.org/pgsql-hackers/2010-10/msg00160.php Although the authors used PostgreSQL as a

Re: [HACKERS] mosbench revisited

2011-08-06 Thread Jeff Janes
On Wed, Aug 3, 2011 at 3:21 PM, Jim Nasby j...@nasby.net wrote: On Aug 3, 2011, at 1:21 PM, Robert Haas wrote: 1. We configure PostgreSQL to use a 2 Gbyte application-level cache because PostgreSQL protects its free-list with a single lock and thus scales poorly with smaller caches.  This is a

Re: [HACKERS] mosbench revisited

2011-08-06 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will. For those who may be curious, the problem is in generic_file_llseek() in fs/read-write.c. On a platform with 8-byte atomic reads, it seems like it

Re: [HACKERS] mosbench revisited

2011-08-06 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: My experiments have shown that the freelist proper is not substantially faster than the freelist clocksweep--and that is even under the assumption that putting things back into the freelist is absolutely free. The freelist isn't there to make buffer

Re: [HACKERS] mosbench revisited

2011-08-04 Thread Robert Haas
On Wed, Aug 3, 2011 at 9:16 PM, Robert Haas robertmh...@gmail.com wrote: Spinlocks seem to have a very ugly tipping point. And on that note, here are oprofile results from pgbench -n -T 300 -S -c 64 -j 64 -M prepared on the latest master branch, compiled with -O2 -fno-omit-frame-pointer.

Re: [HACKERS] mosbench revisited

2011-08-04 Thread Aidan Van Dyk
On Wed, Aug 3, 2011 at 5:04 PM, Robert Haas robertmh...@gmail.com wrote:  And hoping that the Linux guys decide to do something about it.  This isn't really our bug - lseek is quite cheap in the uncontended case. Has anyone tried this on a recent kernel (i.e. 2.6.39 or later), where

Re: [HACKERS] mosbench revisited

2011-08-04 Thread Robert Haas
On Thu, Aug 4, 2011 at 5:09 PM, Aidan Van Dyk ai...@highrise.ca wrote: On Wed, Aug 3, 2011 at 5:04 PM, Robert Haas robertmh...@gmail.com wrote:      And hoping that the Linux guys decide to do something about it.  This isn't really our bug - lseek is quite cheap in the uncontended case. Has

[HACKERS] mosbench revisited

2011-08-03 Thread Robert Haas
About nine months ago, we had a discussion of some benchmarking that was done by the mosbench folks at MIT: http://archives.postgresql.org/pgsql-hackers/2010-10/msg00160.php Although the authors used PostgreSQL as a test harness for driving load, it's pretty clear from reading the paper that

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Martijn van Oosterhout
On Wed, Aug 03, 2011 at 02:21:25PM -0400, Robert Haas wrote: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will. For those who may be curious, the problem is in generic_file_llseek() in fs/read-write.c. On a platform with 8-byte atomic reads,

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Tom Lane
Martijn van Oosterhout klep...@svana.org writes: On Wed, Aug 03, 2011 at 02:21:25PM -0400, Robert Haas wrote: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will. For those who may be curious, the problem is in generic_file_llseek() in

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Robert Haas
On Wed, Aug 3, 2011 at 2:49 PM, Tom Lane t...@sss.pgh.pa.us wrote: Martijn van Oosterhout klep...@svana.org writes: On Wed, Aug 03, 2011 at 02:21:25PM -0400, Robert Haas wrote: It would be nice if the Linux guys would fix this problem for us, but I'm not sure whether they will.  For those who

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On a straight pgbench -S test, you get four system calls per query: recvfrom(), lseek(), lseek(), sendto(). Adding -M prepared eliminates the two lseeks. [ scratches head... ] Two? Is that one for the table and one for its lone index, or are we

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Robert Haas
On Wed, Aug 3, 2011 at 3:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On a straight pgbench -S test, you get four system calls per query: recvfrom(), lseek(), lseek(), sendto().  Adding -M prepared eliminates the two lseeks. [ scratches head... ]  Two?

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Aug 3, 2011 at 3:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: (If the query ended up being a seqscan, I'd expect a second lseek(SEEK_END) when the executor starts up, but I gather from the other complaints that the mosbench people were only

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Robert Haas
On Wed, Aug 3, 2011 at 4:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Aug 3, 2011 at 3:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: (If the query ended up being a seqscan, I'd expect a second lseek(SEEK_END) when the executor starts up, but I gather

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Aug 3, 2011 at 4:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... We could possibly accept stale values for the planner estimates, but I think heapam's number had better be accurate. I think the exact requirement is that, if the relation turns

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Jim Nasby
On Aug 3, 2011, at 1:21 PM, Robert Haas wrote: 1. We configure PostgreSQL to use a 2 Gbyte application-level cache because PostgreSQL protects its free-list with a single lock and thus scales poorly with smaller caches. This is a complaint about BufFreeList lock which, in fact, I've seen as a

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Robert Haas
On Wed, Aug 3, 2011 at 5:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: That still seems utterly astonishing to me.  We're touching each of those files once per query cycle; a cycle that contains two message sends, who knows how many internal spinlock/lwlock/heavyweightlock acquisitions inside

Re: [HACKERS] mosbench revisited

2011-08-03 Thread Robert Haas
On Wed, Aug 3, 2011 at 6:21 PM, Jim Nasby j...@nasby.net wrote: On Aug 3, 2011, at 1:21 PM, Robert Haas wrote: 1. We configure PostgreSQL to use a 2 Gbyte application-level cache because PostgreSQL protects its free-list with a single lock and thus scales poorly with smaller caches.  This is a