Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

On 2015-01-26 18:30:13 -0600, Jim Nasby wrote: > On 12/23/14 11:41 AM, Andres Freund wrote: > >I think it'd generally be useful to have something like errhidecontext() > >akin to errhidestatement() to avoid things like the above. > > > >>> > >>>Under this proposal, do you want to sup

Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

On 12/23/14 11:41 AM, Andres Freund wrote: > >I think it'd generally be useful to have something like errhidecontext() > >akin to errhidestatement() to avoid things like the above. > > > >Under this proposal, do you want to suppress the context/statement >unconditionally or via some hook/variabl

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.10

Hi, Attached is a new version of the patchset which I intend to commit soon. Stuff changed since 0.9: * Greatly simplified locking logic - the whole concept that a lock could be spuriously acquired is gone. That cost a small bit of performance (0.5%, I thought it'd be much bigger) on x86, bu

Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

On 2014-12-22 10:35:35 +0530, Amit Kapila wrote: > On Fri, Dec 19, 2014 at 9:36 PM, Andres Freund > wrote: > > > > Hi, > > > > When debugging lwlock issues I found PRINT_LWDEBUG/LOG_LWDEBUG rather > > painful to use because of the amount of elog contexts/statements > > emitted. Given the number of

Re: Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

On Fri, Dec 19, 2014 at 9:36 PM, Andres Freund wrote: > > Hi, > > When debugging lwlock issues I found PRINT_LWDEBUG/LOG_LWDEBUG rather > painful to use because of the amount of elog contexts/statements > emitted. Given the number of lwlock acquirations that's just not doable. > > To solve that du

Suppressing elog.c context messages (was Re: [HACKERS] Wait free LW_SHARED acquisition)

Hi, When debugging lwlock issues I found PRINT_LWDEBUG/LOG_LWDEBUG rather painful to use because of the amount of elog contexts/statements emitted. Given the number of lwlock acquirations that's just not doable. To solve that during development I've solved that by basically replacing: if (Tra

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Wed, Dec 3, 2014 at 4:03 PM, Michael Paquier wrote: > Ping? This patch is in a stale state for a couple of weeks and still > marked as waiting on author for this CF. Marked as returned with feedback. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Tue, Nov 18, 2014 at 12:33 AM, Robert Haas wrote: > On Mon, Nov 17, 2014 at 10:31 AM, Andres Freund > wrote: >> On 2014-11-17 10:21:04 -0500, Robert Haas wrote: >>> Andres, where are we with this patch? >>> >>> 1. You're going to commit it, but haven't gotten around to it yet. >>> >>> 2. You'

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Mon, Nov 17, 2014 at 10:31 AM, Andres Freund wrote: > On 2014-11-17 10:21:04 -0500, Robert Haas wrote: >> Andres, where are we with this patch? >> >> 1. You're going to commit it, but haven't gotten around to it yet. >> >> 2. You're going to modify it some more and repost, but haven't gotten >>

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-11-17 10:21:04 -0500, Robert Haas wrote: > Andres, where are we with this patch? > > 1. You're going to commit it, but haven't gotten around to it yet. > > 2. You're going to modify it some more and repost, but haven't gotten > around to it yet. > > 3. You're willing to see it modified i

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Sat, Oct 25, 2014 at 1:50 AM, Amit Kapila wrote: > On Fri, Oct 24, 2014 at 4:05 PM, Andres Freund > wrote: >> >> On 2014-10-24 15:59:30 +0530, Amit Kapila wrote: >> > > > and w.r.t performance it can lead extra >> > > > function call, few checks and I think in some cases even can >> > > > acqu

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-30 19:05:06 +0530, Amit Kapila wrote: > On Thu, Oct 30, 2014 at 6:58 PM, Andres Freund > wrote: > > On 2014-10-30 18:54:57 +0530, Amit Kapila wrote: > > > On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund > > > wrote: > > > > Hm. What commit did you apply the series ontop? I managed to >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Thu, Oct 30, 2014 at 6:58 PM, Andres Freund wrote: > On 2014-10-30 18:54:57 +0530, Amit Kapila wrote: > > On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund > > wrote: > > > Hm. What commit did you apply the series ontop? I managed to reproduce a > > > hang, but it was just something that heikki h

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-30 18:54:57 +0530, Amit Kapila wrote: > On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund > wrote: > > On 2014-10-21 12:40:56 +0530, Amit Kapila wrote: > > > I have ran it for half an hour, but it doesn't came out even after > > > ~2 hours. It doesn't get reproduced every time, currently

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Thu, Oct 30, 2014 at 5:52 PM, Andres Freund wrote: > > On 2014-10-21 12:40:56 +0530, Amit Kapila wrote: > > While doing performance tests, I noticed a hang at higher client > > counts with patch. I have tried to check call stack for few of > > processes and it is as below: > > > > #0 0x008

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-21 12:40:56 +0530, Amit Kapila wrote: > While doing performance tests, I noticed a hang at higher client > counts with patch. I have tried to check call stack for few of > processes and it is as below: > > #0 0x008010933e54 in .semop () from /lib64/libc.so.6 > #1 0x10286e4

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Fri, Oct 24, 2014 at 4:05 PM, Andres Freund wrote: > > On 2014-10-24 15:59:30 +0530, Amit Kapila wrote: > > > > and w.r.t performance it can lead extra > > > > function call, few checks and I think in some cases even can > > > > acquire/release spinlock. > > > > > > I fail to see how that could

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-24 15:59:30 +0530, Amit Kapila wrote: > > > and w.r.t performance it can lead extra > > > function call, few checks and I think in some cases even can > > > acquire/release spinlock. > > > > I fail to see how that could be the case. > > Won't it happen incase first backend sets releaseO

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Wed, Oct 22, 2014 at 8:04 PM, Andres Freund wrote: > On 2014-10-21 19:56:05 +0530, Amit Kapila wrote: > > On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund > > wrote: > > spin_delay_count gives > > how much delay has happened to acquire spinlock which when > > combined with other stats gives the c

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Wed, Oct 22, 2014 at 7:12 PM, Andres Freund wrote: > > On 2014-10-22 13:32:07 +0530, Amit Kapila wrote: > > On Tue, Oct 21, 2014 at 7:56 PM, Amit Kapila > > wrote: > > > On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund > > wrote: > > > > > > > > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-21 19:56:05 +0530, Amit Kapila wrote: > On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund > wrote: > > > > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > > > 2. > > > LWLockWakeup() > > > { > > > .. > > > #ifdef LWLOCK_STATS > > > lwstats->spin_delay_count += SpinLockAcquire(&lock->mute

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-22 13:32:07 +0530, Amit Kapila wrote: > On Tue, Oct 21, 2014 at 7:56 PM, Amit Kapila > wrote: > > On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund > wrote: > > > > > > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > > Today, I have verified all previous comments raised by > me and loo

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Tue, Oct 21, 2014 at 7:56 PM, Amit Kapila wrote: > On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund wrote: > > > > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: Today, I have verified all previous comments raised by me and looked at new code and below are my findings: >> >> 4. >> LWLockAcqui

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Wed, Oct 8, 2014 at 6:17 PM, Andres Freund wrote: > > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > > 2. > > LWLockWakeup() > > { > > .. > > #ifdef LWLOCK_STATS > > lwstats->spin_delay_count += SpinLockAcquire(&lock->mutex); > > #else > > SpinLockAcquire(&lock->mutex); > > #endif > > .. >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Fri, Oct 17, 2014 at 11:41 PM, Andres Freund wrote: > On 2014-10-17 17:14:16 +0530, Amit Kapila wrote: > > On Tue, Oct 14, 2014 at 11:34 AM, Amit Kapila > > wrote: > > HEAD – commit 494affb + wait free lw_shared_v2 > > > > Shared_buffers=8GB; Scale Factor = 3000 > > > > Client Count/No. Of

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-17 17:14:16 +0530, Amit Kapila wrote: > On Tue, Oct 14, 2014 at 11:34 AM, Amit Kapila > wrote: > > > > > > I am not sure why we are seeing difference even though running > > on same m/c with same configuration. > > I have tried many times, but I could not get the numbers you have > pos

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Tue, Oct 14, 2014 at 11:34 AM, Amit Kapila wrote: > > > I am not sure why we are seeing difference even though running > on same m/c with same configuration. I have tried many times, but I could not get the numbers you have posted above with HEAD, however now trying with the latest version [1]

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Wed, Oct 15, 2014 at 12:06 AM, Merlin Moncure wrote: > > A while back, I submitted a minor tweak to the clock sweep so that, > instead of spinlocking every single buffer header as it swept it just > did a single TAS as a kind of a trylock and punted to the next buffer > if the test failed on th

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Tue, Oct 14, 2014 at 8:58 AM, Andres Freund wrote: > On 2014-10-14 08:40:49 -0500, Merlin Moncure wrote: >> On Fri, Oct 10, 2014 at 11:00 AM, Andres Freund >> wrote: >> > Which is nearly trivial now that atomics are in. Check out the attached >> > WIP patch which eliminates the spinlock from

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-14 08:40:49 -0500, Merlin Moncure wrote: > On Fri, Oct 10, 2014 at 11:00 AM, Andres Freund > wrote: > > On 2014-10-10 16:41:39 +0200, Andres Freund wrote: > >> FWIW, the profile always looks like > >> - 48.61% postgres postgres [.] s_lock > >>- s_lock > >>

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Fri, Oct 10, 2014 at 11:00 AM, Andres Freund wrote: > On 2014-10-10 16:41:39 +0200, Andres Freund wrote: >> FWIW, the profile always looks like >> - 48.61% postgres postgres [.] s_lock >>- s_lock >> + 96.67% StrategyGetBuffer >> + 1.19% UnpinBuffer >> +

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Sat, Oct 11, 2014 at 7:02 PM, Amit Kapila wrote: > On Sat, Oct 11, 2014 at 6:40 PM, Andres Freund wrote: > > On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: > > > On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund > > > > And since > > > > your general performance numbers are a fair bit lower th

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-11 15:10:45 +0200, Andres Freund wrote: > Hi, > > On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: > > On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund > > > And since > > > your general performance numbers are a fair bit lower than what I see > > > with, hopefully, the same code on the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Sat, Oct 11, 2014 at 6:40 PM, Andres Freund wrote: > On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: > > On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund > > > And since > > > your general performance numbers are a fair bit lower than what I see > > > with, hopefully, the same code on the same

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

Hi, On 2014-10-11 07:26:57 +0530, Amit Kapila wrote: > On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund > > And since > > your general performance numbers are a fair bit lower than what I see > > with, hopefully, the same code on the same machine... > > You have reported numbers at 1000 scale fact

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Sat, Oct 11, 2014 at 7:00 AM, Andres Freund wrote: > On 2014-10-11 06:49:54 +0530, Amit Kapila wrote: > > On Sat, Oct 11, 2014 at 6:29 AM, Andres Freund > > wrote: > > > > > > On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: > > > I've run some short tests on hydra: > > > > > > Could you pleas

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-11 06:49:54 +0530, Amit Kapila wrote: > On Sat, Oct 11, 2014 at 6:29 AM, Andres Freund > wrote: > > > > On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: > > I've run some short tests on hydra: > > > > scale 1000: > > > > base: > > 4GB: > > tps = 296273.004800 (including connections esta

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Sat, Oct 11, 2014 at 6:29 AM, Andres Freund wrote: > > On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: > I've run some short tests on hydra: > > scale 1000: > > base: > 4GB: > tps = 296273.004800 (including connections establishing) > tps = 296373.978100 (excluding connections establishing) >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-11 06:18:11 +0530, Amit Kapila wrote: > On Fri, Oct 10, 2014 at 8:11 PM, Andres Freund > wrote: > > On 2014-10-10 17:18:46 +0530, Amit Kapila wrote: > > > On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund > > > wrote: > > > > > Observations > > > > > -- > > > > > a. Th

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Fri, Oct 10, 2014 at 8:11 PM, Andres Freund wrote: > On 2014-10-10 17:18:46 +0530, Amit Kapila wrote: > > On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund > > wrote: > > > > Observations > > > > -- > > > > a. The patch performs really well (increase upto ~40%) incase all the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-10 16:41:39 +0200, Andres Freund wrote: > FWIW, the profile always looks like > - 48.61% postgres postgres [.] s_lock >- s_lock > + 96.67% StrategyGetBuffer > + 1.19% UnpinBuffer > + 0.90% PinBuffer > + 0.70% hash_search_with_hash_value > +

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-10 17:18:46 +0530, Amit Kapila wrote: > On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund > wrote: > > > Observations > > > -- > > > a. The patch performs really well (increase upto ~40%) incase all the > > > data fits in shared buffers (scale factor -100). > > > b. Inc

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Fri, Oct 10, 2014 at 1:27 PM, Andres Freund wrote: > On 2014-10-10 10:13:03 +0530, Amit Kapila wrote: > > I have done few performance tests for above patches and results of > > same is as below: > > Cool, thanks. > > > Performance Data > > -- > > IBM POWER-7 16 cores

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

Hi Robert, On 2014-10-08 16:01:53 -0400, Robert Haas wrote: > [ comment fixes ] Thanks, I've incorporated these + a bit more. Could you otherwise make sense of the explanation and the algorithm? > +/* yipeyyahee */ > > Although this will be clear to individuals with a good command

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-08 20:07:35 -0400, Robert Haas wrote: > On Wed, Oct 8, 2014 at 2:04 PM, Andres Freund wrote: > > So, what makes it work for me (among other unrelated stuff) seems to be > > the following in .gdbinit, defineing away some things that gdb doesn't > > handle: > > macro define __builtin_offs

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

Hi, On 2014-10-10 10:13:03 +0530, Amit Kapila wrote: > I have done few performance tests for above patches and results of > same is as below: Cool, thanks. > Performance Data > -- > IBM POWER-7 16 cores, 64 hardware threads > RAM = 64GB > max_connections =210 > Databa

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Wed, Oct 8, 2014 at 7:05 PM, Andres Freund wrote: > > Hi, > > Attached you can find the next version of my LW_SHARED patchset. Now > that atomics are committed, it seems like a good idea to also add their > raison d'être. > > Since the last public version I have: > * Addressed lots of Amit's co

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 10/9/14, 4:57 PM, Andres Freund wrote: If you modify either, you better grep for them... I don't think that's going to happen anyway. Requiring it during startup would mean exposing SHARED_LOCK_MASK outside of lwlock.c which'd be ugly. We could possibly stick a StaticAssert() someplace in lwlo

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-09 16:52:46 -0500, Jim Nasby wrote: > On 10/8/14, 8:35 AM, Andres Freund wrote: > >+#define EXCLUSIVE_LOCK (((uint32) 1) << (31 - 1)) > >+ > >+/* Must be greater than MAX_BACKENDS - which is 2^23-1, so we're fine. */ > >+#define SHARED_LOCK_MASK (~EXCLUSIVE_LOCK) > > There should at lea

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 10/8/14, 8:35 AM, Andres Freund wrote: +#define EXCLUSIVE_LOCK (((uint32) 1) << (31 - 1)) + +/* Must be greater than MAX_BACKENDS - which is 2^23-1, so we're fine. */ +#define SHARED_LOCK_MASK (~EXCLUSIVE_LOCK) There should at least be a comment where we define MAX_BACKENDS about the relati

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Wed, Oct 8, 2014 at 2:04 PM, Andres Freund wrote: > So, what makes it work for me (among other unrelated stuff) seems to be > the following in .gdbinit, defineing away some things that gdb doesn't > handle: > macro define __builtin_offsetof(T, F) ((int) &(((T *) 0)->F)) > macro define __extensi

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund wrote: > 2) Implement the wait free LW_SHARED algorithm. + * too high for workloads/locks that were locked in shared mode very s/locked/taken/? + * frequently. Often we were spinning in the (obviously exlusive) spinlock, exclusive. + * acquiration

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On 2014-10-08 15:23:22 -0400, Robert Haas wrote: > On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund wrote: > > 1) Convert PGPROC->lwWaitLink into a dlist. The old code was frail and > >verbose. This also does: > > * changes the logic in LWLockRelease() to release all shared lockers > >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

On Wed, Oct 8, 2014 at 9:35 AM, Andres Freund wrote: > 1) Convert PGPROC->lwWaitLink into a dlist. The old code was frail and >verbose. This also does: > * changes the logic in LWLockRelease() to release all shared lockers > when waking up any. This can yield some significant perform

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-08 14:23:44 -0300, Alvaro Herrera wrote: > Robert Haas wrote: > > On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund > > wrote: > > > I don't see that as being relevant. The difference is an instruction or > > > two - in the slow path we'll enter the kernel and sleep. This doesn't > > > mat

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

Robert Haas wrote: > On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund wrote: > > I don't see that as being relevant. The difference is an instruction or > > two - in the slow path we'll enter the kernel and sleep. This doesn't > > matter in comparison. > > And the code is *so* much more readable. >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-08 13:13:33 -0400, Robert Haas wrote: > On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund wrote: > > I don't see that as being relevant. The difference is an instruction or > > two - in the slow path we'll enter the kernel and sleep. This doesn't > > matter in comparison. > > And the code i

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Wed, Oct 8, 2014 at 8:47 AM, Andres Freund wrote: > I don't see that as being relevant. The difference is an instruction or > two - in the slow path we'll enter the kernel and sleep. This doesn't > matter in comparison. > And the code is *so* much more readable. I find the slist/dlist stuff ac

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.9

Hi, Attached you can find the next version of my LW_SHARED patchset. Now that atomics are committed, it seems like a good idea to also add their raison d'être. Since the last public version I have: * Addressed lots of Amit's comments. Thanks! * Peformed a fair amount of testing. * Rebased the cod

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-10-08 14:47:44 +0200, Andres Freund wrote: > On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > > 5. > > LWLockWakeup() > > { > > .. > > dlist_foreach_modify(iter, (dlist_head *) &wakeup) > > { > > PGPROC *waiter = dlist_container(PGPROC, lwWaitLink, iter.cur); > > LOG_LWDEBUG("LWLockReleas

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-06-25 19:06:32 +0530, Amit Kapila wrote: > 2. > LWLockWakeup() > { > .. > #ifdef LWLOCK_STATS > lwstats->spin_delay_count += SpinLockAcquire(&lock->mutex); > #else > SpinLockAcquire(&lock->mutex); > #endif > .. > } > > Earlier while releasing lock, we don't count it towards LWLock stats >

Re: [HACKERS] Wait free LW_SHARED acquisition

On 01/07/14 23:25, Heikki Linnakangas wrote: On 07/01/2014 01:08 PM, Andres Freund wrote: Hi, Over at -performance Mark Kirkwood tested a recent version of this (http://archives.postgresql.org/message-id/53B283F3.7020005%40catalyst.net.nz) . I thought it's interesting to add the numbers to thi

Re: [HACKERS] Wait free LW_SHARED acquisition

On 07/01/2014 01:08 PM, Andres Freund wrote: Hi, Over at -performance Mark Kirkwood tested a recent version of this (http://archives.postgresql.org/message-id/53B283F3.7020005%40catalyst.net.nz) . I thought it's interesting to add the numbers to this thread: Test: pgbench Options: scale 500

Re: [HACKERS] Wait free LW_SHARED acquisition

Hi, Over at -performance Mark Kirkwood tested a recent version of this (http://archives.postgresql.org/message-id/53B283F3.7020005%40catalyst.net.nz) . I thought it's interesting to add the numbers to this thread: > Test: pgbench > Options: scale 500 > read only > Os: Ubuntu 14.04 > Pg:

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Tue, Jun 24, 2014 at 9:33 AM, Amit Kapila wrote: > On Mon, Jun 23, 2014 at 9:12 PM, Andres Freund wrote: > > On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: > > > 7. > > > LWLockWaitForVar() > > > { > > > .. > > > /* > > > * Add myself to wait queue. Note that this is racy, somebody else > >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Mon, Jun 23, 2014 at 9:12 PM, Andres Freund wrote: > On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: > > 12. > > #ifdef LWLOCK_DEBUG > > lock->owner = MyProc; > > #endif > > > > Shouldn't it be reset in LWLockRelease? > > That's actually intentional. It's quite useful to know the last owner >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Mon, Jun 23, 2014 at 9:12 PM, Andres Freund wrote: > On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: > > On Tue, Jun 17, 2014 at 8:56 PM, Andres Freund > > wrote: > > 2. > > LWLockAcquireCommon() > > { > > .. > > if (!LWLockDequeueSelf(l)) > > { > > /* > > * Somebody else dequeued us and has

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-06-23 19:59:10 +0530, Amit Kapila wrote: > On Tue, Jun 17, 2014 at 8:56 PM, Andres Freund > wrote: > > On 2014-06-17 20:47:51 +0530, Amit Kapila wrote: > > > On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund > > > wrote: > > > > > > You have followed it pretty well as far as I can understand

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Tue, Jun 17, 2014 at 8:56 PM, Andres Freund wrote: > On 2014-06-17 20:47:51 +0530, Amit Kapila wrote: > > On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund > > wrote: > > > > You have followed it pretty well as far as I can understand from your > > replies, as there is no reproducible test (which

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-06-17 20:47:51 +0530, Amit Kapila wrote: > On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund > wrote: > > On 2014-06-17 18:01:58 +0530, Amit Kapila wrote: > > > On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund > > > > On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: > > I unfortunately still c

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Tue, Jun 17, 2014 at 6:35 PM, Andres Freund wrote: > > On 2014-06-17 18:01:58 +0530, Amit Kapila wrote: > > On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund > > > On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: > > > > 2. > > > > Handling of potentialy_spurious case seems to be pending > > > >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-06-17 18:01:58 +0530, Amit Kapila wrote: > On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund > > On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: > > > 2. > > > Handling of potentialy_spurious case seems to be pending > > > in LWLock functions like LWLockAcquireCommon(). > > > > > > LWLockAcq

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Tue, Jun 17, 2014 at 3:56 PM, Andres Freund wrote: > > On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: > > On Fri, May 23, 2014 at 10:01 PM, Amit Kapila > > wrote: > > > On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund > > wrote: > > > > I've pushed a rebased version of the patchset to > > > >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-06-17 12:41:26 +0530, Amit Kapila wrote: > On Fri, May 23, 2014 at 10:01 PM, Amit Kapila > wrote: > > On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund > wrote: > > > I've pushed a rebased version of the patchset to > > > http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git >

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Fri, May 23, 2014 at 10:01 PM, Amit Kapila wrote: > On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund wrote: > > I've pushed a rebased version of the patchset to > > http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git > > branch rwlock contention. > > 220b34331f77effdb46798ddd7cca

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Fri, Jan 31, 2014 at 3:24 PM, Andres Freund wrote: > I've pushed a rebased version of the patchset to > http://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git > branch rwlock contention. > 220b34331f77effdb46798ddd7cca0cffc1b2858 actually was the small problem, > ea9df812d8502fff74

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 01/31/2014 11:54 AM, Andres Freund wrote: Hi, On 2014-01-28 21:27:29 -0800, Peter Geoghegan wrote: On Fri, Nov 15, 2013 at 11:47 AM, Andres Freund wrote: 1) I've added an abstracted atomic ops implementation. Needs a fair amount of work, also submitted as a separate CF entry. (Patch 1

Re: [HACKERS] Wait free LW_SHARED acquisition

On 2014-02-04 13:42:51 -0800, Peter Geoghegan wrote: > On Tue, Feb 4, 2014 at 12:30 PM, Christian Kruse > wrote: > > Ok, benchmark for baseline+alignment patch is running. > > I see that you have enabled latency information. For this kind of > thing I prefer to hack pgbench-tools to not collect t

Re: [HACKERS] Wait free LW_SHARED acquisition

On Tue, Feb 4, 2014 at 12:30 PM, Christian Kruse wrote: > Ok, benchmark for baseline+alignment patch is running. I see that you have enabled latency information. For this kind of thing I prefer to hack pgbench-tools to not collect this (i.e. to not pass the "-l" flag, "Per-Transaction Logging").

Re: [HACKERS] Wait free LW_SHARED acquisition

Hi, On 04/02/14 21:03, Andres Freund wrote: > Christian, could you rerun with master (the commit on which the > branch is based on), the alignment patch, and then the lwlock patch? > Best with max_connections 200. That's probably more important than > the write tests as a first step.. Ok, benchm

Re: [HACKERS] Wait free LW_SHARED acquisition

Hi, On 04/02/14 12:02, Peter Geoghegan wrote: > On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse > wrote: > > I'm doing some benchmarks regarding this problem: one set with > > baseline and one set with your patch. Machine was a 32 core machine (4 > > CPUs with 8 cores), 252 gib RAM. > > What CP

Re: [HACKERS] Wait free LW_SHARED acquisition

On February 4, 2014 8:53:36 PM CET, Peter Geoghegan wrote: >On Tue, Feb 4, 2014 at 11:50 AM, Andres Freund >wrote: >> I think he has applied the patch to hack around the alignment issue I >> pushed to git for both branches. It's not nice enough to be applied >yet, >> but it should fix the issue.

Re: [HACKERS] Wait free LW_SHARED acquisition

On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse wrote: > I'm doing some benchmarks regarding this problem: one set with > baseline and one set with your patch. Machine was a 32 core machine (4 > CPUs with 8 cores), 252 gib RAM. What CPU model? Can you post /proc/cpuinfo? The distinction between

Re: [HACKERS] Wait free LW_SHARED acquisition

On Tue, Feb 4, 2014 at 11:50 AM, Andres Freund wrote: > I think he has applied the patch to hack around the alignment issue I > pushed to git for both branches. It's not nice enough to be applied yet, > but it should fix the issue. > I think the 201 is just a remembrance of debugging the issue. I

Re: [HACKERS] Wait free LW_SHARED acquisition

On 2014-02-04 11:48:14 -0800, Peter Geoghegan wrote: > On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse > wrote: > > I added -M prepared to the pgbench call in the benchwarmer script. > > > > The read-only tests are finished, I come to similiar results as yours: > > > >

Re: [HACKERS] Wait free LW_SHARED acquisition

On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse wrote: > I added -M prepared to the pgbench call in the benchwarmer script. > > The read-only tests are finished, I come to similiar results as yours: > > Note that Christian ran this test with ma

Re: [HACKERS] Wait free LW_SHARED acquisition

On Tue, Feb 4, 2014 at 11:39 AM, Christian Kruse wrote: > I'm doing some benchmarks regarding this problem: one set with > baseline and one set with your patch. Machine was a 32 core machine (4 > CPUs with 8 cores), 252 gib RAM. Both versions have the type align > patch applied. It certainly seem

Re: [HACKERS] Wait free LW_SHARED acquisition

Hi, I'm doing some benchmarks regarding this problem: one set with baseline and one set with your patch. Machine was a 32 core machine (4 CPUs with 8 cores), 252 gib RAM. Both versions have the type align patch applied. pgbench-tools config: SCALES="100" SETCLIENTS="1 4 8 16 32 48 64 96 128" SETT

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-02-03 17:51:20 -0800, Peter Geoghegan wrote: > On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund wrote: > > On 2014-02-01 19:47:29 -0800, Peter Geoghegan wrote: > >> Here are the results of a benchmark on Nathan Boley's 64-core, 4 > >> socket server: > >> http://postgres-benchmarks.s3-website

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund wrote: > On 2014-02-01 19:47:29 -0800, Peter Geoghegan wrote: >> Here are the results of a benchmark on Nathan Boley's 64-core, 4 >> socket server: >> http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/amd-4-socket-rwlocks/ > > That's inter

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund wrote: > The changed algorithm for lwlock imo is an *algorithmic* improvement, > not one for a particular architecture. The advantage being that locking > a lwlock which is primarily taken in shared mode will never need need to > wait or loop. I agree

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Sun, Feb 2, 2014 at 6:00 AM, Andres Freund wrote: > > Some background: > The setups that triggered me into working on the patchset didn't really > have a pgbench like workload, the individual queries were/are more > complicated even though it's still an high throughput OLTP workload. And > the

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

Hi, On 2014-02-01 19:47:29 -0800, Peter Geoghegan wrote: > Here are the results of a benchmark on Nathan Boley's 64-core, 4 > socket server: > http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/amd-4-socket-rwlocks/ That's interesting. The maximum number of what you see here (~293125)

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Sat, Feb 1, 2014 at 1:41 PM, Andres Freund wrote: >> However, I tested the >> most recent revision from your git remote on the AWS instance. > But that was before my fix, right. Except you managed to timetravel :) Heh, okay. So Nathan Boley has generously made available a machine with 4 AMD O

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-02-01 13:40:20 -0800, Peter Geoghegan wrote: > On Sat, Feb 1, 2014 at 4:57 AM, Andres Freund wrote: > >> I'm looking at alternative options, because this is not terribly > >> helpful. With those big caveats in mind, consider the results of the > >> benchmark, which show the patch performin

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Sat, Feb 1, 2014 at 4:57 AM, Andres Freund wrote: >> I'm looking at alternative options, because this is not terribly >> helpful. With those big caveats in mind, consider the results of the >> benchmark, which show the patch performing somewhat worse than the >> master baseline at higher client

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On 2014-01-31 17:52:58 -0800, Peter Geoghegan wrote: > On Fri, Jan 31, 2014 at 1:54 AM, Andres Freund wrote: > > I plan to split the atomics patch into smaller chunks before > > reposting. Imo the "Convert the PGPROC->lwWaitLink list into a dlist > > instead of open coding it." is worth being appl

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

I thought I'd try out what I was in an immediate position to do without having access to dedicated multi-socket hardware: A benchmark on AWS. This was a "c3.8xlarge" instance, which are reportedly backed by Intel Xeon E5-2680 processors. Since the Intel ARK website reports that these CPUs have 16 "

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

On Fri, Jan 31, 2014 at 1:54 AM, Andres Freund wrote: > I plan to split the atomics patch into smaller chunks before > reposting. Imo the "Convert the PGPROC->lwWaitLink list into a dlist > instead of open coding it." is worth being applied independently from > the rest of the series, it simplies

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

Hi, On 2014-01-28 21:27:29 -0800, Peter Geoghegan wrote: > On Fri, Nov 15, 2013 at 11:47 AM, Andres Freund > wrote: > > 1) I've added an abstracted atomic ops implementation. Needs a fair > >amount of work, also submitted as a separate CF entry. (Patch 1 & 2) > > Commit 220b34331f77effdb467

  1   2   >