Re: heavily contended lwlocks with long wait queues scale badly

2024-01-22 Thread Michael Paquier
On Fri, Jan 19, 2024 at 01:49:59PM +0100, Jakub Wartak wrote: > Hi Michael, just to reassure you that it is a good thing. We have a > customer who reported much better performance on 16.x than on 13~15 in > very heavy duty LWLock/lockmanager scenarios (ofc, before that was > committed/released),

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-17 Thread Michael Paquier
On Tue, Jan 16, 2024 at 03:11:48PM +0900, Michael Paquier wrote: > I'd like to apply that, just let me know if you have any comments > and/or objections. And done on 12~15. While on it, I have also looked at source code references on github and debian that involve lwWaiting, and all of them rely

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-16 Thread Michael Paquier
On Tue, Jan 16, 2024 at 11:24:49PM -0500, Jonathan S. Katz wrote: > On 1/16/24 1:11 AM, Michael Paquier wrote: >> I'd like to apply that, just let me know if you have any comments >> and/or objections. > > Looking at the code, I understand an argument for not backpatching given we > modify the

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-16 Thread Jonathan S. Katz
On 1/16/24 1:11 AM, Michael Paquier wrote: On Thu, Jan 11, 2024 at 09:47:33AM -0500, Jonathan S. Katz wrote: I have similar data sources to Nathan/Michael and I'm trying to avoid piling on, but one case that's interesting occurred after a major version upgrade from PG10 to PG14 on a database

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-15 Thread Michael Paquier
On Thu, Jan 11, 2024 at 09:47:33AM -0500, Jonathan S. Katz wrote: > I have similar data sources to Nathan/Michael and I'm trying to avoid piling > on, but one case that's interesting occurred after a major version upgrade > from PG10 to PG14 on a database supporting a very active/highly concurrent

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-11 Thread Jonathan S. Katz
On 1/10/24 10:45 PM, Michael Paquier wrote: On Wed, Jan 10, 2024 at 09:17:47PM -0600, Nathan Bossart wrote: Now that commit a4adc31 has had some time to bake and concerns about unintended consequences may have abated, I wanted to revive this back-patching discussion. I see a few

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-10 Thread Michael Paquier
On Wed, Jan 10, 2024 at 09:17:47PM -0600, Nathan Bossart wrote: > Now that commit a4adc31 has had some time to bake and concerns about > unintended consequences may have abated, I wanted to revive this > back-patching discussion. I see a few possibly-related reports [0] [1] > [2], and I'm now

Re: heavily contended lwlocks with long wait queues scale badly

2024-01-10 Thread Nathan Bossart
On Mon, Nov 21, 2022 at 10:31:14AM -0500, Jonathan S. Katz wrote: > On 11/20/22 2:56 PM, Andres Freund wrote: >> I still think it might be worth to backpatch in a bit, but so far the votes >> on >> that weren't clear enough on that to feel comfortable. > > My general feeling is "yes" on

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-21 Thread Jonathan S. Katz
On 11/20/22 2:56 PM, Andres Freund wrote: Hi, On 2022-11-09 17:03:13 -0800, Andres Freund wrote: On 2022-11-09 09:38:08 -0800, Andres Freund wrote: I'm on a hike, without any connectivity, Thu afternoon - Sun. I think it's OK to push it to HEAD if I get it done in the next few hours. Bigger

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-20 Thread Andres Freund
Hi, On 2022-11-09 17:03:13 -0800, Andres Freund wrote: > On 2022-11-09 09:38:08 -0800, Andres Freund wrote: > > I'm on a hike, without any connectivity, Thu afternoon - Sun. I think it's > > OK > > to push it to HEAD if I get it done in the next few hours. Bigger issues, > > which I do not

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-09 Thread Andres Freund
On 2022-11-09 09:38:08 -0800, Andres Freund wrote: > I'm on a hike, without any connectivity, Thu afternoon - Sun. I think it's OK > to push it to HEAD if I get it done in the next few hours. Bigger issues, > which I do not expect, should show up before tomorrow afternoon. Smaller > things could

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-09 Thread Andres Freund
Hi, On 2022-11-09 15:54:16 +0530, Bharath Rupireddy wrote: > On Tue, Nov 1, 2022 at 12:46 PM Bharath Rupireddy > wrote: > > > > > Updated patch attached. > > > > Thanks. It looks good to me. However, some minor comments on the v3 patch: > > > > 1. > > -if (MyProc->lwWaiting) > > +if

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-09 Thread Bharath Rupireddy
On Tue, Nov 1, 2022 at 12:46 PM Bharath Rupireddy wrote: > > > Updated patch attached. > > Thanks. It looks good to me. However, some minor comments on the v3 patch: > > 1. > -if (MyProc->lwWaiting) > +if (MyProc->lwWaiting != LW_WS_NOT_WAITING) > elog(PANIC, "queueing for lock

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-03 Thread Jonathan S. Katz
On 11/1/22 1:41 PM, Andres Freund wrote: Andres: when you suggested backpatching, were you thinking of the Nov 2022 release or the Feb 2023 release? I wasn't thinking that concretely. Even if we decide to backpatch, I'd be very hesitant to do it in a few days. Yeah this was my thinking (and

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-01 Thread Andres Freund
Hi, On 2022-11-01 11:19:02 -0400, Jonathan S. Katz wrote: > This is the type of fix that would make headlines in a major release > announcement (10x TPS improvement w/4096 connections?!). That is also part > of the tradeoff of backpatching this, is that we may lose some of the higher > visibility

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-01 Thread Andres Freund
Hi, On 2022-11-01 08:37:39 -0400, Robert Haas wrote: > On Tue, Nov 1, 2022 at 3:17 AM Bharath Rupireddy > wrote: > > Below are test results with v3 patch. +1 for back-patching it. > > The problem with back-patching stuff like this is that it can have > unanticipated consequences. I think that

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-01 Thread Jonathan S. Katz
On 11/1/22 8:37 AM, Robert Haas wrote: On Tue, Nov 1, 2022 at 3:17 AM Bharath Rupireddy wrote: Below are test results with v3 patch. +1 for back-patching it. First, awesome find and proposed solution! The problem with back-patching stuff like this is that it can have unanticipated

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-01 Thread Robert Haas
On Tue, Nov 1, 2022 at 3:17 AM Bharath Rupireddy wrote: > Below are test results with v3 patch. +1 for back-patching it. The problem with back-patching stuff like this is that it can have unanticipated consequences. I think that the chances of something like this backfiring are less than for a

Re: heavily contended lwlocks with long wait queues scale badly

2022-11-01 Thread Bharath Rupireddy
On Tue, Nov 1, 2022 at 5:21 AM Andres Freund wrote: > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > > with the same set up [1], I'm not sure if it's really because of the > > patch. I'm unable to reproduce it now

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Zhihong Yu
On Mon, Oct 31, 2022 at 5:19 PM Andres Freund wrote: > Hi, > > On 2022-10-31 17:17:03 -0700, Zhihong Yu wrote: > > On Mon, Oct 31, 2022 at 4:51 PM Andres Freund > wrote: > > > > > Hi, > > > > > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > > > BTW, I've seen a sporadic crash

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 17:17:03 -0700, Zhihong Yu wrote: > On Mon, Oct 31, 2022 at 4:51 PM Andres Freund wrote: > > > Hi, > > > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > > > with the same set up [1], I'm not

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Zhihong Yu
On Mon, Oct 31, 2022 at 4:51 PM Andres Freund wrote: > Hi, > > On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > > with the same set up [1], I'm not sure if it's really because of the > > patch. I'm unable to reproduce

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Andres Freund
Hi, On 2022-10-31 16:21:06 +0530, Bharath Rupireddy wrote: > BTW, I've seen a sporadic crash (SEGV) with the patch in bg writer > with the same set up [1], I'm not sure if it's really because of the > patch. I'm unable to reproduce it now and unfortunately I didn't > capture further details when

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Robert Haas
On Thu, Oct 27, 2022 at 12:59 PM Andres Freund wrote: > After pulling my > hair out for quite a while to try to understand that behaviour, I figured out > that it's just a side-effect of *removing* some other contention. I've seen this kind of pattern on multiple occasions. I don't know if they

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Alexander Korotkov
Hi Andres, Thank you for your patch. The results are impressive. On Mon, Oct 31, 2022 at 2:10 PM Pavel Borisov wrote: > > I was working on optimizing the LWLock queue in a little different way > and I also did a benchmarking of Andres' original patch from this > thread. [1] > The results are

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Pavel Borisov
I was working on optimizing the LWLock queue in a little different way and I also did a benchmarking of Andres' original patch from this thread. [1] The results are quite impressive, indeed. Please feel free to see the results and join the discussion in [1] if you want. Best regards, Pavel [1]

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Bharath Rupireddy
On Thu, Oct 27, 2022 at 10:29 PM Andres Freund wrote: > > But I think we can solve that fairly reasonably nonetheless. We can change > PGPROC->lwWaiting to not just be a boolean, but have three states: > 0: not waiting > 1: waiting in waitlist > 2: waiting to be woken up > > which we then can use

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-31 Thread Dilip Kumar
On Mon, Oct 31, 2022 at 11:03 AM Kyotaro Horiguchi wrote: > > At Thu, 27 Oct 2022 09:59:14 -0700, Andres Freund wrote > in > > But I think we can solve that fairly reasonably nonetheless. We can change > > PGPROC->lwWaiting to not just be a boolean, but have three states: > > 0: not waiting > >

Re: heavily contended lwlocks with long wait queues scale badly

2022-10-30 Thread Kyotaro Horiguchi
At Thu, 27 Oct 2022 09:59:14 -0700, Andres Freund wrote in > But I think we can solve that fairly reasonably nonetheless. We can change > PGPROC->lwWaiting to not just be a boolean, but have three states: > 0: not waiting > 1: waiting in waitlist > 2: waiting to be woken up > > which we then

heavily contended lwlocks with long wait queues scale badly

2022-10-27 Thread Andres Freund
Hi, I am working on posting a patch series making relation extension more scalable. As part of that I was running some benchmarks for workloads that I thought should not or just positively impacted - but I was wrong, there was some very significant degradation at very high client counts. After