Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-18 Thread Arseniy Mukhin
Hi, Thank you for the new version! There are several points I noticed: 1) On Tue, Sep 9, 2025 at 3:08 AM Rishu Bagga wrote: > > ... > On Sat, Sep 6, 2025 at 7:52 AM Matheus Alcantara > wrote: > > > Your patch already aims to fix the issue? On [2] I implemented a TAP > > test that reproduce the

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-18 Thread Matheus Alcantara
On Tue Sep 9, 2025 at 8:14 PM -03, Rishu Bagga wrote: > Oops again - I didn't "git add" the new files, so they weren't showing > up in the patch. I added async_xlog.h as well now, > and tested to make sure the patch applies and compiles. Sorry about > that, it should work now. > Thanks, it's compil

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-12 Thread Masahiko Sawada
On Wed, Sep 10, 2025 at 12:00 PM Arseniy Mukhin wrote: > > > 6) We have fixed size queue entries, so I think we don't need this > "padding" logic at the end of the page anymore, because we know how > many entries we can have on each page. +1 Probably we also no longer need to have a pair of page

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-10 Thread Masahiko Sawada
On Thu, Sep 4, 2025 at 3:53 PM Rishu Bagga wrote: > > On Fri, Jul 18, 2025 at 10:06 AM Tom Lane wrote: > > > After thinking about this for awhile, I have a rough idea of > > something we could do to improve parallelism of NOTIFY. > > As a bonus, this'd allow processes on hot standby servers to >

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-09 Thread Rishu Bagga
On Tue, Sep 9, 2025 at 4:02 PM Matheus Alcantara wrote: > > FAILED: src/backend/postgres_lib.a.p/access_rmgrdesc_asyncdesc.c.o > ccache cc -Isrc/backend/postgres_lib.a.p -Isrc/include -I../src/include > -I/opt/homebrew/Cellar/icu4c@77/77.1/include -I/opt/homebrew/opt/lz4/include > -I/opt/homebr

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-09 Thread Matheus Alcantara
On Tue Sep 9, 2025 at 7:49 PM -03, Rishu Bagga wrote: > On Tue, Sep 9, 2025 at 3:34 PM Matheus Alcantara > wrote: >> >> On Mon Sep 8, 2025 at 9:08 PM -03, Rishu Bagga wrote: >> > Hi Joel, Arseniy, Matheus, thanks for taking a look. I’ve attached an >> > updated patch and rebased on the latest comm

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-09 Thread Matheus Alcantara
On Mon Sep 8, 2025 at 9:08 PM -03, Rishu Bagga wrote: > Hi Joel, Arseniy, Matheus, thanks for taking a look. I’ve attached an > updated patch and rebased on the latest commits that fixes the > correctness issues. > I think that your latest patch miss to include of asyncdesc.c file. I'm getting a co

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-09 Thread Rishu Bagga
On Tue, Sep 9, 2025 at 3:34 PM Matheus Alcantara wrote: > > On Mon Sep 8, 2025 at 9:08 PM -03, Rishu Bagga wrote: > > Hi Joel, Arseniy, Matheus, thanks for taking a look. I’ve attached an > > updated patch and rebased on the latest commits that fixes the > > correctness issues. > > > I think that

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-08 Thread Rishu Bagga
Hi Joel, Arseniy, Matheus, thanks for taking a look. I’ve attached an updated patch and rebased on the latest commits that fixes the correctness issues. On Fri, Sep 5, 2025 at 2:38 AM Joel Jacobson wrote: > What's the definition of the test table? It’s just a one column integer table defined as

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-06 Thread Matheus Alcantara
On Thu Sep 4, 2025 at 7:53 PM -03, Rishu Bagga wrote: > Attached is an initial patch that implements this idea. > Thanks for working on this! I haven't looked at the code yet, but have some questions related with the issue discussed at [1]. > Additionally, this would help solve the issue being dis

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-05 Thread Arseniy Mukhin
Hi, On Fri, Sep 5, 2025 at 1:53 AM Rishu Bagga wrote: > > Attached is an initial patch that implements this idea. Thank you for sharing the patch and for working on this! I briefly read the patch and there are some points: 1) There are some compilation warnings about unused functions. 2) I th

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-05 Thread Joel Jacobson
On Fri, Sep 5, 2025, at 00:53, Rishu Bagga wrote: > With the following sql script, I ran > pgbench -T 100 -c 100 -j 8 -f pgbench_transaction_notify.sql -d postgres > > BEGIN; > INSERT INTO test VALUES(1); > NOTIFY benchmark_channel, 'transaction_completed'; > COMMIT; Thanks for working on this. I

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-09-04 Thread Rishu Bagga
On Fri, Jul 18, 2025 at 10:06 AM Tom Lane wrote: > After thinking about this for awhile, I have a rough idea of > something we could do to improve parallelism of NOTIFY. > As a bonus, this'd allow processes on hot standby servers to > receive NOTIFYs from processes on the primary, which is a > fe

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-30 Thread Joel Jacobson
On Tue, Jul 22, 2025, at 14:48, Joel Jacobson wrote: > Benchmark from original post: ... > For a normal PostgreSQL with the CPU and storage on the same physical machine, > I think the results above clearly demonstrate that the global exclusive lock > is at least not the bottleneck, which I strongly

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-22 Thread Joel Jacobson
On Mon, Jul 21, 2025, at 08:16, Joel Jacobson wrote: > Since there is no point of just doing NOTIFY if nobody is LISTENing, > a realistic benchmark would also need to do LISTEN. > What you will then see is that TPS will be severely impacted, > and the gains from removing the global exclusive lock w

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-20 Thread Joel Jacobson
On Mon, Jul 21, 2025, at 00:06, Rishu Bagga wrote: >> "Joel Jacobson" writes: >> >> Unfortunately, the author of [1] jumped to conclusion and assumed >> the global lock was the problem. I'm quite sure it is probably not, >> because: >> >> We know for sure, that current users do LISTEN and NOTIFY >

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-20 Thread Rishu Bagga
> "Joel Jacobson" writes: > > Unfortunately, the author of [1] jumped to conclusion and assumed > the global lock was the problem. I'm quite sure it is probably not, > because: > > We know for sure, that current users do LISTEN and NOTIFY > in the same database. And there is no point in doing NOTI

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-19 Thread Joel Jacobson
On Fri, Jul 18, 2025, at 19:06, Tom Lane wrote: > "Joel Jacobson" writes: >> My patch improves NOTIFY TPS when many backends are listening on multiple >> channels by eliminating unnecessary syscall wake‑ups, but it doesn't increase >> the internal parallelism of the NOTIFY queue itself. > > After

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-18 Thread Tom Lane
"Joel Jacobson" writes: > My patch improves NOTIFY TPS when many backends are listening on multiple > channels by eliminating unnecessary syscall wake‑ups, but it doesn't increase > the internal parallelism of the NOTIFY queue itself. After thinking about this for awhile, I have a rough idea of s

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-18 Thread Joel Jacobson
On Fri, Jul 18, 2025, at 10:15, Joel Jacobson wrote: > On Fri, Jul 18, 2025, at 02:49, Rishu Bagga wrote: >> Hi all, >> >> There’s been some renewed attention on improving the performance of the >> LISTEN/NOTIFY system, which historically hasn’t scaled well under high >> notify frequency. Joel Jaco

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-18 Thread Joel Jacobson
On Fri, Jul 18, 2025, at 08:46, Joel Jacobson wrote: > However, for cases when up to K backends listen on the same channel > (multicast), > my patch and benchmark in the other thread suggests we can achieve massive > improvements of parallelization of both LISTEN/UNLISTEN as well as NOTIFY > witho

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-18 Thread Joel Jacobson
On Fri, Jul 18, 2025, at 02:49, Rishu Bagga wrote: > Hi all, > > There’s been some renewed attention on improving the performance of the > LISTEN/NOTIFY system, which historically hasn’t scaled well under high > notify frequency. Joel Jacobson recently shared some work on optimizing > the LISTEN pa

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-17 Thread Joel Jacobson
On Fri, Jul 18, 2025, at 03:28, Tom Lane wrote: > So I sympathize with concerns about how well the notification code > scales, but I think you're going to have to do a ton more work than > this to get to anything that would pass muster to get committed. > In particular, I'd really want to see somet

Re: Proposal: Out-of-Order NOTIFY via GUC to Improve LISTEN/NOTIFY Throughput

2025-07-17 Thread Tom Lane
Rishu Bagga writes: > To explore this further, I’ve drafted a patch that introduces a new GUC: > `publish_out_of_order_notifications`. We have generally found that GUCs that change query semantics turn out to be very regrettable choices. They break applications that aren't expecting it, and the