Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Andres Freund
Hi, On 2023-01-28 14:25:38 +1300, Thomas Munro wrote: > The nearby thread about searching for uses of volatile reminded me: we > can now drop a bunch of these in postmaster.c. The patch I originally > wrote to do that as part of this series somehow morphed into an > experimental patch to nuke all

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Tom Lane
Thomas Munro writes: > The nearby thread about searching for uses of volatile reminded me: we > can now drop a bunch of these in postmaster.c. The patch I originally > wrote to do that as part of this series somehow morphed into an > experimental patch to nuke all global variables[1], but of cour

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Thomas Munro
The nearby thread about searching for uses of volatile reminded me: we can now drop a bunch of these in postmaster.c. The patch I originally wrote to do that as part of this series somehow morphed into an experimental patch to nuke all global variables[1], but of course we should at least drop the

Re: Using WaitEventSet in the postmaster

2023-01-12 Thread Thomas Munro
On Fri, Jan 13, 2023 at 7:26 AM Andres Freund wrote: > On 2023-01-12 20:35:43 +1300, Thomas Munro wrote: > > Subject: [PATCH] Fix WaitEventSetWait() buffer overrun. > Makes sense. We should backpatch this, I think? Done.

Re: Using WaitEventSet in the postmaster

2023-01-12 Thread Andres Freund
Hi, On 2023-01-12 20:35:43 +1300, Thomas Munro wrote: > Subject: [PATCH] Fix WaitEventSetWait() buffer overrun. > > The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of > WaitEventSetWaitBlock() confused the size of their internal buffer with > the size of the caller's output buffer, and cou

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Thomas Munro
On Thu, Jan 12, 2023 at 7:57 PM Thomas Munro wrote: > On Thu, Jan 12, 2023 at 7:27 PM Tom Lane wrote: > > skink seems to have found a problem: > > > > ==2011873== VALGRINDERROR-BEGIN > > ==2011873== Syscall param epoll_wait(events) points to unaddressable byte(s) > > ==2011873==at 0x4D8DC73:

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Thomas Munro
On Thu, Jan 12, 2023 at 7:27 PM Tom Lane wrote: > skink seems to have found a problem: > > ==2011873== VALGRINDERROR-BEGIN > ==2011873== Syscall param epoll_wait(events) points to unaddressable byte(s) > ==2011873==at 0x4D8DC73: epoll_wait (epoll_wait.c:30) > ==2011873==by 0x55CA49: WaitEv

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Tom Lane
Thomas Munro writes: > Pushed, after a few very minor adjustments, mostly comments. Thanks > for the reviews and pointers. I think there are quite a lot of > refactoring and refinement opportunities unlocked by this change (I > have some draft proposals already), but for now I'll keep an eye on

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Thomas Munro
On Wed, Jan 11, 2023 at 4:07 PM Thomas Munro wrote: > Thanks. Here's v10. I'll wait a bit longer to see if anyone else has > feedback. Pushed, after a few very minor adjustments, mostly comments. Thanks for the reviews and pointers. I think there are quite a lot of refactoring and refinement

Re: Using WaitEventSet in the postmaster

2023-01-10 Thread Thomas Munro
On Sun, Jan 8, 2023 at 11:55 AM Andres Freund wrote: > On 2023-01-07 18:08:11 +1300, Thomas Munro wrote: > > On Sat, Jan 7, 2023 at 12:25 PM Andres Freund wrote: > > > On 2023-01-07 11:08:36 +1300, Thomas Munro wrote: > > > > 3. Is it OK to clobber the shared pending flag for SIGQUIT, SIGTERM, >

Re: Using WaitEventSet in the postmaster

2023-01-07 Thread Andres Freund
Hi, On 2023-01-07 18:08:11 +1300, Thomas Munro wrote: > On Sat, Jan 7, 2023 at 12:25 PM Andres Freund wrote: > > On 2023-01-07 11:08:36 +1300, Thomas Munro wrote: > > > 3. Is it OK to clobber the shared pending flag for SIGQUIT, SIGTERM, > > > SIGINT? If you send all of these extremely rapidly,

Re: Using WaitEventSet in the postmaster

2023-01-06 Thread Thomas Munro
On Sat, Jan 7, 2023 at 12:25 PM Andres Freund wrote: > On 2023-01-07 11:08:36 +1300, Thomas Munro wrote: > > 3. Is it OK to clobber the shared pending flag for SIGQUIT, SIGTERM, > > SIGINT? If you send all of these extremely rapidly, it's > > indeterminate which one will be seen by handle_shutdo

Re: Using WaitEventSet in the postmaster

2023-01-06 Thread Andres Freund
Hi, On 2023-01-07 11:08:36 +1300, Thomas Munro wrote: > 1. Is it OK that we are now using APIs that might throw, in places > where we weren't? I think so: we don't really expect WaitEventSet > APIs to throw unless something is pretty seriously wrong, and if you > hack things to inject failures t

Re: Using WaitEventSet in the postmaster

2023-01-06 Thread Thomas Munro
On Fri, Dec 23, 2022 at 8:46 PM Thomas Munro wrote: > I pushed the small preliminary patches. Here's a rebase of the main patch. Here are some questions I have considered. Anyone got an opinion on point 3, in particular? 1. Is it OK that we are now using APIs that might throw, in places where

Re: Using WaitEventSet in the postmaster

2022-12-22 Thread Thomas Munro
I pushed the small preliminary patches. Here's a rebase of the main patch. From d23fba75cf693ffabc068a36424b7be22342c1b2 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 9 Nov 2022 22:59:58 +1300 Subject: [PATCH v7] Give the postmaster a WaitEventSet and a latch. Traditionally, the postmas

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
Oops, v5 was broken as visible on cfbot (a last second typo broke it). Here's a better one. From 07b04dc410118ad04fd0006edda7ba80f241357a Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 6 Dec 2022 15:21:11 +1300 Subject: [PATCH v6 1/4] Add WL_SOCKET_ACCEPT event to WaitEventSet API. To be

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Andres Freund
Hi, On 2022-12-07 14:12:37 +1300, Thomas Munro wrote: > On Wed, Dec 7, 2022 at 12:12 PM Andres Freund wrote: > > On 2022-12-07 00:58:06 +1300, Thomas Munro wrote: > > > One way to fix that for the epoll version is to EPOLL_CTL_DEL and > > > EPOLL_CTL_ADD, whenever transitioning to/from a zero eve

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
On Wed, Dec 7, 2022 at 2:08 PM Justin Pryzby wrote: > > + /* > > + * It would probably be safe to re-use the inherited signalfd > > since > > + * signalfds only see the current processes pending signals, > > but it > > I think you mean "current process's", r

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
On Wed, Dec 7, 2022 at 12:12 PM Andres Freund wrote: > On 2022-12-07 00:58:06 +1300, Thomas Munro wrote: > > One way to fix that for the epoll version is to EPOLL_CTL_DEL and > > EPOLL_CTL_ADD, whenever transitioning to/from a zero event mask. > > Tried like that in this version. Another approach

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Justin Pryzby
> From 61480441f67ca7fac96ca4bcfe85f27013a47aa8 Mon Sep 17 00:00:00 2001 > From: Thomas Munro > Date: Tue, 6 Dec 2022 16:13:36 +1300 > Subject: [PATCH v4 2/5] Don't leak a signalfd when using latches in the > postmaster. > > + /* > + * It would probably be safe to re-use

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Andres Freund
On 2022-12-07 00:58:06 +1300, Thomas Munro wrote: > One way to fix that for the epoll version is to EPOLL_CTL_DEL and > EPOLL_CTL_ADD, whenever transitioning to/from a zero event mask. > Tried like that in this version. Another approach would be to > (finally) write DeleteWaitEvent() to do the sam

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
On Tue, Dec 6, 2022 at 7:09 AM Andres Freund wrote: > On 2022-12-05 22:45:57 +1300, Thomas Munro wrote: > > The reason for the existing sleep-based approach was that we didn't > > want to accept any more connections (or spin furiously because the > > listen queue was non-empty). So in this versio

Re: Using WaitEventSet in the postmaster

2022-12-05 Thread Andres Freund
Hi, On 2022-12-05 22:45:57 +1300, Thomas Munro wrote: > On Sat, Dec 3, 2022 at 10:41 AM Thomas Munro wrote: > > Here's an iteration like that. Still WIP grade. It passes, but there > > must be something I don't understand about this computer program yet, > > because if I move the "if (pending_.

Re: Using WaitEventSet in the postmaster

2022-12-05 Thread Thomas Munro
On Sat, Dec 3, 2022 at 10:41 AM Thomas Munro wrote: > Here's an iteration like that. Still WIP grade. It passes, but there > must be something I don't understand about this computer program yet, > because if I move the "if (pending_..." section up into the block > where WL_LATCH_SET has arrived

Re: Using WaitEventSet in the postmaster

2022-12-02 Thread Thomas Munro
On Fri, Dec 2, 2022 at 3:36 PM Thomas Munro wrote: > On Fri, Dec 2, 2022 at 2:40 PM Andres Freund wrote: > > It doesn't seem trivial (but not impossible either) to make SetLatch() > > robust > > against arbitrary corruption. So it seems easier to me to just put the latch > > in process local mem

Re: Using WaitEventSet in the postmaster

2022-12-01 Thread Thomas Munro
On Fri, Dec 2, 2022 at 2:40 PM Andres Freund wrote: > On 2022-12-02 10:12:25 +1300, Thomas Munro wrote: > > with a latch as the wakeup mechanism for "PM signals" (requests from > > backends to do things like start a background worker, etc). > > Hm - is that directly related? ISTM that using a WES

Re: Using WaitEventSet in the postmaster

2022-12-01 Thread Andres Freund
Hi, On 2022-12-02 10:12:25 +1300, Thomas Munro wrote: > Here's a work-in-progress patch that uses WaitEventSet for the main > event loop in the postmaster Wee! > with a latch as the wakeup mechanism for "PM signals" (requests from > backends to do things like start a background worker, etc). H

Using WaitEventSet in the postmaster

2022-12-01 Thread Thomas Munro
Hi, Here's a work-in-progress patch that uses WaitEventSet for the main event loop in the postmaster, with a latch as the wakeup mechanism for "PM signals" (requests from backends to do things like start a background worker, etc). There are still raw signals that are part of the external interfac