Re: Make query cancellation keys longer
On Fri, Aug 16, 2024 at 11:29 AM Robert Haas wrote: > > I'll split this patch like that, to make it easier to compare and merge > > with Jelte's corresponding patches. > > That sounds great. IMHO, comparing and merging the patches is the next > step here and would be great to see. Heikki, do you have any update on this work? -- Robert Haas EDB: http://www.enterprisedb.com
Re: Make query cancellation keys longer
On Thu, Sep 5, 2024 at 9:36 AM Jelte Fennema-Nio wrote: > Totally agreed that it would be good to update psql to use the new > much more secure libpq function introduced in PG17[1]. This is not a > trivial change though because it requires refactoring the way we > handle signals (which is why I didn't do it as part of introducing > these new APIs). Yeah, I figured it wasn't a quick fix. > I had hoped that the work in [2] would either do that > or at least make it a lot easier, but that thread seems to have > stalled. So +1 for doing this, but I think it's a totally separate > change and so should be discussed on a separate thread. As long as the new thread doesn't also stall out/get forgotten, I'm happy. > It sounds to me like we should at least use OpenSSL's CRYPTO_memcmp if > we linked against it and the OS doesn't provide a timingsafe_bcmp. > Would that remove your concerns? If we go that direction, I'd still like to know which platforms we expect to have a suboptimal port, if for no other reason than documenting that those users should try to get OpenSSL into their builds. (I agree that they probably will already, if they care.) And if I'm being really picky, I'm not sure we should call our port "timingsafe_bcmp" (vs. pg_secure_bcmp or something) if we know it's not actually timing-safe for some. But I won't die on that hill. Thanks, --Jacob
Re: Make query cancellation keys longer
On Thu, 5 Sept 2024 at 17:43, Jacob Champion wrote: > Has there been any work/discussion around not sending the cancel key > in plaintext from psql? It's not a prerequisite or anything (the > longer length is a clear improvement either way), but it seems odd > that this longer "secret" is still just going to be exposed on the > wire when you press Ctrl+C. Totally agreed that it would be good to update psql to use the new much more secure libpq function introduced in PG17[1]. This is not a trivial change though because it requires refactoring the way we handle signals (which is why I didn't do it as part of introducing these new APIs). I had hoped that the work in [2] would either do that or at least make it a lot easier, but that thread seems to have stalled. So +1 for doing this, but I think it's a totally separate change and so should be discussed on a separate thread. [1]: https://www.postgresql.org/docs/17/libpq-cancel.html#LIBPQ-CANCEL-FUNCTIONS [2]: https://www.postgresql.org/message-id/flat/20240331222502.03b5354bc6356bc5c388919d%40sraoss.co.jp#1450c8fee45408acaa5b5a1b9a6f70fc > For the cancel key implementation in particular, I agree with you that > it's probably not a serious problem. But if other security code starts > using timingsafe_bcmp() then it might be something to be concerned > about. Are there any platform/architecture combos that don't provide a > native timingsafe_bcmp() *and* need a DIT bit for safety? It sounds to me like we should at least use OpenSSL's CRYPTO_memcmp if we linked against it and the OS doesn't provide a timingsafe_bcmp. Would that remove your concerns? I expect anyone that cares about security to link against some TLS library. That way our "fallback" implementation is only used on the rare systems where that's not the case.
Re: Make query cancellation keys longer
On Thu, Sep 5, 2024 at 9:21 AM Tom Lane wrote: > Wasn't this already addressed in v17, by > > Author: Alvaro Herrera > 2024-03-12 [61461a300] libpq: Add encrypted and non-blocking query > cancellation > > ? Perhaps we need to run around and make sure none of our standard > clients use the old API anymore, but the libpq infrastructure is > there already. Right. From a quick grep, it looks like we have seven binaries using the signal-based cancel handler. (For programs that only send a cancel request right before they break the connection, it's probably not worth a huge amount of effort to change it right away, but for psql in particular I think the status quo is a little weird.) Thanks, --Jacob
Re: Make query cancellation keys longer
Jacob Champion writes: > Has there been any work/discussion around not sending the cancel key > in plaintext from psql? It's not a prerequisite or anything (the > longer length is a clear improvement either way), but it seems odd > that this longer "secret" is still just going to be exposed on the > wire when you press Ctrl+C. Wasn't this already addressed in v17, by Author: Alvaro Herrera 2024-03-12 [61461a300] libpq: Add encrypted and non-blocking query cancellation ? Perhaps we need to run around and make sure none of our standard clients use the old API anymore, but the libpq infrastructure is there already. regards, tom lane
Re: Make query cancellation keys longer
On Thu, Aug 15, 2024 at 10:14 AM Heikki Linnakangas wrote: > I'm back to working on the main patch here, to make cancellation keys > longer. New rebased version attached, with all the FIXMEs and TODOs from > the earlier version fixed. There was a lot of bitrot, too. I have a couple of questions/comments separate from the protocol changes: Has there been any work/discussion around not sending the cancel key in plaintext from psql? It's not a prerequisite or anything (the longer length is a clear improvement either way), but it seems odd that this longer "secret" is still just going to be exposed on the wire when you press Ctrl+C. > The first patch now introduces timingsafe_bcmp(), a function borrowed > from OpenBSD to perform a constant-time comparison. There's a configure > check to use the function from the OS if it's available, and includes a > copy of OpenBSD's implementation otherwise. Similar functions exist with > different names in OpenSSL (CRYPTO_memcmp) and NetBSD > (consttime_memequal), but it's a pretty simple function so I don't think > we need to work too hard to pick up those other native implementations. One advantage to using other implementations is that _they're_ on the hook for keeping constant-time guarantees, which is getting trickier due to weird architectural optimizations [1]. CRYPTO_memcmp() has almost the same implementation as 0001 here, except they made the decision to mark the pointers volatile, and they also provide hand-crafted assembly versions. This patch has OpenBSD's version, but they've also turned on data-independent timing by default across their ARM64 processors [2]. And Intel may require the same tweak, but it doesn't look like userspace has access to that setting yet, and the kernel thread [3] appears to have just withered... For the cancel key implementation in particular, I agree with you that it's probably not a serious problem. But if other security code starts using timingsafe_bcmp() then it might be something to be concerned about. Are there any platform/architecture combos that don't provide a native timingsafe_bcmp() *and* need a DIT bit for safety? Thanks, --Jacob [1] https://github.com/golang/go/issues/66450 [2] https://github.com/openbsd/src/commit/cf1440f11c [3] https://lore.kernel.org/lkml/851920c5-31c9-ddd9-3e2d-57d379aa0...@intel.com/
Re: Make query cancellation keys longer
On Fri, Aug 16, 2024 at 10:37 AM Heikki Linnakangas wrote: > If we envision accepting ranges like that in the future, it would be > good to do now rather than later. Otherwise, if someone wants to require > features from protocol 3.2 today, they will have to put > "protocol_version=3.2" in the connection string, and later when 3.3 > version is released, their connection string will continue to force the > then-old 3.2 version. I'm totally cool with doing it now rather than later if you or someone else is willing to do the work. But I don't see why we'd need a protocol bump to change it later. If you write protocol_version=3.7 or protocol_version=3.2-3.7 we send the same thing to the server either way. It's only a difference in whether we slam the connection shut if the server comes back and say it can only do 3.0. > I'll split this patch like that, to make it easier to compare and merge > with Jelte's corresponding patches. That sounds great. IMHO, comparing and merging the patches is the next step here and would be great to see. -- Robert Haas EDB: http://www.enterprisedb.com
Re: Make query cancellation keys longer
On 16/08/2024 15:31, Robert Haas wrote: On Thu, Aug 15, 2024 at 6:07 PM Heikki Linnakangas wrote: Ok, I've read through that thread now, and opined there too. One difference is with libpq option name: My patch adds "protocol_version", while Jelte proposes "max_protocol_version". I don't have strong opinions on that. I hope the ecosystem catches up to support NegotiateProtocolVersion quickly, so that only few people will need to set this option. In particular, I hope that there will never be need to use "max_protocol_version=3.2", because by the time we introduce version 3.3, all the connection poolers that support 3.2 will also implement NegotiateProtocolVersion. In Jelte's design, there end up being two connection parameters. We tell the server we want max_protocol_version, but we accept that it might give us something older. If, however, it tries to degrade us to something lower than min_protocol_version, we bail out. I see you've gone for a simpler design: you ask the server for protocol_version and you get that or you die. To be honest, right up until exactly now, I was assuming we wanted a two-parameter system like that, just because being able to tolerate a range of protocol versions seems useful. However, maybe we don't need it. Alternatively, we could do this for now, and then later we could adjust the parameter so that you can say protocol_version=3.2-3.7 and the client will ask for 3.7 but tolerate anything >= 3.2. Hmm, I kind of like that idea. Works for me. If we envision accepting ranges like that in the future, it would be good to do now rather than later. Otherwise, if someone wants to require features from protocol 3.2 today, they will have to put "protocol_version=3.2" in the connection string, and later when 3.3 version is released, their connection string will continue to force the then-old 3.2 version. I think it's likely that the ecosystem will catch up with NegotiateProtocolVersion once things start breaking. However, I feel pretty confident that there are going to be glitches. Clients are going to want to force newer protocol versions to make sure they get new features, or to make sure that security features that they want to have (like this one) are enabled. Some users are going to be running old poolers that can't handle 3.2, or there will be weirder things where the pooler says it supports it but it doesn't actually work properly in all cases. There are also non-PG servers that reimplement the PG wire protocol. I can't really enumerate all the things that go wrong, but I think there are a number of wire protocol changes that various people have been wanting for a long while now, and when we start to get the infrastructure in place to make that practical, people are going to take advantage of it. So I think we can expect a number of protocol enhancements and changes -- Peter's transparent column encryption stuff is another example -- and there will be mistakes by us and mistakes by others along the way. Allowing users to specify what protocol version they want is probably an important part of coping with that. Yes, it's a good escape hatch to have. The documentation in the patch you attached still seems to think there's an explicit length field for the cancel key. ok thanks Also, I think it would be good to split this into two patches, one to bump the protocol version and a second to change the cancel key stuff. It would facilitate review, and I also think that bumping the protocol version is a big enough deal that it should have its own entry in the commit log. Right. That's what Jelte's first patches did too. Those changes are more or less the same between this patch and his. These clearly need to be merged into one "introduce protocol version 3.2" patch. I'll split this patch like that, to make it easier to compare and merge with Jelte's corresponding patches. -- Heikki Linnakangas Neon (https://neon.tech)
Re: Make query cancellation keys longer
On Thu, Aug 15, 2024 at 6:07 PM Heikki Linnakangas wrote: > Ok, I've read through that thread now, and opined there too. One > difference is with libpq option name: My patch adds "protocol_version", > while Jelte proposes "max_protocol_version". I don't have strong > opinions on that. I hope the ecosystem catches up to support > NegotiateProtocolVersion quickly, so that only few people will need to > set this option. In particular, I hope that there will never be need to > use "max_protocol_version=3.2", because by the time we introduce version > 3.3, all the connection poolers that support 3.2 will also implement > NegotiateProtocolVersion. In Jelte's design, there end up being two connection parameters. We tell the server we want max_protocol_version, but we accept that it might give us something older. If, however, it tries to degrade us to something lower than min_protocol_version, we bail out. I see you've gone for a simpler design: you ask the server for protocol_version and you get that or you die. To be honest, right up until exactly now, I was assuming we wanted a two-parameter system like that, just because being able to tolerate a range of protocol versions seems useful. However, maybe we don't need it. Alternatively, we could do this for now, and then later we could adjust the parameter so that you can say protocol_version=3.2-3.7 and the client will ask for 3.7 but tolerate anything >= 3.2. Hmm, I kind of like that idea. I think it's likely that the ecosystem will catch up with NegotiateProtocolVersion once things start breaking. However, I feel pretty confident that there are going to be glitches. Clients are going to want to force newer protocol versions to make sure they get new features, or to make sure that security features that they want to have (like this one) are enabled. Some users are going to be running old poolers that can't handle 3.2, or there will be weirder things where the pooler says it supports it but it doesn't actually work properly in all cases. There are also non-PG servers that reimplement the PG wire protocol. I can't really enumerate all the things that go wrong, but I think there are a number of wire protocol changes that various people have been wanting for a long while now, and when we start to get the infrastructure in place to make that practical, people are going to take advantage of it. So I think we can expect a number of protocol enhancements and changes -- Peter's transparent column encryption stuff is another example -- and there will be mistakes by us and mistakes by others along the way. Allowing users to specify what protocol version they want is probably an important part of coping with that. The documentation in the patch you attached still seems to think there's an explicit length field for the cancel key. Also, I think it would be good to split this into two patches, one to bump the protocol version and a second to change the cancel key stuff. It would facilitate review, and I also think that bumping the protocol version is a big enough deal that it should have its own entry in the commit log. -- Robert Haas EDB: http://www.enterprisedb.com
Re: Make query cancellation keys longer
On 15/08/2024 23:20, Robert Haas wrote: On Thu, Aug 15, 2024 at 1:13 PM Heikki Linnakangas wrote: Added a "protocol_version" libpq option for that. It defaults to "auto", but you can set it to "3.1" or "3.0" to force the version. It makes it easier to test that the backwards-compatibility works, too. Over on the "Add new protocol message to change GUCs for usage with future protocol-only GUCs" there is a lot of relevant discussion about how bumping the protocol version should work. This thread shouldn't ignore all that discussion. Just to take one example, Jelte wants to bump the protocol version to 3.2, not 3.1, for some reasons that are in the commit message for the relevant patch over there. Ok, I've read through that thread now, and opined there too. One difference is with libpq option name: My patch adds "protocol_version", while Jelte proposes "max_protocol_version". I don't have strong opinions on that. I hope the ecosystem catches up to support NegotiateProtocolVersion quickly, so that only few people will need to set this option. In particular, I hope that there will never be need to use "max_protocol_version=3.2", because by the time we introduce version 3.3, all the connection poolers that support 3.2 will also implement NegotiateProtocolVersion. -- Heikki Linnakangas Neon (https://neon.tech)
Re: Make query cancellation keys longer
On Thu, Aug 15, 2024 at 1:13 PM Heikki Linnakangas wrote: > Added a "protocol_version" libpq option for that. It defaults to "auto", > but you can set it to "3.1" or "3.0" to force the version. It makes it > easier to test that the backwards-compatibility works, too. Over on the "Add new protocol message to change GUCs for usage with future protocol-only GUCs" there is a lot of relevant discussion about how bumping the protocol version should work. This thread shouldn't ignore all that discussion. Just to take one example, Jelte wants to bump the protocol version to 3.2, not 3.1, for some reasons that are in the commit message for the relevant patch over there. -- Robert Haas EDB: http://www.enterprisedb.com
Re: Make query cancellation keys longer
I'm back to working on the main patch here, to make cancellation keys longer. New rebased version attached, with all the FIXMEs and TODOs from the earlier version fixed. There was a lot of bitrot, too. The first patch now introduces timingsafe_bcmp(), a function borrowed from OpenBSD to perform a constant-time comparison. There's a configure check to use the function from the OS if it's available, and includes a copy of OpenBSD's implementation otherwise. Similar functions exist with different names in OpenSSL (CRYPTO_memcmp) and NetBSD (consttime_memequal), but it's a pretty simple function so I don't think we need to work too hard to pick up those other native implementations. I used it for checking if the cancellation key matches, now that it's not a single word anymore. It feels paranoid to worry about timing attacks here, a few instructions is unlikely to give enough signal to an attacker and query cancellation is not a very interesting target anyway. But better safe than sorry. You can still get information about whether a backend with the given PID exists at all, the constant-time comparison only applies to comparing the key. We probably should be using this in some other places in the backend, but I haven't gone around looking for them. Hmm, looking at the current code, I do agree that not introducing a new message would simplify both client and server implementation. Now clients need to do different things depending on if the server supports 3.1 or 3.0 (sending CancelRequestExtended instead of CancelRequest and having to parse BackendKeyData differently). And I also agree that the extra length field doesn't add much in regards to sanity checking (for the CancelRequest and BackendKeyData message types at least). So, sorry for the back and forth on this, but I now agree with you that we should not add the length field. I think one reason I didn't see the benefit before was because the initial patch 0002 was still introducing a CancelRequestExtended message type. If we can get rid of this message type by not adding a length, then I think that's worth losing the sanity checking. Ok, I went back to the original scheme that just redefines the secret key in the CancelRequest message to be variable length, with the length deduced from the message length. We could teach libpq to disconnect and reconnect with minor protocol version 3.0, if connecting with 3.1 fails, but IMHO it's better to not complicate this and accept the break in backwards-compatibility. Yeah, implementing automatic reconnection seems a bit overkill to me too. But it might be nice to add a connection option that causes libpq to use protocol 3.0. Having to install an old libpq to connect to an old server seems quite annoying. Added a "protocol_version" libpq option for that. It defaults to "auto", but you can set it to "3.1" or "3.0" to force the version. It makes it easier to test that the backwards-compatibility works, too. Especially since I think that many other types of servers that implement the postgres protocol have not implemented the minor version negotiation. I at least know PgBouncer[1] and pgcat[2] have not, but probably other server implementations like CockroachDB and Google Spanner have this problem too. Good point. -- Heikki Linnakangas Neon (https://neon.tech) From 5fb2a442efca75dbc384bafee486b090f91806ba Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Thu, 15 Aug 2024 15:49:46 +0300 Subject: [PATCH v5 1/2] Add timingsafe_bcmp(), for constant-time memory comparison timingsafe_bcmp() should be used instead of memcmp() or a naive for-loop, when comparing passwords or secret tokens, to avoid leaking information about the secret token by timing. This commit just introduces the function but does not change any existing code to use it yet. --- configure | 23 +++ configure.ac | 3 ++- meson.build| 2 ++ src/include/port.h | 4 src/port/meson.build | 1 + src/port/timingsafe_bcmp.c | 35 +++ 6 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 src/port/timingsafe_bcmp.c diff --git a/configure b/configure index 2abbeb2794..1fd832bd5b 100755 --- a/configure +++ b/configure @@ -15758,6 +15758,16 @@ fi cat >>confdefs.h <<_ACEOF #define HAVE_DECL_STRSEP $ac_have_decl _ACEOF +ac_fn_c_check_decl "$LINENO" "timingsafe_bcmp" "ac_cv_have_decl_timingsafe_bcmp" "$ac_includes_default" +if test "x$ac_cv_have_decl_timingsafe_bcmp" = xyes; then : + ac_have_decl=1 +else + ac_have_decl=0 +fi + +cat >>confdefs.h <<_ACEOF +#define HAVE_DECL_TIMINGSAFE_BCMP $ac_have_decl +_ACEOF # We can't use AC_CHECK_FUNCS to detect these functions, because it @@ -15918,6 +15928,19 @@ esac fi +ac_fn_c_check_func "$LINENO" "timingsafe_bcmp" "ac_cv_func_timingsafe_bcmp" +if test "x$ac_cv_func_timingsafe_bcmp" = xyes; then : + $as_echo "#define HAVE_TIMINGSAFE_
Re: Make query cancellation keys longer
+ * See if we have a matching backend. Reading the pss_pid and + * pss_cancel_key fields is racy, a backend might die and remove itself + * from the array at any time. The probability of the cancellation key + * matching wrong process is miniscule, however, so we can live with that. + * PIDs are reused too, so sending the signal based on PID is inherently + * racy anyway, although OS's avoid reusing PIDs too soon. Just BTW, we know that Windows sometimes recycles PIDs very soon, sometimes even immediately, to the surprise of us Unix hackers. It can make for some very confusing build farm animal logs. My patch will propose to change that particular thing to proc numbers anyway so I'm not asking for a change here, I just didn't want that assumption to go un-footnoted. I suppose that's actually (another) good reason to want to widen the cancellation key, so that we don't have to worry about proc number allocation order being less protective than traditional Unix PID allocation...
Re: Make query cancellation keys longer
On 24/07/2024 19:12, Heikki Linnakangas wrote: On 04/07/2024 15:20, Jelte Fennema-Nio wrote: On Thu, 4 Jul 2024 at 12:32, Heikki Linnakangas wrote: We currently don't do any locking on the ProcSignal array. For query cancellations, that's good because a query cancel packet is processed without having a PGPROC entry, so we cannot take LWLocks. We could use spinlocks though. In this patch, I used memory barriers to ensure that we load/store the pid and the cancellation key in a sensible order, so that you cannot e.g. send a cancellation signal to a backend that's just starting up and hasn't advertised its cancellation key in ProcSignal yet. But I think this might be simpler and less error-prone by just adding a spinlock to each ProcSignal slot. That would also fix the existing race condition where we might set the pss_signalFlags flag for a slot, when the process concurrently terminates and the slot is reused for a different process. Because of that, we currently have this caveat: "... all the signals are such that no harm is done if they're mistakenly fired". With a spinlock, we could eliminate that race. I think a spinlock would make this thing a whole concurrency stuff a lot easier to reason about. + slot->pss_cancel_key_valid = false; + slot->pss_cancel_key = 0; If no spinlock is added, I think these accesses should still be made atomic writes. Otherwise data-races on those fields are still possible, resulting in undefined behaviour. The memory barriers you added don't prevent that afaict. With atomic operations there are still race conditions, but no data-races. Actually it seems like that same argument applies to the already existing reading/writing of pss_pid: it's written/read using non-atomic operations so data-races can occur and thus undefined behaviour too. Ok, here's a version with spinlocks. I went back and forth on what exactly is protected by the spinlock. I kept the "volatile sig_atomic_t" type for pss_signalFlags, so that it can still be safely read without holding the spinlock in CheckProcSignal, but all the functions that set the flags now hold the spinlock. That removes the race condition that you might set the flag for wrong slot, if the target backend exits and the slot is recycled. The race condition was harmless and there were comments to note it, but it doesn't occur anymore with the spinlock. (Note: Thomas's "Interrupts vs signals" patch will remove ps_signalFlags altogether. I'm looking forward to that.) -volatile pid_t pss_pid; +pid_tpss_pid; Why remove the volatile modifier? Because I introduced a memory barrier to ensure the reads/writes of pss_pid become visible to other processes in right order. That makes the 'volatile' unnecessary IIUC. With the spinlock, the point is moot however. I: - fixed a few comments, - fixed a straightforward failure with EXEC_BACKEND (ProcSignal needs to be passed down in BackendParameters now), - put back the snippet to signal the whole process group if supported, which you pointed out earlier and committed this refactoring patch. Thanks for the review! -- Heikki Linnakangas Neon (https://neon.tech)
Re: Make query cancellation keys longer
On 04/07/2024 15:20, Jelte Fennema-Nio wrote: On Thu, 4 Jul 2024 at 12:32, Heikki Linnakangas wrote: We currently don't do any locking on the ProcSignal array. For query cancellations, that's good because a query cancel packet is processed without having a PGPROC entry, so we cannot take LWLocks. We could use spinlocks though. In this patch, I used memory barriers to ensure that we load/store the pid and the cancellation key in a sensible order, so that you cannot e.g. send a cancellation signal to a backend that's just starting up and hasn't advertised its cancellation key in ProcSignal yet. But I think this might be simpler and less error-prone by just adding a spinlock to each ProcSignal slot. That would also fix the existing race condition where we might set the pss_signalFlags flag for a slot, when the process concurrently terminates and the slot is reused for a different process. Because of that, we currently have this caveat: "... all the signals are such that no harm is done if they're mistakenly fired". With a spinlock, we could eliminate that race. I think a spinlock would make this thing a whole concurrency stuff a lot easier to reason about. + slot->pss_cancel_key_valid = false; + slot->pss_cancel_key = 0; If no spinlock is added, I think these accesses should still be made atomic writes. Otherwise data-races on those fields are still possible, resulting in undefined behaviour. The memory barriers you added don't prevent that afaict. With atomic operations there are still race conditions, but no data-races. Actually it seems like that same argument applies to the already existing reading/writing of pss_pid: it's written/read using non-atomic operations so data-races can occur and thus undefined behaviour too. Ok, here's a version with spinlocks. I went back and forth on what exactly is protected by the spinlock. I kept the "volatile sig_atomic_t" type for pss_signalFlags, so that it can still be safely read without holding the spinlock in CheckProcSignal, but all the functions that set the flags now hold the spinlock. That removes the race condition that you might set the flag for wrong slot, if the target backend exits and the slot is recycled. The race condition was harmless and there were comments to note it, but it doesn't occur anymore with the spinlock. (Note: Thomas's "Interrupts vs signals" patch will remove ps_signalFlags altogether. I'm looking forward to that.) -volatile pid_t pss_pid; +pid_tpss_pid; Why remove the volatile modifier? Because I introduced a memory barrier to ensure the reads/writes of pss_pid become visible to other processes in right order. That makes the 'volatile' unnecessary IIUC. With the spinlock, the point is moot however. -- Heikki Linnakangas Neon (https://neon.tech) From 2e6035837c028abd3b645a8c038cac7179016eef Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 24 Jul 2024 19:06:32 +0300 Subject: [PATCH v4 1/1] Move cancel key generation to after forking the backend Move responsibility of generating the cancel key to the backend process. The cancel key is now generated after forking, and the backend advertises it in the ProcSignal array. When a cancel request arrives, the backend handling it scans the ProcSignal array to find the target pid and cancel key. This is similar to how this previously worked in the EXEC_BACKEND case with the ShmemBackendArray, just reusing the ProcSignal array. One notable change is that we no longer generate cancellation keys for non-backend processes. We generated them before just to prevent a malicious user from canceling them, the keys for non-backend processes were never actually given to anyone. There is now an explicit flag indicating whether a process has a valid key or not. I wrote this originally in preparation for supporting longer cancel keys, but it's a nice cleanup on its own. Reviewed-by: Jelte Fennema-Nio Discussion: https://www.postgresql.org/message-id/508d0505-8b7a-4864-a681-e7e5edfe3...@iki.fi --- src/backend/postmaster/auxprocess.c | 2 +- src/backend/postmaster/launch_backend.c | 6 - src/backend/postmaster/postmaster.c | 196 +--- src/backend/storage/ipc/ipci.c | 11 -- src/backend/storage/ipc/procsignal.c| 172 - src/backend/tcop/backend_startup.c | 9 +- src/backend/tcop/postgres.c | 21 +++ src/backend/utils/init/globals.c| 3 +- src/backend/utils/init/postinit.c | 2 +- src/include/miscadmin.h | 1 + src/include/postmaster/postmaster.h | 9 -- src/include/storage/procsignal.h| 3 +- 12 files changed, 175 insertions(+), 260 deletions(-) diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c index b21eae7136..74b8a00c94 100644 --- a/src/backend/postmaster/auxprocess.c +++ b/src/backend/postmaster/auxprocess.c @@ -71,7 +71,7 @@ AuxiliaryProcessMainCommon(void) BaseInit
Re: Make query cancellation keys longer
On Thu, 4 Jul 2024 at 14:43, Tomas Vondra wrote: > I don't have any immediate feedback regarding this patch, but I'm > wondering about one thing related to cancellations - we talk cancelling > a query, but we really target a PID (or a particular backend, no matter > how we identify it). > > I occasionally want to only cancel a particular query, but I don't think > that's really possible - the query may complete meanwhile, and the > backend may even get used for a different user connection (e.g. with a > connection pool)? Or am I missing something important? No, you're not missing anything. Having the target of the cancel request be the backend instead of a specific query is really annoying and can cause all kinds of race conditions. I had to redesign and complicate the cancellation logic in PgBouncer significantly, to make sure that one client could not cancel a connection from another client anymore: https://github.com/pgbouncer/pgbouncer/pull/717 > Anyway, I wonder if making the cancellation key longer (or variable > length) might be useful for this too - it would allow including some > sort of optional "query ID" in the request, not just the PID. (Maybe > st_xact_start_timestamp would work?) Yeah, some query ID would be necessary. I think we'd want a dedicated field for it though. Instead of encoding it in the secret. Getting the xact_start_timestamp would require communication with the server to get it, which you would like to avoid since the server might be unresponsive. So I think a command counter that both sides keep track of would be better. This counter could then be incremented after every Query and Sync message. > Obviously, that's not up to this patch, but it's somewhat related. Yeah, let's postpone more discussion on this until we have the currently proposed much simpler change in, which only changes the secret length.
Re: Make query cancellation keys longer
Hi, I don't have any immediate feedback regarding this patch, but I'm wondering about one thing related to cancellations - we talk cancelling a query, but we really target a PID (or a particular backend, no matter how we identify it). I occasionally want to only cancel a particular query, but I don't think that's really possible - the query may complete meanwhile, and the backend may even get used for a different user connection (e.g. with a connection pool)? Or am I missing something important? Anyway, I wonder if making the cancellation key longer (or variable length) might be useful for this too - it would allow including some sort of optional "query ID" in the request, not just the PID. (Maybe st_xact_start_timestamp would work?) Obviously, that's not up to this patch, but it's somewhat related. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: Make query cancellation keys longer
On Thu, 4 Jul 2024 at 12:32, Heikki Linnakangas wrote: > We currently don't do any locking on the ProcSignal array. For query > cancellations, that's good because a query cancel packet is processed > without having a PGPROC entry, so we cannot take LWLocks. We could use > spinlocks though. In this patch, I used memory barriers to ensure that > we load/store the pid and the cancellation key in a sensible order, so > that you cannot e.g. send a cancellation signal to a backend that's just > starting up and hasn't advertised its cancellation key in ProcSignal > yet. But I think this might be simpler and less error-prone by just > adding a spinlock to each ProcSignal slot. That would also fix the > existing race condition where we might set the pss_signalFlags flag for > a slot, when the process concurrently terminates and the slot is reused > for a different process. Because of that, we currently have this caveat: > "... all the signals are such that no harm is done if they're mistakenly > fired". With a spinlock, we could eliminate that race. I think a spinlock would make this thing a whole concurrency stuff a lot easier to reason about. + slot->pss_cancel_key_valid = false; + slot->pss_cancel_key = 0; If no spinlock is added, I think these accesses should still be made atomic writes. Otherwise data-races on those fields are still possible, resulting in undefined behaviour. The memory barriers you added don't prevent that afaict. With atomic operations there are still race conditions, but no data-races. Actually it seems like that same argument applies to the already existing reading/writing of pss_pid: it's written/read using non-atomic operations so data-races can occur and thus undefined behaviour too. -volatile pid_t pss_pid; +pid_tpss_pid; Why remove the volatile modifier?
Re: Make query cancellation keys longer
On 04/07/2024 13:50, Jelte Fennema-Nio wrote: On Thu, 4 Jul 2024 at 12:35, Heikki Linnakangas wrote: On 04/07/2024 13:32, Heikki Linnakangas wrote: Here's a new version of the first patch. Sorry, forgot attachment. It seems you undid the following earlier change. Was that on purpose? If not, did you undo any other earlier changes by accident? +SendCancelRequest(int backendPID, int32 cancelAuthCode) I think this name of the function is quite confusing, it's not sending a cancel request, it is processing one. It sends a SIGINT. Heh, well, it's sending the cancel request signal to the right backend, but I see your point. Renamed to ProcessCancelRequest. Ah, I made that change as part of the second patch earlier, so I didn't consider it now. I don't feel strongly about it, but I think SendCancelRequest() actually feels a little better, in procsignal.c. It's more consistent with the existing SendProcSignal() function. There was indeed another change in the second patch that I missed: + /* If we have setsid(), signal the backend's whole process group */ +#ifdef HAVE_SETSID + kill(-backendPID, SIGINT); +#else kill(backendPID, SIGINT); +#endif I'm not sure that's really required, when sending SIGINT to a backend process. A backend process shouldn't have any child processes, and if it did, it's not clear what good SIGINT will do them. But I guess it makes sense to do it anyway, for consistency with pg_cancel_backend(), which also signals the whole process group. -- Heikki Linnakangas Neon (https://neon.tech)
Re: Make query cancellation keys longer
On Thu, 4 Jul 2024 at 12:35, Heikki Linnakangas wrote: > > On 04/07/2024 13:32, Heikki Linnakangas wrote: > > Here's a new version of the first patch. > > Sorry, forgot attachment. It seems you undid the following earlier change. Was that on purpose? If not, did you undo any other earlier changes by accident? > > +SendCancelRequest(int backendPID, int32 cancelAuthCode) > > > > I think this name of the function is quite confusing, it's not sending > > a cancel request, it is processing one. It sends a SIGINT. > > Heh, well, it's sending the cancel request signal to the right backend, > but I see your point. Renamed to ProcessCancelRequest.
Re: Make query cancellation keys longer
On 04/07/2024 13:32, Heikki Linnakangas wrote:
Here's a new version of the first patch.
Sorry, forgot attachment.
--
Heikki Linnakangas
Neon (https://neon.tech)
From e9fc1f4365077673c71ba138a0ae7fbcebe16a36 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas
Date: Thu, 4 Jul 2024 01:03:15 +0300
Subject: [PATCH v3 1/1] Move cancel key generation to after forking the
backend
Move responsibility of generating the cancel key to the backend
process. The cancel key is now generated after forking, and the
backend advertises it in the ProcSignal array. When a cancel request
arrives, the backend handling it scans the ProcSignal array to find
the target pid and cancel key. This is similar to how this previously
worked in the EXEC_BACKEND case with the ShmemBackendArray, just
reusing the ProcSignal array.
One notable change is that we no longer generate cancellation keys for
non-backend processes. We generated them before just to prevent a
malicious user from canceling them, the keys for non-backend processes
were never actually given to anyone. There is now an explicit flag
indicating whether a process has a valid key or not.
I wrote this originally in preparation for supporting longer cancel
keys, but it's a nice cleanup on its own.
Reviewed-by: Jelte Fennema-Nio
Discussion: https://www.postgresql.org/message-id/508d0505-8b7a-4864-a681-e7e5edfe3...@iki.fi
---
src/backend/postmaster/auxprocess.c | 2 +-
src/backend/postmaster/launch_backend.c | 8 +-
src/backend/postmaster/postmaster.c | 196 +---
src/backend/storage/ipc/ipci.c | 11 --
src/backend/storage/ipc/procsignal.c| 77 +-
src/backend/tcop/backend_startup.c | 9 +-
src/backend/tcop/postgres.c | 21 +++
src/backend/utils/init/globals.c| 3 +-
src/backend/utils/init/postinit.c | 2 +-
src/include/miscadmin.h | 1 +
src/include/postmaster/postmaster.h | 5 -
src/include/storage/procsignal.h| 3 +-
12 files changed, 113 insertions(+), 225 deletions(-)
diff --git a/src/backend/postmaster/auxprocess.c b/src/backend/postmaster/auxprocess.c
index b21eae7136..74b8a00c94 100644
--- a/src/backend/postmaster/auxprocess.c
+++ b/src/backend/postmaster/auxprocess.c
@@ -71,7 +71,7 @@ AuxiliaryProcessMainCommon(void)
BaseInit();
- ProcSignalInit();
+ ProcSignalInit(false, 0);
/*
* Auxiliary processes don't run transactions, but they may need a
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index f9b24b7989..50a5a38526 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -93,7 +93,6 @@ typedef int InheritableSocket;
typedef struct
{
char DataDir[MAXPGPATH];
- int32 MyCancelKey;
int MyPMChildSlot;
#ifndef WIN32
unsigned long UsedShmemSegID;
@@ -103,7 +102,6 @@ typedef struct
#endif
void *UsedShmemSegAddr;
slock_t*ShmemLock;
- struct bkend *ShmemBackendArray;
#ifndef HAVE_SPINLOCKS
PGSemaphore *SpinlockSemaArray;
#endif
@@ -676,7 +674,7 @@ extern slock_t *ProcStructLock;
extern PGPROC *AuxiliaryProcs;
extern PMSignalData *PMSignalState;
extern pg_time_t first_syslogger_file_time;
-extern struct bkend *ShmemBackendArray;
+extern bool redirection_done;
#ifndef WIN32
#define write_inheritable_socket(dest, src, childpid) ((*(dest) = (src)), true)
@@ -708,7 +706,6 @@ save_backend_variables(BackendParameters *param, ClientSocket *client_sock,
strlcpy(param->DataDir, DataDir, MAXPGPATH);
- param->MyCancelKey = MyCancelKey;
param->MyPMChildSlot = MyPMChildSlot;
#ifdef WIN32
@@ -718,7 +715,6 @@ save_backend_variables(BackendParameters *param, ClientSocket *client_sock,
param->UsedShmemSegAddr = UsedShmemSegAddr;
param->ShmemLock = ShmemLock;
- param->ShmemBackendArray = ShmemBackendArray;
#ifndef HAVE_SPINLOCKS
param->SpinlockSemaArray = SpinlockSemaArray;
@@ -967,7 +963,6 @@ restore_backend_variables(BackendParameters *param)
SetDataDir(param->DataDir);
- MyCancelKey = param->MyCancelKey;
MyPMChildSlot = param->MyPMChildSlot;
#ifdef WIN32
@@ -977,7 +972,6 @@ restore_backend_variables(BackendParameters *param)
UsedShmemSegAddr = param->UsedShmemSegAddr;
ShmemLock = param->ShmemLock;
- ShmemBackendArray = param->ShmemBackendArray;
#ifndef HAVE_SPINLOCKS
SpinlockSemaArray = param->SpinlockSemaArray;
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 6f974a8d21..02442a4b85 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -168,7 +168,6 @@
typedef struct bkend
{
pid_t pid; /* process id of backend */
- int32 cancel_key; /* cancel key for cancels for this backend */
int child_slot; /* PMChildSlot for this backend, if any */
int bkend_type; /* child process flavor, see above */
bool dead_end; /* is it going to send an error and quit? */
@@ -178,10 +177,6 @@ typede
Re: Make query cancellation keys longer
Here's a new version of the first patch. In the previous version, I added the pid cancellation key to pmsignal.c, but on second thoughts, I think procsignal.c is a better place. The ProcSignal array already contains the pid, we just need to add the cancellation key there. This first patch just refactors the current code, without changing the protocol or length of the cancellation key. I'd like to get this reviewed and committed first, and get back to the protocol changes after that. We currently don't do any locking on the ProcSignal array. For query cancellations, that's good because a query cancel packet is processed without having a PGPROC entry, so we cannot take LWLocks. We could use spinlocks though. In this patch, I used memory barriers to ensure that we load/store the pid and the cancellation key in a sensible order, so that you cannot e.g. send a cancellation signal to a backend that's just starting up and hasn't advertised its cancellation key in ProcSignal yet. But I think this might be simpler and less error-prone by just adding a spinlock to each ProcSignal slot. That would also fix the existing race condition where we might set the pss_signalFlags flag for a slot, when the process concurrently terminates and the slot is reused for a different process. Because of that, we currently have this caveat: "... all the signals are such that no harm is done if they're mistakenly fired". With a spinlock, we could eliminate that race. Thoughts? -- Heikki Linnakangas Neon (https://neon.tech)
Re: Make query cancellation keys longer
On Fri, Mar 8, 2024 at 5:20 PM Heikki Linnakangas wrote: > The nice thing about relying on the message length was that we could > just redefine the CancelRequest message to have a variable length > secret, and old CancelRequest with 4-byte secret was compatible with the > new definition too. But it doesn't matter much, so added an explicit > length field. I think I liked your original idea better than this one. > One consequence of this patch that I didn't mention earlier is that it > makes libpq incompatible with server version 9.2 and below, because the > minor version negotiation was introduced in version 9.3. We could teach > libpq to disconnect and reconnect with minor protocol version 3.0, if > connecting with 3.1 fails, but IMHO it's better to not complicate this > and accept the break in backwards-compatibility. 9.3 was released in > 2013. We dropped pg_dump support for versions older than 9.2 a few years > ago, this raises the bar for pg_dump to 9.3 as well. I think we shouldn't underestimate the impact of bumping the minor protocol version. Minor version negotiation is probably not supported by all drivers and Jelte says that it's not supported by any poolers, so for anybody but direct libpq users, there will be some breakage. Now, on the one hand, as Jelte has said, there's little value in having a protocol version if we're too afraid to make use of it. But on the other hand, is this problem serious enough to justify the breakage we'll cause? I'm not sure. It seems pretty silly to be using a 32-bit value for this in 2024, but I'm sure some people aren't going to like it, and they may not all have noticed this thread. On the third hand, if we do this, it may help to unblock a bunch of other pending patches that also want to do protocol-related things. -- Robert Haas EDB: http://www.enterprisedb.com
Re: Make query cancellation keys longer
On Fri, 8 Mar 2024 at 23:20, Heikki Linnakangas wrote: > Added some documentation. There's more work to be done there, but at > least the message type descriptions are now up-to-date. Thanks, that's very helpful. > The nice thing about relying on the message length was that we could > just redefine the CancelRequest message to have a variable length > secret, and old CancelRequest with 4-byte secret was compatible with the > new definition too. But it doesn't matter much, so I added an explicit > length field. > > FWIW I don't think that restriction makes sense. Any code that parses > the messages needs to have the message length available, and I don't > think it helps with sanity checking that much. I think the documentation > is a little anachronistic. The real reason that all the message types > include enough information to find the message end is that in protocol > version 2, there was no message length field. The only exception that > doesn't have that property is CopyData, and it's no coincidence that it > was added in protocol version 3. Hmm, looking at the current code, I do agree that not introducing a new message would simplify both client and server implementation. Now clients need to do different things depending on if the server supports 3.1 or 3.0 (sending CancelRequestExtended instead of CancelRequest and having to parse BackendKeyData differently). And I also agree that the extra length field doesn't add much in regards to sanity checking (for the CancelRequest and BackendKeyData message types at least). So, sorry for the back and forth on this, but I now agree with you that we should not add the length field. I think one reason I didn't see the benefit before was because the initial patch 0002 was still introducing a CancelRequestExtended message type. If we can get rid of this message type by not adding a length, then I think that's worth losing the sanity checking. > Unfortunately 1234,5679 already in use by NEGOTIATE_SSL_CODE. That's why > I went the other direction. If we want to add this to "the end", it > needs to be 1234,5681. But I wanted to keep the cancel requests together. Fair enough, I didn't realise that. This whole point is moot anyway if we're not introducing CancelRequestExtended > We could teach > libpq to disconnect and reconnect with minor protocol version 3.0, if > connecting with 3.1 fails, but IMHO it's better to not complicate this > and accept the break in backwards-compatibility. Yeah, implementing automatic reconnection seems a bit overkill to me too. But it might be nice to add a connection option that causes libpq to use protocol 3.0. Having to install an old libpq to connect to an old server seems quite annoying. Especially since I think that many other types of servers that implement the postgres protocol have not implemented the minor version negotiation. I at least know PgBouncer[1] and pgcat[2] have not, but probably other server implementations like CockroachDB and Google Spanner have this problem too. I'll try to add such a fallback connection option to my patchset soon. [1]: https://github.com/pgbouncer/pgbouncer/pull/1007 [2]: https://github.com/postgresml/pgcat/issues/706
Re: Make query cancellation keys longer
On 01/03/2024 07:19, Jelte Fennema-Nio wrote:
I think this behaviour makes more sense as a minor protocol version
bump instead of a parameter.
Ok, the consensus is to bump the minor protocol version for this. Works
for me.
+ if (strcmp(conn->workBuffer.data, "_pq_.extended_query_cancel") == 0)
+ {
+ /* that's ok */
+ continue;
+ }
Please see this thread[1], which in the first few patches makes
pqGetNegotiateProtocolVersion3 actually usable for extending the
protocol. I started that, because very proposed protocol change that's
proposed on the list has similar changes to
pqGetNegotiateProtocolVersion3 and I think we shouldn't make these
changes ad-hoc hacked into the current code, but actually do them once
in a way that makes sense for all protocol changes.
Thanks, I will take a look and respond on that thread.
Documentation is still missing
I think at least protocol message type documentation would be very
helpful in reviewing, because that is really a core part of this
change.
Added some documentation. There's more work to be done there, but at
least the message type descriptions are now up-to-date.
Based on the current code I think it should have a few changes:
+ int cancel_key_len = 5 + msgLength - (conn->inCursor - conn->inStart);
+
+ conn->be_cancel_key = malloc(cancel_key_len);
+ if (conn->be_cancel_key == NULL)
This is using the message length to determine the length of the cancel
key in BackendKey. That is not something we generally do in the
protocol. It's even documented: "Notice that although each message
includes a byte count at the beginning, the message format is defined
so that the message end can be found without reference to the byte
count." So I think the patch should be changed to include the length
of the cancel key explicitly in the message.
The nice thing about relying on the message length was that we could
just redefine the CancelRequest message to have a variable length
secret, and old CancelRequest with 4-byte secret was compatible with the
new definition too. But it doesn't matter much, so added an explicit
length field.
FWIW I don't think that restriction makes sense. Any code that parses
the messages needs to have the message length available, and I don't
think it helps with sanity checking that much. I think the documentation
is a little anachronistic. The real reason that all the message types
include enough information to find the message end is that in protocol
version 2, there was no message length field. The only exception that
doesn't have that property is CopyData, and it's no coincidence that it
was added in protocol version 3.
On 03/03/2024 16:27, Jelte Fennema-Nio wrote:
On Fri, 1 Mar 2024 at 06:19, Jelte Fennema-Nio wrote:
This is a preliminary review. I'll look at this more closely soon.
Some more thoughts after looking some more at the proposed changes
+#define EXTENDED_CANCEL_REQUEST_CODE PG_PROTOCOL(1234,5677)
nit: I think the code should be 1234,5679 because it's the newer
message type, so to me it makes more sense if the code is a larger
number.
Unfortunately 1234,5679 already in use by NEGOTIATE_SSL_CODE. That's why
I went the other direction. If we want to add this to "the end", it
needs to be 1234,5681. But I wanted to keep the cancel requests together.
+ * FIXME: we used to use signal_child. I believe kill() is
+ * maybe even more correct, but verify that.
signal_child seems to be the correct one, not kill. signal_child has
this relevant comment explaining why it's better than plain kill:
* On systems that have setsid(), each child process sets itself up as a
* process group leader. For signals that are generally interpreted in the
* appropriate fashion, we signal the entire process group not just the
* direct child process. This allows us to, for example, SIGQUIT a blocked
* archive_recovery script, or SIGINT a script being run by a backend via
* system().
I changed it to signal the process group if HAVE_SETSID, like
pg_signal_backend() does. We don't need to signal both the process group
and the process itself like signal_child() does, because we don't have
the race condition with recently-forked children that signal_child()
talks about.
+SendCancelRequest(int backendPID, int32 cancelAuthCode)
I think this name of the function is quite confusing, it's not sending
a cancel request, it is processing one. It sends a SIGINT.
Heh, well, it's sending the cancel request signal to the right backend,
but I see your point. Renamed to ProcessCancelRequest.
While we're at it, switch to using atomics in pmsignal.c for the state
field. That feels easier to reason about than volatile
pointers.
I feel like this refactor would benefit from being a separate commit.
That would make it easier to follow which change to pmsignal.c is
necessary for what.
Point taken. I didn't do that yet, but it makes sense.
+ MyCancelKeyLength = (MyProcPort != NULL &&
MyProcPort->extended_query_cancel) ?
Re: Make query cancellation keys longer
On Fri, Mar 1, 2024 at 03:19:23PM +0100, Peter Eisentraut wrote: > On 29.02.24 22:25, Heikki Linnakangas wrote: > > Currently, cancel request key is a 32-bit token, which isn't very much > > entropy. If you want to cancel another session's query, you can > > brute-force it. In most environments, an unauthorized cancellation of a > > query isn't very serious, but it nevertheless would be nice to have more > > protection from it. The attached patch makes it longer. It is an > > optional protocol feature, so it's fully backwards-compatible with > > clients that don't support longer keys. > > My intuition would be to make this a protocol version bump, not an optional > feature. I think this is something that everyone should eventually be > using, not a niche feature that you explicitly want to opt-in for. Agreed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Make query cancellation keys longer
On Sun, 3 Mar 2024 at 15:27, Jelte Fennema-Nio wrote: > + case EOF: > + /* We'll come back when there is more data */ > + return PGRES_POLLING_READING; > > Nice catch, I'll go steal this for my patchset which adds all the > necessary changes to be able to do a protocol bump[1]. Actually, it turns out your change to return PGRES_POLLING_READING on EOF is incorrect (afaict). A little bit above there is this code comment above a check to see if the whole body was received: * Can't process if message body isn't all here yet. * * After this check passes, any further EOF during parsing * implies that the server sent a bad/truncated message. * Reading more bytes won't help in that case, so don't return * PGRES_POLLING_READING after this point. So I'll leave my patchset as is.
Re: Make query cancellation keys longer
On Fri, 1 Mar 2024 at 06:19, Jelte Fennema-Nio wrote: > This is a preliminary review. I'll look at this more closely soon. Some more thoughts after looking some more at the proposed changes +#define EXTENDED_CANCEL_REQUEST_CODE PG_PROTOCOL(1234,5677) nit: I think the code should be 1234,5679 because it's the newer message type, so to me it makes more sense if the code is a larger number. + * FIXME: we used to use signal_child. I believe kill() is + * maybe even more correct, but verify that. signal_child seems to be the correct one, not kill. signal_child has this relevant comment explaining why it's better than plain kill: * On systems that have setsid(), each child process sets itself up as a * process group leader. For signals that are generally interpreted in the * appropriate fashion, we signal the entire process group not just the * direct child process. This allows us to, for example, SIGQUIT a blocked * archive_recovery script, or SIGINT a script being run by a backend via * system(). +SendCancelRequest(int backendPID, int32 cancelAuthCode) I think this name of the function is quite confusing, it's not sending a cancel request, it is processing one. It sends a SIGINT. > While we're at it, switch to using atomics in pmsignal.c for the state > field. That feels easier to reason about than volatile > pointers. I feel like this refactor would benefit from being a separate commit. That would make it easier to follow which change to pmsignal.c is necessary for what. + MyCancelKeyLength = (MyProcPort != NULL && MyProcPort->extended_query_cancel) ? MAX_CANCEL_KEY_LENGTH : 4; I think we should be doing this check the opposite way, i.e. only fall back to the smaller key when explicitly requested: + MyCancelKeyLength = (MyProcPort != NULL && MyProcPort->old_query_cancel) ? 4 : MAX_CANCEL_KEY_LENGTH; That way we'd get the more secure, longer key length for non-backend processes such as background workers. + case EOF: + /* We'll come back when there is more data */ + return PGRES_POLLING_READING; Nice catch, I'll go steal this for my patchset which adds all the necessary changes to be able to do a protocol bump[1]. + int be_pid; /* PID of backend --- needed for XX cancels */ Seems like you accidentally added XX to the comment in this line. [1]: https://www.postgresql.org/message-id/flat/CAGECzQSr2%3DJPJHNN06E_jTF2%2B0E60K%3DhotyBw5wY%3Dq9Wvmt7DQ%40mail.gmail.com#359e4222eb161da37124be1a384f8d92
Re: Make query cancellation keys longer
On Fri, 1 Mar 2024 at 15:19, Peter Eisentraut wrote: > > One complication with this was that because we no longer know how long > > the key should be, 4-bytes or something longer, until the backend has > > performed the protocol negotiation, we cannot generate the key in the > > postmaster before forking the process anymore. > > Maybe this would be easier if it's a protocol version number change, > since that is sent earlier than protocol extensions? Protocol version and protocol extensions are both sent in the StartupMessage, so the same complication applies. (But I do agree that a protocol version bump is more appropriate for this type of change)
Re: Make query cancellation keys longer
On 29.02.24 22:25, Heikki Linnakangas wrote: Currently, cancel request key is a 32-bit token, which isn't very much entropy. If you want to cancel another session's query, you can brute-force it. In most environments, an unauthorized cancellation of a query isn't very serious, but it nevertheless would be nice to have more protection from it. The attached patch makes it longer. It is an optional protocol feature, so it's fully backwards-compatible with clients that don't support longer keys. My intuition would be to make this a protocol version bump, not an optional feature. I think this is something that everyone should eventually be using, not a niche feature that you explicitly want to opt-in for. One complication with this was that because we no longer know how long the key should be, 4-bytes or something longer, until the backend has performed the protocol negotiation, we cannot generate the key in the postmaster before forking the process anymore. Maybe this would be easier if it's a protocol version number change, since that is sent earlier than protocol extensions?
Re: Make query cancellation keys longer
This is a preliminary review. I'll look at this more closely soon.
On Thu, 29 Feb 2024 at 22:26, Heikki Linnakangas wrote:
> If the client requests the "_pq_.extended_query_cancel" protocol
> feature, the server will generate a longer 256-bit cancellation key.
Huge +1 for this general idea. This is a problem I ran into with
PgBouncer, and had to make some concessions when fitting the
information I wanted into the available bits. Also from a security
perspective I don't think the current amount of bits have stood the
test of time.
+ ADD_STARTUP_OPTION("_pq_.extended_query_cancel", "");
Since this parameter doesn't actually take a value (just empty
string). I think this behaviour makes more sense as a minor protocol
version bump instead of a parameter.
+ if (strcmp(conn->workBuffer.data, "_pq_.extended_query_cancel") == 0)
+ {
+ /* that's ok */
+ continue;
+ }
Please see this thread[1], which in the first few patches makes
pqGetNegotiateProtocolVersion3 actually usable for extending the
protocol. I started that, because very proposed protocol change that's
proposed on the list has similar changes to
pqGetNegotiateProtocolVersion3 and I think we shouldn't make these
changes ad-hoc hacked into the current code, but actually do them once
in a way that makes sense for all protocol changes.
> Documentation is still missing
I think at least protocol message type documentation would be very
helpful in reviewing, because that is really a core part of this
change. Based on the current code I think it should have a few
changes:
+ int cancel_key_len = 5 + msgLength - (conn->inCursor - conn->inStart);
+
+ conn->be_cancel_key = malloc(cancel_key_len);
+ if (conn->be_cancel_key == NULL)
This is using the message length to determine the length of the cancel
key in BackendKey. That is not something we generally do in the
protocol. It's even documented: "Notice that although each message
includes a byte count at the beginning, the message format is defined
so that the message end can be found without reference to the byte
count." So I think the patch should be changed to include the length
of the cancel key explicitly in the message.
[1]:
https://www.postgresql.org/message-id/flat/CAGECzQSr2%3DJPJHNN06E_jTF2%2B0E60K%3DhotyBw5wY%3Dq9Wvmt7DQ%40mail.gmail.com#359e4222eb161da37124be1a384f8d92
Make query cancellation keys longer
Currently, cancel request key is a 32-bit token, which isn't very much
entropy. If you want to cancel another session's query, you can
brute-force it. In most environments, an unauthorized cancellation of a
query isn't very serious, but it nevertheless would be nice to have more
protection from it. The attached patch makes it longer. It is an
optional protocol feature, so it's fully backwards-compatible with
clients that don't support longer keys.
If the client requests the "_pq_.extended_query_cancel" protocol
feature, the server will generate a longer 256-bit cancellation key.
However, the new longer key length is no longer hardcoded in the
protocol. The client is expected to deal with variable length keys, up
to some reasonable upper limit (TODO: document the maximum). This
flexibility allows e.g. a connection pooler to add more information to
the cancel key, which could be useful. If the client doesn't request the
protocol feature, the server generates a 32-bit key like before.
One complication with this was that because we no longer know how long
the key should be, 4-bytes or something longer, until the backend has
performed the protocol negotiation, we cannot generate the key in the
postmaster before forking the process anymore. The first patch here
changes things so that the cancellation key is generated later, in the
backend, and the backend advertises the key in the PMSignalState array.
This is similar to how this has always worked in EXEC_BACKEND mode with
the ShmemBackendArray, but instead of having a separate array, I added
fields to the PMSignalState slots. This removes a bunch of
EXEC_BACKEND-specific code, which is nice.
Any thoughts on this? Documentation is still missing, and there's one
TODO on adding a portable time-constant memcmp() function; I'll add
those if there's agreement on this otherwise.
--
Heikki Linnakangas
Neon (https://neon.tech)From f871e915fee08b3cc27e732646f3a62ec1c3024b Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas
Date: Thu, 29 Feb 2024 23:20:52 +0200
Subject: [PATCH 1/2] Move cancel key generation to after forking the backend
Move responsibility of generating the cancel key to the backend
process. The cancel key is now generated after forking, and the
backend advertises it in the PMSignalState array. When a cancel
request arrives, the backend handling it scans the PMSignalState array
to find the target pid and cancel key. This is similar to how this
previously worked in the EXEC_BACKEND case with the ShmemBackendArray,
but instead of having the separate array, we store the cancel keys in
PMSignalState array, and use it also in the !EXEC_BACKEND case.
While we're at it, switch to using atomics in pmsignal.c for the state
field. That feels easier to reason about than volatile
pointers. (TODO: also switch to pg_atomic_flag for PMSignalFlags
array. Weirdly we don't have "pg_atomic_test_clear_flag" nor
"pg_atomic_set_flag". Those would be the natural opertions for
PMSignalFlags. Maybe we should add those, but I didn't want to go that
deep down the rabbit hole in this patch.)
This is needed by the next patch, so that we can generate the cancel
key later, after negotiating the protocol version with the client.
---
src/backend/postmaster/postmaster.c | 185 +---
src/backend/storage/ipc/ipci.c | 11 --
src/backend/storage/ipc/pmsignal.c | 124 ++-
src/backend/storage/lmgr/proc.c | 16 ++-
src/include/postmaster/postmaster.h | 3 -
src/include/storage/pmsignal.h | 3 +-
src/tools/pgindent/typedefs.list| 1 +
7 files changed, 116 insertions(+), 227 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index da0c627107e..408f8fb9b6d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -177,7 +177,6 @@
typedef struct bkend
{
pid_t pid; /* process id of backend */
- int32 cancel_key; /* cancel key for cancels for this backend */
int child_slot; /* PMChildSlot for this backend, if any */
int bkend_type; /* child process flavor, see above */
bool dead_end; /* is it going to send an error and quit? */
@@ -187,10 +186,6 @@ typedef struct bkend
static dlist_head BackendList = DLIST_STATIC_INIT(BackendList);
-#ifdef EXEC_BACKEND
-static Backend *ShmemBackendArray;
-#endif
-
BackgroundWorker *MyBgworkerEntry = NULL;
@@ -430,7 +425,6 @@ static void SendNegotiateProtocolVersion(List *unrecognized_protocol_options);
static void processCancelRequest(Port *port, void *pkt);
static void report_fork_failure_to_client(Port *port, int errnum);
static CAC_state canAcceptConnections(int backend_type);
-static bool RandomCancelKey(int32 *cancel_key);
static void signal_child(pid_t pid, int signal);
static void sigquit_child(pid_t pid);
static bool SignalSomeChildren(int signal, int target);
@@ -517,7 +511,6 @@ typedef struct
#endif
void *UsedShmemSegAddr;
slock_
