Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 13:19:34 -0700, Peter Geoghegan wrote: > On Fri, Aug 11, 2023 at 12:26 PM Andres Freund wrote: > > > For example, dealing with core dumps left behind by the regression > > > tests can be annoying. > > > > Hm. I don't have a significant pro

Re: walsender "wakeup storm" on PG16, likely because of bc971f4025c (Optimize walsender wake up logic using condition variables)

2023-08-11 Thread Andres Freund
d. Hypothetical users of ConditionVariableSignal() would > then still have a way to deal with rare lost signals if they are > concerned about that problem. Sounds like a plan to me. Greetings, Andres Freund

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 11:56:27 -0700, Peter Geoghegan wrote: > On Fri, Aug 11, 2023 at 11:23 AM Andres Freund wrote: > > > Couldn't you say the same thing about defensive "can't happen" ERRORs? > > > They are essentially a form of assertion that isn

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 11:14:34 -0700, Peter Geoghegan wrote: > On Fri, Aug 11, 2023 at 10:57 AM Andres Freund wrote: > > I am quite strongly against this. This will lead to assertions being hit in > > tests without that being noticed, e.g. because they happen in a background > &

Re: [PATCH] Support static linking against LLVM

2023-08-11 Thread Andres Freund
KNX_%3Df%2B1C4r06WETKTq0G4Z_7q4L4Fxn5WWpMycDj9Fw%40mail.gmail.com > Migrating to the new APIs will have PostgreSQL require at the minimum > version 8 of LLVM (released in March 2019). I think Thomas' patch abstract it enough so that older versions continue to work... Greetings, Andres Freund

Re: LLVM 16 (opaque pointers)

2023-08-11 Thread Andres Freund
t); > +bool > +ExecEvalBoolSubroutineTemplate(ExprState *state, > +struct ExprEvalStep > *op, > +ExprContext > *econtext) > +{ > + ExecEvalBoolSubroutine fp PG_USED_FOR_ASSERTS_ONLY; > + > + fp = &ExecEvalBoolSubroutineTemplate; > + return false; > +} > + Thanks for working on this! Greetings, Andres Freund

Re: LLVM 16 (opaque pointers)

2023-08-11 Thread Andres Freund
hairier > most notably due to to the change in fcinfo args representation. But I guess > that's also a topic for another day :-) Given that 11 is about to be EOL, I don't think it's worth spending the time to support a new LLVM version for it. Greetings, Andres Freund

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
ss that just restarts. Greetings, Andres Freund

Re: walsender "wakeup storm" on PG16, likely because of bc971f4025c (Optimize walsender wake up logic using condition variables)

2023-08-11 Thread Andres Freund
. I've been "solving" it by adding a ConditionVariableCancelSleepEx(), which has a only_broadcasts argument. I'm inclined to think that any code that needs that needs the forwarding behaviour is pretty much buggy. Greetings, Andres Freund

Re: [PATCH] Support static linking against LLVM

2023-08-11 Thread Andres Freund
ronment variable > `LLVM_CONFIG_LINK_ARGS`. > > To statically link against LLVM it suffices, then, to call `configure` > with environment variable `LLVM_CONFIG_LINK_ARGS=--link-static`. I'm not opposed to it, but I'm not sure I see much need for it either. Do you have a specific use case for this? Greetings, Andres Freund

Re: Add PG CI to older PG releases

2023-08-10 Thread Andres Freund
nd it much less stressful to backpatch if I can test the patches via CI first. I don't think I'm alone in that - Alvaro for example has previously done this in a more limited fashion (no windows): https://postgr.es/m/20220928192226.4c6zeenujaoqq4bq%40alvherre.pgsql Greetings, Andres Freund

Re: Add PG CI to older PG releases

2023-08-10 Thread Andres Freund
be combined with commit 950e64fa46b164df87b5eb7c6e15213ab9880f87 Author: Andres Freund Date: 2022-07-18 17:06:34 -0700 Use STDOUT/STDERR_FILENO in most of syslogger. Greetings, Andres Freund

Re: Support to define custom wait events for extensions

2023-08-09 Thread Andres Freund
Hi, On 2023-08-09 20:10:42 +0900, Masahiro Ikeda wrote: > * Is there any way to not force extensions that don't use shared > memory for their use like dblink to acquire AddinShmemInitLock?; I think the caller shouldn't need to do deal with AddinShmemInitLock at all. Greetings, Andres Freund

Re: Remove distprep

2023-08-09 Thread Andres Freund
The situation for configure is somewhat different, due to being maintained in the repository, rather than just being included in the tarball... Greetings, Andres Freund

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-08 Thread Andres Freund
crosoft, fwiw. Perhaps Joe and Jonathan know how to start within AWS? And perhaps Noah inside GCP? It'd be the least work to get it up and running in GCP, as it's already running there, but should be quite doable at the others as well. Greetings, Andres Freund

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-08 Thread Andres Freund
Hi, On 2023-08-08 11:58:25 +0300, Heikki Linnakangas wrote: > On 08/08/2023 05:15, Andres Freund wrote: > > With the improvements detailed below, cirrus' free CI would last > > about ~65 runs / month. > > I think that's plenty. Not so sure, I would regularly exceed

Re: Support to define custom wait events for extensions

2023-08-08 Thread Andres Freund
Hi, On 2023-08-09 08:03:29 +0900, Michael Paquier wrote: > On Tue, Aug 08, 2023 at 03:59:54PM -0700, Andres Freund wrote: > > On 2023-08-08 08:54:10 +0900, Michael Paquier wrote: > >> - WaitEventExtensionShmemInit() should gain a dshash_create(), to make > >> sure that

Re: Support to define custom wait events for extensions

2023-08-08 Thread Andres Freund
Hi, On 2023-08-08 08:54:10 +0900, Michael Paquier wrote: > - WaitEventExtensionShmemInit() should gain a dshash_create(), to make > sure that the shared table is around, and we are going to have a > reference to it in WaitEventExtensionCounterData, saved from > dshash_get_hash_table_handle(). I'm

Re: Configurable FP_LOCK_SLOTS_PER_BACKEND

2023-08-08 Thread Andres Freund
Hi, On 2023-08-08 16:44:37 -0400, Robert Haas wrote: > On Mon, Aug 7, 2023 at 6:05 PM Andres Freund wrote: > > I think the biggest flaw of the locking scheme is that the LockHash locks > > protect two, somewhat independent, things: > > 1) the set of currently lockable obje

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-08 Thread Andres Freund
Hi, On 2023-08-08 18:34:58 +0200, Alvaro Herrera wrote: > On 2023-Aug-08, Andres Freund wrote: > > > Given the cost of macos, it seems like it'd be by far the most of affordable > > to just buy 1-2 mac minis (2x ~660USD) and stick them in a shelf somewhere, > > as &

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-08 Thread Andres Freund
Hi, On 2023-08-08 10:25:52 -0500, Tristan Partin wrote: > On Mon Aug 7, 2023 at 9:15 PM CDT, Andres Freund wrote: > > FWIW: with the patches applied, the "credit costs" in cirrus CI are roughly > > like the following (depends on caching etc): > > > > task

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-08 Thread Andres Freund
Hi, On 2023-08-08 16:28:49 +0200, Peter Eisentraut wrote: > On 08.08.23 04:15, Andres Freund wrote: > > Potential paths forward for cfbot, in addition to the above: > > > > - Pay for compute / ask the various cloud providers to grant us compute > >credits. At least

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-07 Thread Andres Freund
Hi, On 2023-08-07 19:15:41 -0700, Andres Freund wrote: > The set of free CI providers has shrunk since we chose cirrus, as have the > "free" resources provided. I started, quite incomplete as of now, wiki page at > [4]. Oops, as Thomas just noticed, I left off th

Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-07 Thread Andres Freund
nity: 0.01 linux-compiler-warnings: 0.05 linux-meson: 0.07 freebsd : 0.08 linux-autoconf: 0.09 windows : 0.18 macos : 0.28 total task runtime is 40.8 cost in credits is 0.76, monthly credits of 50 allow approx 66.10 runs/month Greetings, And

Re: cpluspluscheck vs ICU

2023-08-07 Thread Andres Freund
Hi, On 2023-03-10 20:10:30 -0800, Andres Freund wrote: > On 2023-03-10 19:37:27 -0800, Andres Freund wrote: > > I just hit this once more - and I figured out a fairly easy fix: > > > > We just need a > > #ifndef U_DEFAULT_SHOW_DRAFT > > #define U_DEFAULT_SHO

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-08-07 Thread Andres Freund
got: '0' > # expected: '1' Hm, that could just be a "harmless" race. Does it still happen if you apply the attached patch in addition? Greetings, Andres Freund diff --git i/src/backend/utils/activity/pgstat_database.c w/src/backend/utils/activity/pgstat_da

Re: Configurable FP_LOCK_SLOTS_PER_BACKEND

2023-08-07 Thread Andres Freund
Hi, On 2023-08-07 14:36:48 -0700, Andres Freund wrote: > What if fast path locks entered PROCLOCK into the shared hashtable, just like > with normal locks, the first time a lock is acquired by a backend. Except that > we'd set a flag indicating the lock is a fastpath lock. Wh

Re: Configurable FP_LOCK_SLOTS_PER_BACKEND

2023-08-07 Thread Andres Freund
e anymore. Acquiring a fast path lock would just require atomically modifying the PROCLOCK to indicate that the lock is held. On a first blush, this sounds like it could end up being fairly clean and generic? Greetings, Andres Freund

Re: Configurable FP_LOCK_SLOTS_PER_BACKEND

2023-08-07 Thread Andres Freund
x27;m not sure if we're seeing this behavior in production It might be worth for you to backpatch commit 92daeca45df Author: Andres Freund Date: 2022-11-21 20:34:17 -0800 Add wait event for pg_usleep() in perform_spin_delay() into 12. That should be low risk and have only trivially resol

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-07 Thread Andres Freund
43 2 80 4155 8280 16 369 32 405 64 344 Any chance you could your benchmark? I don't see as much of a regression vs 16 as you... Greetings, Andres Freund [1] COPY (SELECT generate_series(1, 10)) TO '/tmp/data.copy'

Re: Configurable FP_LOCK_SLOTS_PER_BACKEND

2023-08-06 Thread Andres Freund
r partitions... So if there's a bout of real lock contention (for longer than deadlock_timeout)... Given that most of your lock manager traffic comes from query planning - have you evaluated using prepared statements more heavily? Greetings, Andres Freund

Re: initdb caching during tests

2023-08-05 Thread Andres Freund
Hi, On 2023-08-05 16:58:38 -0400, Tom Lane wrote: > Andres Freund writes: > > Times for running all tests under meson, on my workstation (20 cores / 40 > > threads): > > > cassert build -O2: > > > Before: > > real0m44.638s > > user

ci: Improve macos startup using a cached macports installation

2023-08-05 Thread Andres Freund
build anyway). That gain is a bit smaller without that, but still significant. An alternative implementation would be to create the "base" .dmg file outside of CI and download it onto the CI instances. But I didn't want to figure out the hosting situation for such files, so I thoug

initdb caching during tests

2023-08-05 Thread Andres Freund
7;s not worth worrying about that. Greetings, Andres Freund [1] https://postgr.es/m/20220120021859.3zpsfqn4z7ob7afz%40alap3.anarazel.de >From 0fd431c277f01284a91999a04368de6b59b6691e Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Thu, 2 Feb 2023 21:51:53 -0800 Subject: [PATCH v3 1/2] U

Re: SIGQUIT handling, redux

2023-08-02 Thread Andres Freund
Hi, On 2023-08-02 12:35:19 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-09-11 11:52:55 -0400, Tom Lane wrote: > >> It's simple enough that maybe we could back-patch it, once it's > >> aged awhile in HEAD. OTOH, given the lack of field repor

Re: SIGQUIT handling, redux

2023-08-02 Thread Andres Freund
the commit? I've not heard of issues due to the checks in check_on_shmem_exit_lists_are_empty(). Greetings, Andres Freund

Re: stats test intermittent failure

2023-08-01 Thread Andres Freund
e ring. This is considered an > eviction and not a reuse. I integrated the suggested change of the comment and tweaked it a bit more. And finally pushed the fix. Sorry that it took so long. Greetings, Andres Freund

Re: Support to define custom wait events for extensions

2023-07-31 Thread Andres Freund
e. It'd be much easier to use if we had a shared hashtable with the identifiers than what's been merged now. In plenty of cases it's not realistic for an extension library to run in each backend, while still needing to wait for things. Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-31 Thread Andres Freund
\ pgbench -n -P1 -f <( echo "COPY lotsaints_wide FROM '/tmp/lotsaints_wide.copy' WHERE false") -t 5 15: 2992.605 HEAD: 3325.201 fastpath1.patch 2932.606 fastpath2.patch 2783.915 Interestingly fastpath1 is slower now, even though it wasn't with earlier patches (which still is repeatable). I do not have the foggiest as to why. Greetings, Andres Freund

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 19:11:38 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-07-31 18:33:37 -0400, Tom Lane wrote: > >> (And could it be that we had one in the predecessor 13.1 image?) > > > No, I checked, and it's not in there either... It looks like

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Andres Freund
u mean with "commit LSN4-4000" here. > This makes sense to me, but just to be extra clear, I will never receive a > transaction commit before receiving all other events for that transaction. Correct. Greetings, Andres Freund

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 18:33:37 -0400, Tom Lane wrote: > Andres Freund writes: > > I saw that CI image builds for freebsd were failing, because 13.1, used > > until > > now, is EOL. Update to 13.2, no problem, I thought. Ran a CI run against > > 13.2 > > - unfortuna

Re: CDC/ETL system on top of logical replication with pgoutput, custom client

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 14:16:22 +, José Neves wrote: > Hi Amit, thanks for the reply. > > In our worker (custom pg replication client), we care only about INSERT, > UPDATE, and DELETE operations, which - sure - may be part of the issue. That seems likely. Postgres streams out changes in commit ord

Re: pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
Hi, On 2023-07-31 12:15:10 -0700, Andres Freund wrote: > It sure looks like freebsd 13.2 tcl is just busted. Notably it can't even > parse what it generates: > > echo 'puts [clock scan [clock format [clock seconds] -format "%Y/%m/%d"] > -format "%Y/%m

pltcl tests fail with FreeBSD 13.2

2023-07-31 Thread Andres Freund
;UTC"]'|tclsh8.6 Which, surprise, works. So does specifying the timezone via the TZ='UTC' environment variable. I guess there could be a libc behaviour change or such around timezones? I do see https://www.freebsd.org/releases/13.2R/relnotes/ "tzcode has been upgraded to version 2022g with improved timezone change detection and reliability fixes." Greetings, Andres Freund

Postmaster doesn't correctly handle crashes in PM_STARTUP state

2023-07-29 Thread Andres Freund
x27;t fully dug through the history, this looks to be a quite old problem. Arguably we might also be missing PM_SHUTDOWN_2, but I can't really see a bad consequence of that. Greetings, Andres Freund

Re: Support worker_spi to execute the function dynamically.

2023-07-28 Thread Andres Freund
at in the patch I have posted today for the custom > wait events at [1] and it enforces the startup sequences of the > workers in a stricter way. Is that very meaningful? ISTM the interesting thing to check for would be that the state is idle? Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-07-28 Thread Andres Freund
Hi, On 2023-07-28 16:19:47 -0400, Robert Haas wrote: > On Fri, Jul 28, 2023 at 3:00 PM Peter Geoghegan wrote: > > > Or are we trying to determine how likely > > > the freeze record is to emit an FPI and avoid eager freezing when it > > > isn't worth the cost? > > > > No, that's not something that

Re: Support worker_spi to execute the function dynamically.

2023-07-27 Thread Andres Freund
the problem on master. Note that without the sleep(3) in the test the workers don't actually finish starting, the test shuts down the cluster before that happens... Greetings, Andres Freund diff --git i/src/test/modules/worker_spi/t/001_worker_spi.pl w/src/test/modules/worker_spi/t/001_

Re: Buildfarm animal grassquit is failing

2023-07-25 Thread Andres Freund
e seen them locally. Newer versions of gcc and clang (or libasan?) default to enabling "stack use after return" checks - for some reason that implies using a shadow stack *sometimes*. Which can confuse our stack depth checking terribly, not so surprisingly. I've been meaning to look into what we could do to fix that, but I haven't found the cycles... For now I added :detect_stack_use_after_return=0 to ASAN_OPTIONS, which I think should fix the issue. Greetings, Andres Freund

Re: WAL Insertion Lock Improvements

2023-07-25 Thread Andres Freund
Hi, On 2023-07-26 07:40:31 +0900, Michael Paquier wrote: > On Tue, Jul 25, 2023 at 09:49:01AM -0700, Andres Freund wrote: > > FWIW, I'm working on a patch that replaces WAL insert locks as a whole, > > because they don't scale all that well. > > What were you l

Re: WAL Insertion Lock Improvements

2023-07-25 Thread Andres Freund
e possible that this is AIO specific, but I didn't see anything in stacks to suggest that. I do wonder if this possibly exposed an undocumented prior dependency on the value update always happening under the list lock. Greetings, Andres Freund

Re: Logical walsenders don't process XLOG_CHECKPOINT_SHUTDOWN

2023-07-25 Thread Andres Freund
t there is a variant that seems better: We could just scan the end of the WAL for records that should have been streamed out? Greetings, Andres Freund

Re: WAL Insertion Lock Improvements

2023-07-25 Thread Andres Freund
te of read and/or write operations?). FWIW, I'm working on a patch that replaces WAL insert locks as a whole, because they don't scale all that well. If there's no very clear improvements, I'm not sure it's worth putting too much effort into polishing them all that much. Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-25 Thread Andres Freund
Hi, On 2023-07-25 08:50:19 -0700, Andres Freund wrote: > One idea I had was to add a fastpath that won't parse all strings, but will > parse the strings that we would generate, and fall back to the more general > variant if it fails. See the attached, rough, prototype: > > fi

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-25 Thread Andres Freund
Hi, On 2023-07-25 23:37:08 +1200, David Rowley wrote: > On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > I've not really studied the fix_COPY_DEFAULT.patch patch. Is there a > reason to delay committing that? It would be good to eliminate that > as a variable for the cu

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-24 Thread Andres Freund
) > + /* process hex digits */ > + else if (ptr[1] == 'x' || ptr[1] == 'X') > { > > firstdigit = ptr += 2; I find this unnecessarily hard to read. I realize it's been added in 6fcda9aba83, but I don't see a reason to use such a construct here. I find it somewhat grating how much duplication there now is in this code due to being repeated for all the bases... Greetings, Andres Freund

Re: [BUG] Crash on pgbench initialization.

2023-07-24 Thread Andres Freund
Hi, On 2023-07-24 09:42:44 -0700, Andres Freund wrote: > > I don't know this code at all, but I hope that this can be solved with > > just Anton's proposed patch. > > Yes, it's just that off-by-one. I need to check if there's a similar bug for > lo

Re: [BUG] Crash on pgbench initialization.

2023-07-24 Thread Andres Freund
this can be solved with > just Anton's proposed patch. Yes, it's just that off-by-one. I need to check if there's a similar bug for local / temp table buffers though. Greetings, Andres Freund

Re: Avoid stack frame setup in performance critical routines using tail calls

2023-07-19 Thread Andres Freund
ms 02: 29.934 ms The server was pinned to the one core, turbo mode disabled. That's a pretty nice win, I'd say. And I don't think this is actually the most allocator bound workload, I just tried something fairly random... Greetings, Andres Freund >From 4b97070e32ed323ae0f0

Re: Extension Enhancement: Buffer Invalidation in pg_buffercache

2023-07-18 Thread Andres Freund
re give up. > + */ > + > + if ((buf_state & (BM_DIRTY | BM_VALID)) != (BM_VALID)) > + { > + UnlockBufHdr(bufHdr, buf_state); > + return false; > + } > + > + } > + > + InvalidateBuffer(bufHdr); I'm wary of using InvalidateBuffer() here, it's typically used for different purposes, including throwing valid contents away. That seems a bit scary. I think you should be able to just use InvalidateVictimBuffer()? Greetings, Andres Freund

Re: Atomic ops for unlogged LSN

2023-07-17 Thread Andres Freund
; It's still in needs-review status but there's been a number of > > comments/reviews (drive-by, at least) but without any real ask for any > > changes to be made. > > LGTM Why don't we just use a barrier when around reading the value? It's not like CreateCheckPoint() is frequent? Greetings, Andres Freund

Report distinct wait events when waiting for WAL "operation"

2023-07-17 Thread Andres Freund
lient backend │ (null) │ (null)│ 1 │ └┴─┴───┴───┘ even though they are very different FWIW, the former is bottlenecked by the number of WAL insertion locks, the second is bottlenecked by copying WAL into buffers due to needing to flush them. Greetings, Andres Freund >From 9d

Re: Improve heapgetpage() performance, overhead from serializable

2023-07-17 Thread Andres Freund
>    block, lines, all_visible, check_serializable); I think that makes it less likely that the compiler actually generates a constant-folded version for each of the branches. Perhaps worth some experimentation. Greetings, Andres Freund

Re: Autogenerate some wait events code and documentation

2023-07-16 Thread Andres Freund
ber to chop that off. I don't think we need to be particularly consistent with wait events across major versions. They're necessarily tied to how the code works, and we've yanked that around plenty. Greetings, Andres Freund

Improve heapgetpage() performance, overhead from serializable

2023-07-15 Thread Andres Freund
ageIsAllVisible() in heap_hot_search_buffer() so far? Greetings, Andres Freund >From 92ca2c15ccf2adb9b7f2da9cf86b571534287136 Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Sat, 15 Jul 2023 15:13:30 -0700 Subject: [PATCH v1] optimize heapgetpage() --- src/backend/access/heap/heapam.c

Re: Inefficiency in parallel pg_restore with many tables

2023-07-15 Thread Andres Freund
all array into the big array, once it has gotten large enough that sorting becomes expensive. We could go for a heap of N>2 such arrays, but I doubt it would be worth much. Greetings, Andres Freund

XLogSaveBufferForHint() correctness and more

2023-07-14 Thread Andres Freund
nextval_internal() is also wrong. I think the uses in slot.c, snapbuild.c, rewriteheap.c are fine. Greetings, Andres Freund [1] https://www.postgresql.org/message-id/20230714151626.rhgae7taigk2xrq7%40awork3.anarazel.de

Re: New WAL record to detect the checkpoint redo location

2023-07-14 Thread Andres Freund
I suspect we might be able to get rid of the need for exclusive inserts here. If we rid of that, we could determine the redoa location based on the LSN determined by the XLogInsert(). Alternatively, I think we should split XLogInsertRecord() so that the part with the insertion locks held is a separate function, that we could use here. Greetings, Andres Freund

Re: WAL Insertion Lock Improvements

2023-07-13 Thread Andres Freund
Hi, On 2023-07-13 14:04:31 -0700, Andres Freund wrote: > From b74b6e953cb5a7e7ea1a89719893f6ce9e231bba Mon Sep 17 00:00:00 2001 > From: Bharath Rupireddy > Date: Fri, 19 May 2023 15:00:21 + > Subject: [PATCH v8] Optimize WAL insertion lock acquisition and release > > Thi

Re: WAL Insertion Lock Improvements

2023-07-13 Thread Andres Freund
On 2023-07-11 09:20:45 +0900, Michael Paquier wrote: > On Mon, Jun 05, 2023 at 08:00:00AM +0530, Bharath Rupireddy wrote: > > On Wed, May 31, 2023 at 5:05 PM Michael Paquier wrote: > >> @Andres: Were there any extra tests you wanted to be run for more > >> input? >

Re: DROP DATABASE is interruptible

2023-07-13 Thread Andres Freund
Hi, On 2023-07-12 11:54:18 +0200, Daniel Gustafsson wrote: > > On 12 Jul 2023, at 03:59, Andres Freund wrote: > > On 2023-07-07 14:09:08 +0200, Daniel Gustafsson wrote: > >>> On 25 Jun 2023, at 19:03, Andres Freund wrote: > >>> On 2023-06-21 12:02:04 -0700,

Re: vac_truncate_clog()'s bogus check leads to bogusness

2023-07-13 Thread Andres Freund
On 2023-06-22 22:29:12 -0700, Noah Misch wrote: > On Thu, Jun 22, 2023 at 09:45:18AM -0700, Andres Freund wrote: > > On 2023-06-21 21:50:39 -0700, Noah Misch wrote: > > > On Wed, Jun 21, 2023 at 03:12:08PM -0700, Andres Freund wrote: > > > > When vac

Re: Meson build updates

2023-07-12 Thread Andres Freund
Hi, On 2023-06-29 13:34:42 -0500, Tristan Partin wrote: > On Thu Jun 29, 2023 at 12:35 PM CDT, Andres Freund wrote: > > Hm. One minor disadvantage of this is that if no c++ compiler was found, you > > can't really see anything about llvm in the the output, nor in > > me

Re: Remove distprep

2023-07-12 Thread Andres Freund
; continue > test "$f" = src/test/isolation/specparse.h && continue > > + # FIXME > + test "$f" = src/backend/utils/adt/jsonpath_internal.h && continue > + > # ppport.h is not under our control, so we can't make it standalone. > test "$f" = src/pl/plperl/ppport.h && continue Hm, what's that about? We really ought to replace these scripts with something better, which understands concurrency... Greetings, Andres Freund

Re: Forgive trailing semicolons inside of config files

2023-07-11 Thread Andres Freund
f. Oops. Looks like we could make this easier in core postgres by adding one more sd_notify() call, with something like STATUS=reload failed due to syntax error in file "/srv/dev/pgdev-dev/postgresql.conf" line 821, near end of line Greetings, Andres Freund

Re: DROP DATABASE is interruptible

2023-07-11 Thread Andres Freund
Hi, On 2023-07-07 14:09:08 +0200, Daniel Gustafsson wrote: > > On 25 Jun 2023, at 19:03, Andres Freund wrote: > > On 2023-06-21 12:02:04 -0700, Andres Freund wrote: > >> I'm hacking on this bugfix again, > > This patch LGTM from reading through and testing (m

Re: Support to define custom wait events for extensions

2023-07-11 Thread Andres Freund
x27;postgresql.conf', > + "shared_preload_libraries = 'test_custom_wait_events'" > +); > +$node->start; I think this should also test registering a wait event later. > @@ -0,0 +1,188 @@ > +/*-- > + * > + * test_custom_wait_events.c > + * Code for testing custom wait events > + * > + * This code initializes a custom wait event in shmem_request_hook() and > + * provide a function to launch a background worker waiting forever > + * with the custom wait event. Isn't this vast overkill? Why not just have a simple C function that waits with a custom wait event, until cancelled? That'd maybe 1/10th of the LOC. Greetings, Andres Freund

Re: Cleaning up threading code

2023-07-11 Thread Andres Freund
On 2023-07-12 08:58:29 +1200, Thomas Munro wrote: > On Mon, Jul 10, 2023 at 10:45 AM Thomas Munro wrote: > > * defined ENABLE_THREAD_SAFETY 1 in c.h, for the benefit of extensions > > I may lack imagination but I couldn't think of any use for that > vestigial macro in backend/extension code, and

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-11 Thread Andres Freund
ered=500026107 14550 8644605049434766 HEAD+fixes,buffered=100025875 14505 8200490035653433 HEAD+fixes,buffered=500025830 12975 6527359427392642 Greetings, Andres Freund [1] https://postgr.es/m/CAD21AoAEwHTLYhuQ6PaBRPXKWN-CgW9iw%2B4hm%3D2EOFXbJQ3tOg%40mail.gmail.com

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-11 Thread Andres Freund
Hi, On 2023-07-11 09:09:43 +0200, Jakub Wartak wrote: > On Mon, Jul 10, 2023 at 6:24 PM Andres Freund wrote: > > > > Hi, > > > > On 2023-07-03 11:53:56 +0200, Jakub Wartak wrote: > > > Out of curiosity I've tried and it is reproducible as you have st

Re: Refactoring backend fork+exec code

2023-07-10 Thread Andres Freund
> + } Hm, shouldn't we error out when called without --forkchild? > +/* Save critical backend variables into the BackendParameters struct */ > +#ifndef WIN32 > +static bool > +save_backend_variables(BackendParameters *param, ClientSocket *client_sock) > +#else There's so m

Re: ResourceOwner refactoring

2023-07-10 Thread Andres Freund
Hi, On 2023-07-10 12:14:46 -0700, Andres Freund wrote: > > /* > > - * Initially allocated size of a ResourceArray. Must be power of two since > > - * we'll use (arraysize - 1) as mask for hashing. > > + * Size of the fixed-size array to hold most-

Re: ResourceOwner refactoring

2023-07-10 Thread Andres Freund
owner->releasing = true; Is it a problem than a possible error would lead to ->releasing = true to be "leaked"? I think it might be, because we haven't called ResourceOwnerSort(), which means we'd not process resources in the right order during abort processing. > > +/* > + * Hash function for value+kind combination. > + */ > static inline uint32 > hash_resource_elem(Datum value, ResourceOwnerFuncs *kind) > { > - Datum data[2]; > - > - data[0] = value; > - data[1] = PointerGetDatum(kind); > - > - return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM); > + /* > + * Most resource kinds store a pointer in 'value', and pointers are > unique > + * all on their own. But some resources store plain integers (Files and > + * Buffers as of this writing), so we want to incorporate the 'kind' in > + * the hash too, otherwise those resources will collide a lot. But > + * because there are only a few resource kinds like that - and only a > few > + * resource kinds to begin with - we don't need to work too hard to mix > + * 'kind' into the hash. Just add it with hash_combine(), it perturbs > the > + * result enough for our purposes. > + */ > +#if SIZEOF_DATUM == 8 > + return hash_combine64(murmurhash64((uint64) value), (uint64) kind); > +#else > + return hash_combine(murmurhash32((uint32) value), (uint32) kind); > +#endif > } Why are we using a 64bit hash to then just throw out the high bits? Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-10 Thread Andres Freund
ase, because the table is so narrow that we never end up extending by a sufficient number of blocks in RelationAddBlocks() to reach that path. Yet there's a regression at 1 client. I don't yet have a RHEL8 system at hand, could either of you send the result of a strace -c -p $pid_of_backend_doing_copy Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-10 Thread Andres Freund
x27;t think I can reproduce your problem on that system... I also tried adding a fdatasync() into the loop, but that just made things uniformly slow. I guess I'll try to dig up whether this is a problem in older upstream kernels, or whether it's been introduced in RHEL. Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-10 Thread Andres Freund
. FWIW, I had extensively tested this with XFS, just with a newer kernel. Have you tested this on RHEL9 as well by any chance? Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-10 Thread Andres Freund
to you, because of commit 00d1e02be2. I'll take a look - I wasn't even aware of this thread until now. Greetings, Andres Freund

Re: Autogenerate some wait events code and documentation

2023-07-06 Thread Andres Freund
Hi, On 2023-07-06 09:36:12 +0900, Michael Paquier wrote: > On Wed, Jul 05, 2023 at 02:59:39PM -0700, Andres Freund wrote: > > Rebasing a patch over this I was a bit confused because I got a bunch of > > ""unable to parse wait_event_names.txt" errors. Took me a whi

Re: Fix last unitialized memory warning

2023-07-06 Thread Andres Freund
sn't seem wise to me for the project to basically say that that's not advisable due to the level of warnings created. I've also found bugs with -O3 that -O2 didn't find. And often -O3 warnings end up showing up with -O2 a compiler major version or three down the line, so it's often just deferring work. Greetings, Andres Freund

Re: [PATCH] Add GitLab CI to PostgreSQL

2023-07-06 Thread Andres Freund
ses 2 tmp_install > cube/regress pg_ctl/001_start_stop' > +# Run all tests > +- su postgres -c 'meson test $MTEST_ARGS --num-processes 2' > + artifacts: > +when: always # Must be able to see logs > +paths: > + - build/meson-logs/testlog.txt FWIW, that's not enough to be able to debug problems. You really also need the log files created by failing tests. Greetings, Andres Freund

Re: Avoid overflow with simplehash

2023-07-06 Thread Andres Freund
i; > - uint64 startelem = PG_UINT64_MAX; > + uint32 i; > + uint32 startelem = PG_UINT32_MAX; The startelem type change doesn't strike me as a good idea. Currently PG_UINT32_MAX is a valid element. Greetings, Andres Freund

Re: Add more sanity checks around callers of changeDependencyFor()

2023-07-06 Thread Andres Freund
ly or via search_path), somebody else can create a function there, which might be called from a more privileged context. Obviously permissions limit the likelihood of this being a real issue. I also don't think pg_dump will dump the changed schema, which means a dump/restore leads to a different schema - IMO something to avoid. Greetings, Andres Freund

Re: Avoid overflow with simplehash

2023-07-06 Thread Andres Freund
, That should never be reachable - which the assert tries to ensure. > then will it iterate forwards? No, it'd still iterate backwards, but starting from the wrong place - but there is no correct place to start iterating from if there is no unused element. Greetings, Andres Freund

Re: Avoid overflow with simplehash

2023-07-06 Thread Andres Freund
too-large than SH_MAX_SIZE, but ... > And I agree that declaring "i" as int is wrong. Yea, that's definitely not right, not sure how I ended up with that. Will push a fix. I guess it should be backpatched... Greetings, Andres Freund

Re: Autogenerate some wait events code and documentation

2023-07-05 Thread Andres Freund
n is always the camel-cased version of what follows WAIT_EVENT_. Feels pretty tedious to write that out. Perhaps we should just change the case of the upper-cased names (DSM, SSL, WAL, ...) to follow the other names? Greetings, Andres Freund

Re: Optionally using a better backtrace library?

2023-07-03 Thread Andres Freund
Hi, On 2023-07-03 11:58:25 +0200, Alvaro Herrera wrote: > On 2023-Jul-02, Andres Freund wrote: > > I like that we now have a builtin backtrace ability. Unfortunately I think > > the > > backtraces are often not very useful, because only externally visible > &g

Optionally using a better backtrace library?

2023-07-02 Thread Andres Freund
tforward to add support for other object file and debugging formats. The state I currently have is very hacky, but if there's interest in upstreaming something like this, I could clean it up. Greetings, Andres Freund

Re: Replacing abort() with __builtin_trap()?

2023-07-02 Thread Andres Freund
Hi, On 2023-07-02 13:55:53 -0400, Tom Lane wrote: > Andres Freund writes: > > I'd like to propose that we do a configure test for __builtin_trap() and use > > it, if available, before the abort() in ExceptionalCondition(). Perhaps also > > for PANIC, but it's not a

Replacing abort() with __builtin_trap()?

2023-07-02 Thread Andres Freund
me/andres/src/postgresql/src/backend/postmaster/postmaster.c:1463 #6 0x55e7e80ce2e2 in main (argc=73, argv=0x55e7e9e2f3f0) at ../../../../home/andres/src/postgresql/src/backend/main/main.c:198 Maybe I crash things too often, but I like to not have to deal with 4 pointless frames at the top... Greetings, Andres Freund

<    5   6   7   8   9   10   11   12   13   14   >