Re: Exit walsender before confirming remote flush in logical replication

2023-02-06 Thread Andres Freund
Hi, On 2023-02-06 12:23:54 +0530, Amit Kapila wrote: > On Mon, Feb 6, 2023 at 10:33 AM Andres Freund wrote: > > Smart shutdown is practically unusable. I don't think it makes sense to tie > > behavior of walsender to it in any way. > > > > So, we have the follow

Re: SLRUs in the main buffer pool - Page Header definitions

2023-02-06 Thread Andres Freund
l APIs across the tree. Why is that required for the topic at hand? Why is it worth doing that, given it'll break loads of extensions? Greetings, Andres Freund

Re: [PATCH] Compression dictionaries for JSONB

2023-02-06 Thread Andres Freund
ble and a > > dictionary for this table that you refresh from time to time. Very > > simple. > > You cannot "just" refresh a dictionary used once to compress an > object, because you need it to decompress the object too. Right. That's what I was trying to refer to wh

Re: Pluggable toaster

2023-02-06 Thread Andres Freund
think we should accept such hooks. I don't think I am alone in that. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-02-06 Thread Andres Freund
chine is now on run ~2700 without any further lockups :-/ > > Seems it was quite lucky we hit it twice in a handful of attempts. Did you cut down the workload before you reproduced it the first time, or after? It's quite possible that it's not reproducible in isolation. Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-02-06 Thread Andres Freund
Hi, On 2023-02-06 14:07:39 -0500, Robert Haas wrote: > On Fri, Feb 3, 2023 at 3:47 AM Andres Freund wrote: > > On 2023-02-02 09:28:03 -0500, Robert Haas wrote: > > > I don't know what you mean by this. DML doesn't confer privileges. If > > > code gets executed an

Re: undersized unions

2023-02-06 Thread Andres Freund
arrays allow the compiler to understand the variable size, but we don't use it for all variability. Hence the warnings. Greetings, Andres Freund

Re: undersized unions

2023-02-06 Thread Andres Freund
Hi On 2023-02-06 11:42:57 -0500, Robert Haas wrote: > On Sun, Feb 5, 2023 at 6:28 AM Andres Freund wrote: > > On the other hand, it also just seems risky from a code writing > > perspective. It's not immediate obvious that it'd be unsafe to create an > > on-stack Nume

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-02-05 Thread Andres Freund
Hi, On February 5, 2023 9:12:17 PM PST, Tom Lane wrote: >Damir Belyalov writes: >>> I don't think this is the right approach. Creating a subtransaction for >>> each row will cause substantial performance issues. > >> Subtransactions aren't created for each row. The block of rows in one >>

Re: Exit walsender before confirming remote flush in logical replication

2023-02-05 Thread Andres Freund
Hi, On February 5, 2023 8:29:19 PM PST, Amit Kapila wrote: >On Sat, Feb 4, 2023 at 6:31 PM Andres Freund wrote: >> >> On 2023-02-02 11:21:54 +0530, Amit Kapila wrote: >> > The main problem we want to solve here is to avoid shutdown failing in >> > case walrec

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-02-05 Thread Andres Freund
Hi, On 2023-02-01 14:20:19 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-02-01 09:49:00 -0800, Andres Freund wrote: > >> On 2023-02-01 12:23:27 -0500, Tom Lane wrote: > >>> And the minimum version appears to be newer than RHEL8's 1.8.2, which &

Re: Weird failure with latches in curculio on v15

2023-02-05 Thread Andres Freund
nce, centrally. > I mean, we could introduce a GUC that tells us how far ahead to > restore and have a background worker (or multiple background workers) > asynchronously pull files into a staging directory via the callbacks. > Is that the sort of scope you are envisioning? Closer, at least. Greetings, Andres Freund

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-02-05 Thread Andres Freund
Safe(). Use that to avoid throwing an error instead of catching it. Greetings, Andres Freund

Re: Temporary tables versus wraparound... again

2023-02-05 Thread Andres Freund
b4520992c5ed7a3136c7103b7a668b591542bba58a32f5a84cb21bbeeafea09dc525d1631a5f413a0f98df43cc90ebf6c4206e6df61606bc634c3a8116c53d7c6dd4bc5b26547cd7d1a1633839ace694b73426267a9f434317350905b905b9c88132be14a7762c2f204b8072a3bd7e4e1d30217d9e60102d525b08e28bcfaabae80fba734a1015d8eb0a7": http2: request body larger than specified content length Hm, I suspect the problem is that we didn't shut down the server due to the error, so the log file was changing while cirrus was trying to upload. Greetings, Andres Freund

Re: Pluggable toaster

2023-02-05 Thread Andres Freund
he whole Pluggable TOAST API to make it > available as an extension and based on hooks to minimize changes in > the core. It will be available soon. I don't think we should accept that either. It still doesn't improve anything about toast, it just allows you to do such improvements out of core. Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-05 Thread Andres Freund
nflict, would make sense. For the segment files, we'd likely need a parameter to indicate whether the restore is random or not. Greetings, Andres Freund

Re: [PATCH] Compression dictionaries for JSONB

2023-02-05 Thread Andres Freund
Hi, On 2023-02-05 20:05:51 +0300, Aleksander Alekseev wrote: > > I don't think we'd want much of the infrastructure introduced in the > > patch for type agnostic cross-row compression. A dedicated "dictionary" > > type as a wrapper around other types IMO is the wrong direction. This > > should be

Re: File descriptors in exec'd subprocesses

2023-02-05 Thread Andres Freund
Hi, On 2023-02-05 11:06:13 -0500, Tom Lane wrote: > Andres Freund writes: > > On February 5, 2023 1:00:50 AM GMT+01:00, Thomas Munro > > wrote: > >> Are there any more descriptors we need to think about? > > > Postmaster's listen sockets? > > I

Re: [PATCH] Compression dictionaries for JSONB

2023-02-05 Thread Andres Freund
ely ignore it. Especially > considering the fact that the implementation was proven to be fairly > simple and there was even no need to rebase the patch since November > :) I don't think a prototype-y patch not needing a rebase two months is a good measure of complexity :) Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-05 Thread Andres Freund
Hi, On 2023-02-04 10:03:54 -0800, Nathan Bossart wrote: > On Sat, Feb 04, 2023 at 03:30:29AM -0800, Andres Freund wrote: > > That's kind of my problem with these changes. They try to introduce new > > abstraction layers, but don't provide real abstraction, because they're > &g

Re: File descriptors in exec'd subprocesses

2023-02-05 Thread Andres Freund
Hi, Unsurprisingly I'm in favor of this. On February 5, 2023 1:00:50 AM GMT+01:00, Thomas Munro wrote: >Are there any more descriptors we need to think about? Postmaster's listen sockets? Saves a bunch of syscalls, at least. Logging collector pipe write end, in backends? Greetings,

Re: undersized unions

2023-02-05 Thread Andres Freund
Hi, On 2023-02-05 10:18:14 +0900, Michael Paquier wrote: > On Sat, Feb 04, 2023 at 05:07:08AM -0800, Andres Freund wrote: > > : In function 'assign': > > :9:6: warning: array subscript 'foo[0]' is partly outside array > > bounds of 'unsigned char[4]' [-Warray-bounds=] >

Re: undersized unions

2023-02-05 Thread Andres Freund
Hi, On February 5, 2023 6:16:55 AM GMT+01:00, Tom Lane wrote: >Michael Paquier writes: >> On Sat, Feb 04, 2023 at 05:07:08AM -0800, Andres Freund wrote: >>> We actually have a fair amount of code like that, but currently are >>> escaping most of the warning

Re: [PATCH] Compression dictionaries for JSONB

2023-02-04 Thread Andres Freund
Hi, On 2023-02-03 14:39:31 +0400, Pavel Borisov wrote: > On Fri, 3 Feb 2023 at 14:04, Alvaro Herrera wrote: > > > > This patch came up at the developer meeting in Brussels yesterday. > > https://wiki.postgresql.org/wiki/FOSDEM/PGDay_2023_Developer_Meeting#v16_Patch_Triage > > > > First, as far

undersized unions

2023-02-04 Thread Andres Freund
actually undefined behaviour to ever dereference a Numeric pointer, when the pointer points to something smaller than sizeof(NumericData). Greetings, Andres Freund

Re: Exit walsender before confirming remote flush in logical replication

2023-02-04 Thread Andres Freund
pointer all can continue to write WAL. Without being able to check that the last LSN has indeed been sent out, how do you know that you didn't miss something? Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-02-04 Thread Andres Freund
than perltidy. At the same time, I think the issue of C code in HEAD not being indented is more pressing - IME it's much more common to have to touch a lot of C code than to have to touch a lot fo perl files. So perhaps we should just start with being more stringent with C code, and once we made perltidy less noisy, add that? Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-04 Thread Andres Freund
ts that it is only ever to be called in the startup process. And outside of the startup process none of the above actually makes sense. That's kind of my problem with these changes. They try to introduce new abstraction layers, but don't provide real abstraction, because they're very tightly bound to the way the functions were called before the refactoring. And none of these restrictions are actually documented. Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-04 Thread Andres Freund
ProcPid == (int) getpid()) > + proc_exit(1); > + else > + _exit(1); I think it might be worth adding something like const char msg[] = "StartupProcShutdownHandler() called in child process"; write(STDERR_FILENO, msg, sizeof(msg)); to this path. Otherwise it might end up being a very hard to debug path. Greetings, Andres Freund

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-02-04 Thread Andres Freund
Hi, Heikki, Andrey, CCing you because you wrote commit 6655a7299d835dea9e8e0ba69cc5284611b96f29 Author: Heikki Linnakangas Date: 2019-07-24 20:24:07 +0300 Use full 64-bit XID for checking if a deleted GiST page is old enough. On 2023-01-07 19:09:56 -0800, Andres Freund wrote: >

Re: Weird failure with latches in curculio on v15

2023-02-03 Thread Andres Freund
Hi, On February 3, 2023 9:19:23 AM GMT+01:00, Thomas Munro wrote: >On Fri, Feb 3, 2023 at 9:09 PM Andres Freund wrote: >> Thinking about popen() suggests that we have a similar problem with COPY >> FROM PROGRAM as we have in pgarch (i.e. not as bad as the startup >> pro

Re: Non-superuser subscription owners

2023-02-03 Thread Andres Freund
s influenced to some degree by the amount of privileges that user has. Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-03 Thread Andres Freund
Hi, On 2023-02-03 02:50:38 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-02-03 02:24:03 -0500, Tom Lane wrote: > >> setsid(2) is required since SUSv2, so I'm not sure which systems > >> are of concern here ... other than Redmond's of course. > >

Re: Weird failure with latches in curculio on v15

2023-02-02 Thread Andres Freund
en the child is > terminated, but without running the handler. Isn't that what we want > here? Yep, I think so. Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-02 Thread Andres Freund
Hi, On 2023-02-03 02:24:03 -0500, Tom Lane wrote: > Andres Freund writes: > > Ugh, I think I might understand what's happening: > > > The signal arrives just after the fork() (within system()). Because we > > have all our processes configure themselves as process g

Re: Weird failure with latches in curculio on v15

2023-02-02 Thread Andres Freund
sid(), there's no guarantee that the executed process reacts to SIGTERM and that the child didn't create its own process group (and thus isn't reached by the signal to the process group, sent in signal_child()). Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-02 Thread Andres Freund
Hi, On 2023-02-02 10:23:21 +0900, Michael Paquier wrote: > On Wed, Feb 01, 2023 at 10:18:27AM -0800, Andres Freund wrote: > > On 2023-02-01 12:27:19 -0500, Tom Lane wrote: > >> Andres Freund writes: > >> The main thing that system() brings to the table is platf

Re: Use windows VMs instead of windows containers on the CI

2023-02-02 Thread Andres Freund
Hi, On 2023-02-02 17:47:37 +0300, Nazir Bilal Yavuz wrote: > On 1/12/2023 3:30 AM, Andres Freund wrote: > > Yea, there's a problem where packer on windows doesn't seem to abort after a > > powershell script error out. The reason isn't yet quiete clear. I think > >

Re: Error "initial slot snapshot too large" in create replication slot

2023-02-02 Thread Andres Freund
backpatch the change. Greetings, Andres Freund

Re: XMAX_LOCK_ONLY and XMAX_COMMITTED (fk/multixact code)

2023-02-02 Thread Andres Freund
pgraded from pre-v9.3 versions. Given that fact, that aspect at least seems to be not viable? Greetings, Andres Freund

Re: recovery modules

2023-02-01 Thread Andres Freund
I'd always pass ArchiveModuleState *state as the first arg, but it's not important. Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-02-01 Thread Andres Freund
a password."), > + errhint("Target server's authentication method > must be changed. or set password_required=false in the subscription > attributes\ ."))); > + > if (logical) > { > PGresult *res; This still leaks the connection on error, no? Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 09:43:39 -0500, Robert Haas wrote: > On Tue, Jan 31, 2023 at 7:01 PM Andres Freund wrote: > > I don't really understand that - the run-as approach seems like a > > necessary piece of improving the security model. > > > > I think it's perfectly reas

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 09:49:00 -0800, Andres Freund wrote: > On 2023-02-01 12:23:27 -0500, Tom Lane wrote: > > And the minimum version appears to be newer than RHEL8's 1.8.2, which > > I find pretty unfortunate. On RHEL8, it fails with > > $ ninja > > ninja: error: build.n

Re: Weird failure with latches in curculio on v15

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 12:27:19 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-02-01 12:08:24 -0500, Robert Haas wrote: > >> I like the idea of not relying on system(). In most respects, doing > >> fork() + exec() ourselves seems superior. We can control where

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 12:23:27 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-02-01 13:05:32 +0300, Aleksander Alekseev wrote: > >> It works. Perhaps we should add: > >> ninja -C build alldocs > > > FWIW, just 'docs' would build just the multi-pa

Re: Weird failure with latches in curculio on v15

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 12:08:24 -0500, Robert Haas wrote: > On Wed, Feb 1, 2023 at 11:58 AM Andres Freund wrote: > > > 9a740f81e clearly made things a lot worse, but it wasn't great > > > before. Can we see a way forward to removing the problem entirely? > > > > Yea

Re: Weird failure with latches in curculio on v15

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 10:12:26 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-02-01 16:21:16 +1300, Thomas Munro wrote: > >> It's always in proc_exit() in StartupProcShutdownHandler(), a SIGTERM > >> handler which is allowed to call that while in_restore_comm

Re: recovery modules

2023-02-01 Thread Andres Freund
rivate_data } that's passed to basic_archive_startup() and all the other callbacks. Greetings, Andres Freund

Re: Weird failure with latches in curculio on v15

2023-02-01 Thread Andres Freund
the recent changes. They need a fairly large rewrite. Greetings, Andres Freund

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-02-01 Thread Andres Freund
`` > ninja -C build alldocs > ``` FWIW, just 'docs' would build just the multi-page html/man pages, alldocs takes a lot longer... And yes, adding that to the docs is a good idea. Greetings, Andres Freund

Re: meson: pkgconfig difference

2023-01-31 Thread Andres Freund
Hi, On January 31, 2023 11:40:52 PM PST, Peter Eisentraut wrote: >I think there is a tiny typo in src/interfaces/ecpg/ecpglib/meson.build: > >diff --git a/src/interfaces/ecpg/ecpglib/meson.build >b/src/interfaces/ecpg/ecpglib/meson.build >index dba9e3c3d9..da8d304f54 100644 >---

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-31 Thread Andres Freund
Hi, On 2023-01-30 12:00:55 -0800, Nathan Bossart wrote: > On Mon, Jan 30, 2023 at 11:49:37AM -0800, Andres Freund wrote: > > Why don't the dblink tests catch this? Any chance you or Robins could > > prepare > > a patch with fix and test, given that you know how to tr

Re: Weird failure with latches in curculio on v15

2023-01-31 Thread Andres Freund
es. > The information available in standby2.log shows that 30681 is the > startup process. I am not sure what all that means, yet. > > Thoughts or comments welcome. Perhaps a wild write overwriting shared memory state? Greetings, Andres Freund

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-31 Thread Andres Freund
I wonder if we should provide a build target to download the stylesheets ourselves. The amount of packages our instructions download is quite substantial. We could perhaps trim them a bit, but we intentionally are including things to build pdfs etc as well, which does make sense... Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-01-31 Thread Andres Freund
X etc). > And if we suppose that that already works and is safe, well then > what's the case where I do need a run-as user? It's not at all safe today, IMO. You need to trust that nothing bad will be replicated, otherwise the owner of the subscription has to be considered compromised. Greetings, Andres Freund

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-31 Thread Andres Freund
Hi, On 2023-01-31 15:05:17 +0100, Matthias van de Meent wrote: > On Mon, 30 Jan 2023 at 21:19, Andres Freund wrote: > > In an earlier, not posted, version I had an vacuum_defer_cleanup_age > > specific > > helper function for this, but it seems likely we'll need it in othe

Re: pg_upgrade test failure

2023-01-31 Thread Andres Freund
Hi, On January 31, 2023 12:54:42 PM PST, Thomas Munro wrote: >On Wed, Feb 1, 2023 at 6:28 AM Justin Pryzby wrote: >> > I pushed the rmtree() change. Let's see if that helps, or tells us >> > something new. >> >> I found a few failures since then: >> >>

Re: Making background psql nicer to use in tap tests

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 15:06:46 -0500, Tom Lane wrote: > Andres Freund writes: > > It's annoyingly hard to wait for the result of a query in a generic way with > > background_psql(), and more generally for psql. background_psql() uses > > -XAtq, > > which means that we

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-30 Thread Andres Freund
Hi, On 2023-01-10 21:32:54 +0100, Matthias van de Meent wrote: > On Tue, 10 Jan 2023 at 20:14, Andres Freund wrote: > > On 2023-01-10 15:03:42 +0100, Matthias van de Meent wrote: > > What precisely do you mean with "skew" here? Do you just mean that it'd >

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 11:30:08 -0800, Nathan Bossart wrote: > On Mon, Jan 23, 2023 at 07:28:06PM -0800, Andres Freund wrote: > > After a tiny bit further polishing, and after separately pushing a resource > > leak fix for walrcv_connect(), I pushed this. > > My colleague

Re: recovery modules

2023-01-30 Thread Andres Freund
ment? Here is the idea, simply: > typedef struct ArchiveModule { > ArchiveCallbacks *routines; > void *private_data; > /* Potentially more here, like some flags? */ > } ArchiveModule; I don't like this much. This still basically ends up with the module callbacks not being sufficient to instantiate an archive module. Greetings, Andres Freund

Making background psql nicer to use in tap tests

2023-01-30 Thread Andres Freund
Hi, Plenty tap tests require a background psql. But they're pretty annoying to use. I think the biggest improvement would be an easy way to run a single query and get the result of that query. Manually having to pump_until() is awkward and often leads to hangs/timeouts, instead of test failures,

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-30 Thread Andres Freund
; I wondered if there was interest in knowing if the tuple had to get > TOASTed, and further wondered if we would be interested in the number of > updates that had to wait on a lock. Again, more counters. Those seem a lot less actionable / related to the topic at hand to me. Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-30 Thread Andres Freund
propriate version. Won't even be used on HEAD, but before that it can make a difference. Depending on how you built clang, it could be that the above recipe ends up using the system lld, which might be too old. What are the crashes you're getting? Greetings, Andres Freund

Re: meson: Optionally disable installation of test modules

2023-01-30 Thread Andres Freund
quite likely to result in plenty people not running the tests. Sending a link instead of attaching, in case you already registered a cfbot entry: https://github.com/anarazel/postgres/commit/d1d192a860da39af9aa63d7edf643eed07c4 Probably worth adding an install-test-modules target for manual use. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-29 Thread Andres Freund
own latch (a second time) when someone sets our latch in > releases 9.something to 13. But this part does indeed put a crimp on some potential theories. TBH, I'd be in favor of just adding the barriers for good measure, even if we don't know if it's a live bug today - it seems incredibly fragile. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-29 Thread Andres Freund
can amount to "probabilistic barriers" because that'll be a barrier. At the same time it can make it more likely that the relatively narrow window in WaitEventSetWait() is hit, or lead to larger delays processing signals. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-29 Thread Andres Freund
need a barrier in SetLatch(), after is_set = true. We have that in some of the newer branches (due to the maybe_sleeping logic), but not in the older branches. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
828.1646000688%40sss.pgh.pa.us Such problems are quite annoying. Last time I hit such a case was https://postgr.es/m/20220325052654.3xpbmntatyofau2w%40alap3.anarazel.de but I can't see anything like that being the issue here. Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
t in 2pc, and I'm not sure ProcArrayLock is > held). Maybe there are other edge cases. Unless I am missing something, that would be very bad [TM], completely independent of txid_status(). The underlying functions like TransactionIdIsInProgress() are used for MVCC. Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 19:49:17 -0800, Andres Freund wrote: > It's quite commonly used as part of trigger based replication tools (IIRC > that's its origin), monitoring, as part of client side logging, as part of > snapshot management. Forgot one: Queues. The way it's used for trig

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 22:39:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: > >> FWIW, my vote goes for a more expensive but reliable function even in > >> stable branches. > > > I very strenuously objec

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
order model than x86. IIUC elver previously hit this on 12? Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
table branches. I very strenuously object. If we make txid_current() (by way of pg_current_xact_id()) flush WAL, we'll cause outages. Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
t's a good idea to make a function that was quite cheap for 15 years, suddenly be several orders of magnitude more expensive... Greetings, Andres Freund

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
out of page, and how often it causes potential HOT updates to go out of page. If you just have a single index, it's not too hard, but after that seems decidedly nontrival. But I might just be missing what you're suggesting. Greetings, Andres Freund

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 20:39:04 -0500, Tom Lane wrote: > Andres Freund writes: > > Tom, is there a reason we run the various sanity tests early-ish in the > > schedule? It does seem to reduce their effectiveness a bit... > > Originally, those tests were mainly needed to sa

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Andres Freund
atch to nuke all global variables[1], Hah. > but of course we should at least drop the now redundant use of volatile and > sigatomic_t. See attached. +1 Greetings, Andres Freund

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: > >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, > >> +

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
N ('S', 'v', 'f', 'c', 'p') and > +relam != 0; > + oid | relname | relkind | relam > +-+-+-+--- > (0 rows) Particularly because you include them again here :) Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 18:28:16 -0500, Tom Lane wrote: > Andres Freund writes: > > I'm sure we have a few places that aren't that careful, but I would hope > > it's > > not a large number. Are you thinking of specific "patterns" we've repeated > > all >

Re: recovery modules

2023-01-27 Thread Andres Freund
ckMutuallyExclusiveGUCs(restoreLibrary, "restore_library", > +recoveryEndCommand, > "recovery_end_command"); > + before_shmem_exit(call_recovery_module_shutdown_cb, 0); > + LoadRecoveryCallbacks(); This kind of sequence is duplicated into several places. Greetings, Andres Freund

Re: recovery modules

2023-01-27 Thread Andres Freund
ustness. But given that it's pretty much required for performance that the module runs in the startup process, that ship probably has sailed. Greetings, Andres Freund

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
nt that the old page could use prune/defrag */ > PageSetFull(page); > } > @@ -3875,7 +3877,7 @@ l2: >*/ > PageSetPrunable(page, xid); > > - if (use_hot_update) > + if (update_type == PGSTAT_HEAPUPDATE_HOT) It's a bit weird that heap_update() uses a pgstat type to decide what to do. But not sure there's a much better alternative. Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
. Are you thinking of specific "patterns" we've repeated all over, or just a few cases you recall? Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
s. IIRC it was >15% gain in concurrent readonly pgbench. I dimly recall failing to get some benefit out of bolt for some reason that I unfortunately don't even vaguely recall. Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
155656e1b7a8d26715bb94. To really fix that, we'd need an > analogue of RangeVarGetRelidExtended. Yea, we really should have something like RangeVarGetRelidExtended() for other kinds of objects. It'd take a fair bit of work / time to use it widely, but it'll take even longer if we start in 5 yea

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
ps you that much. If there's e.g. a huge VACUUM / COPY emitting loads of WAL you'll suddenly see commit latency of a concurrently committing transactions spike into oblivion. Whereas a general WAL throttling mechanism would throttle the VACUUM, without impacting the commit latency of normal transactions. Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
.com. I think something narrower and easier to achieve is a good thing. We've already had loads of discussion for the more general problem, without a lot of actual progress. Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:06:49 +0100, Jakub Wartak wrote: > On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: > > > Huh? Why did you remove the GUC? > > After reading previous threads, my optimism level of getting it ever > in shape of being widely accepted degraded sign

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
tion will raise > + * an error and will not return. If it can, it will return true if this > + * connection string specifies a password and false otherwise. > */ > -static void > +static bool > libpqrcv_check_conninfo(const char *conninfo) That is a somewhat odd API. Why does it throw for some things, but not others? Seems a bit cleaner to pass in a parameter indicating whether it should throw when not finding a password? Particularly because you already pass that to walrcv_connect(). Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:53:58 -0500, Robert Haas wrote: > On Fri, Jan 27, 2023 at 12:58 AM Andres Freund wrote: > > Essentially the "any fpi" logic is a very coarse grained way of using the > > page > > LSN as a measurement. As I said, I don't think "has a ch

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 00:51:59 -0800, Andres Freund wrote: > One use-case where the eager strategy is particularly useful is > [nearly-]append-only tables - and it's also the one workload that's reasonably > easy to detect using stats. Maybe something like > (dead_tuples_since

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
still haven't changed the fact that you get rather different > behavior with checksums/wal_log_hints. I think that that's good, but > you didn't seem to. I think that, if we had something like the recency test I was talking about, we could afford to alway freeze when the page is already dirty and not very recently modified. I.e. not even insist on a WAL record having been generated during pruning/HTSV. But I need to think through the dangers of that more. Greetings, Andres Freund

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-26 Thread Andres Freund
ed for correctness too. */ if (!XLogIsNeeded() && !AmWalReceiverProcess()) o_direct_flag = PG_O_DIRECT; Even if that weren't the case, splitting up bigger commits in incrementally committable chunks is a good idea. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 19:01:03 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 6:37 PM Andres Freund wrote: > > I also don't really see how that is responsive to anything else in my > > email. That's just as true for the current gating condition (the issuance of &g

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 15:36:52 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 12:45 PM Andres Freund wrote: > > > Most of the overhead of FREEZE WAL records (with freeze plan > > > deduplication and page-level freezing in) is generic WAL record header > > > ove

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
ess than more recent data. But that's hard/expensive to collect. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
reducing the > WAL overhead of aggressive and more opportunistic freezing. Yep. I think we should also seriously consider setting all-visible during on-access pruning, and freezing rows during on-access pruning. Greetings, Andres Freund

<    12   13   14   15   16   17   18   19   20   21   >