Re: Add recovery to pg_control and remove backup_label

2023-11-21 Thread Andres Freund
upported compression methods - that's gona be very rare in practice. > I can't understate how valuable checksums are in finding corruption, > especially in long-lived backups. I agree! But I think we need faster checksum algorithms or a faster implementation of the existing

Re: Add recovery to pg_control and remove backup_label

2023-11-21 Thread Andres Freund
Hi, On 2023-11-21 13:41:15 -0400, David Steele wrote: > On 11/20/23 16:41, Andres Freund wrote: > > > > On 2023-11-20 15:56:19 -0400, David Steele wrote: > > > I understand this is an option -- but does it need to be? What is the > > > benefit of excluding the m

Re: simplehash: SH_OPTIMIZE_REPEAT for optimizing repeated lookups of the same key

2023-11-21 Thread Andres Freund
make it always on - there's plenty cases where it just would make things slower because the hit rate is low. A equal comparison is far from free. I am not quite sure this kind of cache best lives in simplehash - ISTM that quite often it'd be more beneficial to have a cache that you can test more cheaply higher up. Greetings, Andres Freund

Re: Add recovery to pg_control and remove backup_label

2023-11-21 Thread Andres Freund
Hi, On 2023-11-21 07:42:42 -0400, David Steele wrote: > On 11/20/23 19:58, Andres Freund wrote: > > On 2023-11-21 08:52:08 +0900, Michael Paquier wrote: > > > On Mon, Nov 20, 2023 at 12:37:46PM -0800, Andres Freund wrote: > > > > Given that, I wonder if what we

Re: Annoying build warnings from latest Apple toolchain

2023-11-20 Thread Andres Freund
/ in its search path, but those libraries aren't from homebrew, they're referencing macos frameworks. Apparently meson isn't able to understand which files those .pc files link to and gives up on the deduplicating. If I add to the pkg-config search path, e.g. via meson configure -Dpkg_config_path=$OTHER_STUFF:/opt/homebrew/opt/zlib/lib/pkgconfig/:/opt/homebrew/opt/libxml2/lib/pkgconfig/ the warnings about duplicate libraries vanish. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-20 Thread Andres Freund
Hi, On 2023-11-20 08:27:48 +0100, Peter Eisentraut wrote: > On 17.11.23 19:53, Andres Freund wrote: > > I pushed the first two commits (the selinux stuff) and worked a bit more on > > the subsequent ones. > > Patches 0001 through 0004 look good to me. Cool, I pushed them n

Re: Hide exposed impl detail of wchar.c

2023-11-20 Thread Andres Freund
tling this in the past, independent of rust :( What I don't quite get is why SIMD headers are particularly more problematic than others - there's other headers that are compiler specific? Greetings, Andres Freund

Re: Add recovery to pg_control and remove backup_label

2023-11-20 Thread Andres Freund
Hi, On 2023-11-21 08:52:08 +0900, Michael Paquier wrote: > On Mon, Nov 20, 2023 at 12:37:46PM -0800, Andres Freund wrote: > > Given that, I wonder if what we should do is to just add a new field to > > pg_control that says "error out if backup_label does not exist", that

Re: Why is hot_standby_feedback off by default?

2023-11-20 Thread Andres Freund
On 2023-11-20 16:34:47 +0700, John Naylor wrote: > Sounds like a TODO? WFM. I don't personally use or update TODO, as I have my doubts about its usefulness or state of maintenance. But please feel free to add this as a TODO from my end...

Re: Add recovery to pg_control and remove backup_label

2023-11-20 Thread Andres Freund
Hi, On 2023-11-20 14:18:15 -0700, David G. Johnston wrote: > On Mon, Nov 20, 2023 at 1:37 PM Andres Freund wrote: > > > > > Given that, I wonder if what we should do is to just add a new field to > > pg_control that says "error out if backup_label does not exi

Re: PANIC serves too many masters

2023-11-20 Thread Andres Freund
e there aren't many PANICs in extensions). If that weren't the case, something like this could make sense: PANIC: crash-restart CRITICAL: crash-shutdown BUG: crash-restart, abort() Greetings, Andres Freund

Re: Add recovery to pg_control and remove backup_label

2023-11-20 Thread Andres Freund
Hi, On 2023-11-20 15:56:19 -0400, David Steele wrote: > I understand this is an option -- but does it need to be? What is the > benefit of excluding the manifest? It's not free to create the manifest, particularly if checksums are enabled. Also, for external backups, there's no manifest... - An

Re: Add recovery to pg_control and remove backup_label

2023-11-20 Thread Andres Freund
t;error out if backup_label does not exist", that we set when creating a streaming base backup Greetings, Andres Freund

Re: Annoying build warnings from latest Apple toolchain

2023-11-20 Thread Andres Freund
Hi, On 2023-11-20 14:46:13 -0500, Robert Haas wrote: > On Mon, Nov 20, 2023 at 2:35 PM Andres Freund wrote: > > > Is there still outstanding work on this thread? Because I'm just now > > > using a new MacBook (M2, Ventura 13.6.2) and I'm getting a lot of this >

Re: Annoying build warnings from latest Apple toolchain

2023-11-20 Thread Andres Freund
s > kind of thing in a meson build: Ventura? In that case I assume you installed newer developer tools? Because otherwise I think we were talking about issues introduced in Sonoma. Greetings, Andres Freund

Re: Use of backup_label not noted in log

2023-11-20 Thread Andres Freund
om base backup with redo LSN E/AFF07F20, checkpoint LSN E/B01B17F0, on timeline ID 1 LOG: entering standby mode LOG: redo starts at E/AFF07F20 LOG: completed recovery from base backup with redo LSN E/AFF07F20 LOG: consistent recovery state reached at E/B420FC80 Besides the phrasing and the add

Re: Use of backup_label not noted in log

2023-11-20 Thread Andres Freund
Hi, On 2023-11-20 17:30:31 +0900, Michael Paquier wrote: > On Sat, Nov 18, 2023 at 01:49:15PM -0800, Andres Freund wrote: > > Note that the LSN in the "continuing" case is the one the backup started at, > > not where recovery will start. > > > > I've

Re: Use of backup_label not noted in log

2023-11-20 Thread Andres Freund
> internal difference which we could use to drive a similar log message? I think we absolutely have to. If there's no way to tell whether an "external" pg_backup_start/stop() procedure actually used the proper pg_control, it'd make the situation substantially worse compared to today's, already bad, situation. Greetings, Andres Freund

Re: long-standing data loss bug in initial sync of logical replication

2023-11-18 Thread Andres Freund
On 2023-11-19 02:15:33 +0100, Tomas Vondra wrote: > > > On 11/18/23 22:05, Andres Freund wrote: > > Hi, > > > > On 2023-11-18 21:45:35 +0100, Tomas Vondra wrote: > >> On 11/18/23 19:12, Andres Freund wrote: > >>>> If we increase the l

Re: Relation bulk write facility

2023-11-18 Thread Andres Freund
uffered pages to disk. > + */ > +static void > +bulkw_flush(BulkWriteState *bulkw) > +{ > + int nbuffered = bulkw->nbuffered; > + BulkWriteBuffer *buffers = bulkw->buffers; > + > + if (nbuffered == 0) > + return; > + > + if (nbuffered > 1) > + { > + int o; > + > + qsort(buffers, nbuffered, sizeof(BulkWriteBuffer), buffer_cmp); > + > + /* > + * Eliminate duplicates, keeping the last write of each block. > + * (buffer_cmp uses 'order' as the last sort key) > + */ Huh - which use cases would actually cause duplicate writes? Greetings, Andres Freund

reindexing an invalid index should not use ERRCODE_INDEX_CORRUPTED

2023-11-18 Thread Andres Freund
is an interrupted CREATE INDEX CONCURRENTLY, which I don't think can be fairly characterized as "corruption". ISTM something like ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE would be more appropriate? Greetings, Andres Freund

errcode_for_file_access() maps EROFS to INSUFFICIENT_PRIVILEGE

2023-11-18 Thread Andres Freund
be more appropriate than ERRCODE_INSUFFICIENT_PRIVILEGE, but not exactly great. The only real downside would be a slightly odd sqlstate for postmaster's creation of a lock file. If a tablespace were mounted read-only, IO_ERROR actually seems fine. Greetings, Andres Freund

PANIC serves too many masters

2023-11-18 Thread Andres Freund
ld trigger exiting. We do have something like this for the startup process, but only when errors happen "early enough", and without being able to distinguish between "retryable" and "should exit" type errors. But ISTM that that requires adding more and more knowledge to postmaster.c, instead of leaving it with the code that raises the error. Greetings, Andres Freund

Re: Use of backup_label not noted in log

2023-11-18 Thread Andres Freund
Hi, On 2023-11-18 10:01:42 -0800, Andres Freund wrote: > > What about adding it to the "redo starts at" message, something like > > > > redo starts at 12/12345678, taken from control file > > > > or > > > > redo starts at 12/12345678, tak

Re: long-standing data loss bug in initial sync of logical replication

2023-11-18 Thread Andres Freund
Hi, On 2023-11-18 21:45:35 +0100, Tomas Vondra wrote: > On 11/18/23 19:12, Andres Freund wrote: > >> If we increase the locks from ShareUpdateExclusive to ShareRowExclusive, > >> we're making it conflict with RowExclusive. Which is just DML, and I > >> think we

Re: long-standing data loss bug in initial sync of logical replication

2023-11-18 Thread Andres Freund
catalogs, or something like that? (However I hate those > weird differences in behavior, it can easily lead to bugs.) > > But it feels more like a band-aid than actually fixing the issue. Agreed. Greetings, Andres Freund

Re: Use of backup_label not noted in log

2023-11-18 Thread Andres Freund
fy that new standbys and PITRs needs to check that backup_label was actually used, just to remember that there's nothing they could realistically use (using DEBUG1 in production imo isn't realistic). Greetings, Andres Freund

Re: Use of backup_label not noted in log

2023-11-18 Thread Andres Freund
Hi, On 2023-11-17 06:41:46 +0100, Laurenz Albe wrote: > On Thu, 2023-11-16 at 20:18 -0800, Andres Freund wrote: > > I've often had to analyze what caused corruption in PG instances, where the > > symptoms match not having had backup_label in place when bringing on the >

Re: long-standing data loss bug in initial sync of logical replication

2023-11-17 Thread Andres Freund
Hi, On 2023-11-17 17:54:43 -0800, Andres Freund wrote: > On 2023-11-17 15:36:25 +0100, Tomas Vondra wrote: > > Overall, this looks, walks and quacks like a cache invalidation issue, > > likely a missing invalidation somewhere in the ALTER PUBLICATION code. I can confirm that some

Re: long-standing data loss bug in initial sync of logical replication

2023-11-17 Thread Andres Freund
;s odd. There's cases where changing the lock level can cause invalidation processing to happen because there is no pre-existing lock for the "new" lock level, but there was for the old. But OpenTableList() is used when altering the publications, so I don't see how that connects. Greetings, Andres Freund

Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
Hi, On 2023-11-17 16:01:31 -0800, Jeff Davis wrote: > On Fri, 2023-11-17 at 15:27 -0800, Andres Freund wrote: > > At > > first I thought that that's largely because you aren't using > > SH_STORE_HASH. > > I might want to use that in the search_path cache, then

Re: On non-Windows, hard depend on uselocale(3)

2023-11-17 Thread Andres Freund
ls, but functions just call into some thunk that does necessary magic, but that option doesn't exist for variables, so the compiler/linker have to do stuff, hence needing __declspec(dllimport). Greetings, Andres Freund

Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
e_hash("andres.c3") $15 = 3798554170 Fixing that makes simplehash always faster, but still doesn't win on memory usage at the upper end - the two pointers in GUCHashEntry make it too big. I think, independent of this patch, it might be worth requiring that hash table lookups applied the transformation before the lookup. A comparison function this expensive is not great... Greetings, Andres Freund

Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
asters. I doubt it's good idea to use the same code for highly contended, partitioned, shared memory hashtables and many tiny local memory hashtables. The design goals are just very different. Greetings, Andres Freund

Re: Change GUC hashtable to use simplehash?

2023-11-17 Thread Andres Freund
thing to port over - whether it's worth it I don't quite know, but I'd not be against it on principle or such. Greetings, Andres Freund

Re: simplehash: preserve consistency in case of OOM

2023-11-17 Thread Andres Freund
On 2023-11-17 13:00:19 -0800, Jeff Davis wrote: > Please tell me if you think the use of simplehash for a search_path > cache is the wrong tool for the job. No, seems fine. I just was curious - as you said, the older existing users won't ever care about this case.

Re: Lifetime of commit timestamps

2023-11-17 Thread Andres Freund
mechanism to clean up data in order > to avoid creating a new one. I don't really understand this part - independent of the mechanism (i.e. an slru), at some point we need to remove old data, just for space reasons. Greetings, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-17 Thread Andres Freund
Hi, On 2023-11-15 16:21:45 -0500, Melanie Plageman wrote: > On Tue, Nov 14, 2023 at 7:15 PM Andres Freund wrote: > > On 2023-11-14 07:46:10 -0500, Melanie Plageman wrote: > > > > FreeSpaceMapVacuumRange()'s comment says: > > > > * As above, but assume that

Re: ResourceOwner refactoring

2023-11-17 Thread Andres Freund
Hi, On 2023-11-17 12:44:41 -0800, Andres Freund wrote: > On 2023-11-07 13:28:28 +0200, Heikki Linnakangas wrote: > > I feel pretty good about this overall. Barring objections or new cfbot > > failures, I will commit this in the next few days. > > I am working on rebasing th

Re: ResourceOwner refactoring

2023-11-17 Thread Andres Freund
sowner in ResourceOwnerRememberLock(). It's currently not strictly required, but that seems like it's just leaking an implementation detail out? Greetings, Andres Freund

Re: simplehash: preserve consistency in case of OOM

2023-11-17 Thread Andres Freund
S(tb, size); > return tb; > } Maybe add a comment explaining why it's important to update parameters after allocating? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-17 Thread Andres Freund
Hi, On 2023-11-14 16:30:24 -0800, Andres Freund wrote: > On 2023-11-14 16:22:31 -0800, Andres Freund wrote: > > > v2-0004-meson-Add-world-target.patch > > > > > > AFAICT, this world target doesn't include the man target. (Again, this > > > woul

Use of backup_label not noted in log

2023-11-16 Thread Andres Freund
still emit a useful message when starting in a state that's "equivalent" to having used the backup_label. Thoughts? Greetings, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-16 Thread Andres Freund
Hi, On 2023-11-16 15:29:38 -0500, Robert Haas wrote: > On Wed, Nov 15, 2023 at 6:17 PM Andres Freund wrote: > > Thoughts on whether to backpatch? It'd probably be at least a bit painful, > > there have been a lot of changes in the surrounding code in the last 5 > > ye

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-15 Thread Andres Freund
Hi, On 2023-11-15 16:32:48 -0500, Robert Haas wrote: > On Mon, Nov 13, 2023 at 8:26 PM Andres Freund wrote: > > I think this undersells the situation a bit. We right now do > > FreeSpaceMapVacuumRange() for 8GB of data (VACUUM_FSM_EVERY_PAGES) in the > > main >

Re: Some performance degradation in REL_16 vs REL_15

2023-11-15 Thread Andres Freund
flush those pages eventually, which may contend for IO and increases page faults. Rebuilding an optimized build generates something like 1GB of dirty data. Particularly with ccache, that'll typically not yet be flushed by the time you run a benchmark. That's not nothing, even with a decent NVMe SSD. - many more, unfortunately Greetings, Andres Freund

Re: Some performance degradation in REL_16 vs REL_15

2023-11-15 Thread Andres Freund
inish (you're clearly not using a very fast disk) the CPU would have plenty time to clock down. Benchmarking sucks. Modern hardware realities suck. Greetings, Andres Freund

Re: remaining sql/json patches

2023-11-15 Thread Andres Freund
On 2023-11-15 09:11:19 -0800, Andres Freund wrote: > On 2023-11-15 22:00:41 +0900, Amit Langote wrote: > > > This causes a nontrivial increase in the size of the parser (~5% in an > > > optimized build here), I wonder if we can do better. > > > > Hmm, sorry if

Re: remaining sql/json patches

2023-11-15 Thread Andres Freund
;t much slowdown in the > raw-parsing speed. What does "isn't much slowdown" mean in numbers? Greetings, Andres Freund

Potential use-after-free in partion related code

2023-11-15 Thread Andres Freund
x27;t quickly enough identify what partConstraint, defPartConstraint, partBoundConstraint are, so I don't don't really know what the fix here is. Greetings, Andres Freund

Re: Explicitly skip TAP tests under Meson if disabled

2023-11-15 Thread Andres Freund
On 2023-11-15 11:02:19 +0100, Peter Eisentraut wrote: > On 04.11.23 01:51, Andres Freund wrote: > > I'd just use a single test() invocation here, and add an argument to > > testwrap > > indicating that it should print out the skipped message. That way we a) &

Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 14:42:13 +0200, Alexander Korotkov wrote: > It's possibly dumb option, but what about just removing the assert? That's not at all an option - the in-place bms_* functions can free their input. So a dangling pointer to the "old" version is a use-after-free waiting to happen - you

Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 19:14:57 +0800, Richard Guo wrote: > While working on BUG #18187 [1], I noticed that we also have issues with > how SJE replaces join clauses involving the removed rel. As an example, > consider the query below, which would trigger an Assert. > > create table t (a int primary ke

Re: typo in fallback implementation for pg_atomic_test_set_flag()

2023-11-14 Thread Andres Freund
some faster atomic-exchange > implementations that may activate this code for certain > compiler/architecture combinations. Hm. I don't really see how adding a faster atomic-exchange implementation could trigger this implementation being used? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 16:22:31 -0800, Andres Freund wrote: > > v2-0004-meson-Add-world-target.patch > > > > AFAICT, this world target doesn't include the man target. (Again, this > > would all work better if we added "man" to "docs".) > > I

Re: meson documentation build open issues

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 11:58:53 +0100, Peter Eisentraut wrote: > On 09.11.23 18:52, Andres Freund wrote: > > I think if we want to do this, we'd need a configure time check for being > > able > > to validate a document with > > > V4.5//EN""http://ww

Re: meson documentation build open issues

2023-11-14 Thread Andres Freund
know > how that affects Debian packaging, but this patch might not be the right > one. I'm not really seeing why that'd affect pkglibdir being adjustable, besides needing to tweak the description of pkglibdir? > I suggest we leave this patch for a separate discussion. Fair enough. Thanks for the review, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-14 Thread Andres Freund
ockBufferForCleanup() locks the buffer conditionally, *before* even looking at the refcount and returns false if not. And writing out a buffer takes a content lock. Which made me realize that "tuples missed: 5848 dead from 89 pages not removed due to cleanup lock contention" is often ki

Re: pgsql: doc: fix wording describing the checkpoint_flush_after GUC

2023-11-14 Thread Andres Freund
Hi, On 2023-11-14 17:49:59 +0100, Alvaro Herrera wrote: > On 2023-Nov-13, Andres Freund wrote: > > On 2023-11-13 12:31:42 +0100, Alvaro Herrera wrote: > > > On 2023-Nov-09, Bruce Momjian wrote: > > > > > > > doc: fix wording describing the checkpoint_flush_

Re: building with meson on windows with ssl

2023-11-13 Thread Andres Freund
st that? The problem could be related to the fact that on windows you (I think) can have binaries with both 32bit and 64bit libraries loaded. Was this with msvc or gcc/mingw? Greetings, Andres Freund

Re: mxid_age() and age(xid) appear undocumented

2023-11-13 Thread Andres Freund
ething in this area, we'd > definitely need to provide for the needs of those monitoring queries I > mentioned. I think it'd be a bad idea to "deprecate" existing working queries, with the replacement being a more complicated way to represent the same information. Greetings, Andres Freund

Re: lazy_scan_heap() should release lock on buffer before vacuuming FSM

2023-11-13 Thread Andres Freund
───┴┴───┴┴─┴───┴─┴─┘ =# DELETE FROM copytest_0; =# VACUUM (VERBOSE) copytest_0; ... INFO: 0: table "copytest_0": truncated 146264 to 110934 pages ... tuples missed: 5848 dead from 89 pages not removed due to cleanup lock contention ... A bit of debugging later I figured out that this is due to the background writer. If I SIGSTOP bgwriter, the whole relation is truncated... Greetings, Andres Freund

Re: pgsql: doc: fix wording describing the checkpoint_flush_after GUC

2023-11-13 Thread Andres Freund
k after a configurable number of bytes. Otherwise, these [...] > I'm not suggesting to revert this change, but rather I'd like to prevent > future changes of this type. Just saying it'd be sad to turn all the > Postgres documentation to using Basic English or whatever. +1 for the general notion. Greetings, Andres Freund

Re: Requiring recovery.signal or standby.signal when recovering with a backup_label

2023-11-13 Thread Andres Freund
Hi, On 2023-11-14 09:13:44 +0900, Michael Paquier wrote: > On Mon, Nov 13, 2023 at 03:41:44PM -0800, Andres Freund wrote: > > On 2023-11-09 12:16:52 +0900, Michael Paquier wrote: > >> On Thu, Nov 09, 2023 at 12:04:19PM +0900, Michael Paquier wrote: > >> > Sure, sorr

Re: [PATCH] pgbench log file headers

2023-11-13 Thread Andres Freund
gether are a wall-clock timestamp. Without differentiating between those, the column headers seem not very useful, because one needs to look in the documentation to understand the fields. I don't think there's all that strong a need for backward compatibility in pgbench. We could just change the columns as I suggest above and then always emit the header in the "main" log file. Greetings, Andres Freund

Re: Requiring recovery.signal or standby.signal when recovering with a backup_label

2023-11-13 Thread Andres Freund
in the label > file because we are missing some stuff in pg_wal/, which is something > that the code is currently able to handle. "able to handle" as in detect and error out? Because that's the only possible sane thing to do, correct? Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-13 Thread Andres Freund
Hi, On 2023-11-14 00:02:13 +0100, Tomas Vondra wrote: > On 11/13/23 23:12, Andres Freund wrote: > > On 2023-11-13 22:36:24 +0100, Tomas Vondra wrote: > >> ISTM it's about how complex the rules implemented by the collation are, > >> so I agree the cost shoul

Re: archive modules loose ends

2023-11-13 Thread Andres Freund
xlog, pathname); > + > + /* Remove our exception handler */ > + PG_exception_stack = NULL; > + > + /* Reset our memory context and switch back to the original one > */ > + MemoryContextSwitchTo(oldcontext); > + MemoryContextReset(archive_context); > + } It could be worth setting up an errcontext providing the module and file that's being processed. I personally find that at least as important as setting up a ps string detailing the log file... But I guess that could be a separate patch. It'd be nice to add a comment explaining why pgarch_archiveXlog() is the right place to handle errors. Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-13 Thread Andres Freund
Even if we initially perhaps just get the > > default cost from the provider such, it structurally seems the sanest place > > to > > locate the cost. > > > > ISTM it's about how complex the rules implemented by the collation are, > so I agree the cost should be a feature of collations not providers. I'm not sure analysing the complexity in detail is worth it. ISTM there's a few "levels" of costliness: 1) memcmp() suffices 2) can safely use strxfrm() (i.e. ICU), possibly limited to when we sort 3) deterministic collations 4) non-deterministic collations I'm sure there are graduations, particularly within 3), but I'm not sure it's realistic / worthwhile to go to that detail. I think a cost model like the above would provide enough detail to make better decisions than today... Greetings, Andres Freund

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread Andres Freund
Hi, On 2023-11-13 14:37:47 -0600, David Christensen wrote: > On Mon, Nov 13, 2023 at 2:27 PM Andres Freund wrote: > > On 2023-11-08 18:47:56 -0800, Peter Geoghegan wrote: > > > On Wed, Nov 8, 2023 at 6:04 AM Stephen Frost wrote: > > > > In conversations with folk

Re: Split index and table statistics into different types of stats

2023-11-13 Thread Andres Freund
gt; + /* links to other structs for same relation: */ > + struct PgStat_IndexXactStatus *upper; /* next higher subxact if any */ > + PgStat_IndexStatus *parent; /* per-table status */ > + /* structs of same subxact level are linked here: */ > + struct PgStat_IndexXactStatus *next;/* next of same subxact */ > +} PgStat_IndexXactStatus; I don't think much of this is used? It doesn't look like you're using most of the fields. Which makes sense - there's not really the same transactional behaviour for indexes as there is for tables. Greetings, Andres Freund

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread Andres Freund
e. Storing redundant data in each page header, when we've wanted space in the page header for plenty other things, just doesn't seem a good use of said space. Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-13 Thread Andres Freund
Hi, On 2023-11-11 23:19:55 -0800, Jeff Davis wrote: > On Fri, 2023-11-10 at 17:19 -0800, Andres Freund wrote: > > I guess you are arguing that the user didn't intend to create an > > index here? > > No, obviously the user should expect an index when a primary key is &

Re: pg_walfile_name_offset can return inconsistent values

2023-11-13 Thread Andres Freund
17 | 0 | > 00010017 > + 0/1701 | 00010017 | 1 | > 00010017 > +(3 rows) These would break when testing with a different segment size. Today that's not the case... Greetings, Andres Freund

Re: autovectorize page checksum code included elsewhere

2023-11-11 Thread Andres Freund
apsed 1431%CPU (0avgtext+0avgdata 278660maxresident)k 82624inputs+4571480outputs (571major+14004898minor)pagefaults 0swaps Greetings, Andres Freund

Re: pgsql: Don't trust unvalidated xl_tot_len.

2023-11-11 Thread Andres Freund
, '<', "$stdout/$header_path" or die "$!"; +my $fname = "$stdout/$header_path"; +open my $header_h, '<', "$fname" or die "could not open $fname: $!"; my @match = undef; while (<$header_h>) would be helpful. I guess we should also apply something like that to our tree - printing a stringified errno without even a path/filename isn't very useful. Greetings, Andres Freund

Re: pg_walfile_name_offset can return inconsistent values

2023-11-10 Thread Andres Freund
_lsn + :segment_size + 1), pg_split_walfile_name(file_name); SELECT segment_number, file_offset = :segment_size - 1 FROM pg_walfile_name_offset('0/0'::pg_lsn + :segment_size - 1), pg_split_walfile_name(file_name); Greetings, Andres Freund

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 20:38:13 -0600, Nathan Bossart wrote: > On Fri, Nov 10, 2023 at 03:11:50PM -0800, Andres Freund wrote: > > On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: > >> + * This read is guaranteed to read the current value, > > > > It doesn'

Re: Adding facility for injection points (or probe points?) for more advanced tests

2023-11-10 Thread Andres Freund
d, then perhaps wait inside the "modified" code, and do something else in the test script. But as-is a decent amount of C code would need to be written to write such a test, from what I can tell? Greetings, Andres Freund

Re: Why do indexes and sorts use the database collation?

2023-11-10 Thread Andres Freund
reasons other than the user's direct request (e.g. DISTINCT/GROUP BY, merge joins). I think we should also explain in our docs that C can be considerably faster - I couldn't find anything in a quick look. Greetings, Andres Freund

Re: Force the old transactions logs cleanup even if checkpoint is skipped

2023-11-10 Thread Andres Freund
f space. And if that doesn't happen - do we really want to wait till a checkpoint finishes to free up space? What if we instead made archiver delete WAL files after archiving, if they're old enough? Some care would be needed to avoid checkpointer and archiver trampling on each other, but that doesn't seem too hard. Greetings, Andres Freund

Re: ResourceOwner refactoring

2023-11-10 Thread Andres Freund
acility. When you make > allocations with dsa_allocate() or just call dsa_get_address() on an > existing dsa_pointer, you wouldn't expect the current resource owner to > matter. I think dsa_create/attach() should store the current resource owner > in the dsa_area, for use in subsequent operations on the DSA, per attached > patch (0002-Fix-dsa.c-with-different-resource-owners.patch). Yea, that seems the right direction from here. Greetings, Andres Freund

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
not* tied to individual addresses. And the fence specification is just about unreadable (although I think they did fix some of the worst issues). Greetings, Andres Freund

Re: locked reads for atomics

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 14:51:28 -0600, Nathan Bossart wrote: > Moving this to a new thread and adding it to the January commitfest. > > On Thu, Nov 09, 2023 at 03:27:33PM -0600, Nathan Bossart wrote: > > On Tue, Nov 07, 2023 at 04:58:16PM -0800, Andres Freund wrote: > >>

Re: Failure during Building Postgres in Windows with Meson

2023-11-10 Thread Andres Freund
Hi, On November 10, 2023 10:53:12 AM PST, Tristan Partin wrote: >On Thu Nov 9, 2023 at 9:31 AM CST, Nazir Bilal Yavuz wrote: >> Hi, >> >> On Thu, 9 Nov 2023 at 18:27, Tristan Partin wrote: >> > >> > Can you try with Meson v1.2.3? >> >> I tried with Meson v1.2.3 and upstream, both failed with

Re: AdvanceXLInsertBuffers() vs wal_sync_method=open_datasync

2023-11-10 Thread Andres Freund
Hi, On 2023-11-10 17:16:35 +0200, Heikki Linnakangas wrote: > On 10/11/2023 05:54, Andres Freund wrote: > > In this case I had used wal_sync_method=open_datasync - it's often faster > > and > > if we want to scale WAL writes more we'll have to use it more wide

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-09 Thread Andres Freund
Hi, On November 8, 2023 11:28:08 PM PST, Michael Paquier wrote: >On Thu, Nov 09, 2023 at 01:50:34PM +0900, torikoshia wrote: >> PGSTAT_KIND_SLRU cannot be reset by pg_stat_reset_shared(), so I feel >> uncomfortable to delete it all together. >> It might be better after pg_stat_reset_shared() ha

AdvanceXLInsertBuffers() vs wal_sync_method=open_datasync

2023-11-09 Thread Andres Freund
early all pending WAL in XLogFlush(), I'm not sure there's much point in not also being somewhat aggressive in GetXLogBuffer()->AdvanceXLInsertBuffer(). Greetings, Andres Freund

Re: Failure during Building Postgres in Windows with Meson

2023-11-09 Thread Andres Freund
Hi, On 2023-11-10 08:17:48 +0530, vignesh C wrote: > On Thu, 9 Nov 2023 at 21:12, Tristan Partin wrote: > > > > On Thu Nov 9, 2023 at 9:31 AM CST, Nazir Bilal Yavuz wrote: > > > Hi, > > > > > > On Thu, 9 Nov 2023 at 18:27, Tristan Partin wrote: > > > > > > > > Can you try with Meson v1.2.3? > >

Re: pg_upgrade failing for 200+ million Large Objects

2023-11-09 Thread Andres Freund
Hi, On November 9, 2023 10:41:01 AM PST, Tom Lane wrote: >Also, pg_upgrade is often invoked indirectly via scripts, so I do >not especially buy the idea that we're going to get useful control >input from some human somewhere. I think we'd be better off to >assume that pg_upgrade is on its own t

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-11-09 Thread Andres Freund
f the page header is already valid, I don't think it's ok to just read/parse WAL data that's concurrently being modified. We can never allow WAL being read that's past XLogBytePosToRecPtr(XLogCtl->Insert->CurrBytePos) as it does not exist. And if the to-be-read LSN is between XLogCtl->LogwrtResult->Write and XLogBytePosToRecPtr(Insert->CurrBytePos) we need to call WaitXLogInsertionsToFinish() before copying the data. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-09 Thread Andres Freund
Hi, On 2023-11-09 15:32:39 +0100, Peter Eisentraut wrote: > On 09.11.23 01:59, Andres Freund wrote: > > > I think we could build doc/src/sgml/postgres-full.xml by default. That > > > takes less than 0.5 seconds here and it's an intermediate target for html > >

Re: make pg_ctl more friendly

2023-11-08 Thread Andres Freund
Hi, On 2023-11-09 09:29:32 +0800, Crisp Lee wrote: > How to judge from 'DB_SHUTDOWNED' that PITR ends normally? 'DB_SHUTDOWNED' > is just a state, it could not give more meaning, so I reuse the > recovery.done. DB_SHUTDOWNED cannot be encountered while recovery is ongoing. If there was a hard cra

Re: Add new option 'all' to pg_stat_reset_shared()

2023-11-08 Thread Andres Freund
e time the individual kind of stats is reset. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
we want to build it by default, otherwise we'll cause errors on plenty systems that don't get them today. The docbook dts aren't a huge dependency, but still. Some OSs might not have a particularly install source for them, e.g. windows. I don't think that'd detect the missing ids? Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
too long, we could still offer > alternatives that exclude some areas. If it's the other way round, > some areas will never be checked widely. The 'test' target (generated by meson, otherwise I'd have named it check), runs all enabled tests. You obviously can run a subset if you so desire. Greetings, Andres Freund

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-11-08 Thread Andres Freund
Hi, On 2023-11-08 19:00:01 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-11-08 13:18:39 -0500, Tom Lane wrote: > >> I think an actually usable feature of this sort would involve > >> copying all the failed lines to some alternate output medium, > &g

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2023-11-08 Thread Andres Freund
to handle failing rows and also to abort if the number of errors or the type of errors gets to be too big. Greetings, Andres Freund

Re: meson documentation build open issues

2023-11-08 Thread Andres Freund
rter than in the preceding commit. But I don't think that hurts much. Greetings, Andres Freund >From 9b0b5cd952880ecebbd157c05698125755bc53ed Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Wed, 8 Nov 2023 09:29:38 -0800 Subject: [PATCH v2 1/7] meson: Change default of 'selinux&#x

Re: Syncrep and improving latency due to WAL throttling

2023-11-08 Thread Andres Freund
eceive the data, wake up a user space process, send back 100 bytes, wait for the data be transmitted, and then wake up a process, there are periods where the link in one direction is largely idle. But in case of a 10kB packet on the 1gbit network, yes, we are bandwidth limited for ~80us (or perhaps more interestingly, we are bandwidth limited for 0.8ms when sending 100kB). Greetings, Andres Freund

<    2   3   4   5   6   7   8   9   10   11   >