Re: Making background psql nicer to use in tap tests

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 15:06:46 -0500, Tom Lane wrote: > Andres Freund writes: > > It's annoyingly hard to wait for the result of a query in a generic way with > > background_psql(), and more generally for psql. background_psql() uses > > -XAtq, > > which means that we

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-30 Thread Andres Freund
Hi, On 2023-01-10 21:32:54 +0100, Matthias van de Meent wrote: > On Tue, 10 Jan 2023 at 20:14, Andres Freund wrote: > > On 2023-01-10 15:03:42 +0100, Matthias van de Meent wrote: > > What precisely do you mean with "skew" here? Do you just mean that it'd >

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 11:30:08 -0800, Nathan Bossart wrote: > On Mon, Jan 23, 2023 at 07:28:06PM -0800, Andres Freund wrote: > > After a tiny bit further polishing, and after separately pushing a resource > > leak fix for walrcv_connect(), I pushed this. > > My colleague

Re: recovery modules

2023-01-30 Thread Andres Freund
ment? Here is the idea, simply: > typedef struct ArchiveModule { > ArchiveCallbacks *routines; > void *private_data; > /* Potentially more here, like some flags? */ > } ArchiveModule; I don't like this much. This still basically ends up with the module callbacks not being sufficient to instantiate an archive module. Greetings, Andres Freund

Making background psql nicer to use in tap tests

2023-01-30 Thread Andres Freund
Hi, Plenty tap tests require a background psql. But they're pretty annoying to use. I think the biggest improvement would be an easy way to run a single query and get the result of that query. Manually having to pump_until() is awkward and often leads to hangs/timeouts, instead of test failures,

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-30 Thread Andres Freund
; I wondered if there was interest in knowing if the tuple had to get > TOASTed, and further wondered if we would be interested in the number of > updates that had to wait on a lock. Again, more counters. Those seem a lot less actionable / related to the topic at hand to me. Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-30 Thread Andres Freund
propriate version. Won't even be used on HEAD, but before that it can make a difference. Depending on how you built clang, it could be that the above recipe ends up using the system lld, which might be too old. What are the crashes you're getting? Greetings, Andres Freund

Re: meson: Optionally disable installation of test modules

2023-01-30 Thread Andres Freund
quite likely to result in plenty people not running the tests. Sending a link instead of attaching, in case you already registered a cfbot entry: https://github.com/anarazel/postgres/commit/d1d192a860da39af9aa63d7edf643eed07c4 Probably worth adding an install-test-modules target for manual use. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-29 Thread Andres Freund
own latch (a second time) when someone sets our latch in > releases 9.something to 13. But this part does indeed put a crimp on some potential theories. TBH, I'd be in favor of just adding the barriers for good measure, even if we don't know if it's a live bug today - it seems incredibly fragile. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-29 Thread Andres Freund
can amount to "probabilistic barriers" because that'll be a barrier. At the same time it can make it more likely that the relatively narrow window in WaitEventSetWait() is hit, or lead to larger delays processing signals. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-29 Thread Andres Freund
need a barrier in SetLatch(), after is_set = true. We have that in some of the newer branches (due to the maybe_sleeping logic), but not in the older branches. Greetings, Andres Freund

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
828.1646000688%40sss.pgh.pa.us Such problems are quite annoying. Last time I hit such a case was https://postgr.es/m/20220325052654.3xpbmntatyofau2w%40alap3.anarazel.de but I can't see anything like that being the issue here. Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
t in 2pc, and I'm not sure ProcArrayLock is > held). Maybe there are other edge cases. Unless I am missing something, that would be very bad [TM], completely independent of txid_status(). The underlying functions like TransactionIdIsInProgress() are used for MVCC. Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 19:49:17 -0800, Andres Freund wrote: > It's quite commonly used as part of trigger based replication tools (IIRC > that's its origin), monitoring, as part of client side logging, as part of > snapshot management. Forgot one: Queues. The way it's used for trig

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 22:39:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: > >> FWIW, my vote goes for a more expensive but reliable function even in > >> stable branches. > > > I very strenuously objec

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
order model than x86. IIUC elver previously hit this on 12? Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
table branches. I very strenuously object. If we make txid_current() (by way of pg_current_xact_id()) flush WAL, we'll cause outages. Greetings, Andres Freund

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
t's a good idea to make a function that was quite cheap for 15 years, suddenly be several orders of magnitude more expensive... Greetings, Andres Freund

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
out of page, and how often it causes potential HOT updates to go out of page. If you just have a single index, it's not too hard, but after that seems decidedly nontrival. But I might just be missing what you're suggesting. Greetings, Andres Freund

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 20:39:04 -0500, Tom Lane wrote: > Andres Freund writes: > > Tom, is there a reason we run the various sanity tests early-ish in the > > schedule? It does seem to reduce their effectiveness a bit... > > Originally, those tests were mainly needed to sa

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Andres Freund
atch to nuke all global variables[1], Hah. > but of course we should at least drop the now redundant use of volatile and > sigatomic_t. See attached. +1 Greetings, Andres Freund

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: > >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, > >> +

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
N ('S', 'v', 'f', 'c', 'p') and > +relam != 0; > + oid | relname | relkind | relam > +-+-+-+--- > (0 rows) Particularly because you include them again here :) Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 18:28:16 -0500, Tom Lane wrote: > Andres Freund writes: > > I'm sure we have a few places that aren't that careful, but I would hope > > it's > > not a large number. Are you thinking of specific "patterns" we've repeated > > all >

Re: recovery modules

2023-01-27 Thread Andres Freund
ckMutuallyExclusiveGUCs(restoreLibrary, "restore_library", > +recoveryEndCommand, > "recovery_end_command"); > + before_shmem_exit(call_recovery_module_shutdown_cb, 0); > + LoadRecoveryCallbacks(); This kind of sequence is duplicated into several places. Greetings, Andres Freund

Re: recovery modules

2023-01-27 Thread Andres Freund
ustness. But given that it's pretty much required for performance that the module runs in the startup process, that ship probably has sailed. Greetings, Andres Freund

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
nt that the old page could use prune/defrag */ > PageSetFull(page); > } > @@ -3875,7 +3877,7 @@ l2: >*/ > PageSetPrunable(page, xid); > > - if (use_hot_update) > + if (update_type == PGSTAT_HEAPUPDATE_HOT) It's a bit weird that heap_update() uses a pgstat type to decide what to do. But not sure there's a much better alternative. Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
. Are you thinking of specific "patterns" we've repeated all over, or just a few cases you recall? Greetings, Andres Freund

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
s. IIRC it was >15% gain in concurrent readonly pgbench. I dimly recall failing to get some benefit out of bolt for some reason that I unfortunately don't even vaguely recall. Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
155656e1b7a8d26715bb94. To really fix that, we'd need an > analogue of RangeVarGetRelidExtended. Yea, we really should have something like RangeVarGetRelidExtended() for other kinds of objects. It'd take a fair bit of work / time to use it widely, but it'll take even longer if we start in 5 yea

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
ps you that much. If there's e.g. a huge VACUUM / COPY emitting loads of WAL you'll suddenly see commit latency of a concurrently committing transactions spike into oblivion. Whereas a general WAL throttling mechanism would throttle the VACUUM, without impacting the commit latency of normal transactions. Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
.com. I think something narrower and easier to achieve is a good thing. We've already had loads of discussion for the more general problem, without a lot of actual progress. Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:06:49 +0100, Jakub Wartak wrote: > On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: > > > Huh? Why did you remove the GUC? > > After reading previous threads, my optimism level of getting it ever > in shape of being widely accepted degraded sign

Re: Non-superuser subscription owners

2023-01-27 Thread Andres Freund
tion will raise > + * an error and will not return. If it can, it will return true if this > + * connection string specifies a password and false otherwise. > */ > -static void > +static bool > libpqrcv_check_conninfo(const char *conninfo) That is a somewhat odd API. Why does it throw for some things, but not others? Seems a bit cleaner to pass in a parameter indicating whether it should throw when not finding a password? Particularly because you already pass that to walrcv_connect(). Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 12:53:58 -0500, Robert Haas wrote: > On Fri, Jan 27, 2023 at 12:58 AM Andres Freund wrote: > > Essentially the "any fpi" logic is a very coarse grained way of using the > > page > > LSN as a measurement. As I said, I don't think "has a ch

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 00:51:59 -0800, Andres Freund wrote: > One use-case where the eager strategy is particularly useful is > [nearly-]append-only tables - and it's also the one workload that's reasonably > easy to detect using stats. Maybe something like > (dead_tuples_since

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-27 Thread Andres Freund
still haven't changed the fact that you get rather different > behavior with checksums/wal_log_hints. I think that that's good, but > you didn't seem to. I think that, if we had something like the recency test I was talking about, we could afford to alway freeze when the page is already dirty and not very recently modified. I.e. not even insist on a WAL record having been generated during pruning/HTSV. But I need to think through the dangers of that more. Greetings, Andres Freund

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-26 Thread Andres Freund
ed for correctness too. */ if (!XLogIsNeeded() && !AmWalReceiverProcess()) o_direct_flag = PG_O_DIRECT; Even if that weren't the case, splitting up bigger commits in incrementally committable chunks is a good idea. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 19:01:03 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 6:37 PM Andres Freund wrote: > > I also don't really see how that is responsive to anything else in my > > email. That's just as true for the current gating condition (the issuance of &g

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 15:36:52 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 12:45 PM Andres Freund wrote: > > > Most of the overhead of FREEZE WAL records (with freeze plan > > > deduplication and page-level freezing in) is generic WAL record header > > > ove

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
ess than more recent data. But that's hard/expensive to collect. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
reducing the > WAL overhead of aggressive and more opportunistic freezing. Yep. I think we should also seriously consider setting all-visible during on-access pruning, and freezing rows during on-access pruning. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 10:44:45 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 9:53 AM Andres Freund wrote: > > > That's going to be very significantly more aggressive. For example > > > it'll impact small tables very differently. > > > > Maybe it would be t

Re: Minimal logical decoding on standbys

2023-01-26 Thread Andres Freund
dead, if you declare the cursor before the DELETE). So you'd need overlapping cursors in a concurrent session... Too complicated. Greetings, Andres Freund

Re: meson oddities

2023-01-26 Thread Andres Freund
On 2023-01-26 10:20:58 +0100, Peter Eisentraut wrote: > On 19.01.23 21:45, Andres Freund wrote: > > Hi, > > > > On 2023-01-19 21:37:15 +0100, Peter Eisentraut wrote: > > > On 11.01.23 12:05, Peter Eisentraut wrote: > > > > I think there is also

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 08:54:55 -0800, Peter Geoghegan wrote: > On Thu, Jan 26, 2023 at 8:35 AM Andres Freund wrote: > > I think it's probably ok, but perhaps deserves a bit more thought about when > > to "opportunistically" freeze. Perhaps to make it *more* aggressi

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-26 Thread Andres Freund
Hi, On 2023-01-26 09:20:57 -0500, Robert Haas wrote: > On Wed, Jan 25, 2023 at 10:56 PM Andres Freund wrote: > > but that's only true because page level freezing neutered > > vacuum_freeze_min_age. Compared to <16, it's a *huge* change. > > Do you think

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Andres Freund
> wal_distance > wal_throttle_threshold && replication_lag > > wal_throttle_replication_lag_threshold. I think my idea of only forcing to flush/wait an LSN some distance in the past would automatically achieve that? Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Andres Freund
atile sig_atomic_t LogMemoryContextPending = false; > volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false; > +volatile sig_atomic_t XLogDelayPending = false; > volatile uint32 InterruptHoldoffCount = 0; > volatile uint32 QueryCancelHoldoffCount = 0; > volatile uint32 CritSectionCount = 0; I don't think the new variable needs to be volatile, or even sig_atomic_t. We'll just manipulate it from outside signal handlers. Greetings, Andres Freund

Re: Syncrep and improving latency due to WAL throttling

2023-01-26 Thread Andres Freund
gt; up with when you have a large number of sessions. But if you'd limit the > > flushing to be done to synchronous_commit_flush_wal_after / 2 boundaries, > > and > > only ever to a prior boundary, the amount of unnecessary WAL flushes would > > be > > proportional to synchronous_commit_flush_wal_after. > > > > True, that's kinda what I suggested above as a solution for partially > filled WAL pages. I think flushing only up to a point further in the past than the last page boundary is somewhat important to prevent unnecessary flushes. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hi, On 2023-01-25 18:43:10 -0800, Peter Geoghegan wrote: > On Wed, Jan 25, 2023 at 6:33 PM Andres Freund wrote: > > As far as I can tell, with the eager strategy, the only thing > > vacuum_freeze_min_age really influences is whether we'll block waiting for a > > clean

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hk, On 2023-01-25 18:31:16 -0800, Peter Geoghegan wrote: > On Wed, Jan 25, 2023 at 5:49 PM Andres Freund wrote: > > Sure. But significantly regressing plausible if not common workloads is > > different than knowing that there'll be some edge case where we'll do > > somethi

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hi, On 2023-01-25 17:28:48 -0800, Peter Geoghegan wrote: > On Wed, Jan 25, 2023 at 5:15 PM Andres Freund wrote: > > However, it significantly increases the overall work when rows have a > > somewhat > > limited lifetime. The documented reason why vacuum_freeze_min_age e

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hi, On 2023-01-25 17:37:17 -0800, Peter Geoghegan wrote: > On Wed, Jan 25, 2023 at 5:26 PM Andres Freund wrote: > > Another bad scenario: Some longrunning / hung transaction caused us to get > > close to the xid wraparound. Problem was resolved, autovacuum runs. > > Previ

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hi, On 2023-01-25 17:22:32 -0800, Peter Geoghegan wrote: > On Wed, Jan 25, 2023 at 4:43 PM Andres Freund wrote: > > I unfortunately haven't been able to keep up with the thread and saw this > > just > > now. But I've expressed the concern below several times before, so

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hi, On 2023-01-25 16:43:47 -0800, Andres Freund wrote: > I think, as committed, this will cause serious issues for some reasonably > common workloads, due to substantially increased WAL traffic. > > > The most common problematic scenario I see are tables full of rows with >

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
pages. That's only true because vacuum_freeze_min_age being has been fairly radically redefined recently. Greetings, Andres Freund

Re: New strategies for freezing, advancing relfrozenxid early

2023-01-25 Thread Andres Freund
Hi, On 2023-01-24 14:49:38 -0800, Peter Geoghegan wrote: > On Mon, Jan 16, 2023 at 5:55 PM Peter Geoghegan wrote: > > 0001 (the freezing strategies patch) is now committable IMV. Or at > > least will be once I polish the docs a bit more. I plan on committing > > 0001 some time next week, barring

Re: [PATCH] Make ON CONFLICT DO NOTHING and ON CONFLICT DO UPDATE consistent

2023-01-25 Thread Andres Freund
urrent DO NOTHING behavior but rather makes DO UPDATE work the same > way DO NOTHING does. I see that now - I somehow thought you were recommending to error out in both cases, rather than the other way round. Greetings, Andres Freund

Re: Set arbitrary GUC options during initdb

2023-01-25 Thread Andres Freund
because just about > every other initdb switch has a long form. The server itself > doesn't have that spelling, so I'm not wedded to that part. Fine with me, but also fine to leave out. Greetings, Andres Freund

Re: pgsql: Rename contrib module basic_archive to basic_wal_module

2023-01-25 Thread Andres Freund
k it'd be just fine to keep the name as basic_archive and use it for both archiving and restoring. Restoring from an archive still deals with archiving. I agree that basic_wal_module isn't a good name. Greetings, Andres Freund

Re: plpython vs _POSIX_C_SOURCE

2023-01-25 Thread Andres Freund
ut afterward that this > caused some subtle and horrible problem for which we had no > back-patchable fix, but that seems pretty unlikely. Agreed. Greetings, Andres Freund

Re: Improve WALRead() to suck data directly from WAL buffers when possible

2023-01-25 Thread Andres Freund
Hi, On 2023-01-14 12:34:03 -0800, Andres Freund wrote: > On 2023-01-14 00:48:52 -0800, Jeff Davis wrote: > > On Mon, 2022-12-26 at 14:20 +0530, Bharath Rupireddy wrote: > > > Please review the attached v2 patch further. > > > > I'm still unclear on the performa

Re: Syncrep and improving latency due to WAL throttling

2023-01-25 Thread Andres Freund
end-local accounting mechanism, I think that's inevitably what you'd end up with when you have a large number of sessions. But if you'd limit the flushing to be done to synchronous_commit_flush_wal_after / 2 boundaries, and only ever to a prior boundary, the amount of unnecessary WAL flushes would be proportional to synchronous_commit_flush_wal_after. Greetings, Andres Freund

Re: [PATCH] Make ON CONFLICT DO NOTHING and ON CONFLICT DO UPDATE consistent

2023-01-25 Thread Andres Freund
likely that a behavioural change will cause trouble for some of the uses of DO NOTHING out there. Greetings, Andres Freund

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-25 Thread Andres Freund
d from unordered indexes for building pathkeys.) +1 Certainly seems confusing to me. Greetings, Andres Freund

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-25 Thread Andres Freund
Hi, On 2023-01-24 20:59:04 -0800, Andres Freund wrote: > I find this to be awkward code. The booleans are kinda pointless, and the > tableagevac case is hard to follow because trigger is set elsewhere. > > I can give reformulating it a go. Need to make some food first. Here's a dra

Re: plpython vs _POSIX_C_SOURCE

2023-01-25 Thread Andres Freund
Hi, On 2023-01-24 23:37:44 -0500, Tom Lane wrote: > Andres Freund writes: > > Patches attached. > > +1 for 0001. Cool, will push tomorrow. > I'm still nervous about 0002. However, maybe the cases that we had trouble > with are legacy issues that nobody cares about any

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-24 Thread Andres Freund
Hi, On 2023-01-23 19:22:18 -0800, Peter Geoghegan wrote: > On Mon, Jan 23, 2023 at 6:56 PM Andres Freund wrote: > > Why is there TABLE_ in AUTOVACUUM_TABLE_XID_AGE but not > > AUTOVACUUM_DEAD_TUPLES? Both are on tables. > > Why does vacuum_freeze_table_age contain th

Re: plpython vs _POSIX_C_SOURCE

2023-01-24 Thread Andres Freund
Hi, On 2023-01-24 18:32:46 -0800, Andres Freund wrote: > > I ran out of energy to test on aix with xlc, I spent way more time on this > > than I have already. I'll pick it up later. > > Building in the background now. Also passes. Thus I think getting rid of the #undefine

Re: plpython vs _POSIX_C_SOURCE

2023-01-24 Thread Andres Freund
Hi, On 2023-01-24 17:48:56 -0800, Andres Freund wrote: > Also for autoconf, I needed to link > $prefix/lib/python3.11/config-3.11/libpython3.11.a > to > $prefix/lib/libpython3.11.a > That might be a python version difference or be related to building python > with --enable-shar

Re: plpython vs _POSIX_C_SOURCE

2023-01-24 Thread Andres Freund
Hi, On 2023-01-24 16:16:06 -0500, Tom Lane wrote: > Andres Freund writes: > > Python's _POSIX_C_SOURCE value is set to a specific value in their configure > > script: > > > if test $define_xopen_source = yes > > then > > ... > > AC_DEFINE(_POSIX_C_SO

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2023-01-24 Thread Andres Freund
ot;bytes hadled at an operation". Can't it > be op_blocksize or just block_size? > > + b.io_object, > + b.io_context, No, block wouldn't be helpful - we'd like to use this for something that isn't uniform blocks. Greetings, Andres Freund

Re: plpython vs _POSIX_C_SOURCE

2023-01-24 Thread Andres Freund
Hi, On 2023-01-24 12:55:15 -0500, Tom Lane wrote: > Andres Freund writes: > > The background for the undefines is that _POSIX_C_SOURCE needs to be defined > > the same for the whole compilation, not change in the middle, and Python.h > > defines it. To protect "our&quo

plpython vs _POSIX_C_SOURCE

2023-01-24 Thread Andres Freund
ised? That's from d8f75d41315 in 2006. python scribbling over our macros: /* * Sometimes python carefully scribbles on our *printf macros. * So we undefine them here and redefine them after it's done its dirty deed. I didn't find code in recent-ish python to do that. Perhaps we should try to

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-23 Thread Andres Freund
Hi, On 2023-01-20 19:00:08 -0800, Andres Freund wrote: > Updated patch attached. I split it into multiple pieces. > 1) A fix for [1], included here because I encountered it while testing > 2) Introduction of libpq-be-fe-helpers.h > 3) Convert dblink and postgres_fdw to the helper

Test failures of 100_bugs.pl

2023-01-23 Thread Andres Freund
for_subscription_sync() should dump the set of rel states to make something like this more debuggable? The fact that the synchronization for t finished just before the refresh makes me wonder if a wakeup or a cache invalidation got lost? Greetings, Andres Freund

Re: libpqrcv_connect() leaks PGconn

2023-01-23 Thread Andres Freund
On 2023-01-21 23:14:08 -0800, Noah Misch wrote: > On Sat, Jan 21, 2023 at 12:04:53PM -0800, Andres Freund wrote: > > On 2023-01-21 08:16:42 -0800, Noah Misch wrote: > > > On Fri, Jan 20, 2023 at 06:50:37PM -0800, Andres Freund wrote: > > > > I seems we don

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-23 Thread Andres Freund
; Other cases are reported as autovacuum trigger when the tuple insert > thresholds or the dead tuple thresholds were crossed. > > Author: Peter Geoghegan > Reviewed-By: Andres Freund > Reviewed-By: Jeff Davis > Discussion: > https://postgr.es/m/CAH2-Wz=s-r_2ro49hm94nuvhu

Re: Improve logging when using Huge Pages

2023-01-23 Thread Andres Freund
me. Hm. Should this be a GUC? There's a reason shared_memory_size_in_huge_pages is one - it's so it's accessible via -C. But here we could make it a function or whatnot as well. I'm ok with this being a GUC, it doesn't feel elegant, but I suspect there's no realistic elegant answer. Greetings, Andres Freund

Re: Minimal logical decoding on standbys

2023-01-23 Thread Andres Freund
rio 2 but with "vacuum full pg_class" would not really add value > to the tests, right? A database wide VACUUM FULL is also just a row removal test, no? I think it makes sense to test that both VACUUM and VACUUM FULL both trigger conflicts, because they internally use *very* different mechanisms. It'd probably be good to test at least conflicts triggered due to row removal via on-access pruning as well. And perhaps also for btree killtuples. I think those are the common cases for catalog tables. Greetings, Andres Freund

Re: Cygwin cleanup

2023-01-23 Thread Andres Freund
Hi, On 2023-01-23 17:28:14 -0600, Justin Pryzby wrote: > On Thu, Jan 12, 2023 at 10:17:55PM -0600, Justin Pryzby wrote: > > On Thu, Jan 12, 2023 at 06:43:54PM -0800, Andres Freund wrote: > > > > It looks like logical decoding may be the "most wrong" place that &

Re: Add the ability to limit the amount of memory that can be allocated to backends.

2023-01-23 Thread Andres Freund
patch attached I think it's basically still waiting on author, until the O(N) cost is gone from the overflow limit check. Greetings, Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

2023-01-23 Thread Andres Freund
I haven't yet pushed the pg_test_timing (nor it's small prerequisite) > > patch. > > > > I've attached those two patches. Feel free to include them in your series if > > you want, then the CF entry (and thus cfbot) makes sense again... > I'll include them in my new patch set

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

2023-01-23 Thread Andres Freund
Hi, On 2023-01-23 18:49:37 +0100, David Geier wrote: > On 1/21/23 05:12, Andres Freund wrote: > > We do currently do the conversion quite frequently. Admittedly I was > > partially motivated by trying to get the per-loop overhead in pg_test_timing > > down ;) > >

Re: Non-superuser subscription owners

2023-01-23 Thread Andres Freund
t might be possible to teach libpq to differentiate peer from trust (by disabling passing the current user), or we could tell the server via an option to disable peer. But as you say, I don't think it'd buy us much. Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-01-23 Thread Andres Freund
Hi, On 2023-01-23 12:39:50 -0500, Robert Haas wrote: > On Sun, Jan 22, 2023 at 8:52 PM Andres Freund wrote: > > > Perhaps we should have a way to directly turn on/off authentication > > > methods in libpq through API functions and/or options? > > > > Yes. There

Re: Non-superuser subscription owners

2023-01-23 Thread Andres Freund
there's nothing you can do about the local-user case. > > If I'm asked to attempt to connect to a PostgreSQL server, and I > choose to do that, and the connection succeeds, all I know is that the > connection actually succeeded. Well, there is PQconnectionUsedPassword()... Not that it's a great answer. Greetings, Andres Freund

Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

2023-01-23 Thread Andres Freund
Hi, On 2023-01-23 18:23:17 +0100, David Geier wrote: > On 1/21/23 05:14, Andres Freund wrote: > > The elapsed time is already inherently unstable, so we shouldn't have any > > test > > output showing the time. > > > > But I doubt showing it in every explain i

Re: run pgindent on a regular basis / scripted manner

2023-01-23 Thread Andres Freund
; happen, but I haven't experimented. I think we should exempt it, initially at least. If somebody decides to invest a substantial amount of time in pgindent, let's change it, but I'm somewhat doubtful that'll happen anytime soon. Greetings, Andres Freund

Re: Non-superuser subscription owners

2023-01-22 Thread Andres Freund
Hi, On 2023-01-22 09:05:27 -0800, Jeff Davis wrote: > On Sat, 2023-01-21 at 14:01 -0800, Andres Freund wrote: > > There are good reasons to have 'peer' authentication set up for the > > user > > running postgres, so admin scripts can connect without issues. Which > > un

Re: run pgindent on a regular basis / scripted manner

2023-01-22 Thread Andres Freund
Hi, On 2023-01-22 19:28:42 -0500, Tom Lane wrote: > Andres Freund writes: > > I think I've proposed this before, but I still think that as long as we rely > > on pg_bsd_indent, we should have it be part of our source tree and > > automatically built. It's no wonder that b

Re: run pgindent on a regular basis / scripted manner

2023-01-22 Thread Andres Freund
the server side. Which obviously would be project wide... Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-01-22 Thread Andres Freund
Hi, On 2023-01-22 18:28:27 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-01-22 18:20:49 +0100, Jelte Fennema wrote: > >> I don't think the amount of pain is really much lower if we reformat > >> 10,000 or 300,000 lines of code, without automation both

Re: pgindent vs variable declaration across multiple lines

2023-01-22 Thread Andres Freund
to change that we'd like it to. I think it's a significant improvement, even if it turns out that there's other cases it misses. Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-01-22 Thread Andres Freund
ial value possibly wrapped). The fixed alignment doesn't have that issue. Personally I think the cost of trying to align local variable names is way way higher than the benefit. Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-01-22 Thread Andres Freund
ce changes could indicate a problem. Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-01-22 Thread Andres Freund
ther changes isn't going to be fun. Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-01-21 Thread Andres Freund
not fulfulling precisely our needs is the same as low quality requirements. Greetings, Andres Freund

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-21 Thread Andres Freund
Hi, On 2023-01-21 18:37:59 -0800, Peter Geoghegan wrote: > On Tue, Jan 17, 2023 at 10:02 AM Andres Freund wrote: > > I think with a bit of polish "Add autovacuum trigger instrumentation." ought > > to be quickly mergeable. > > Any thoughts on v5-0001-

<    13   14   15   16   17   18   19   20   21   22   >