Re: Adding wait events statistics

2025-07-28 Thread Bertrand Drouvot
Hi, On Thu, Jul 24, 2025 at 10:38:49AM -0400, Robert Haas wrote: > On Thu, Jul 24, 2025 at 7:52 AM Bertrand Drouvot > wrote: > > Well, the idea was more: as we speak about "wait" events then it would make > > sense to add their duration. And then, to have meaningfu

Re: Add os_page_num to pg_buffercache

2025-07-25 Thread Bertrand Drouvot
r to the above > (SET ROLE pg_database_owner/pg_monitor ... RESET role), with a couple of > different SELECT statements within. Should we rather add the above new > SELECTs there, instead of in the new pg_buffercache_os_pages.sql? Yeah, that probably makes more sense, done in the attached.

Re: Custom pgstat support performance regression for simple queries

2025-07-25 Thread Bertrand Drouvot
Hi, On Fri, Jul 25, 2025 at 10:38:59AM +0900, Michael Paquier wrote: > On Thu, Jul 24, 2025 at 07:38:46AM +0000, Bertrand Drouvot wrote: > > What about? > > > > - create a global variable but that should be used only by extensions? Say, > > pgstat_report_custom_fixed &

Re: Adding wait events statistics

2025-07-24 Thread Bertrand Drouvot
Hi, On Wed, Jul 23, 2025 at 11:38:07AM -0400, Robert Haas wrote: > On Tue, Jul 22, 2025 at 8:24 AM Bertrand Drouvot > wrote: > > So based on the cycles metric I think it looks pretty safe to implement for > > the > > whole majority of classes. > > I'm not

Re: Adding wait events statistics

2025-07-24 Thread Bertrand Drouvot
Hi, On Tue, Jul 22, 2025 at 10:07:30AM -0400, Andres Freund wrote: > Hi, > > On 2025-07-22 12:24:46 +, Bertrand Drouvot wrote: > > Anyway, let's forget about eBPF, I ran another experiment by counting the > > cycles > > with: > > > > static inlin

Re: Custom pgstat support performance regression for simple queries

2025-07-24 Thread Bertrand Drouvot
stat_report_stat(): - do the check on have_iostats, have_slrustats and friends and on pgstat_report_custom_fixed - reset pgstat_report_custom_fixed That way I think it's less error prone for the builtin stats and more clear for the extensions (as at least "custom" is also part of the global variable name and the check in pgstat_report_stat() would make it clear that we rely on the custom global variable). Thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Custom pgstat support performance regression for simple queries

2025-07-23 Thread Bertrand Drouvot
On Wed, Jul 23, 2025 at 05:09:54PM +0900, Michael Paquier wrote: > On Jul 23, 2025, at 16:33, Bertrand Drouvot > wrote: > > Maybe we could use a flag, say: > > > > #define PGSTAT_PENDING_IO (1 << 0) > > #define PGSTAT_PENDING_WAL(1 << 1)

Re: Custom pgstat support performance regression for simple queries

2025-07-23 Thread Bertrand Drouvot
_pending_mask accordingly when they flush, have pending stats though. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Verify predefined LWLocks tranches have entries in wait_event_names.txt

2025-07-22 Thread Bertrand Drouvot
for example. I don't think that's a big deal though. Also, I think we don't need to read lwlocklist.h twice in generate-lwlocknames.pl. PFA v4 where the only change compared to v3 is that it reads lwlocklist.h once in generate-lwlocknames.pl. Regards, -- Bertrand Drouvot PostgreS

Re: Adding wait events statistics

2025-07-22 Thread Bertrand Drouvot
Hi, On Fri, Jul 18, 2025 at 11:43:47AM -0400, Andres Freund wrote: > Hi, > > On 2025-07-18 06:04:38 +, Bertrand Drouvot wrote: > > Here what I’ve done: > > > > 1. Added 2 probes: one in pgstat_report_wait_start() and one in > > pgstat_report_wait_end

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-07-22 Thread Bertrand Drouvot
hub.com/torvalds/linux/commit/10d04c26ab2b7 > > Yay! I like how the first response is "you sent the patch wrong" ;-) > > I would have thought that coming from two major projects that use > email extensively (Debian, PostgreSQL), I would navigate that process > better.

Re: Verify predefined LWLocks tranches have entries in wait_event_names.txt

2025-07-21 Thread Bertrand Drouvot
Hi, On Mon, Jul 21, 2025 at 03:20:55PM -0500, Nathan Bossart wrote: > On Fri, Jul 18, 2025 at 01:39:15PM +0000, Bertrand Drouvot wrote: > > +#define PG_BUILTIN_LWTRANCHE(id, name) [id] = name, > > +#include "storage/lwlocktranchelist.h" > > +#undef PG_BUILT

Re: Verify predefined LWLocks tranches have entries in wait_event_names.txt

2025-07-21 Thread Bertrand Drouvot
Hi, On Mon, Jul 21, 2025 at 03:28:14PM -0500, Nathan Bossart wrote: > On Mon, Jul 21, 2025 at 03:20:55PM -0500, Nathan Bossart wrote: > > On Fri, Jul 18, 2025 at 01:39:15PM +, Bertrand Drouvot wrote: > >> +#define PG_BUILTIN_LWTRANCHE(id, name) [id] = name, >

Re: Add os_page_num to pg_buffercache

2025-07-21 Thread Bertrand Drouvot
he wording is almost the same as in pg_buffercache_numa_pages(). Also I think that it is not correct in pg_get_shmem_allocations_numa(), I think that it should be something like proposed in [1]. [1]: https://www.postgresql.org/message-id/aH4DDhdiG9Gi0rG7%40ip-10-97-1-34.eu-west-3.compute.internal Regards

Fix incorrect comment in pg_get_shmem_allocations_numa()

2025-07-21 Thread Bertrand Drouvot
Hi hackers, One comment in pg_get_shmem_allocations_numa() incorrectly describes the function as dealing with database blocks when it actually processes shared memory allocations from the shmem index. PFA, a patch to $SUBJECT. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS

Re: Fix lwlock.c and wait_event_names.txt discrepancy

2025-07-18 Thread Bertrand Drouvot
Hi, On Thu, Jul 17, 2025 at 02:25:46PM +, Bertrand Drouvot wrote: > Hi, > > On Thu, Jul 17, 2025 at 09:50:12AM +0900, Michael Paquier wrote: > > On Wed, Jul 16, 2025 at 11:41:09AM +, Bertrand Drouvot wrote: > > > > > As a way to prevent this to occur we

Verify predefined LWLocks tranches have entries in wait_event_names.txt

2025-07-18 Thread Bertrand Drouvot
; (in lwlock.h) from lwlocktranchelist.h with generate-lwlocknames.pl. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 830e9dce82d42eddbd7281f4267bb771162dbd47 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot Date: Fr

Re: Adding wait events statistics

2025-07-17 Thread Bertrand Drouvot
Hi, On Tue, Jul 15, 2025 at 04:13:49PM +0200, Bertrand Drouvot wrote: > Hi, > > On Fri, Jul 11, 2025 at 11:59:39AM -0400, Andres Freund wrote: > > Hi, > > > > On 2025-07-10 08:21:56 +, Bertrand Drouvot wrote: > > > - Let's define a list of &quo

Re: Fix lwlock.c and wait_event_names.txt discrepancy

2025-07-17 Thread Bertrand Drouvot
Hi, On Thu, Jul 17, 2025 at 09:50:12AM +0900, Michael Paquier wrote: > On Wed, Jul 16, 2025 at 11:41:09AM +0000, Bertrand Drouvot wrote: > > > As a way to prevent this to occur we might want to add extra input file(s) > > parameter to generate-wait_event_types.pl

Re: Fix lwlock.c and wait_event_names.txt discrepancy

2025-07-16 Thread Bertrand Drouvot
Hi, On Wed, Jul 16, 2025 at 01:33:07PM +0200, Álvaro Herrera wrote: > On 2025-Jul-16, Michael Paquier wrote: > > > On Wed, Jul 16, 2025 at 09:30:31AM +0000, Bertrand Drouvot wrote: > > > > Indeed, the wait event names are MultixactOffsetSLRU and > > > Multix

Fix lwlock.c and wait_event_names.txt discrepancy

2025-07-16 Thread Bertrand Drouvot
postgresql.org/message-id/aHZiHZ8sSQdHpyM6%40bdt-Laptop-13th-Gen-Intel-Core Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From ceef3dea35a28546bd4b6e4b5a78878640815c77 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot

Re: Improve LWLock tranche name visibility across backends

2025-07-15 Thread Bertrand Drouvot
nd Sami have in mind. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Adding wait events statistics

2025-07-15 Thread Bertrand Drouvot
Hi, On Fri, Jul 11, 2025 at 11:59:39AM -0400, Andres Freund wrote: > Hi, > > On 2025-07-10 08:21:56 +, Bertrand Drouvot wrote: > > - Let's define a list of "cheap" and a list of "expensive" wait events > > - Change the code path to in

Re: Improve LWLock tranche name visibility across backends

2025-07-15 Thread Bertrand Drouvot
t; wait_event_info = classId | eventId; > ``` > Do you think collision can still be possible? I meant to say collision between the trancheID and WaitEventCustomCounter->nextId Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Improve LWLock tranche name visibility across backends

2025-07-11 Thread Bertrand Drouvot
use the passed in value, otherwise generate the next > eventId. > > I do like the latter approach more, what do you think? I think I do prefer it too, but in both cases we'll have to make sure there is no collision on the eventID (LWTRANCHE_FIRST_USER_DEFINED is currently 95). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Adding basic NUMA awareness

2025-07-10 Thread Bertrand Drouvot
er;/* lock group leader, if I'm a member */ dlist_head lockGroupMembers; /* list of members, if I'm a leader */ dlist_node lockGroupLink; /* my member link, if I'm a member */ - - /* NUMA node */ - int numa_node; }; That could be back to 832 (the order does not make sense logically speaking though). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Improve LWLock tranche name visibility across backends

2025-07-10 Thread Bertrand Drouvot
okup now proceeds in > the order of built-in names, the local list, and finally the shared memory. > The fallback name "extension" can still be returned if an extension does > not register a tranche. I did not look in details, but do you think we could make use of Wai

Re: Adding wait events statistics

2025-07-10 Thread Bertrand Drouvot
Hi, On Wed, Jul 09, 2025 at 11:19:00AM -0400, Andres Freund wrote: > Hi, > > On 2025-07-09 07:26:05 +, Bertrand Drouvot wrote: > > On Tue, Jul 08, 2025 at 10:19:07PM -0400, Andres Freund wrote: > > > On 2025-06-30 13:36:12 +, Bertrand Drouvot wrote: > > TB

Re: Adding basic NUMA awareness - Preliminary feedback and outline for an extensible approach

2025-07-09 Thread Bertrand Drouvot
b's one)) could help to get performance gains depending on a "known" context (so less generic). So, probably having both could make sense but I think that they serve different purposes. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Add os_page_num to pg_buffercache

2025-07-09 Thread Bertrand Drouvot
ut a high-level "skeleton" > function, > with a couple of slots which would then be provided by the 3 variants. Yeah, I tried to avoid code duplication for the "os pages" related stuff in v1. I can check if more can be done (outside of the "os pages" related stuff).

Re: Adding wait events statistics

2025-07-09 Thread Bertrand Drouvot
Hi, On Wed, Jul 09, 2025 at 09:56:10AM +0900, Michael Paquier wrote: > On Tue, Jul 08, 2025 at 03:25:26PM +0000, Bertrand Drouvot wrote: > > So, something like: > > > > ClassInfo LWLockClassInfo = { > > .numberOfEvents = NB_LWLOCK_WAIT_EVE

Re: Adding wait events statistics

2025-07-09 Thread Bertrand Drouvot
Hi, On Tue, Jul 08, 2025 at 10:19:07PM -0400, Andres Freund wrote: > On 2025-06-30 13:36:12 +0000, Bertrand Drouvot wrote: > > * we don't want to allocate memory in some places where the counters are > > incremented (see 4feba03d8b9). We could still use the same implem

Re: Adding wait events statistics

2025-07-08 Thread Bertrand Drouvot
Hi, On Tue, Jul 08, 2025 at 04:46:28AM +, Bertrand Drouvot wrote: > On Tue, Jul 08, 2025 at 09:36:53AM +0900, Michael Paquier wrote: > > Yes, you may need some level of meta-data generated for the wait > > classes generated when the perl script generating this data is run.

Re: Adding wait events statistics

2025-07-07 Thread Bertrand Drouvot
Hi, On Tue, Jul 08, 2025 at 09:36:53AM +0900, Michael Paquier wrote: > On Mon, Jul 07, 2025 at 03:07:12PM +0000, Bertrand Drouvot wrote: > > > I think the question is: if the extension owner does not increment it, do > > we want > > to do it in core on their behalf? I&

Re: Adding wait events statistics

2025-07-07 Thread Bertrand Drouvot
Hi, On Mon, Jul 07, 2025 at 02:56:29PM +0900, Michael Paquier wrote: > On Mon, Jun 30, 2025 at 01:36:12PM +0000, Bertrand Drouvot wrote: > > * Do we need to serialize the stats based on their names (as for > > PGSTAT_KIND_REPLSLOT)? This question is the same as "is the ord

Re: Remove unused wait_event_info parameter in FileStartReadV()

2025-07-04 Thread Bertrand Drouvot
you're suggesting) but I think the closer to pg_preadv() the better. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Remove unused wait_event_info parameter in FileStartReadV()

2025-07-04 Thread Bertrand Drouvot
t might need a deeper analysis though (to check if there are some outside of the callback cases). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 367acfe3608f4450e32ea306068d61d2c74389b9 Mon Sep 17 00:00:00

Re: Add os_page_num to pg_buffercache

2025-07-02 Thread Bertrand Drouvot
Hi, On Wed, Jul 02, 2025 at 06:39:25AM +, Bertrand Drouvot wrote: > Hi, > > On Tue, Jul 01, 2025 at 07:46:30PM +0200, Tomas Vondra wrote: > > On 7/1/25 19:20, Bertrand Drouvot wrote: > > > Now it's just a matter of extracting the necessary pieces from &g

Re: A assert failure when initdb with track_commit_timestamp=on

2025-07-02 Thread Bertrand Drouvot
tstrapProcessingMode() is the right thing to do here. === 1 +* Don't bother to record commit_ts for Booststrap mode. typo: s/Booststrap/Bootstrap/ Also, grep on "Bootstrap mode" and "bootstrap mode" gives much more occurrences for the later, so maybe use "bo

Re: Fix inconsistency in the pg_buffercache documentation

2025-07-02 Thread Bertrand Drouvot
Hi, On Wed, Jul 02, 2025 at 09:55:52AM +0200, Daniel Gustafsson wrote: > > On 2 Jul 2025, at 08:23, Bertrand Drouvot > > wrote: > > > > Hi hackers, > > > > While working on [1], I noticed an inconsistency in the pg_buffercache > > documentation: b

Re: Add os_page_num to pg_buffercache

2025-07-01 Thread Bertrand Drouvot
Hi, On Tue, Jul 01, 2025 at 07:46:30PM +0200, Tomas Vondra wrote: > On 7/1/25 19:20, Bertrand Drouvot wrote: > > Now it's just a matter of extracting the necessary pieces from > > pg_buffercache_numa_pages() > > so that: > > > > * the new view could make

Fix inconsistency in the pg_buffercache documentation

2025-07-01 Thread Bertrand Drouvot
cache_numa() has been introduced. [1]: https://www.postgresql.org/message-id/flat/Z/fFA2heH6lpSLlt%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon

Re: Add os_page_num to pg_buffercache

2025-07-01 Thread Bertrand Drouvot
Hi, On Tue, Jul 01, 2025 at 06:45:37PM +0200, Tomas Vondra wrote: > On 7/1/25 18:34, Bertrand Drouvot wrote: > > But isn't the _numa view good enough for this? Sure, you need NUMA > support for it, and it may take a fair amount of time, but how often you > need to do suc

Re: Add os_page_num to pg_buffercache

2025-07-01 Thread Bertrand Drouvot
Hi, On Tue, Jul 01, 2025 at 04:31:01PM +0200, Tomas Vondra wrote: > On 7/1/25 15:45, Bertrand Drouvot wrote: > > I took a quick look on this, Thanks for looking at it! > and I doubt we want to change the schema of > pg_buffercache like this. Adding columns is fine, but it seems

Re: Add os_page_num to pg_buffercache

2025-07-01 Thread Bertrand Drouvot
Hi, On Thu, Apr 10, 2025 at 03:05:29PM +, Bertrand Drouvot wrote: > Hi, > > On Thu, Apr 10, 2025 at 09:58:18AM -0500, Nathan Bossart wrote: > > On Thu, Apr 10, 2025 at 03:35:24PM +0200, Tomas Vondra wrote: > > > This seems like a good idea in principle, but at this po

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-30 Thread Bertrand Drouvot
port or mentioned it can be found in the discussion? For 0003: " But with the chunking, introduced to work around the do_pages_stat() bug" Do you have in mind to quote the hex commit object name that will be generated by 0001? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-30 Thread Bertrand Drouvot
the new API > > and automatic behavior at this stage. If we find out there are certain > > use cases, we can revisit this idea. > > > > Okay, sounds good to me. Same here. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Adding wait events statistics

2025-06-30 Thread Bertrand Drouvot
", but with the additional code added here, compilers may ignore the inline keyword. Need to check if that's the case and the impact (see above). * After it goes in: add timings, add into per-backend stats too [1]: https://www.postgresql.org/message-id/aBhuTqNhMN3prcqe%40ip-10-97-1-34

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-27 Thread Bertrand Drouvot
ECK_FOR_INTERRUPTS() into the batch loop. This querying can take > quite a bit of time, so letting people to interrupt it seems reasonable. > It wasn't possible with just one call into the kernel, but with the > batching we can add a CFI. Yeah, LGTM. [1]: https://www.postgresql.org/

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-25 Thread Bertrand Drouvot
gt; you want me to do that, or shall one of you take care of that? I suppose > >> that'd be better, as you already wrote a fix / know the code better. > > > > Submitted: https://marc.info/?l=linux-mm&m=175077821909222&w=2 > > > > Thanks! Now we wait ...

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-25 Thread Bertrand Drouvot
Hi, On Wed, Jun 25, 2025 at 11:00:38AM +0200, Christoph Berg wrote: > Re: Bertrand Drouvot > > +/* > > + * Work around Linux kernel bug in 32-bit compat mode: do_pages_stat() has > > + * incorrect pointer arithmetic for more than DO_PAGES_STAT_CHUNK_NR pages. > > + */

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-25 Thread Bertrand Drouvot
Hi, On Wed, Jun 25, 2025 at 09:15:04AM +0530, shveta malik wrote: > On Wed, Jun 25, 2025 at 9:12 AM shveta malik wrote: > > > > On Tue, Jun 24, 2025 at 2:12 PM Bertrand Drouvot > > wrote: > > > > > > Yeah, I think that sounds reasonable and that would av

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-25 Thread Bertrand Drouvot
the parameters to kernel. And it'll > interpret them per kernel pointer size. > I also suspect something in this area... > If this is what's happening, I'm not sure what to do about it ... We could work by chunks (16?) on 32 bits but would probably produce performance degra

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-24 Thread Bertrand Drouvot
, maybe: - add a rule for Memcheck:Addr4? - have the same parameters name for the macro and the function? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-24 Thread Bertrand Drouvot
the kernel people. Do > > you want me to do that, or shall one of you take care of that? I suppose > > that'd be better, as you already wrote a fix / know the code better. > > Submitted: https://marc.info/?l=linux-mm&m=175077821909222&w=2 Thanks! I had in mind to

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-24 Thread Bertrand Drouvot
Hi, On Tue, Jun 24, 2025 at 04:41:33PM +0200, Christoph Berg wrote: > Re: Bertrand Drouvot > > Yes, I think compat_uptr_t usage is missing in do_pages_stat() (while it's > > used > > in do_pages_move()). > > I was also reading the kernel source around that pla

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-24 Thread Bertrand Drouvot
Hi, On Tue, Jun 24, 2025 at 02:33:59PM +0200, Tomas Vondra wrote: > > > On 6/24/25 13:10, Bertrand Drouvot wrote: > > So, if we look at do_pages_stat() ([1]), we can see that it uses an > > hardcoded > > "#define DO_PAGES_STAT_CHUNK_NR 16UL&qu

Re: pgsql: Introduce pg_shmem_allocations_numa view

2025-06-24 Thread Bertrand Drouvot
Hi, On Tue, Jun 24, 2025 at 11:20:15AM +0200, Tomas Vondra wrote: > On 6/24/25 10:24, Bertrand Drouvot wrote: > > Yeah, same for me with pg_get_shmem_allocations_numa(). It works if > > pg_numa_query_pages() is done on chunks <= 16 pages but fails if done on > >

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-24 Thread Bertrand Drouvot
Hi, On Tue, Jun 24, 2025 at 12:13:32AM +0900, Masahiko Sawada wrote: > On Mon, Jun 23, 2025 at 7:01 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Mon, Jun 23, 2025 at 05:10:37PM +0900, Masahiko Sawada wrote: > > > On Thu, Jun 19, 2025 a

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-23 Thread Bertrand Drouvot
Hi, On Mon, Jun 23, 2025 at 05:10:37PM +0900, Masahiko Sawada wrote: > On Thu, Jun 19, 2025 at 6:00 PM Bertrand Drouvot > wrote: > > > > - pg_activate_logical_decoding() is needed only if there is not already a > > logical > > slot on the primary > > - the d

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-19 Thread Bertrand Drouvot
Hi, On Fri, Jun 20, 2025 at 09:48:47AM +0530, shveta malik wrote: > On Thu, Jun 19, 2025 at 2:30 PM Bertrand Drouvot > wrote: > > > > I wonder if a way to address the concerns that we shared above is to use a > > mixed approach like: > > > > - Forget the imme

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-19 Thread Bertrand Drouvot
Hi, On Wed, Jun 18, 2025 at 03:22:59PM +0530, shveta malik wrote: > On Wed, Jun 18, 2025 at 2:39 PM Bertrand Drouvot > wrote: > > > > IIUC the idea is to "just" increase WAL level to 'logical' so that one > > could then > > be allowed to

Re: confusing message in check_tuple

2025-06-18 Thread Bertrand Drouvot
gt; > "number of attributes %u exceeds maximum expected for table (%u)" I thought about adding the table name in the message but it looks like it's already there: " Expected corruption message output stdout /(?^:(?^ms:heap table "postgres\.public\.test", block 0, o

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-06-18 Thread Bertrand Drouvot
better reflect this behavior (if we move that way). Thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2025-06-05 Thread Bertrand Drouvot
Hi, On Tue, Jun 03, 2025 at 01:27:29PM -0400, Robert Haas wrote: > On Mon, Jun 2, 2025 at 10:02 AM Bertrand Drouvot > wrote: > > So, we currently have 2 patterns: > > > > P1: permission checks on a referenced object is done before we call > > recordMultipleDep

Re: Avoid orphaned objects dependencies, take 3

2025-06-02 Thread Bertrand Drouvot
ies is called (see [1]). Does that make sense to you? [1]: https://www.postgresql.org/message-id/aD2u/GR/yaRkBJdJ%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2025-06-02 Thread Bertrand Drouvot
Hi, On Thu, May 22, 2025 at 09:40:47AM -0400, Robert Haas wrote: > On Thu, May 22, 2025 at 8:15 AM Bertrand Drouvot > wrote: > > That would probably address Robert's concern [1] "acquiring two locks on > > the same > > object with different lock modes where

Re: Add AioUringCompletion in wait_event_names.txt and a safeguard in generate-wait_event_types.pl

2025-06-01 Thread Bertrand Drouvot
Hi, On Tue, May 27, 2025 at 08:30:56AM +0900, Michael Paquier wrote: > On Mon, May 26, 2025 at 07:49:34AM +0000, Bertrand Drouvot wrote: > > That makes me think that it is easy to miss adding a new LWLock in > > wait_event_names.txt and generate-wait_event_types.pl does

Add AioUringCompletion in wait_event_names.txt and a safeguard in generate-wait_event_types.pl

2025-05-26 Thread Bertrand Drouvot
() in generate-wait_event_types.pl (in v1-0002) to ensure that the number of LWLocks description generated in the documentation match the ones defined in lwlock.h and lwlocklist.h. Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases A

Re: Avoid orphaned objects dependencies, take 3

2025-05-22 Thread Bertrand Drouvot
uld really only acquire one" but probably not this one "I think it might result in acquiring the locks on those other objects at a subtly wrong time (leading to race conditions)". For the latter I'm not sure how that could be a subtly wrong time or how could we determine what

Re: Avoid orphaned objects dependencies, take 3

2025-05-20 Thread Bertrand Drouvot
existing lock (and if so, skip acquiring a new one) but your proposal sounds better. Indeed it would make the locking behavior explicit and also be flexible (allowing the callers to choose the LOCKMODE). I'll prepare a new version implementing your proposal. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Remove unnecessary "lmgr.h" in stat_utils.c

2025-05-07 Thread Bertrand Drouvot
Hi, On Wed, May 07, 2025 at 07:28:01AM +, Bertrand Drouvot wrote: > Hi, > > On Wed, May 07, 2025 at 09:10:17AM +0900, Michael Paquier wrote: > > On Mon, May 05, 2025 at 04:33:20PM +0300, Ilia Evdokimov wrote: > > > While reviewing the import/export statistics,

Re: Remove unnecessary "lmgr.h" in stat_utils.c

2025-05-07 Thread Bertrand Drouvot
would make sense to work an a "larger" cleanup instead? (in the same vein as [1] did) [1]: https://www.postgresql.org/message-id/af837490-6b2f-46df-ba05-37ea6a6653fc%40eisentraut.org " Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-05-06 Thread Bertrand Drouvot
we could permit logical decoding on the standby even > with wal_level set to 'replica'. Yeah, technically speaking we could as the WALs are coming from the primary (that has wal_level set to logical). > However, this would necessitate > invalidating all logical replication slots during promotion, > potentially extending downtime during failover. Yeah, I'm tempted to vote to not allow logical decoding on the standby if the wal_level is not logical. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: PostgreSQL 18 Beta 1 release announcement draft

2025-05-05 Thread Bertrand Drouvot
k we can add WAL to the game, something like: " and now shows per-connection's I/O and WAL statistics " Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: PG 18 release notes draft committed

2025-05-05 Thread Bertrand Drouvot
one comment related to "per backend" statistics. We have this: " Add per-backend I/O statistics reporting (Bertrand Drouvot) The statistics are accessed via pg_stat_get_backend_io(). Per-backend statistics can be cleared via pg_stat_reset_backend_stats(). " and " Add

Re: Fix a race condition in ConditionVariableTimedSleep()

2025-05-05 Thread Bertrand Drouvot
the condition variable and trigger the failed assertion. I agree that you could still consider this as wrong thing if you think that we should not do anything that cancels condition variable during WaitLatch though. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Fix a race condition in ConditionVariableTimedSleep()

2025-05-05 Thread Bertrand Drouvot
Hi, On Mon, May 05, 2025 at 12:15:39PM +0300, Nazir Bilal Yavuz wrote: > Hi, > > Thanks for the report and the patch! > > On Mon, 5 May 2025 at 10:53, Bertrand Drouvot > wrote: > > > > The proposed fix attached is done in ConditionVariableTimedSleep() as t

Fix a race condition in ConditionVariableTimedSleep()

2025-05-05 Thread Bertrand Drouvot
signs cv_sleep_target to cv and then ensures that cv_sleep_target accurately describes which condition variable we’re prepared to wait on. Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid circular header file dependency

2025-04-27 Thread Bertrand Drouvot
Hi, On Sat, Apr 26, 2025 at 02:55:51PM -0400, Tom Lane wrote: > Bertrand Drouvot writes: > > On Sat, Apr 26, 2025 at 01:20:56AM -0400, Tom Lane wrote: > >> Whatever it contains, we need to kill it with fire before the problem > >> metastasizes like it did the last time

Re: Avoid circular header file dependency

2025-04-26 Thread Bertrand Drouvot
ot a simpler > "wait_classes.h" that gets included by wait_event.h and > wait_event_types.h? Yeah, better. Done that way in the attached. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From c6d1883b6fef09e4c351

Re: Avoid circular header file dependency

2025-04-26 Thread Bertrand Drouvot
Hi, On Sat, Apr 26, 2025 at 01:20:56AM -0400, Tom Lane wrote: > Bertrand Drouvot writes: > > While working on wait events I faced some compilation issues due to circular > > header file dependency (introduced in fa88928470b5) between wait_event.h and > > wait_event_types.h.

Avoid circular header file dependency

2025-04-25 Thread Bertrand Drouvot
2 | #include "a.h" " So it looks like that our code base only contains those 2: plpy_util.h and wait_event_types.h cases. Thoughts? [1]: https://clang.llvm.org/extra/clang-tidy/checks/misc/header-include-cycle.html Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Op

Avoid core dump in pgstat_read_statsfile()

2025-04-23 Thread Bertrand Drouvot
back-patch as the issue is unlikely to occur without custom stats. Adding a v18 open item then. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From b80bcbcf05e44d31cc90c1de2e3200899db1294f Mon Sep 17 00:00:00 200

Re: Recent pg_rewind test failures in buildfarm

2025-04-21 Thread Bertrand Drouvot
Hi, On Mon, Apr 21, 2025 at 10:17:40AM +0900, Michael Paquier wrote: > On Tue, Apr 15, 2025 at 05:50:32AM +0000, Bertrand Drouvot wrote: > > Sorry, can't look at the details right now but it might be linked to > > 039549d70f6 which is recent enough and in this area. Will give

Re: pending_since assertion failure on skink

2025-04-18 Thread Bertrand Drouvot
Hi, On Sat, Apr 12, 2025 at 10:33:33AM -0400, Andres Freund wrote: > I suspect that this is related to > > commit 039549d70f6aa2daa3714a13752a08fa8ca2fb05 +1 > We might just have to give up on that assertion, I guess? Probably. I'll have a look at it next week. Regards, --

Re: NUMA shared memory interleaving

2025-04-18 Thread Bertrand Drouvot
numa nodes and accessing their "local" memory first (and help with "remote" memory access if there is still more work to do "remotely"). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: NUMA shared memory interleaving

2025-04-18 Thread Bertrand Drouvot
same host. One could ensure that numa nodes are not shared across instances (cpu and memory resource isolation per instance). Bonus point, adding Direct I/O into the game would ensure that the OS page cache is not shared too. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Recent pg_rewind test failures in buildfarm

2025-04-14 Thread Bertrand Drouvot
body have an idea? Sorry, can't look at the details right now but it might be linked to 039549d70f6 which is recent enough and in this area. Will give it a look once I've time. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-10 Thread Bertrand Drouvot
Hi, On Tue, Apr 08, 2025 at 03:27:41PM +0900, Michael Paquier wrote: > On Tue, Apr 08, 2025 at 06:19:02AM +0000, Bertrand Drouvot wrote: > > - A new injection_points_wakeup_detach() function that is holding the > > spinlock > > during the whole duration to ensure that n

Re: Add os_page_num to pg_buffercache

2025-04-10 Thread Bertrand Drouvot
From a glance, this seems to fall in the "new feature" bucket and > should likely wait for v19. Thank you both for providing your thoughts that confirm my initial doubt. Let's come back to that one later then. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Sour

Re: Add pg_buffercache_evict_all() and pg_buffercache_mark_dirty[_all]() functions

2025-04-10 Thread Bertrand Drouvot
for buffers that are spread across NUMA nodes though. Does that sound like an idea worth to spend time on? (If so, I'd be happy to work on it). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Add os_page_num to pg_buffercache

2025-04-10 Thread Bertrand Drouvot
te for v18 post freeze (it looks more a feature enhancement than anything else) - It's currently doing the changes in pg_buffercache v1.6 but will need to create v1.7 for 19 (if the above stands true) Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributor

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-09 Thread Bertrand Drouvot
to use an empty one. This is not needed here anymore (as we're using another injection point that the one initially prooposed) but I'll open a dedicated thread for that for 19 when the timing will be appropriate. [1]: https://www.postgresql.org/message-id/Z6oQXc8LmiTLfwLA%40ip-10-97-1-34.e

Re: Draft for basic NUMA observability

2025-04-09 Thread Bertrand Drouvot
bvious but OTOH I think that would not hurt. Something like in the attached? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From c22533a6c8a8526b75a95ccf0eb39b5d10cca1f2 Mon Sep 17 00:00:00 2001 From: Bertrand

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-08 Thread Bertrand Drouvot
Hi, On Wed, Apr 09, 2025 at 12:03:06PM +0900, Michael Paquier wrote: > On Tue, Apr 08, 2025 at 06:42:53AM +0000, Bertrand Drouvot wrote: > > Fully agree. Will need to find another way to prevent a process to wait > > between the > > wakeup and the detach. I'll open a

Re: Draft for basic NUMA observability

2025-04-07 Thread Bertrand Drouvot
om pg_buffercache); count --- 65472 (1 row) So it looks like that the new way to iterate on the buffers that has been introduced in v26/v27 has some issue? [1]: https://www.postgresql.org/message-id/Z64Pr8CTG0RTrGR3%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Dr

Re: Draft for basic NUMA observability

2025-04-07 Thread Bertrand Drouvot
ifetime */ if (firstNumaTouch) - pg_numa_touch_mem_if_required(touch, os_page_ptrs[idx]); + pg_numa_touch_mem_if_required(touch, buffptr + (j * os_page_size)); what do you think? Regards,

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-07 Thread Bertrand Drouvot
on my side, let's keep it that way and don't make the slot active. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Fix 035_standby_logical_decoding.pl race conditions

2025-04-07 Thread Bertrand Drouvot
x27;s reports in [1]. So yeah, let's do as you suggested and do not make the slot active for the vacuum full case too. [1]: https://www.postgresql.org/message-id/386386.1737736...@sss.pgh.pa.us Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

  1   2   3   4   5   6   7   8   9   >