Re: Comments on Custom RMGRs

2024-08-04 Thread Jeff Davis
On Tue, 2024-07-23 at 16:21 +0300, Heikki Linnakangas wrote: > So, I got a feeling that adding this to the rmgr interface is not > quite > right. The rmgr callbacks are for things that run when WAL is > *replayed*, while checkpoints are related to how WAL is generated. > Let's > design this as a

Re: Comments on Custom RMGRs

2024-07-23 Thread Heikki Linnakangas
On 27/05/2024 21:20, Michael Paquier wrote: On Fri, May 17, 2024 at 04:25:15PM -0400, Robert Haas wrote: On Fri, May 17, 2024 at 4:20 PM Jeff Davis wrote: Regarding this particular change: the checkpointing hook seems more like a table AM feature, so I agree with you that we should have a good

Re: Comments on Custom RMGRs

2024-05-27 Thread Tristan Partin
On Fri May 17, 2024 at 3:20 PM CDT, Jeff Davis wrote: ... Obviously it would be better to have a nice example table AM in /contrib, different enough from heap, but nobody has done that yet. You could argue that we never should have exposed the API without something like this in the first place,

Re: Comments on Custom RMGRs

2024-05-27 Thread Michael Paquier
On Fri, May 17, 2024 at 04:25:15PM -0400, Robert Haas wrote: > On Fri, May 17, 2024 at 4:20 PM Jeff Davis wrote: >> Regarding this particular change: the checkpointing hook seems more >> like a table AM feature, so I agree with you that we should have a good >> idea how a real table AM might use t

Re: Comments on Custom RMGRs

2024-05-17 Thread Robert Haas
On Fri, May 17, 2024 at 4:20 PM Jeff Davis wrote: > Regarding this particular change: the checkpointing hook seems more > like a table AM feature, so I agree with you that we should have a good > idea how a real table AM might use this, rather than only > pg_stat_statements. I would even be OK wi

Re: Comments on Custom RMGRs

2024-05-17 Thread Jeff Davis
On Fri, 2024-05-17 at 14:56 -0400, Robert Haas wrote: > (2) a detailed > description of how a non-core table AM or index AM is expected to be > able to make use of this. Bonus points if the person providing that > rationale can say credibly that they've actually implemented this and > it works grea

Re: Comments on Custom RMGRs

2024-05-17 Thread Robert Haas
On Fri, Mar 29, 2024 at 1:09 PM Jeff Davis wrote: > I am fine with this. > > You've moved the discussion forward in two ways: > > 1. Changes to pg_stat_statements to actually use the API; and > 2. The hook is called at multiple points. > > Those at least partially address the concerns raised b

Re: Comments on Custom RMGRs

2024-03-29 Thread Jeff Davis
On Fri, 2024-03-29 at 18:20 +0700, Danil Anisimow wrote: > > In [rmgr_003.v3.patch] I added a phase argument to RmgrCheckpoint(). > Currently it is only called in two places: before and after > CheckPointBuffers(). I am fine with this. You've moved the discussion forward in two ways: 1. Chang

Re: Comments on Custom RMGRs

2024-03-29 Thread Danil Anisimow
On Fri, Mar 22, 2024 at 2:02 AM Jeff Davis wrote: > On Thu, 2024-03-21 at 19:47 +0700, Danil Anisimow wrote: > > [pgss_001.v1.patch] adds a custom resource manager to the > > pg_stat_statements extension. > > Did you consider moving the logic for loading the initial contents from > disk from pgss_

Re: Comments on Custom RMGRs

2024-03-21 Thread Jeff Davis
On Thu, 2024-03-21 at 19:47 +0700, Danil Anisimow wrote: > The proposed patch is not a complete solution for pgss and may not > work correctly with replication. Also, what is the desired behavior during replication? Should queries on the primary be represented in pgss on the replica? If the answer

Re: Comments on Custom RMGRs

2024-03-21 Thread Jeff Davis
On Thu, 2024-03-21 at 19:47 +0700, Danil Anisimow wrote: > [pgss_001.v1.patch] adds a custom resource manager to the > pg_stat_statements extension. Did you consider moving the logic for loading the initial contents from disk from pgss_shmem_startup to .rmgr_startup? > The rm_checkpoint hook allo

Re: Comments on Custom RMGRs

2024-03-21 Thread Danil Anisimow
On Fri, Mar 1, 2024 at 2:06 AM Jeff Davis wrote: > Added to March CF. > > I don't have an immediate use case in mind for this, so please drive > that part of the discussion. I can't promise this for 17, but if the > patch is simple enough and a quick consensus develops, then it's > possible. [pgs

Re: Comments on Custom RMGRs

2024-03-04 Thread Andrey M. Borodin
> On 29 Feb 2024, at 19:47, Danil Anisimow wrote: > > Answering your questions might take some time as I want to write a sample > patch for pg_stat_statements and make some tests. > What do you think about putting the patch to commitfest as it closing in a > few hours? I’ve switched the pat

Re: Comments on Custom RMGRs

2024-02-29 Thread Jeff Davis
On Thu, 2024-02-29 at 21:47 +0700, Danil Anisimow wrote: > Answering your questions might take some time as I want to write a > sample patch for pg_stat_statements and make some tests. > What do you think about putting the patch to commitfest as it closing > in a few hours? Added to March CF. I d

Re: Comments on Custom RMGRs

2024-02-29 Thread Danil Anisimow
On Tue, Feb 27, 2024 at 2:56 AM Jeff Davis wrote: > Let's pick this discussion back up, then. Where should the hook go? > Does it need to be broken into phases like resource owners? What > guidance can we provide to extension authors to use it correctly? > > Simon's right that these things don't n

Re: Comments on Custom RMGRs

2024-02-26 Thread Jeff Davis
On Mon, 2024-02-26 at 23:29 +0700, Danil Anisimow wrote: > Hi, > > The checkpoint hook looks very useful, especially for extensions that > have their own storage, like pg_stat_statements. > For example, we can keep work data in shared memory and save it only > during checkpoints. > When recovering

Re: Comments on Custom RMGRs

2024-02-26 Thread Danil Anisimow
Hi, The checkpoint hook looks very useful, especially for extensions that have their own storage, like pg_stat_statements. For example, we can keep work data in shared memory and save it only during checkpoints. When recovering, we need to read all the data from the disk and then repeat the latest

Re: Comments on Custom RMGRs

2022-05-13 Thread Robert Haas
On Fri, May 13, 2022 at 8:47 AM Simon Riggs wrote: > > Note that I'm *not* against making checkpoint extensible - I just think it > > needs a good bit of design work around when the hook is called etc. > > When was any such work done previously for any other hook?? That isn't needed. I think almo

Re: Comments on Custom RMGRs

2022-05-13 Thread Jeff Davis
On Fri, 2022-05-13 at 13:31 +0100, Simon Riggs wrote: > The first was a discussion about a data structure needed by BDR about > 4 years ago. In the absence of a pluggable checkpoint, the solution > was forced to use a normal table, which wasn't very satisfactory. I'm interested to hear more about

Re: Comments on Custom RMGRs

2022-05-13 Thread Simon Riggs
On Fri, 13 May 2022 at 00:42, Andres Freund wrote: > > On 2022-05-12 22:26:51 +0100, Simon Riggs wrote: > > On Thu, 12 May 2022 at 04:40, Andres Freund wrote: > > > I'm not happy with the idea of random code being executed in the middle of > > > CheckPointGuts(), without any documentation of what

Re: Comments on Custom RMGRs

2022-05-13 Thread Simon Riggs
On Fri, 13 May 2022 at 05:13, Jeff Davis wrote: > > On Thu, 2022-05-12 at 22:26 +0100, Simon Riggs wrote: > > I see multiple uses for the rm_checkpoint() point proposed and I've > > been asked multiple times for a checkpoint hook. > > Can you elaborate and/or link to a discussion? Those were conv

Re: Comments on Custom RMGRs

2022-05-12 Thread Jeff Davis
On Thu, 2022-05-12 at 22:26 +0100, Simon Riggs wrote: > I see multiple uses for the rm_checkpoint() point proposed and I've > been asked multiple times for a checkpoint hook. Can you elaborate and/or link to a discussion? Regards, Jeff Davis

Re: Comments on Custom RMGRs

2022-05-12 Thread Andres Freund
Hi, On 2022-05-12 22:26:51 +0100, Simon Riggs wrote: > On Thu, 12 May 2022 at 04:40, Andres Freund wrote: > > I'm not happy with the idea of random code being executed in the middle of > > CheckPointGuts(), without any documentation of what is legal to do at that > > point. > > The "I'm not happ

Re: Comments on Custom RMGRs

2022-05-12 Thread Simon Riggs
On Thu, 12 May 2022 at 04:40, Andres Freund wrote: > > On 2022-05-11 09:39:48 -0700, Jeff Davis wrote: > > On Wed, 2022-05-11 at 15:24 +0100, Simon Riggs wrote: > > > [PATCH: rmgr_001.v1.patch] > > > > > > [PATCH: rmgr_002.v1.patch] > > > > Thank you. Both of these look like good ideas, and I will

Re: Comments on Custom RMGRs

2022-05-11 Thread Andres Freund
Hi, On 2022-05-11 09:39:48 -0700, Jeff Davis wrote: > On Wed, 2022-05-11 at 15:24 +0100, Simon Riggs wrote: > > [PATCH: rmgr_001.v1.patch] > > > > [PATCH: rmgr_002.v1.patch] > > Thank you. Both of these look like good ideas, and I will commit them > in a few days assuming that nobody else sees a

Re: Comments on Custom RMGRs

2022-05-11 Thread Jeff Davis
On Wed, 2022-05-11 at 15:24 +0100, Simon Riggs wrote: > [PATCH: rmgr_001.v1.patch] > > [PATCH: rmgr_002.v1.patch] Thank you. Both of these look like good ideas, and I will commit them in a few days assuming that nobody else sees a problem. > It occurs to me that any use of WAL presumes that Chec

Comments on Custom RMGRs

2022-05-11 Thread Simon Riggs
New rmgr stuff looks interesting. I've had a detailed look through it and tried to think about how it might be used in practice. Spotted a minor comment that needs adjustment for new methods... [PATCH: rmgr_001.v1.patch] I notice rm_startup() and rm_cleanup() presume that this only works in recov