On Fri, Dec 3, 2021 at 11:50 PM Bossart, Nathan <bossa...@amazon.com> wrote: > > On 12/3/21, 5:57 AM, "Bharath Rupireddy" > <bharath.rupireddyforpostg...@gmail.com> wrote: > > On Fri, Dec 3, 2021 at 3:01 AM Bossart, Nathan <bossa...@amazon.com> wrote: > >> > >> On 12/1/21, 6:48 PM, "Bharath Rupireddy" > >> <bharath.rupireddyforpostg...@gmail.com> wrote: > >> > +1 for the overall idea of making the checkpoint faster. In fact, we > >> > here at our team have been thinking about this problem for a while. If > >> > there are a lot of files that checkpoint has to loop over and remove, > >> > IMO, that task can be delegated to someone else (maybe a background > >> > worker called background cleaner or bg cleaner, of course, we can have > >> > a GUC to enable or disable it). The checkpoint can just write some > >> > >> Right. IMO it isn't optimal to have critical things like startup and > >> checkpointing depend on somewhat-unrelated tasks. I understand the > >> desire to avoid adding additional processes, and maybe it is a bigger > >> hammer than what is necessary to reduce the impact, but it seemed like > >> a natural solution for this problem. That being said, I'm all for > >> exploring other ways to handle this. > > > > Having a generic background cleaner process (controllable via a few > > GUCs), which can delete a bunch of files (snapshot, mapping, old WAL, > > temp files etc.) or some other task on behalf of the checkpointer, > > seems to be the easiest solution. > > > > I'm too open for other ideas. > > I might hack something together for the separate worker approach, if > for no other reason than to make sure I really understand how these > functions work. If/when a better idea emerges, we can alter course.
Thanks. As I said upthread we've been discussing the approach of offloading some of the checkpoint tasks like (deleting snapshot files) internally for quite some time and I would like to share a patch that adds a new background cleaner process (currently able to delete the logical replication snapshot files, if required can be extended to do other tasks as well). I don't mind if it gets rejected. Please have a look. Regards, Bharath Rupireddy.
v1-0001-background-cleaner-to-offload-checkpoint-tasks.patch
Description: Binary data
