Re: recovery modules

2024-05-15 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 046d6e4b13d3a6b15df1245f3154969f7553594d Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v22 1/5] introduce routine for checking mutually exclusive string GUCs ---

Re: recovery modules

2024-04-10 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 5897631d5f09032565d92d5b8547baf3d24eef87 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v21 1/5] introduce routine for checking mutually exclusive string GUCs ---

Re: recovery modules

2024-03-26 Thread Nathan Bossart
another rebase for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 0ccdbb0a010830380e6ff4b7a052198d50f0680f Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v20 1/5] introduce routine for checking mutually exclusive

Re: recovery modules

2024-03-15 Thread Nathan Bossart
rebased for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From d3f497906daf1c405059b2c292f1eeb5cfeb742b Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v19 1/5] introduce routine for checking mutually exclusive string

Re: recovery modules

2024-03-06 Thread Nathan Bossart
Here is another rebase. Given the size of this patch set and the lack of review, I am going to punt this one to v18. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From dd69e5987b477818f60b202af5fb9b2603dc8acb Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023

Re: recovery modules

2024-01-11 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 6cee7d220b886e9be309929da5274c5770e59845 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v17 1/5] introduce routine for checking mutually exclusive string GUCs ---

Re: recovery modules

2023-10-23 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 0934f23773a612051c36070ed1f3b8ab100c4c65 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 15 Feb 2023 14:28:53 -0800 Subject: [PATCH v16 1/5] introduce routine for checking mutually exclusive string GUCs ---

Re: recovery modules

2023-04-17 Thread Nathan Bossart
rebased -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From b8856e61d775bc248a292163facc0b227abdde97 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Thu, 23 Feb 2023 14:27:48 -0800 Subject: [PATCH v15 1/7] Move extra code out of the Pre/PostRestoreCommand() block. If

Re: recovery modules

2023-04-05 Thread Michael Paquier
On Fri, Feb 17, 2023 at 11:41:32AM -0800, Andres Freund wrote: > I don't fully now, it's not entirely clear to me what the goals here were. I > think you'd likely need to do a bit of infrastructure work to do this > sanely. So far we just didn't have the need to handle files being released in > a

Re: recovery modules

2023-03-15 Thread Michael Paquier
On Tue, Mar 14, 2023 at 09:13:09PM -0700, Nathan Bossart wrote: > However, that's not what happens when hot_standby is off. In that case, > the postmaster.pid file is updated with PM_STATUS_STANDBY once recovery > starts, which wait_for_postmaster_start() interprets as "ready." I see > this was

Re: recovery modules

2023-03-14 Thread Nathan Bossart
I noticed that the new TAP test for basic_archive was failing intermittently for cfbot. It looks like the query for checking that the post-backup WAL is restored sometimes executes before archive recovery is complete (because hot_standby is on). To fix this, I adjusted the test to use

Re: recovery modules

2023-03-10 Thread Nathan Bossart
Here is a rebased version of the restore modules patch set. I swapped the patch for the stopgap fix for restore_command with the latest version [0], and I marked the restore/ headers as installable (as was recently done for archive/ [1]). There are no other changes. [0]

Re: recovery modules

2023-02-17 Thread Nathan Bossart
Here is a new revision of the restore modules patch set. In this patch set, the interface looks similar to the recent archive modules redesign, and there are separate callbacks for retrieving different types of files. I've attempted to address all the feedback I've received, but there was a lot

Re: recovery modules

2023-02-17 Thread Nathan Bossart
On Fri, Feb 17, 2023 at 05:01:47PM +0900, Michael Paquier wrote: > All that had better > be put into their own threads, IMO, to bring more visibility on these > subjects. I created a new thread for these [0]. > Saying that, I have spent more time on the revamped version of the > archive modules

Re: recovery modules

2023-02-17 Thread Andres Freund
Hi, On 2023-02-16 13:58:10 -0800, Nathan Bossart wrote: > On Thu, Feb 16, 2023 at 01:17:54PM -0800, Andres Freund wrote: > > I'm quite baffled by: > > /* Close any files left open by copy_file() or compare_files() > > */ > > AtEOSubXact_Files(false,

Re: recovery modules

2023-02-17 Thread Michael Paquier
On Thu, Feb 16, 2023 at 01:17:54PM -0800, Andres Freund wrote: > On 2023-02-16 12:15:12 -0800, Nathan Bossart wrote: >> On Thu, Feb 16, 2023 at 11:29:56AM -0800, Andres Freund wrote: >>> Not related the things changed here, but this should never have been pushed >>> down into individual archive

Re: recovery modules

2023-02-16 Thread Nathan Bossart
On Thu, Feb 16, 2023 at 01:17:54PM -0800, Andres Freund wrote: > On 2023-02-16 12:15:12 -0800, Nathan Bossart wrote: >> On Thu, Feb 16, 2023 at 11:29:56AM -0800, Andres Freund wrote: >> > On 2023-02-15 10:44:07 -0800, Nathan Bossart wrote: >> >> @@ -144,10 +170,12 @@ basic_archive_configured(void)

Re: recovery modules

2023-02-16 Thread Andres Freund
Hi, On 2023-02-16 12:15:12 -0800, Nathan Bossart wrote: > Thanks for reviewing. > > On Thu, Feb 16, 2023 at 11:29:56AM -0800, Andres Freund wrote: > > On 2023-02-15 10:44:07 -0800, Nathan Bossart wrote: > >> @@ -144,10 +170,12 @@ basic_archive_configured(void) > >> * Archives one file. > >>

Re: recovery modules

2023-02-16 Thread Nathan Bossart
Thanks for reviewing. On Thu, Feb 16, 2023 at 11:29:56AM -0800, Andres Freund wrote: > On 2023-02-15 10:44:07 -0800, Nathan Bossart wrote: >> @@ -144,10 +170,12 @@ basic_archive_configured(void) >> * Archives one file. >> */ >> static bool >> -basic_archive_file(const char *file, const char

Re: recovery modules

2023-02-16 Thread Andres Freund
Hi, On 2023-02-15 10:44:07 -0800, Nathan Bossart wrote: > On Wed, Feb 15, 2023 at 03:38:21PM +0900, Michael Paquier wrote: > > I may tweak a bit the comments, but nothing more. And I don't think I > > have more to add. Andres, do you have anything you would like to > > mention? Just some minor

Re: recovery modules

2023-02-15 Thread Nathan Bossart
On Wed, Feb 15, 2023 at 03:38:21PM +0900, Michael Paquier wrote: > On Mon, Feb 13, 2023 at 05:02:37PM -0800, Nathan Bossart wrote: >> Sorry for then noise, cfbot alerted me to a missing #include, which I've >> added in v13. Sorry for more noise. I noticed that I missed updating the

Re: recovery modules

2023-02-14 Thread Michael Paquier
On Mon, Feb 13, 2023 at 05:02:37PM -0800, Nathan Bossart wrote: > Sorry for then noise, cfbot alerted me to a missing #include, which I've > added in v13. + basic_archive_context = data->context; + Assert(CurrentMemoryContext != basic_archive_context); So this is what it means to document

Re: recovery modules

2023-02-13 Thread Nathan Bossart
On Mon, Feb 13, 2023 at 04:55:58PM -0800, Nathan Bossart wrote: > Okay. I've added it back in v12 with the suggested adjustment for the > memory context stuff. Sorry for then noise, cfbot alerted me to a missing #include, which I've added in v13. -- Nathan Bossart Amazon Web Services:

Re: recovery modules

2023-02-13 Thread Nathan Bossart
On Mon, Feb 13, 2023 at 03:37:33PM -0800, Andres Freund wrote: > On 2023-02-13 14:56:47 -0800, Nathan Bossart wrote: >> On Mon, Feb 13, 2023 at 04:37:10PM +0900, Michael Paquier wrote: >> > + basic_archive_context = data->context; >> > + if (CurrentMemoryContext == basic_archive_context) >> >

Re: recovery modules

2023-02-13 Thread Andres Freund
Hi, On 2023-02-13 14:56:47 -0800, Nathan Bossart wrote: > On Mon, Feb 13, 2023 at 04:37:10PM +0900, Michael Paquier wrote: > > + basic_archive_context = data->context; > > + if (CurrentMemoryContext == basic_archive_context) > > + MemoryContextSwitchTo(TopMemoryContext); > > + > > +

Re: recovery modules

2023-02-13 Thread Nathan Bossart
On Mon, Feb 13, 2023 at 04:37:10PM +0900, Michael Paquier wrote: > + basic_archive_context = data->context; > + if (CurrentMemoryContext == basic_archive_context) > + MemoryContextSwitchTo(TopMemoryContext); > + > + if (MemoryContextIsValid(basic_archive_context)) > +

Re: recovery modules

2023-02-12 Thread Michael Paquier
On Thu, Feb 09, 2023 at 02:48:26PM -0800, Nathan Bossart wrote: > On Thu, Feb 09, 2023 at 12:18:55PM -0800, Andres Freund wrote: >>> >>> -typedef bool (*ArchiveCheckConfiguredCB) (void); >>> +typedef bool (*ArchiveCheckConfiguredCB) (ArchiveModuleState *state); >>> >>> >>> If true is

Re: recovery modules

2023-02-09 Thread Michael Paquier
On Thu, Feb 09, 2023 at 12:18:55PM -0800, Andres Freund wrote: > I think this nearly ready. Michael, are you planning to commit this? I could take a stab at it, now if you feel strongly about doing it yourselfof course feel free :) > Personally I'd probably squash these into a single commit.

Re: recovery modules

2023-02-09 Thread Nathan Bossart
On Thu, Feb 09, 2023 at 12:18:55PM -0800, Andres Freund wrote: > On 2023-02-09 11:39:17 -0800, Nathan Bossart wrote: > Personally I'd probably squash these into a single commit. done > I'd probably mention that this should typically be of server lifetime / a > 'static const' struct. Tableam

Re: recovery modules

2023-02-09 Thread Andres Freund
Hi, On 2023-02-09 11:39:17 -0800, Nathan Bossart wrote: > rebased for cfbot I think this nearly ready. Michael, are you planning to commit this? Personally I'd probably squash these into a single commit. > diff --git a/doc/src/sgml/archive-modules.sgml > b/doc/src/sgml/archive-modules.sgml >

Re: recovery modules

2023-02-09 Thread Nathan Bossart
rebased for cfbot -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From ea6339276c6863f260572dfc816f9dd27ac7b516 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Fri, 27 Jan 2023 21:01:22 -0800 Subject: [PATCH v9 1/3] s/ArchiveContext/ArchiveCallbacks ---

Re: recovery modules

2023-02-09 Thread Nathan Bossart
On Wed, Feb 08, 2023 at 09:16:19PM -0800, Andres Freund wrote: > Pushed. Thanks! Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: recovery modules

2023-02-08 Thread Andres Freund
On 2023-02-08 10:55:44 -0800, Nathan Bossart wrote: > done Pushed. Thanks!

Re: recovery modules

2023-02-08 Thread Nathan Bossart
On Wed, Feb 08, 2023 at 10:24:18AM -0800, Andres Freund wrote: > If you'd reorder it so that 0004 applies independently from the other changes, > I'd just push that now. done -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 688566e23519f173a805696e4f00345f30d5d8e6 Mon Sep 17

Re: recovery modules

2023-02-08 Thread Andres Freund
Hi, On 2023-02-08 09:57:56 -0800, Nathan Bossart wrote: > On Wed, Feb 08, 2023 at 09:33:44AM -0800, Andres Freund wrote: > > And yes, I'd be for a patch to remove all of those assertions. > > done If you'd reorder it so that 0004 applies independently from the other changes, I'd just push that

Re: recovery modules

2023-02-08 Thread Nathan Bossart
On Wed, Feb 08, 2023 at 09:33:44AM -0800, Andres Freund wrote: > And yes, I'd be for a patch to remove all of those assertions. done -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 09fcca03d4a91be5757201cc311d3dad08463cd0 Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date:

Re: recovery modules

2023-02-08 Thread Andres Freund
Hi, On 2023-02-08 09:27:05 -0800, Nathan Bossart wrote: > On Wed, Feb 08, 2023 at 08:27:13AM -0800, Andres Freund wrote: > > One minor thing: I don't think we really need the AssertVariableIsOfType() > > for > > anything but the Init() one? > > This is another part that was borrowed from

Re: recovery modules

2023-02-08 Thread Nathan Bossart
On Wed, Feb 08, 2023 at 08:27:13AM -0800, Andres Freund wrote: > One minor thing: I don't think we really need the AssertVariableIsOfType() for > anything but the Init() one? This is another part that was borrowed from logical decoding output plugins. I'm not sure this adds much since f2b73c8

Re: recovery modules

2023-02-08 Thread Andres Freund
Hi, On 2023-02-08 16:23:34 +0900, Michael Paquier wrote: > On Sat, Feb 04, 2023 at 10:14:36AM -0800, Nathan Bossart wrote: > > On Sat, Feb 04, 2023 at 11:59:20AM +0900, Michael Paquier wrote: > >> + ArchiveModuleCallbacks struct filled with the callback function > >> pointers for > >> This

Re: recovery modules

2023-02-07 Thread Michael Paquier
On Sat, Feb 04, 2023 at 10:14:36AM -0800, Nathan Bossart wrote: > On Sat, Feb 04, 2023 at 11:59:20AM +0900, Michael Paquier wrote: >> + ArchiveModuleCallbacks struct filled with the callback function pointers >> for >> This needs a structname markup. >> >> + can use state->private_data to

Re: recovery modules

2023-02-04 Thread Nathan Bossart
On Sat, Feb 04, 2023 at 11:59:20AM +0900, Michael Paquier wrote: > + ArchiveModuleCallbacks struct filled with the callback function pointers > for > This needs a structname markup. > > + can use state->private_data to store it. > And here it would be structfield. fixed -- Nathan Bossart

Re: recovery modules

2023-02-03 Thread Michael Paquier
On Thu, Feb 02, 2023 at 11:37:00AM -0800, Nathan Bossart wrote: > On Thu, Feb 02, 2023 at 02:34:17PM +0900, Michael Paquier wrote: >> Okay, the changes done here look straight-forward seen from here. I >> got one small-ish comment. >> >> +basic_archive_startup(ArchiveModuleState *state) >> +{ >>

Re: recovery modules

2023-02-02 Thread Nathan Bossart
On Thu, Feb 02, 2023 at 02:34:17PM +0900, Michael Paquier wrote: > Okay, the changes done here look straight-forward seen from here. I > got one small-ish comment. > > +basic_archive_startup(ArchiveModuleState *state) > +{ > + BasicArchiveData *data = palloc0(sizeof(BasicArchiveData)); > >

Re: recovery modules

2023-02-01 Thread Michael Paquier
On Wed, Feb 01, 2023 at 01:23:26PM -0800, Nathan Bossart wrote: > Yeah, that's nicer. cfbot is complaining about a missing #include, so I > need to send a new revision anyway. Okay, the changes done here look straight-forward seen from here. I got one small-ish comment.

Re: recovery modules

2023-02-01 Thread Nathan Bossart
On Wed, Feb 01, 2023 at 01:06:06PM -0800, Andres Freund wrote: > On 2023-02-01 12:15:29 -0800, Nathan Bossart wrote: >> Here's a new patch set in which I've attempted to address this feedback and >> Michael's feedback. > > Looks better! Thanks! >> @@ -25,12 +34,14 @@ extern PGDLLIMPORT char

Re: recovery modules

2023-02-01 Thread Andres Freund
Hi, On 2023-02-01 12:15:29 -0800, Nathan Bossart wrote: > Here's a new patch set in which I've attempted to address this feedback and > Michael's feedback. Looks better! > @@ -25,12 +34,14 @@ extern PGDLLIMPORT char *XLogArchiveLibrary; > * For more information about the purpose of each

Re: recovery modules

2023-02-01 Thread Nathan Bossart
On Wed, Feb 01, 2023 at 03:54:26AM -0800, Andres Freund wrote: > I'd make basic_archive's private data a struct, with a member for the > context, but it's not that important. > > I'd also be inclined to do the same for the private_state you're passing > around for each module. Even if it's just

Re: recovery modules

2023-02-01 Thread Andres Freund
Hi, On 2023-01-31 15:30:13 -0800, Nathan Bossart wrote: > +/* > + * basic_archive_startup > + * > + * Creates the module's memory context. > + */ > +void * > +basic_archive_startup(void) > +{ > + return (void *) AllocSetContextCreate(TopMemoryContext, > +

Re: recovery modules

2023-01-31 Thread Michael Paquier
On Tue, Jan 31, 2023 at 03:30:13PM -0800, Nathan Bossart wrote: > Okay, here is a new patch set with the aforementioned adjustments and > documentation updates. So, it looks like you have addressed the feedback received here, as of: - Rename of Context to Callback. - Move of the definition into

Re: recovery modules

2023-01-31 Thread Nathan Bossart
On Tue, Jan 31, 2023 at 08:13:11AM +0900, Michael Paquier wrote: > On Mon, Jan 30, 2023 at 12:04:22PM -0800, Nathan Bossart wrote: >> On Mon, Jan 30, 2023 at 11:48:10AM -0800, Andres Freund wrote: >>> I don't think _PG_archive_module_init() should actually allocate a memory >>> context and do

Re: recovery modules

2023-01-30 Thread Michael Paquier
On Mon, Jan 30, 2023 at 12:04:22PM -0800, Nathan Bossart wrote: > On Mon, Jan 30, 2023 at 11:48:10AM -0800, Andres Freund wrote: >> I don't think _PG_archive_module_init() should actually allocate a memory >> context and do other similar initializations. Instead it should just return >> 'const

Re: recovery modules

2023-01-30 Thread Nathan Bossart
On Mon, Jan 30, 2023 at 11:48:10AM -0800, Andres Freund wrote: > I don't think _PG_archive_module_init() should actually allocate a memory > context and do other similar initializations. Instead it should just return > 'const ArchiveModuleCallbacks*', typically a single line. > > Allocations etc

Re: recovery modules

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 16:51:38 +0900, Michael Paquier wrote: > On Fri, Jan 27, 2023 at 10:27:29PM -0800, Nathan Bossart wrote: > > Here is a work-in-progress patch set for adjusting the archive modules > > interface. Is this roughly what you had in mind? > > I have been catching up with what is

Re: recovery modules

2023-01-30 Thread Nathan Bossart
On Mon, Jan 30, 2023 at 04:51:38PM +0900, Michael Paquier wrote: > Now, I find this part, where we use a double pointer to allow the > module initialization to create and give back a private area, rather > confusing, and I think that this could be bug-prone, as well. Once > you incorporate some

Re: recovery modules

2023-01-29 Thread Michael Paquier
On Fri, Jan 27, 2023 at 10:27:29PM -0800, Nathan Bossart wrote: > Here is a work-in-progress patch set for adjusting the archive modules > interface. Is this roughly what you had in mind? I have been catching up with what is happening here. I can get behind the idea to use the term "callbacks"

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 08:17:46PM -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 05:55:42PM -0800, Andres Freund wrote: >> On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: >>> I think it would be weird for the archive module and >>> recovery module interfaces to look so different, but

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 05:55:42PM -0800, Andres Freund wrote: > On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: >> I think it would be weird for the archive module and >> recovery module interfaces to look so different, but if that's okay, I can >> change it. > > I'm a bit sad about the

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: > >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, > >> + const char > >>

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, >> + const char >> *lastRestartPointFileName); >> +typedef void (*RecoveryArchiveCleanupCB)

Re: recovery modules

2023-01-27 Thread Michael Paquier
s picture with the checkpointer or a different process. There are a bunch of deployments where WAL archive retention is controlled by the age of the backups, not by the backend deciding when these should be removed as a checkpoint runs depending on the computed redo LSN, so recovery modules wo

Re: recovery modules

2023-01-27 Thread Andres Freund
e defined by recovery libraries and > + * returned via _PG_recovery_module_init(). For more information about the > + * purpose of each callback, refer to the recovery modules documentation. > + */ > +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, > +

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 15:28:21 -0800, Nathan Bossart wrote: > The more I think about this, the more I wonder whether we really need to > include archive_cleanup_command and recovery_end_command in recovery > modules. I think it would be hard to write a good module that isn't just imp

Re: recovery modules

2023-01-27 Thread Michael Paquier
On Thu, Jan 26, 2023 at 09:40:58PM -0800, Nathan Bossart wrote: > Yeah, there are some problems here. If we ERROR, we'll just bounce back to > the sigsetjmp() block once a second, and we'll never pick up configuration > reloads, shutdown signals, etc. If we FATAL, we'll just rapidly restart >

Re: recovery modules

2023-01-27 Thread Nathan Bossart
> aformentioned problems, this approach might be more suitable for the > checkpointer than it is for the archiver. The more I think about this, the more I wonder whether we really need to include archive_cleanup_command and recovery_end_command in recovery modules. Another weird thing wi

Re: recovery modules

2023-01-26 Thread Nathan Bossart
On Wed, Jan 25, 2023 at 04:34:21PM +0900, Michael Paquier wrote: > The loop part is annoying.. I've never been a fan of adding this > cross-value checks for the archiver modules in the first place, and it > would make things much simpler in the checkpointer if we need to think > about that as we

Re: recovery modules

2023-01-24 Thread Michael Paquier
could it be better to have two different shutdown callbacks for the checkpointer and the startup process? Having some tests for both, like shell_archive.c, would be nice, actually. -- Michael From 7d4425744e2bc96418d6ef85c9408bed9e91aa9b Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Wed, 25 Jan 20

Re: recovery modules

2023-01-23 Thread Nathan Bossart
On Mon, Jan 23, 2023 at 11:44:57AM +0900, Michael Paquier wrote: > Thanks for the rebase. Thanks for the detailed review. > The final state of the documentation is as follows: > 51. Archive and Recovery Modules > 51.1. Archive Module Initialization Functions > 51.2.

Re: recovery modules

2023-01-22 Thread Michael Paquier
On Tue, Jan 17, 2023 at 08:44:27PM -0800, Nathan Bossart wrote: > Thanks. Here's a rebased version of the last patch. Thanks for the rebase. The final state of the documentation is as follows: 51. Archive and Recovery Modules 51.1. Archive Module Initialization Functions 51.2. Arch

Re: recovery modules

2023-01-17 Thread Nathan Bossart
rchive'"); +$restore->append_conf('postgresql.conf', "basic_archive.archive_directory = '$archive_dir'"); +$restore->start; + +# ensure post-backup WAL was replayed +my $result = $restore->safe_psql("postgres", "SELECT count(*) FROM test;"); +is($result

Re: recovery modules

2023-01-17 Thread Michael Paquier
On Tue, Jan 17, 2023 at 10:23:56AM -0800, Nathan Bossart wrote: > Yeah, this seems cleaner. I removed BuildRestoreCommand() in v8. if (*sp == *lp) { - if (val) - { - appendStringInfoString(, val); - found = true; - } -

Re: recovery modules

2023-01-17 Thread Nathan Bossart
ql('postgres', "SELECT pg_switch_wal();"); +$node->safe_psql('postgres', "INSERT INTO test VALUES (1);"); + +# shut down the node (this should archive all WAL files) +$node->stop; + +# restore from the backup +my $restore = PostgreSQL::Test::Cluster->new('restore'); +$res

Re: recovery modules

2023-01-16 Thread Michael Paquier
On Mon, Jan 16, 2023 at 02:40:40PM -0800, Nathan Bossart wrote: > On Mon, Jan 16, 2023 at 04:36:01PM +0900, Michael Paquier wrote: > > Once this issue was fixed, nothing else stood out, so applied this > > part. > > Thanks! I've attached a rebased version of the rest of the patch set. When it

Re: recovery modules

2023-01-16 Thread Nathan Bossart
st::Cluster->new('restore'); +$restore->init_from_backup($node, $backup, has_restoring => 1, standby => 0); +$restore->append_conf('postgresql.conf', "restore_command = ''"); +$restore->append_conf('postgresql.conf', "restore_library = 'basic_archive'"); +

Re: recovery modules

2023-01-15 Thread Michael Paquier
On Thu, Jan 12, 2023 at 10:17:21AM -0800, Nathan Bossart wrote: > On Thu, Jan 12, 2023 at 03:30:40PM +0900, Michael Paquier wrote: > IMHO I don't think there's an urgent need to rename it, but if there's a > better name that people like, I'm happy to do so. Okay. >> Saying that, 0001 seems fine

Re: recovery modules

2023-01-12 Thread Nathan Bossart
gt;start; + +# backup the node +my $backup = 'backup'; +$node->backup($backup); + +# generate some new WAL files +$node->safe_psql('postgres', "CREATE TABLE test (a INT);"); +$node->safe_psql('postgres', "SELECT pg_switch_wal();"); +$node->safe_psql('postgres', "

Re: recovery modules

2023-01-11 Thread Michael Paquier
On Wed, Jan 11, 2023 at 11:29:01AM -0800, Nathan Bossart wrote: > I'm having trouble thinking of any practical advantage of providing the > redo LSN and TLI. If the main use-case is removing older archives as the > documentation indicates, it seems better to provide the file name so that > you

Re: recovery modules

2023-01-11 Thread Nathan Bossart
ABLE test (a INT);"); +$node->safe_psql('postgres', "SELECT pg_switch_wal();"); +$node->safe_psql('postgres', "INSERT INTO test VALUES (1);"); + +# shut down the node (this should archive all WAL files) +$node->stop; + +# restore from the backup +my $r

Re: recovery modules

2023-01-10 Thread Michael Paquier
On Tue, Jan 03, 2023 at 11:05:38AM -0800, Nathan Bossart wrote: > I noticed that cfbot's Windows tests are failing because the backslashes in > the archive directory path are causing escaping problems. Here is an > attempt to fix that by converting all backslashes to forward slashes, which > is

Re: recovery modules

2023-01-03 Thread Nathan Bossart
gres', "SELECT pg_switch_wal();"); +$node->safe_psql('postgres', "INSERT INTO test VALUES (1);"); + +# shut down the node (this should archive all WAL files) +$node->stop; + +# restore from the backup +my $restore = PostgreSQL::Test::Cluster->new('restore'); +$re

Re: recovery modules

2023-01-03 Thread Nathan Bossart
de->backup($backup); + +# generate some new WAL files +$node->safe_psql('postgres', "CREATE TABLE test (a INT);"); +$node->safe_psql('postgres', "SELECT pg_switch_wal();"); +$node->safe_psql('postgres', "INSERT INTO test VALUES (1);"); + +# shut down the node (this

Re: recovery modules

2022-12-29 Thread Nathan Bossart
ql('postgres', "CREATE TABLE test (a INT);"); +$node->safe_psql('postgres', "SELECT pg_switch_wal();"); +$node->safe_psql('postgres', "INSERT INTO test VALUES (1);"); + +# shut down the node (this should archive all WAL files) +$node->stop; + +# restore fr

Re: recovery modules

2022-12-27 Thread Andres Freund
Hi, On 2022-12-27 15:04:28 -0800, Nathan Bossart wrote: > I'm sorry, I'm still lost here. Wouldn't restoration via library tend to > improve latency? Is your point that clusters may end up depending on this > improvement so much that a shell command would no longer be able to keep > up? Yes.

Re: recovery modules

2022-12-27 Thread Michael Paquier
On Tue, Dec 27, 2022 at 02:11:11PM -0800, Andres Freund wrote: > On 2022-12-27 11:24:49 -0800, Nathan Bossart wrote: >> * Unlike archive modules, recovery libraries cannot be changed at runtime. >> There isn't a safe way to unload a library, and archive libraries work >> around this restriction by

Re: recovery modules

2022-12-27 Thread Nathan Bossart
On Tue, Dec 27, 2022 at 02:45:30PM -0800, Andres Freund wrote: > On 2022-12-27 14:37:11 -0800, Nathan Bossart wrote: >> On Tue, Dec 27, 2022 at 02:11:11PM -0800, Andres Freund wrote: >> > On 2022-12-27 11:24:49 -0800, Nathan Bossart wrote: >> >> * pg_rewind uses restore_command, but there isn't a

Re: recovery modules

2022-12-27 Thread Andres Freund
Hi, On 2022-12-27 14:37:11 -0800, Nathan Bossart wrote: > On Tue, Dec 27, 2022 at 02:11:11PM -0800, Andres Freund wrote: > > On 2022-12-27 11:24:49 -0800, Nathan Bossart wrote: > >> I've attached a patch set that adds the restore_library, > >> archive_cleanup_library, and recovery_end_library

Re: recovery modules

2022-12-27 Thread Nathan Bossart
On Tue, Dec 27, 2022 at 02:11:11PM -0800, Andres Freund wrote: > On 2022-12-27 11:24:49 -0800, Nathan Bossart wrote: >> I've attached a patch set that adds the restore_library, >> archive_cleanup_library, and recovery_end_library parameters to allow >> archive recovery via loadable modules. This

Re: recovery modules

2022-12-27 Thread Andres Freund
Hi, On 2022-12-27 11:24:49 -0800, Nathan Bossart wrote: > I've attached a patch set that adds the restore_library, > archive_cleanup_library, and recovery_end_library parameters to allow > archive recovery via loadable modules. This is a follow-up to the > archive_library parameter added in v15

recovery modules

2022-12-27 Thread Nathan Bossart
overy Modules" chapter. They are similar enough that it felt silly to write a separate chapter for recovery modules. However, I've still split them up within the chapter, and they have separate initialization functions. This retains backward compatibility with v15 archive modules, keeps them