Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-20 Thread Stephen Frost
Greetings, * Martín Marqués (martin.marq...@gmail.com) wrote: > The typo is in `exist in in a running cluster`. There's two `in` in a row. Oops, thanks for catching (and thanks to Michael for committing the fix!). > P.D.: I was looking at this just because I was looking at an issue > where someo

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-19 Thread Michael Paquier
On Tue, Apr 19, 2022 at 10:12:32PM -0300, Martín Marqués wrote: > The typo is in `exist in in a running cluster`. There's two `in` in a row. Thanks, fixed. -- Michael signature.asc Description: PGP signature

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-19 Thread Martín Marqués
Hi, I was looking at the commit for this patch and noticed this small typo in the comment in `basebackup.c`: ``` diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 6884cad2c00af1632eacec07903098e7e1874393..815681ada7dd0c135af584ad5da2dd13c9a12465 10064

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-06 Thread Nathan Bossart
On Wed, Apr 06, 2022 at 03:29:15PM -0400, Stephen Frost wrote: > This has now been committed- thanks again to everyone for their help! Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-06 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Tue, 2022-04-05 at 13:06 -0700, Nathan Bossart wrote: > > On Tue, Apr 05, 2022 at 11:25:36AM -0400, Stephen Frost wrote: > > > Please find attached an updated patch + commit message.  Mostly, I just > > > went through and did a bit m

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-06 Thread Laurenz Albe
On Tue, 2022-04-05 at 13:06 -0700, Nathan Bossart wrote: > On Tue, Apr 05, 2022 at 11:25:36AM -0400, Stephen Frost wrote: > > Please find attached an updated patch + commit message.  Mostly, I just > > went through and did a bit more in terms of updating the documentation > > and improving the comm

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread Nathan Bossart
On Tue, Apr 05, 2022 at 11:25:36AM -0400, Stephen Frost wrote: > Please find attached an updated patch + commit message. Mostly, I just > went through and did a bit more in terms of updating the documentation > and improving the comments (there were some places that were still > worrying about the

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread Magnus Hagander
On Tue, Apr 5, 2022 at 5:25 PM Stephen Frost wrote: > Greetings, > > * David Steele (da...@pgmasters.net) wrote: > > On 4/4/22 11:42 AM, Nathan Bossart wrote: > > >I noticed a couple of other things that can be removed. Since we no > longer > > >wait on exclusive backup mode during smart shutdow

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread David Steele
On 4/5/22 11:25 AM, Stephen Frost wrote: Please find attached an updated patch + commit message. Mostly, I just went through and did a bit more in terms of updating the documentation and improving the comments (there were some places that were still worrying about the chance of a 'stray' backup

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-05 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 4/4/22 11:42 AM, Nathan Bossart wrote: > >I noticed a couple of other things that can be removed. Since we no longer > >wait on exclusive backup mode during smart shutdown, we can change > >connsAllowed (in postmaster.c) to a boolean and

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-04 Thread David Steele
On 4/4/22 11:42 AM, Nathan Bossart wrote: I noticed a couple of other things that can be removed. Since we no longer wait on exclusive backup mode during smart shutdown, we can change connsAllowed (in postmaster.c) to a boolean and remove CAC_SUPERUSER. We can also remove a couple of related no

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-04 Thread Nathan Bossart
I noticed a couple of other things that can be removed. Since we no longer wait on exclusive backup mode during smart shutdown, we can change connsAllowed (in postmaster.c) to a boolean and remove CAC_SUPERUSER. We can also remove a couple of related notes in the documentation. I've done all thi

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-04 Thread Nathan Bossart
On Mon, Apr 04, 2022 at 09:56:26AM -0400, David Steele wrote: > Minor typo in the docs: > > + * capable of doing an online backup, but exclude then just in case. > > Should be: > > capable of doing an online backup, but exclude them just in case. fixed -- Nathan Bossart Amazon Web Servic

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-04-04 Thread David Steele
On 3/28/22 10:09 PM, Nathan Bossart wrote: On Mon, Mar 28, 2022 at 04:30:27PM -0400, Stephen Frost wrote: On a once-over of the rest of the code, I definitely like how much we're able to simplify things in this area and remove various hacks in things like pg_basebackup and pg_rewind where we pr

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-28 Thread Nathan Bossart
On Mon, Mar 28, 2022 at 04:30:27PM -0400, Stephen Frost wrote: >> - By default, pg_start_backup can take a long time >> to finish. >> + By default, pg_backup_start can take a long time >> to finish. >> This is because it performs a checkpoint, and the I/O >> required for the c

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-28 Thread Stephen Frost
Greetings, * Nathan Bossart (nathandboss...@gmail.com) wrote: > On Thu, Mar 10, 2022 at 07:13:14PM -0500, Chapman Flack wrote: > > Looks like this change to an example in func.sgml is not quite right: > > > > -postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); > > +postgres=# SELE

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-11 Thread Nathan Bossart
On Fri, Mar 11, 2022 at 02:00:37PM -0500, Chapman Flack wrote: > v7 looks good to me. I have repeated the installcheck-world and given > the changed documentation sections one last read-through, and will > change the status to RfC. Thanks for reviewing! -- Nathan Bossart Amazon Web Services: htt

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-11 Thread Chapman Flack
On 03/10/22 19:38, Nathan Bossart wrote: > On Thu, Mar 10, 2022 at 07:13:14PM -0500, Chapman Flack wrote: >> +postgres=# SELECT * FROM pg_walfile_name_offset((pg_backup_stop()).lsn); > > Ah, good catch. I made this change in v7. I considered doing something v7 looks good to me. I have repeated

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-10 Thread Nathan Bossart
On Thu, Mar 10, 2022 at 07:13:14PM -0500, Chapman Flack wrote: > Looks like this change to an example in func.sgml is not quite right: > > -postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); > +postgres=# SELECT * FROM pg_walfile_name_offset(pg_backup_stop()); > > pg_backup_stop r

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-10 Thread Chapman Flack
On 03/09/22 19:06, Nathan Bossart wrote: > Done. I went ahead and added "label => 'label'" for consistency. Looks like this change to an example in func.sgml is not quite right: -postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); +postgres=# SELECT * FROM pg_walfile_name_offset(p

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread David Steele
On 3/9/22 18:06, Nathan Bossart wrote: On Wed, Mar 09, 2022 at 06:11:19PM -0500, Chapman Flack wrote: I think the listitem In the same connection as before, issue the command: SELECT * FROM pg_backup_stop(true); would be clearer if it used named-parameter form, wait_for_archive => true.

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Nathan Bossart
On Wed, Mar 09, 2022 at 06:11:19PM -0500, Chapman Flack wrote: > I think the listitem > > In the same connection as before, issue the command: > SELECT * FROM pg_backup_stop(true); > > would be clearer if it used named-parameter form, wait_for_archive => true. > > This is not strictly necess

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Chapman Flack
On 03/09/22 17:21, Nathan Bossart wrote: > Great. Is there any additional feedback on this patch? Should we add an > example of using pg_basebackup in the "Standalone Hot Backups" section, or > should we leave all documentation additions like this for Chap's new > thread? I'm composing something

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Nathan Bossart
On Wed, Mar 09, 2022 at 02:32:24PM -0500, Chapman Flack wrote: > On 03/09/22 12:19, Stephen Frost wrote: >> Let's avoid hijacking this thread, which is about this >> patch, for an independent debate about what our documentation should or >> shouldn't include. > > Agreed. New thread here: > > http

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Chapman Flack
On 03/09/22 12:19, Stephen Frost wrote: > Let's avoid hijacking this thread, which is about this > patch, for an independent debate about what our documentation should or > shouldn't include. Agreed. New thread here: https://www.postgresql.org/message-id/6228FFE4.3050309%40anastigmatix.net Regar

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Stephen Frost
Greetings, * Chapman Flack (c...@anastigmatix.net) wrote: > On 03/09/22 11:22, Magnus Hagander wrote: > >> It's more than just too confusing, it's actively bad because people will > >> actually use it and then end up with backups that don't work. > > > > +1. > > > > Or even worse, backups that s

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Chapman Flack
On 03/09/22 11:22, Magnus Hagander wrote: >> It's more than just too confusing, it's actively bad because people will >> actually use it and then end up with backups that don't work. > > +1. > > Or even worse, backups that sometimes work, but not reliably and not > every time. > ... > Pretending

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Magnus Hagander
On Wed, Mar 9, 2022 at 4:42 PM Stephen Frost wrote: > > Greetings, > > * Chapman Flack (c...@anastigmatix.net) wrote: > > On 03/08/22 17:12, Nathan Bossart wrote: > > > I spent some time trying to come up with a workable script to replace the > > > existing one. I think the main problem is that y

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-09 Thread Stephen Frost
Greetings, * Chapman Flack (c...@anastigmatix.net) wrote: > On 03/08/22 17:12, Nathan Bossart wrote: > > I spent some time trying to come up with a workable script to replace the > > existing one. I think the main problem is that you need to write out both > > the backup label file and the tables

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-08 Thread Chapman Flack
On 03/08/22 17:12, Nathan Bossart wrote: > I spent some time trying to come up with a workable script to replace the > existing one. I think the main problem is that you need to write out both > the backup label file and the tablespace map file, but I didn't find an > easy way to write the differe

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-08 Thread Nathan Bossart
On Tue, Mar 08, 2022 at 03:09:50PM -0600, David Steele wrote: > On 3/8/22 14:01, Nathan Bossart wrote: >> On Wed, Mar 02, 2022 at 02:23:51PM -0500, Chapman Flack wrote: >> > I did not notice this earlier (sorry), but there seems to remain in >> > backup.sgml a programlisting example that shows a ps

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-08 Thread David Steele
On 3/8/22 14:01, Nathan Bossart wrote: On Wed, Mar 02, 2022 at 02:23:51PM -0500, Chapman Flack wrote: I did not notice this earlier (sorry), but there seems to remain in backup.sgml a programlisting example that shows a psql invocation for pg_backup_start, then a tar command, then another psql i

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-08 Thread Nathan Bossart
On Wed, Mar 02, 2022 at 02:23:51PM -0500, Chapman Flack wrote: > I did not notice this earlier (sorry), but there seems to remain in > backup.sgml a programlisting example that shows a psql invocation > for pg_backup_start, then a tar command, then another psql invocation > for pg_backup_stop. > >

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-02 Thread Chapman Flack
On 03/01/22 20:03, Nathan Bossart wrote: > Here is a new version of the patch with the following changes: I did not notice this earlier (sorry), but there seems to remain in backup.sgml a programlisting example that shows a psql invocation for pg_backup_start, then a tar command, then another psql

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Nathan Bossart
Here is a new version of the patch with the following changes: 1. Addressed Chap's feedback from upthread. 2. Renamed pg_start/stop_backup() to pg_backup_start/stop() as suggested by David. 3. A couple of other small documentation adjustments. -- Nathan Bossart

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Stephen Frost
Greetings, * Chapman Flack (c...@anastigmatix.net) wrote: > On 03/01/22 14:14, Stephen Frost wrote: > >> There can't really be many teams out there thinking "we'll just ignore > >> these scripts forever, and nothing bad will happen." They all know they'll > >> have to do stuff sometimes. But it ma

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Chapman Flack
On 03/01/22 14:14, Stephen Frost wrote: >> There can't really be many teams out there thinking "we'll just ignore >> these scripts forever, and nothing bad will happen." They all know they'll >> have to do stuff sometimes. But it matters how we allow them to schedule it. > > We only make these cha

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Stephen Frost
Greetings, * Chapman Flack (c...@anastigmatix.net) wrote: > On 03/01/22 13:22, David Steele wrote: > > I think people are going to complain no matter what. If scripts are being > > maintained changing the name is not a big deal (though moving from exclusive > > to non-exclusive may be). If they ar

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Stephen Frost
Greetings, * David Steele (da...@pgmasters.net) wrote: > On 3/1/22 11:32, Nathan Bossart wrote: > >On Tue, Mar 01, 2022 at 11:09:13AM -0500, Chapman Flack wrote: > >>On 03/01/22 09:44, David Steele wrote: > >>>Personally, I am in favor of removing it. We change/rename > >>>functions/tables/views w

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Chapman Flack
On 03/01/22 13:22, David Steele wrote: > I think people are going to complain no matter what. If scripts are being > maintained changing the name is not a big deal (though moving from exclusive > to non-exclusive may be). If they aren't being maintained then they'll just > blow up a few versions do

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Chapman Flack
On 03/01/22 12:32, Nathan Bossart wrote: > On Tue, Mar 01, 2022 at 11:09:13AM -0500, Chapman Flack wrote: >> That way, at least, there would be a period of time where procedures >> that currently work (by passing exclusive => false) would continue to work, >> and could be adapted as time permits by

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread David Steele
On 3/1/22 11:32, Nathan Bossart wrote: On Tue, Mar 01, 2022 at 11:09:13AM -0500, Chapman Flack wrote: On 03/01/22 09:44, David Steele wrote: Personally, I am in favor of removing it. We change/rename functions/tables/views when we need to, and this happens in almost every release. For clarifi

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Nathan Bossart
On Tue, Mar 01, 2022 at 11:09:13AM -0500, Chapman Flack wrote: > On 03/01/22 09:44, David Steele wrote: >> Personally, I am in favor of removing it. We change/rename >> functions/tables/views when we need to, and this happens in almost every >> release. > > For clarification, is that a suggestion

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Stephen Frost
Greetings, * Nathan Bossart (nathandboss...@gmail.com) wrote: > On Tue, Mar 01, 2022 at 08:44:51AM -0600, David Steele wrote: > > Personally, I am in favor of removing it. We change/rename > > functions/tables/views when we need to, and this happens in almost every > > release. > > > > What we ne

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Chapman Flack
On 03/01/22 09:44, David Steele wrote: > Personally, I am in favor of removing it. We change/rename > functions/tables/views when we need to, and this happens in almost every > release. For clarification, is that a suggestion to remove the 'exclusive' parameter in some later release, after using t

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread Nathan Bossart
On Tue, Mar 01, 2022 at 08:44:51AM -0600, David Steele wrote: > Personally, I am in favor of removing it. We change/rename > functions/tables/views when we need to, and this happens in almost every > release. > > What we need to do is make sure that an older installation won't silently > work in a

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-03-01 Thread David Steele
On 2/28/22 23:51, Nathan Bossart wrote: On Sat, Feb 26, 2022 at 02:06:14PM -0800, Nathan Bossart wrote: On Sat, Feb 26, 2022 at 04:48:52PM +, Chapman Flack wrote: Assuming the value is false, so no error is thrown, is it practical to determine from flinfo->fn_expr whether the value was defa

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-28 Thread Nathan Bossart
On Sat, Feb 26, 2022 at 02:06:14PM -0800, Nathan Bossart wrote: > On Sat, Feb 26, 2022 at 04:48:52PM +, Chapman Flack wrote: >> Assuming the value is false, so no error is thrown, is it practical to >> determine >> from flinfo->fn_expr whether the value was defaulted or supplied? If so, I >>

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-26 Thread Nathan Bossart
On Sat, Feb 26, 2022 at 05:03:04PM -0500, Chapman Flack wrote: > I've now looked through the rest, and the only further thing I noticed > was that xlog.c's do_pg_start_backup still has a tablespaces parameter > to receive a List* of tablespaces if the caller wants, but this patch > removes the comm

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-26 Thread Nathan Bossart
On Sat, Feb 26, 2022 at 04:48:52PM +, Chapman Flack wrote: > My biggest concerns are the changes to the SQL-visible pg_start_backup > and pg_stop_backup functions. When the non-exclusive API was implemented > (in 7117685), that was done with care (with a new optional argument to > pg_start_back

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-26 Thread Chapman Flack
On 02/26/22 11:48, Chapman Flack wrote: > This patch applies cleanly for me and passes installcheck-world. > I have not yet studied all of the changes in detail. I've now looked through the rest, and the only further thing I noticed was that xlog.c's do_pg_start_backup still has a tablespaces para

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-26 Thread Chapman Flack
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested This patch applies cleanly for me and passes installcheck-world. I ha

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-21 Thread Nathan Bossart
I've attached an updated patch. On Fri, Feb 18, 2022 at 10:48:10PM +0530, Ashutosh Sharma wrote: > +* runningBackups is a counter indicating the number of backups currently > in > +* progress. forcePageWrites is set to true when either of these is > +* non-zero. lastBackupStart is the

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-18 Thread Ashutosh Sharma
On Sat, Feb 19, 2022 at 2:24 AM Nathan Bossart wrote: > > On Fri, Feb 18, 2022 at 10:48:10PM +0530, Ashutosh Sharma wrote: > > Some review comments on the latest version: > > Thanks for the feedback! Before I start spending more time on this one, I > should probably ask if this has any chance of

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-18 Thread Nathan Bossart
On Fri, Feb 18, 2022 at 10:48:10PM +0530, Ashutosh Sharma wrote: > Some review comments on the latest version: Thanks for the feedback! Before I start spending more time on this one, I should probably ask if this has any chance of making it into v15. -- Nathan Bossart Amazon Web Services: https

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-18 Thread Ashutosh Sharma
Some review comments on the latest version: +* runningBackups is a counter indicating the number of backups currently in +* progress. forcePageWrites is set to true when either of these is +* non-zero. lastBackupStart is the latest checkpoint redo location used as +* a starting poi

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-02-16 Thread Nathan Bossart
Here is a rebased patch. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 8fc744a0b22a1fff9f3470bfdc0e08d9dd5da5be Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 1 Dec 2021 23:50:49 + Subject: [PATCH v2 1/1] remove exclusive backup mode --- doc/src/sgml/backup.

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-07 Thread Bossart, Nathan
On 1/7/22, 5:52 AM, "David Steele" wrote: > On 1/6/22 20:20, Euler Taveira wrote: >> On Thu, Jan 6, 2022, at 9:48 PM, Bossart, Nathan wrote: >>> After a quick glance, I didn't see an easy way to hold a session open >>> while the test does other things. If there isn't one, modifying >>> backup_fs_

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-07 Thread David Steele
On 1/6/22 20:20, Euler Taveira wrote: On Thu, Jan 6, 2022, at 9:48 PM, Bossart, Nathan wrote: After a quick glance, I didn't see an easy way to hold a session open while the test does other things.  If there isn't one, modifying backup_fs_hot() to work with non-exclusive mode might be more troub

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-06 Thread Euler Taveira
On Thu, Jan 6, 2022, at 9:48 PM, Bossart, Nathan wrote: > After a quick glance, I didn't see an easy way to hold a session open > while the test does other things. If there isn't one, modifying > backup_fs_hot() to work with non-exclusive mode might be more trouble > than it is worth. You can use

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2022-01-06 Thread Bossart, Nathan
On 12/2/21, 1:34 PM, "Bossart, Nathan" wrote: > On 12/2/21, 9:50 AM, "David Steele" wrote: >> On 12/2/21 12:38, David Steele wrote: >>> On 12/2/21 11:00, Andrew Dunstan wrote: Should we really be getting rid of PostgreSQL::Test::Cluster::backup_fs_hot() ? >>> >>> Agreed, it would b

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-02 Thread Bossart, Nathan
On 12/2/21, 9:50 AM, "David Steele" wrote: > On 12/2/21 12:38, David Steele wrote: >> On 12/2/21 11:00, Andrew Dunstan wrote: >>> >>> Should we really be getting rid of >>> PostgreSQL::Test::Cluster::backup_fs_hot() ? >> >> Agreed, it would be better to update backup_fs_hot() to use exclusive >> m

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-02 Thread David Steele
On 12/2/21 12:38, David Steele wrote: On 12/2/21 11:00, Andrew Dunstan wrote: Should we really be getting rid of PostgreSQL::Test::Cluster::backup_fs_hot() ? Agreed, it would be better to update backup_fs_hot() to use exclusive mode and save out backup_label instead. Oops, of course I mean

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-02 Thread David Steele
On 12/2/21 11:00, Andrew Dunstan wrote: On 12/1/21 19:30, Bossart, Nathan wrote: On 12/1/21, 10:37 AM, "Bossart, Nathan" wrote: On 12/1/21, 8:27 AM, "David Steele" wrote: On 11/30/21 18:31, Bossart, Nathan wrote: Do you think it's still worth trying to make it safe, or do you think we shou

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-02 Thread Andrew Dunstan
On 12/1/21 19:30, Bossart, Nathan wrote: > On 12/1/21, 10:37 AM, "Bossart, Nathan" wrote: >> On 12/1/21, 8:27 AM, "David Steele" wrote: >>> On 11/30/21 18:31, Bossart, Nathan wrote: Do you think it's still worth trying to make it safe, or do you think we should just remove exclusive m

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-01 Thread Bossart, Nathan
On 12/1/21, 8:27 AM, "David Steele" wrote: > On 11/30/21 18:31, Bossart, Nathan wrote: >> Do you think it's still worth trying to make it safe, or do you think >> we should just remove exclusive mode completely? > > My preference would be to remove it completely, but I haven't gotten a > lot of tr

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-01 Thread David Steele
On 11/30/21 18:31, Bossart, Nathan wrote: On 11/30/21, 2:58 PM, "David Steele" wrote: I did figure out how to keep the safe part of exclusive backup (not having to maintain a connection) while removing the dangerous part (writing backup_label into PGDATA), but it was a substantial amount of wor

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-01 Thread Andrew Dunstan
On 11/30/21 17:26, Tom Lane wrote: > "Bossart, Nathan" writes: >> It looks like the exclusive way has been marked deprecated in all >> supported versions along with a note that it will eventually be >> removed. If it's not going to be removed out of fear of breaking >> backward compatibility, I

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-12-01 Thread David Steele
On 11/30/21 19:54, Michael Paquier wrote: On Tue, Nov 30, 2021 at 05:58:15PM -0500, David Steele wrote: I did figure out how to keep the safe part of exclusive backup (not having to maintain a connection) while removing the dangerous part (writing backup_label into PGDATA), but it was a substant

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread SATYANARAYANA NARLAPURAM
On Tue, Nov 30, 2021 at 4:54 PM Michael Paquier wrote: > On Tue, Nov 30, 2021 at 05:58:15PM -0500, David Steele wrote: > > The main objections as I recall are that it is much harder for simple > backup > > scripts and commercial backup integrations to hold a connection to > postgres > > open and

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Michael Paquier
On Tue, Nov 30, 2021 at 05:58:15PM -0500, David Steele wrote: > The main objections as I recall are that it is much harder for simple backup > scripts and commercial backup integrations to hold a connection to postgres > open and write the backup label separately into the backup. I don't quite und

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 2:58 PM, "David Steele" wrote: > I did figure out how to keep the safe part of exclusive backup (not > having to maintain a connection) while removing the dangerous part > (writing backup_label into PGDATA), but it was a substantial amount of > work and I felt that it had little chanc

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread David Steele
On 11/30/21 17:26, Tom Lane wrote: "Bossart, Nathan" writes: It looks like the exclusive way has been marked deprecated in all supported versions along with a note that it will eventually be removed. If it's not going to be removed out of fear of breaking backward compatibility, I think the do

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 2:27 PM, "Tom Lane" wrote: > If we're willing to outright remove it, I don't have any great objection. > My original two cents was that we shouldn't put effort into improving it; > but removing it isn't that. I might try to put a patch together for the January commitfest, given there

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Tom Lane
"Bossart, Nathan" writes: > It looks like the exclusive way has been marked deprecated in all > supported versions along with a note that it will eventually be > removed. If it's not going to be removed out of fear of breaking > backward compatibility, I think the documentation should be updated

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Bossart, Nathan
On 11/30/21, 9:51 AM, "Stephen Frost" wrote: > I disagree that that’s a satisfactory approach. It certainly wasn’t > intended or documented as part of the original feature and therefore > to call it satisfactory strikes me quite strongly as revisionist > history. It looks like the exclusive way

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Stephen Frost
Greetings, On Tue, Nov 30, 2021 at 11:47 Laurenz Albe wrote: > On Tue, 2021-11-30 at 09:20 -0500, Stephen Frost wrote: > > Greetings, > > > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > > Michael Paquier writes: > > > > On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM > wrote:

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Laurenz Albe
On Tue, 2021-11-30 at 09:20 -0500, Stephen Frost wrote: > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > Michael Paquier writes: > > > On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: > > > > If we are keeping it then why not make it better? > > > > > Well, no

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-30 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Michael Paquier writes: > > On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: > >> If we are keeping it then why not make it better? > > > Well, non-exclusive backups are better by design in many aspects, so I > > don't

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-29 Thread Bossart, Nathan
On 11/26/21, 7:33 AM, "Tom Lane" wrote: > Michael Paquier writes: >> On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: >>> If we are keeping it then why not make it better? > >> Well, non-exclusive backups are better by design in many aspects, so I >> don't quite see the p

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-26 Thread Tom Lane
Michael Paquier writes: > On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: >> If we are keeping it then why not make it better? > Well, non-exclusive backups are better by design in many aspects, so I > don't quite see the point in spending time on something that has more

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-25 Thread Michael Paquier
On Thu, Nov 25, 2021 at 06:19:03PM -0800, SATYANARAYANA NARLAPURAM wrote: > Is there a plan in place to remove the exclusive backup option from the > core in PG 15/16? This was discussed, but removing it could also harm users relying on it. Perhaps it could be revisited, but I am not sure if this

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-25 Thread SATYANARAYANA NARLAPURAM
Thanks Michael! This is a known issue with exclusive backups, which is a reason why > non-exclusive backups have been implemented. pg_basebackup does that, > and using "false" as the third argument of pg_start_backup() would > have the same effect. So I would recommend to switch to that. > Is t

Re: Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-24 Thread Michael Paquier
On Wed, Nov 24, 2021 at 02:12:19PM -0800, SATYANARAYANA NARLAPURAM wrote: > While an exclusive backup is in progress if Postgres restarts, postgres > runs the recovery from the checkpoint identified by the label file instead > of the control file. This can cause long recovery or even sometimes fail

Postgres restart in the middle of exclusive backup and the presence of backup_label file

2021-11-24 Thread SATYANARAYANA NARLAPURAM
Hi Hackers, While an exclusive backup is in progress if Postgres restarts, postgres runs the recovery from the checkpoint identified by the label file instead of the control file. This can cause long recovery or even sometimes fail to recover as the WAL records corresponding to that checkpoint loc