Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Mon, Apr 25, 2016 at 9:11 AM, Fujii Masao wrote: > PostgreSQL resource agent for Pacemaker removes backup_label in > the case of failover to prevent the standby server from failing > to recover from the crash. Yikes! I hope that the providers of Pacemaker document

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 4/24/16 11:49 AM, Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: >> On Sun, Apr 24, 2016 at 5:37 AM, Bruce Momjian wrote: >> >>> On Fri, Apr 22, 2016 at 11:53:46AM -0400, Robert Haas wrote: >>> Frankly, I think that's right. It is one thing to say

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Mon, Apr 25, 2016 at 4:11 PM, Fujii Masao wrote: > On Thu, Apr 21, 2016 at 2:32 AM, Magnus Hagander > wrote: > > On Wed, Apr 20, 2016 at 1:12 AM, Fujii Masao > wrote: > >> > >> On Sun, Apr 17, 2016 at 1:22 AM, Magnus

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Thu, Apr 21, 2016 at 2:32 AM, Magnus Hagander wrote: > On Wed, Apr 20, 2016 at 1:12 AM, Fujii Masao wrote: >> >> On Sun, Apr 17, 2016 at 1:22 AM, Magnus Hagander >> wrote: >> > On Wed, Apr 13, 2016 at 4:07 AM, Noah Misch

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 24 April 2016 at 23:49, Stephen Frost wrote: > > Fixing that means using something more complicated than the old method > and that's a bit of a pain in psql, but that doesn't mean we should tell > people that the old method is an acceptable approach. > +1 Frankly, I

Re: [HACKERS] Updated backup APIs for non-exclusive backups

* Magnus Hagander (mag...@hagander.net) wrote: > On Sun, Apr 24, 2016 at 5:37 AM, Bruce Momjian wrote: > > > On Fri, Apr 22, 2016 at 11:53:46AM -0400, Robert Haas wrote: > > > On Wed, Apr 20, 2016 at 1:32 PM, Magnus Hagander > > wrote: > > > > Note that we

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Sun, Apr 24, 2016 at 5:37 AM, Bruce Momjian wrote: > On Fri, Apr 22, 2016 at 11:53:46AM -0400, Robert Haas wrote: > > On Wed, Apr 20, 2016 at 1:32 PM, Magnus Hagander > wrote: > > > Note that we have not marked them as deprecated. We're just giving >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Fri, Apr 22, 2016 at 11:53:46AM -0400, Robert Haas wrote: > On Wed, Apr 20, 2016 at 1:32 PM, Magnus Hagander wrote: > > Note that we have not marked them as deprecated. We're just giving warnings > > that they will be deprecated. > > But I think that is being said here is

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Apr 20, 2016 at 1:32 PM, Magnus Hagander wrote: > Note that we have not marked them as deprecated. We're just giving warnings > that they will be deprecated. But I think that is being said here is that maybe they won't be deprecated, at least not any time soon. And

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Apr 20, 2016 at 1:12 AM, Fujii Masao wrote: > On Sun, Apr 17, 2016 at 1:22 AM, Magnus Hagander > wrote: > > On Wed, Apr 13, 2016 at 4:07 AM, Noah Misch wrote: > >> > >> On Tue, Apr 12, 2016 at 10:08:23PM +0200, Magnus

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Sun, Apr 17, 2016 at 1:22 AM, Magnus Hagander wrote: > On Wed, Apr 13, 2016 at 4:07 AM, Noah Misch wrote: >> >> On Tue, Apr 12, 2016 at 10:08:23PM +0200, Magnus Hagander wrote: >> > On Tue, Apr 12, 2016 at 8:39 AM, Noah Misch wrote:

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Sat, Apr 16, 2016 at 06:22:47PM +0200, Magnus Hagander wrote: > > On Tue, Apr 12, 2016 at 10:08:23PM +0200, Magnus Hagander wrote: > > > I won't have time to do the bigger rewrite/reordeirng by then, but I can > > > certainly commit to having the smaller updates done to cover the new > > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Apr 13, 2016 at 4:07 AM, Noah Misch wrote: > On Tue, Apr 12, 2016 at 10:08:23PM +0200, Magnus Hagander wrote: > > On Tue, Apr 12, 2016 at 8:39 AM, Noah Misch wrote: > > > On Mon, Apr 11, 2016 at 11:22:27AM +0200, Magnus Hagander wrote: > > > > Well,

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Apr 12, 2016 at 10:08:23PM +0200, Magnus Hagander wrote: > On Tue, Apr 12, 2016 at 8:39 AM, Noah Misch wrote: > > On Mon, Apr 11, 2016 at 11:22:27AM +0200, Magnus Hagander wrote: > > > Well, if we *don't* do the rewrite before we release it, then we have to > > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Apr 12, 2016 at 8:39 AM, Noah Misch wrote: > On Mon, Apr 11, 2016 at 11:22:27AM +0200, Magnus Hagander wrote: > > On Thu, Apr 7, 2016 at 5:15 AM, Noah Misch wrote: > > > Unless you especially want to self-impose the same tight resolution > > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Mon, Apr 11, 2016 at 11:22:27AM +0200, Magnus Hagander wrote: > On Thu, Apr 7, 2016 at 5:15 AM, Noah Misch wrote: > > Unless you especially want to self-impose the same tight resolution > > schedule > > that 9.6 regressions experience, let's move this to the "Non-bugs"

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Mon, Apr 11, 2016 at 5:22 AM, Magnus Hagander wrote: > Well, if we *don't* do the rewrite before we release it, then we have to > instead put information about the new version of the functions into the old > structure I think. I think that you should have done that in the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Thu, Apr 7, 2016 at 5:15 AM, Noah Misch wrote: > On Wed, Apr 06, 2016 at 09:17:22AM +0200, Magnus Hagander wrote: > > On Wed, Apr 6, 2016 at 6:42 AM, Noah Misch wrote: > > > On Tue, Apr 05, 2016 at 08:15:16PM +0200, Magnus Hagander wrote: > > > > I've

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Apr 06, 2016 at 09:17:22AM +0200, Magnus Hagander wrote: > On Wed, Apr 6, 2016 at 6:42 AM, Noah Misch wrote: > > On Tue, Apr 05, 2016 at 08:15:16PM +0200, Magnus Hagander wrote: > > > I've pushed this version, and also added the item from the Brussels > > > developer

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Apr 6, 2016 at 6:42 AM, Noah Misch wrote: > On Tue, Apr 05, 2016 at 08:15:16PM +0200, Magnus Hagander wrote: > > I've pushed this version, and also added the item from the Brussels > > developer meeting to actually rewrite the main backup docs to the open > > items so

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 6 April 2016 at 12:42, Noah Misch wrote: > > The chapter already does describe pg_basebackup before describing > pg_start_backup; what else did the plan entail? > http://www.postgresql.org/docs/current/static/backup.html 24.1. SQL Dump 24.1.1. Restoring the Dump 24.1.2.

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Apr 05, 2016 at 08:15:16PM +0200, Magnus Hagander wrote: > I've pushed this version, and also added the item from the Brussels > developer meeting to actually rewrite the main backup docs to the open > items so they are definitely not forgotten for 9.6. Here's that PostgreSQL 9.6 open

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Apr 5, 2016 at 5:57 PM, Amit Kapila wrote: > On Tue, Apr 5, 2016 at 5:35 PM, Magnus Hagander > wrote: > >> >> >> On Mon, Apr 4, 2016 at 3:15 PM, Amit Kapila >> wrote: >> >>> On Mon, Apr 4, 2016 at 4:31 PM, Magnus

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Apr 5, 2016 at 5:35 PM, Magnus Hagander wrote: > > > On Mon, Apr 4, 2016 at 3:15 PM, Amit Kapila > wrote: > >> On Mon, Apr 4, 2016 at 4:31 PM, Magnus Hagander >> wrote: >> >>> On Fri, Apr 1, 2016 at 6:47 AM, Amit Kapila

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 4/5/16 8:05 AM, Magnus Hagander wrote: PFA a better one (I think), also with the rename and added comments that David was asking for. Barring objections, I will apply this version. This version looks good to me. -- -David da...@pgmasters.net -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Mon, Apr 4, 2016 at 3:15 PM, Amit Kapila wrote: > On Mon, Apr 4, 2016 at 4:31 PM, Magnus Hagander > wrote: > >> On Fri, Apr 1, 2016 at 6:47 AM, Amit Kapila >> wrote: >> >> >>> Also, I think below part of documentation

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Mon, Apr 4, 2016 at 4:31 PM, Magnus Hagander wrote: > On Fri, Apr 1, 2016 at 6:47 AM, Amit Kapila > wrote: > > >> Also, I think below part of documentation for pg_start_backup() needs to >> be modified: >> >> >> >> pg_start_backup accepts an

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 3/19/16 8:15 AM, Magnus Hagander wrote: I've attached an updated patch, which is rebased on current master and includes the oid fix. I've now had a chance to go through this in detail and test thoroughly. We had a lot of discussion about copying pg_control last and that led me to

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Fri, Apr 1, 2016 at 6:47 AM, Amit Kapila wrote: > On Sat, Mar 19, 2016 at 5:45 PM, Magnus Hagander > wrote: > >> On Wed, Mar 2, 2016 at 6:49 PM, Marco Nenciarini < >> marco.nenciar...@2ndquadrant.it> wrote: >> >>> >>> I've attached an updated

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Sat, Mar 19, 2016 at 5:45 PM, Magnus Hagander wrote: > On Wed, Mar 2, 2016 at 6:49 PM, Marco Nenciarini < > marco.nenciar...@2ndquadrant.it> wrote: > >> >> I've attached an updated patch, which is rebased on current master and > includes the oid fix. > > + Finish

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Thu, Mar 31, 2016 at 5:49 PM, Magnus Hagander wrote: > On Thu, Mar 31, 2016 at 4:00 AM, David Steele wrote: > So maybe we should at least start that way. And just document that > clearly, and then use the patch that we have right now? > > Except we

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Thu, Mar 31, 2016 at 2:19 PM, Magnus Hagander wrote: > On Thu, Mar 31, 2016 at 4:00 AM, David Steele wrote: > >> On 3/30/16 4:18 AM, Magnus Hagander wrote: >> > >> > On Wed, Mar 30, 2016 at 4:10 AM, David Steele > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Thu, Mar 31, 2016 at 4:00 AM, David Steele wrote: > On 3/30/16 4:18 AM, Magnus Hagander wrote: > > > > On Wed, Mar 30, 2016 at 4:10 AM, David Steele > > wrote: > > > > This certainly looks like it would work but it

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 3/30/16 4:18 AM, Magnus Hagander wrote: > > On Wed, Mar 30, 2016 at 4:10 AM, David Steele > wrote: > > This certainly looks like it would work but it raises the barrier for > implementing backups by quite a lot. It's fine for

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Mar 30, 2016 at 10:09 AM, Amit Kapila wrote: > On Tue, Mar 29, 2016 at 11:39 PM, Magnus Hagander > wrote: > >> On Tue, Mar 29, 2016 at 6:40 PM, Magnus Hagander >> wrote: >> >>> So - I can definitely see the argument for

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Mar 30, 2016 at 4:10 AM, David Steele wrote: > On 3/29/16 2:09 PM, Magnus Hagander wrote: > > > I had a chat with Heikki, and here's another suggestion: > > > > 1. We don't touch the current exclusive backups at all, as previously > > discussed, other than

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Mar 29, 2016 at 11:39 PM, Magnus Hagander wrote: > On Tue, Mar 29, 2016 at 6:40 PM, Magnus Hagander > wrote: > >> So - I can definitely see the argument for returning the stop wal >> *location*. But I'm still not sure what the definition of the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 3/29/16 2:09 PM, Magnus Hagander wrote: > I had a chat with Heikki, and here's another suggestion: > > 1. We don't touch the current exclusive backups at all, as previously > discussed, other than deprecating their use. For backwards compat. > > 2. For new backups, we return the contents of

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Mar 29, 2016 at 6:40 PM, Magnus Hagander wrote: > > > On Tue, Mar 29, 2016 at 4:36 PM, David Steele wrote: > >> On 3/22/16 12:31 PM, Magnus Hagander wrote: >> >> On Tue, Mar 22, 2016 at 5:27 PM, David Steele >>

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Mar 29, 2016 at 4:36 PM, David Steele wrote: > On 3/22/16 12:31 PM, Magnus Hagander wrote: > > On Tue, Mar 22, 2016 at 5:27 PM, David Steele > > wrote: >> > > > >> > Adding the stop time column should be a simple

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 3/22/16 12:31 PM, Magnus Hagander wrote: On Tue, Mar 22, 2016 at 5:27 PM, David Steele > wrote: > > Adding the stop time column should be a simple addition and I don't see > a problem with that. I think I misunderstood your original

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 23 March 2016 at 00:14, Magnus Hagander wrote: > Doing it in the backup label file is obviously a different target, where > we might need to consider backwards compatibility, Should we? > As part of the failover slots a few folks at 2ndQ looked into whether tools would

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Mar 22, 2016 at 5:27 PM, David Steele wrote: > On 3/22/16 12:14 PM, Magnus Hagander wrote: > > On Tue, Mar 22, 2016 at 5:08 PM, David Steele > > wrote: > > > > On 3/19/16 8:15 AM, Magnus Hagander wrote: > > > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 3/22/16 12:14 PM, Magnus Hagander wrote: > On Tue, Mar 22, 2016 at 5:08 PM, David Steele > wrote: > > On 3/19/16 8:15 AM, Magnus Hagander wrote: > > > I've attached an updated patch, which is rebased on current master and > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Tue, Mar 22, 2016 at 5:08 PM, David Steele wrote: > > On 3/19/16 8:15 AM, Magnus Hagander wrote: > > > I've attached an updated patch, which is rebased on current master and > > includes the oid fix. > > Before doing a thorough review of this patch there are a few points

Re: [HACKERS] Updated backup APIs for non-exclusive backups

Hi Magnus, On 3/19/16 8:15 AM, Magnus Hagander wrote: > I've attached an updated patch, which is rebased on current master and > includes the oid fix. Before doing a thorough review of this patch there are a few points I would like to consider: * I think it's really important to provide the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Mar 2, 2016 at 6:49 PM, Marco Nenciarini < marco.nenciar...@2ndquadrant.it> wrote: > Hi Magnus, > Hi! First, again my apologies for completely missing that you had posted this review! > I've finally found some time to take a look to the patch. > > It applies with some fuzziness on

Re: [HACKERS] Updated backup APIs for non-exclusive backups

Hi Magnus, On 3/2/16 12:49 PM, Marco Nenciarini wrote: > I've finally found some time to take a look to the patch. > > It applies with some fuzziness on master, but the result looks correct. > Unfortunately the OID of the new pg_stop_backup function conflicts with > "pg_blocking_pids()" patch

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Mar 16, 2016 at 5:06 PM, David Steele wrote: > Hi Magnus, > > On 3/2/16 12:49 PM, Marco Nenciarini wrote: > > > I've finally found some time to take a look to the patch. > > > > It applies with some fuzziness on master, but the result looks correct. > > Unfortunately

Re: [HACKERS] Updated backup APIs for non-exclusive backups

Hi Magnus, I've finally found some time to take a look to the patch. It applies with some fuzziness on master, but the result looks correct. Unfortunately the OID of the new pg_stop_backup function conflicts with "pg_blocking_pids()" patch (52f5d578d6c29bf254e93c69043b817d4047ca67). After

Re: [HACKERS] Updated backup APIs for non-exclusive backups

Hi Magnus, thanks for working on this topic. What it does is very similar to what Barman's pgespresso extension does and I'd like to see it implemented soon in the core. I've added myself as reviewer for the patch on commitfest site. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy

Re: [HACKERS] Updated backup APIs for non-exclusive backups

* David Steele (da...@pgmasters.net) wrote: > On 2/10/16 9:44 AM, Stephen Frost wrote: > > Hrmmm. If that's the case then perhaps you're right. I liked the > > general idea of not having to maintain a TCP connection during the > > entire backup (TCP connections can be annoyingly finicky in

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Feb 10, 2016 at 4:38 PM, David Steele wrote: > On 2/10/16 7:46 AM, Magnus Hagander wrote: > > Per discussion at the developer meeting in Brussels, here's a patch that > > makes some updates to the backup APIs, to support non-exclusive backups > > without using

Re: [HACKERS] Updated backup APIs for non-exclusive backups

Hi, On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > Per discussionat the developer meeting in Brussels, here's a patch that > makes some updates to the backup APIs, to support non-exclusive backups > without using pg_basebackup. Thanks for following through with this! > * If the client

Re: [HACKERS] Updated backup APIs for non-exclusive backups

* Andres Freund (and...@anarazel.de) wrote: > On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > > * If the client disconnects with a non-exclusive backup running, the backup > > is automatically aborted. This is the same thing that pg_basebackup does. > > To use these non-exclusive backups

Re: [HACKERS] Updated backup APIs for non-exclusive backups

* Magnus Hagander (mag...@hagander.net) wrote: > On Wed, Feb 10, 2016 at 2:46 PM, Andres Freund wrote: > > On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > > > Per discussionat the developer meeting in Brussels, here's a patch that > > > makes some updates to the backup

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2/10/16 7:46 AM, Magnus Hagander wrote: > Per discussion at the developer meeting in Brussels, here's a patch that > makes some updates to the backup APIs, to support non-exclusive backups > without using pg_basebackup. <...> This sounds like a great idea and I have signed up to review. > * A

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Feb 10, 2016 at 2:46 PM, Andres Freund wrote: > Hi, > > On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: > > Per discussionat the developer meeting in Brussels, here's a patch that > > makes some updates to the backup APIs, to support non-exclusive backups > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2/10/16 9:44 AM, Stephen Frost wrote: > * Magnus Hagander (mag...@hagander.net) wrote: >> On Wed, Feb 10, 2016 at 2:46 PM, Andres Freund wrote: >>> On 2016-02-10 13:46:05 +0100, Magnus Hagander wrote: * If the client disconnects with a non-exclusive backup running, the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2/10/16 11:01 AM, Andres Freund wrote: > On 2016-02-10 16:50:26 +0100, Magnus Hagander wrote: >>> I would be happy to see the time-stamp returned from the >>> pg_start_backup() function as well. It's a bigger change, but once >>> pg_start_backup() returns multiple columns it will be easier to

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2016-02-10 11:06:01 -0500, David Steele wrote: > That makes sense. The backup_label "as is" could be output at the > beginning but if we want to add the minimum recovery point it would need > to be output at the end. > > It seems like tablespace_map could still be output at the beginning,

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2016-02-10 16:50:26 +0100, Magnus Hagander wrote: > > I would be happy to see the time-stamp returned from the > > pg_start_backup() function as well. It's a bigger change, but once > > pg_start_backup() returns multiple columns it will be easier to add more > > columns in the future. > > > >

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2/10/16 10:50 AM, Magnus Hagander wrote: > On Wed, Feb 10, 2016 at 4:38 PM, David Steele > This information is handy for automating selection of a backup when > doing time-based PITR (or so the user can manually select). For > exclusive backups it is possible

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On 2/10/16 11:12 AM, Andres Freund wrote: > On 2016-02-10 11:06:01 -0500, David Steele wrote: >> That makes sense. The backup_label "as is" could be output at the >> beginning but if we want to add the minimum recovery point it would need >> to be output at the end. >> >> It seems like

[HACKERS] Updated backup APIs for non-exclusive backups

Per discussionat the developer meeting in Brussels, here's a patch that makes some updates to the backup APIs, to support non-exclusive backups without using pg_basebackup. The idea is to fix at least three main issues that are there today -- that you cannot run concurrent backups, that the

Re: [HACKERS] Updated backup APIs for non-exclusive backups

On Wed, Feb 10, 2016 at 7:46 AM, Magnus Hagander wrote: > Per discussionat the developer meeting in Brussels, here's a patch that > makes some updates to the backup APIs, to support non-exclusive backups > without using pg_basebackup. The idea is to fix at least three main