Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread David Fetter
On Mon, Oct 06, 2014 at 07:24:32PM +0300, Heikki Linnakangas wrote: > On 10/06/2014 07:00 PM, Gabriele Bartolini wrote: > >Hello, > > > >2014-10-06 17:51 GMT+02:00 Marco Nenciarini >>: > > > >>I agree that a full backup does not need to include a profile. > >> > >>I've added the option to require

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Heikki Linnakangas
On 10/06/2014 07:00 PM, Gabriele Bartolini wrote: Hello, 2014-10-06 17:51 GMT+02:00 Marco Nenciarini : I agree that a full backup does not need to include a profile. I've added the option to require the profile even for a full backup, as it can be useful for backup softwares. We could remov

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 12:18 PM, Marco Nenciarini wrote: >> - Ship a table-of-contents file with a list relation files currently >> present and the length of each in blocks. > > Having the size in bytes allow you to use the same format for non-block > files. Am I missing any advantage of having th

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 12:06 PM, Marco Nenciarini wrote: > Il 06/10/14 17:55, Robert Haas ha scritto: >> On Mon, Oct 6, 2014 at 11:51 AM, Marco Nenciarini >> wrote: >>> I agree that a full backup does not need to include a profile. >>> >>> I've added the option to require the profile even for a f

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Heikki Linnakangas
On 10/06/2014 07:06 PM, Marco Nenciarini wrote: Il 06/10/14 17:55, Robert Haas ha scritto: On Mon, Oct 6, 2014 at 11:51 AM, Marco Nenciarini wrote: I agree that a full backup does not need to include a profile. I've added the option to require the profile even for a full backup, as it can be

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Marco Nenciarini
Il 06/10/14 17:50, Robert Haas ha scritto: > On Mon, Oct 6, 2014 at 11:33 AM, Marco Nenciarini > wrote: >>> 2. Take a differential backup. In the backup label file, note the LSN >>> of the fullback to which the differential backup is relative, and the >>> newest LSN guaranteed to be present in th

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Heikki Linnakangas
On 10/06/2014 06:33 PM, Marco Nenciarini wrote: Il 03/10/14 22:47, Robert Haas ha scritto: 2. Take a differential backup. In the backup label file, note the LSN of the fullback to which the differential backup is relative, and the newest LSN guaranteed to be present in the differential backup.

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Marco Nenciarini
Il 06/10/14 17:55, Robert Haas ha scritto: > On Mon, Oct 6, 2014 at 11:51 AM, Marco Nenciarini > wrote: >> I agree that a full backup does not need to include a profile. >> >> I've added the option to require the profile even for a full backup, as >> it can be useful for backup softwares. We could

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Gabriele Bartolini
Hello, 2014-10-06 17:51 GMT+02:00 Marco Nenciarini : > I agree that a full backup does not need to include a profile. > > I've added the option to require the profile even for a full backup, as > it can be useful for backup softwares. We could remove the option and > build the profile only during

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 11:51 AM, Marco Nenciarini wrote: > I agree that a full backup does not need to include a profile. > > I've added the option to require the profile even for a full backup, as > it can be useful for backup softwares. We could remove the option and > build the profile only dur

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Marco Nenciarini
Il 06/10/14 16:51, Robert Haas ha scritto: > On Mon, Oct 6, 2014 at 8:59 AM, Marco Nenciarini > wrote: >> Il 04/10/14 08:35, Michael Paquier ha scritto: >>> On Sat, Oct 4, 2014 at 12:31 AM, Marco Nenciarini >>> wrote: Compared to first version, we switched from a timestamp+checksum based >>>

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 11:33 AM, Marco Nenciarini wrote: >> 1. Take a full backup. Basically, we already have this. In the >> backup label file, make sure to note the newest LSN guaranteed to be >> present in the backup. > > Don't we already have it in "START WAL LOCATION"? Yeah, probably. I w

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Marco Nenciarini
Il 03/10/14 22:47, Robert Haas ha scritto: > On Fri, Oct 3, 2014 at 12:08 PM, Marco Nenciarini > wrote: >> Il 03/10/14 17:53, Heikki Linnakangas ha scritto: >>> If we're going to need a profile file - and I'm not convinced of that - >>> is there any reason to not always include it in the backup? >

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Robert Haas
On Mon, Oct 6, 2014 at 8:59 AM, Marco Nenciarini wrote: > Il 04/10/14 08:35, Michael Paquier ha scritto: >> On Sat, Oct 4, 2014 at 12:31 AM, Marco Nenciarini >> wrote: >>> Compared to first version, we switched from a timestamp+checksum based >>> approach to one based on LSN. >> Cool. >> >>> This

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Marco Nenciarini
Il 04/10/14 08:35, Michael Paquier ha scritto: > On Sat, Oct 4, 2014 at 12:31 AM, Marco Nenciarini > wrote: >> Compared to first version, we switched from a timestamp+checksum based >> approach to one based on LSN. > Cool. > >> This patch adds an option to pg_basebackup and to replication protoco

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-06 Thread Marco Nenciarini
Il 03/10/14 23:12, Andres Freund ha scritto: > On 2014-10-03 17:31:45 +0200, Marco Nenciarini wrote: >> I've updated the wiki page >> https://wiki.postgresql.org/wiki/Incremental_backup following the result >> of discussion on hackers. >> >> Compared to first version, we switched from a timestamp+c

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Michael Paquier
On Sat, Oct 4, 2014 at 12:31 AM, Marco Nenciarini wrote: > Compared to first version, we switched from a timestamp+checksum based > approach to one based on LSN. Cool. > This patch adds an option to pg_basebackup and to replication protocol > BASE_BACKUP command to generate a backup_profile file.

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Andres Freund
On 2014-10-03 17:31:45 +0200, Marco Nenciarini wrote: > I've updated the wiki page > https://wiki.postgresql.org/wiki/Incremental_backup following the result > of discussion on hackers. > > Compared to first version, we switched from a timestamp+checksum based > approach to one based on LSN. > >

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Robert Haas
On Fri, Oct 3, 2014 at 12:08 PM, Marco Nenciarini wrote: > Il 03/10/14 17:53, Heikki Linnakangas ha scritto: >> If we're going to need a profile file - and I'm not convinced of that - >> is there any reason to not always include it in the backup? > > The main reason is to have a centralized list o

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Bruce Momjian
On Fri, Oct 3, 2014 at 06:08:47PM +0200, Marco Nenciarini wrote: > >> Any comment will be appreciated. In particular I'd appreciate comments > >> on correctness of relnode files detection and LSN extraction code. > > > > I didn't look at it in detail, but one future problem comes to mind: > > Onc

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Claudio Freire
On Fri, Oct 3, 2014 at 1:08 PM, Marco Nenciarini wrote: >>> Any comment will be appreciated. In particular I'd appreciate comments >>> on correctness of relnode files detection and LSN extraction code. >> >> I didn't look at it in detail, but one future problem comes to mind: >> Once you implement

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Marco Nenciarini
Il 03/10/14 17:53, Heikki Linnakangas ha scritto: > If we're going to need a profile file - and I'm not convinced of that - > is there any reason to not always include it in the backup? > The main reason is to have a centralized list of files that need to be present. Without a profile, you have t

Re: [HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Heikki Linnakangas
On 10/03/2014 06:31 PM, Marco Nenciarini wrote: Hi Hackers, I've updated the wiki page https://wiki.postgresql.org/wiki/Incremental_backup following the result of discussion on hackers. Compared to first version, we switched from a timestamp+checksum based approach to one based on LSN. This pa

[HACKERS] [RFC] Incremental backup v2: add backup profile to base backup

2014-10-03 Thread Marco Nenciarini
Hi Hackers, I've updated the wiki page https://wiki.postgresql.org/wiki/Incremental_backup following the result of discussion on hackers. Compared to first version, we switched from a timestamp+checksum based approach to one based on LSN. This patch adds an option to pg_basebackup and to replica