Re: [HACKERS] On markers of changed data

2017-10-15 Thread Andrey Borodin
Hello! > 9 окт. 2017 г., в 10:23, Andrey Borodin написал(а): > > Thanks, Stephen, this actually pointed what to look for > VM is WAL-logged [0] > FSM is not [1] > > [0] >

Re: [HACKERS] On markers of changed data

2017-10-12 Thread Robert Haas
On Fri, Oct 6, 2017 at 10:34 AM, Michael Paquier wrote: > On Fri, Oct 6, 2017 at 11:22 PM, Tom Lane wrote: >> I'd say no: >> >> 1. You don't know the granularity of the filesystem's timestamps, at least >> not without making unportable assumptions.

Re: [HACKERS] On markers of changed data

2017-10-11 Thread Greg Stark
On 10 October 2017 at 23:50, Stephen Frost wrote: > Yeah, it sounds interesting, but I was just chatting w/ David about it > and we were thinking about how checkpoints are really rather often done, > so you end up with quite a few of these lists being out there. > > Now, if

Re: [HACKERS] On markers of changed data

2017-10-10 Thread Stephen Frost
Alvaro, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > Greg Stark wrote: > > > The general shape of what I would like to see is some log which lists > > where each checkpoint starts and ends and what blocks are modified > > since the previous checkpoint. Then to generate an incremental

Re: [HACKERS] On markers of changed data

2017-10-10 Thread Alvaro Herrera
Greg Stark wrote: > The general shape of what I would like to see is some log which lists > where each checkpoint starts and ends and what blocks are modified > since the previous checkpoint. Then to generate an incremental backup > from any point in time to the current you union all the block

Re: [HACKERS] On markers of changed data

2017-10-10 Thread Greg Stark
On 8 October 2017 at 08:52, Andrey Borodin wrote: > > 1. Any other marker would be better (It can be WAL scan during archiving, > some new LSN-based mechanics* et c.) The general shape of what I would like to see is some log which lists where each checkpoint starts and

Re: [HACKERS] On markers of changed data

2017-10-09 Thread Peter Geoghegan
On Sat, Oct 7, 2017 at 6:34 AM, Stephen Frost wrote: >> > That’s actually what pg_rman is doing for what it calls incremental >> > backups (perhaps that would be differential backup in PG >> > terminology?), and the performance is bad as you can imagine. We could >> > have a

Re: [HACKERS] On markers of changed data

2017-10-09 Thread Andrey Borodin
Hi Michael! > 9 окт. 2017 г., в 17:28, Michael Paquier > написал(а): >> VM is WAL-logged [0] >> FSM is not [1] > > If you are willing to go down this road, here are my takes on the matter: > - Any LSN map should be in a different file than FSM and VM. The VM > uses 2

Re: [HACKERS] On markers of changed data

2017-10-09 Thread Michael Paquier
On Mon, Oct 9, 2017 at 2:23 PM, Andrey Borodin wrote: >> I haven't gone and audited it myself, but I would certainly expect you >> to be able to use the LSN for everything which is WAL'd. If you have >> cases where that's not the case, it'd be useful to see them. > >

Re: [HACKERS] On markers of changed data

2017-10-08 Thread Andrey Borodin
> 8 окт. 2017 г., в 20:11, Stephen Frost написал(а): > * Andrey Borodin (x4...@yandex-team.ru) wrote: >> But my other question still seems unanswered: can I use LSN logic for >> incrementing FSM and VM? Seems like most of the time there is valid LSN > > I haven't gone and

Re: [HACKERS] On markers of changed data

2017-10-08 Thread Stephen Frost
Andrey, * Andrey Borodin (x4...@yandex-team.ru) wrote: > But my other question still seems unanswered: can I use LSN logic for > incrementing FSM and VM? Seems like most of the time there is valid LSN I haven't gone and audited it myself, but I would certainly expect you to be able to use the

Re: [HACKERS] On markers of changed data

2017-10-08 Thread Andrey Borodin
Tom, Alvaro, Michael, and especially Septhen, thank you for your valuable comments. I feel enlightened about mtime. My takeaway is: 1. Any other marker would be better (It can be WAL scan during archiving, some new LSN-based mechanics* et c.) 2. mtime could be used, with precautions described

Re: [HACKERS] On markers of changed data

2017-10-07 Thread Stephen Frost
Alvaro, Michael, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > Michael Paquier wrote: > > That’s actually what pg_rman is doing for what it calls incremental > > backups (perhaps that would be differential backup in PG > > terminology?), and the performance is bad as you can imagine. We

Re: [HACKERS] On markers of changed data

2017-10-07 Thread Alvaro Herrera
Michael Paquier wrote: > That’s actually what pg_rman is doing for what it calls incremental > backups (perhaps that would be differential backup in PG > terminology?), and the performance is bad as you can imagine. We could > have a dedicated LSN map to do such things with 4 bytes per page. I am

Re: [HACKERS] On markers of changed data

2017-10-07 Thread Michael Paquier
> Le 6 oct. 2017 à 23:44, Alvaro Herrera a écrit : > > Michael Paquier wrote: > >> The only sane method for Postgres is really to scan the >> page header LSNs, and of course you already know that. > > I hope the idea is not to have to scan every single page in the >

Re: [HACKERS] On markers of changed data

2017-10-06 Thread Stephen Frost
Tom, Michael, * Michael Paquier (michael.paqu...@gmail.com) wrote: > On Fri, Oct 6, 2017 at 11:22 PM, Tom Lane wrote: > > Andrey Borodin writes: > >> Is it safe to use file modification time to track that file were changes > >> since previous backup? >

Re: [HACKERS] On markers of changed data

2017-10-06 Thread Alvaro Herrera
Michael Paquier wrote: > The only sane method for Postgres is really to scan the > page header LSNs, and of course you already know that. I hope the idea is not to have to scan every single page in the database, because that would be too slow. It should be possible to build this so that a

Re: [HACKERS] On markers of changed data

2017-10-06 Thread Michael Paquier
On Fri, Oct 6, 2017 at 11:22 PM, Tom Lane wrote: > Andrey Borodin writes: >> Is it safe to use file modification time to track that file were changes >> since previous backup? > > I'd say no: > > 1. You don't know the granularity of the filesystem's

Re: [HACKERS] On markers of changed data

2017-10-06 Thread Tom Lane
Andrey Borodin writes: > Is it safe to use file modification time to track that file were changes > since previous backup? I'd say no: 1. You don't know the granularity of the filesystem's timestamps, at least not without making unportable assumptions. 2. There's no

[HACKERS] On markers of changed data

2017-10-06 Thread Andrey Borodin
Hi, hackers! Currently I'm working on page-level incremental backups using WAL-G codebase[0]. And I have two questions that I cannot resolve myself. Incremental backup is a set of changes, that should be applied over preexisting backup. I use page LSN to understand should page be backup`ed or