Re: Replication failure, slave requesting old segments

2018-08-13 Thread Phil Endecott
Adrian Klaver wrote: "If you set up a WAL archive that's accessible from the standby, these solutions are not required, since the standby can always use the archive to catch up provided it retains enough segments. *This is dependent on verification that the archiving is working properly. A belt

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Stephen Frost
Greetings, * Adrian Klaver (adrian.kla...@aklaver.com) wrote: > On 08/13/2018 05:08 AM, Phil Endecott wrote: > >Adrian Klaver wrote: > >Really?  I thought the intention was that the system should be > >able to recover reliably when the slave reconnects after a > >period of downtime, subject only t

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Adrian Klaver
On 08/13/2018 05:08 AM, Phil Endecott wrote: Adrian Klaver wrote: On 08/12/2018 02:56 PM, Phil Endecott wrote: Anyway.  Do others agree that my issue was the result of wal_keep_segments=0 ? Only as a sub-issue of the slave losing contact with the master. The basic problem is maintaining two

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Stephen Frost
Greetings, * Adrian Klaver (adrian.kla...@aklaver.com) wrote: > On 08/13/2018 05:39 AM, Stephen Frost wrote: > >* Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > >>Adrian Klaver wrote: > >>>On 08/12/2018 02:56 PM, Phil Endecott wrote: > Anyway.  Do others agree that my issue was th

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Adrian Klaver
On 08/13/2018 05:39 AM, Stephen Frost wrote: Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: Adrian Klaver wrote: On 08/12/2018 02:56 PM, Phil Endecott wrote: Anyway.  Do others agree that my issue was the result of wal_keep_segments=0 ? Only as a sub-issue of the sla

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Stephen Frost
Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > Adrian Klaver wrote: > >On 08/12/2018 02:56 PM, Phil Endecott wrote: > >>Anyway.  Do others agree that my issue was the result of > >>wal_keep_segments=0 ? > > > >Only as a sub-issue of the slave losing contact with the maste

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Stephen Frost
Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > Adrian Klaver wrote: > >On 08/12/2018 03:54 PM, Stephen Frost wrote: > >>Greetings, > >> > >>* Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > >>>OK. I think this is perhaps a documentation bug, maybe a missing >

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Phil Endecott
Adrian Klaver wrote: On 08/12/2018 03:54 PM, Stephen Frost wrote: Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: OK. I think this is perhaps a documentation bug, maybe a missing warning when the master reads its configuration, and maybe (as you say) a bad default value

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Phil Endecott
Adrian Klaver wrote: On 08/12/2018 02:56 PM, Phil Endecott wrote: Anyway.  Do others agree that my issue was the result of wal_keep_segments=0 ? Only as a sub-issue of the slave losing contact with the master. The basic problem is maintaining two separate operations, archiving and streaming,

Re: Replication failure, slave requesting old segments

2018-08-13 Thread Stephen Frost
Greetings, * Adrian Klaver (adrian.kla...@aklaver.com) wrote: > On 08/12/2018 03:54 PM, Stephen Frost wrote: > >* Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > >>OK. I think this is perhaps a documentation bug, maybe a missing > >>warning when the master reads its configuration, and

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Adrian Klaver
On 08/12/2018 03:54 PM, Stephen Frost wrote: Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: OK. I think this is perhaps a documentation bug, maybe a missing warning when the master reads its configuration, and maybe (as you say) a bad default value. If we consider it

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Adrian Klaver
On 08/12/2018 02:56 PM, Phil Endecott wrote: Stephen Frost wrote: Specifically, section 26.2.5 of the docs says: "If you use streaming replication without file-based continuous archiving, the server might recycle old WAL segments before the standby has received them. If this occurs, the stan

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Stephen Frost
Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > OK. I think this is perhaps a documentation bug, maybe a missing > warning when the master reads its configuration, and maybe (as you say) > a bad default value. If we consider it to be an issue worthy of a change then we

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Phil Endecott
Stephen Frost wrote: * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: Stephen Frost wrote: >* Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: >>2018-08-11 00:12:15.536 UTC [7954] LOG: restored log file "0001000700D0" from archive >>2018-08-11 00:12:15.797 UTC [

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Stephen Frost
Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > Stephen Frost wrote: > >* Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > >>archive_command = 'ssh backup test ! -f backup/postgresql/archivedir/%f && > >> scp %p backup:backup/postgresql/archive

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Adrian Klaver
On 08/12/2018 12:53 PM, Phil Endecott wrote: Phil Endecott wrote: On the master, I have: wal_level = replica archive_mode = on archive_command = 'ssh backup test ! -f backup/postgresql/archivedir/%f && scp %p backup:backup/postgresql/archivedir/%f' On the slave I have: sta

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Adrian Klaver
On 08/12/2018 12:25 PM, Phil Endecott wrote: Hi Adrian, Adrian Klaver wrote: On 08/11/2018 12:42 PM, Phil Endecott wrote: Hi Adrian, Adrian Klaver wrote: Looks like the master recycled the WAL's while the slave could not connect. Yes but... why is that a problem?  The master is copying the

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Phil Endecott
Phil Endecott wrote: > On the master, I have: > > wal_level = replica > archive_mode = on > archive_command = 'ssh backup test ! -f backup/postgresql/archivedir/%f && >scp %p backup:backup/postgresql/archivedir/%f' > > On the slave I have: > > standby_mode = 'on' > primary_conni

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Phil Endecott
Hi Adrian, Adrian Klaver wrote: On 08/11/2018 12:42 PM, Phil Endecott wrote: Hi Adrian, Adrian Klaver wrote: Looks like the master recycled the WAL's while the slave could not connect. Yes but... why is that a problem?  The master is copying the WALs to the backup server using scp, where th

Re: Replication failure, slave requesting old segments

2018-08-12 Thread Phil Endecott
Hi Stephen, Stephen Frost wrote: * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: archive_command = 'ssh backup test ! -f backup/postgresql/archivedir/%f && scp %p backup:backup/postgresql/archivedir/%f' This is really not a sufficient or particularly intelligent

Re: Replication failure, slave requesting old segments

2018-08-11 Thread Stephen Frost
Greetings, * Phil Endecott (spam_from_pgsql_li...@chezphil.org) wrote: > Dear Experts, Since you're asking ... > I recently set up replication for the first time. It seemed to be > working OK in my initial tests, but didn't cope when the slave was > down for a longer period. This is all with

Re: Replication failure, slave requesting old segments

2018-08-11 Thread Adrian Klaver
On 08/11/2018 12:42 PM, Phil Endecott wrote: Hi Adrian, Adrian Klaver wrote: Looks like the master recycled the WAL's while the slave could not connect. Yes but... why is that a problem?  The master is copying the WALs to the backup server using scp, where they remain forever.  The slave gets

Re: Replication failure, slave requesting old segments

2018-08-11 Thread Phil Endecott
Hi Adrian, Adrian Klaver wrote: Looks like the master recycled the WAL's while the slave could not connect. Yes but... why is that a problem? The master is copying the WALs to the backup server using scp, where they remain forever. The slave gets them from there before it starts streaming.

Re: Replication failure, slave requesting old segments

2018-08-11 Thread Adrian Klaver
On 08/11/2018 08:18 AM, Phil Endecott wrote: Dear Experts, I recently set up replication for the first time. It seemed to be working OK in my initial tests, but didn't cope when the slave was down for a longer period. This is all with the Debian stable packages of PostgreSQL 9.6. My replicati