Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-29 Thread Nick B
Greetings, I also would like to thank everyone for looking into this. On Sat, Jan 26, 2019 at 01:45:46PM +0100, Magnus Hagander wrote: > One workaround you could perhaps look at here is to run pg_basebackup > with --no-sync. That way there will be no fsyncs issued while running. You > will then

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-29 Thread Oleksii Kliukin
> On 29. Jan 2019, at 10:45, Magnus Hagander wrote: > > On Tue, Jan 29, 2019 at 6:19 AM Michael Paquier > wrote: > On Mon, Jan 28, 2019 at 02:00:59PM +0100, Alex Kliukin wrote: > > While reading the doc page for the pg_basebackup, I've been confused > > by the fact

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-29 Thread Michael Paquier
On Tue, Jan 29, 2019 at 10:45:34AM +0100, Magnus Hagander wrote: > Since you also agreed on it, I went ahead and pushed (with backpatch). Thanks for taking care of it, Magnus. -- Michael signature.asc Description: PGP signature

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-29 Thread Magnus Hagander
On Tue, Jan 29, 2019 at 6:19 AM Michael Paquier wrote: > On Mon, Jan 28, 2019 at 02:00:59PM +0100, Alex Kliukin wrote: > > While reading the doc page for the pg_basebackup, I've been confused > > by the fact that it says WAL files will be written to .tarballs > > (either base.tar or pg_wal.tar)

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-28 Thread Michael Paquier
On Mon, Jan 28, 2019 at 02:00:59PM +0100, Alex Kliukin wrote: > While reading the doc page for the pg_basebackup, I've been confused > by the fact that it says WAL files will be written to .tarballs > (either base.tar or pg_wal.tar) when pg_basebackup is instructed to > stream WALs alongside the

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-28 Thread Alex Kliukin
On Mon, Jan 28, 2019, at 10:25, Michael Paquier wrote: > On Mon, Jan 28, 2019 at 09:05:26AM +0100, Magnus Hagander wrote: > > And for plain format, we'd do the same -- sync after each file segment, and > > then a final one of the directory when done, right? > > Well, the code is doing a double

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-28 Thread Michael Paquier
On Mon, Jan 28, 2019 at 09:05:26AM +0100, Magnus Hagander wrote: > Yeah, that could be done without giving up any of the guarantees -- we only > give the guarantee at the end of the completed backup. I wouldn't > necessarily say we're wrong now, but it could definitely be a nice > performance

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-28 Thread Magnus Hagander
On Sun, Jan 27, 2019 at 1:59 PM Michael Paquier wrote: > On Sat, Jan 26, 2019 at 01:45:46PM +0100, Magnus Hagander wrote: > > One workaround you could perhaps look at here is to run pg_basebackup > > with --no-sync. That way there will be no fsyncs issued while running. > You > > will then of

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-27 Thread Michael Paquier
On Sat, Jan 26, 2019 at 01:45:46PM +0100, Magnus Hagander wrote: > One workaround you could perhaps look at here is to run pg_basebackup > with --no-sync. That way there will be no fsyncs issued while running. You > will then of course have to take care of syncing all the files to disk > after

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-26 Thread Magnus Hagander
On Sat, Jan 26, 2019 at 1:35 PM Nick B wrote: > On Sat, Jan 26, 2019 at 4:23 AM Michael Paquier > wrote: > > These are a bit unregular. Which files are taking that long to > > complete while others are way faster? It may be something that we > > could improve on the base backup side as there

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-26 Thread Nick B
On Sat, Jan 26, 2019 at 4:23 AM Michael Paquier wrote: > These are a bit unregular. Which files are taking that long to > complete while others are way faster? It may be something that we > could improve on the base backup side as there is no actual point in > syncing segments while the backup

Re: pg_basebackup, walreceiver and wal_sender_timeout

2019-01-25 Thread Michael Paquier
On Fri, Jan 25, 2019 at 03:26:38PM +0100, Nick B wrote: > On server we see this error firing: "terminating walsender process due to > replication timeout" > The problem occurs during a network or file system acting very slow. One > example of such case looks like this (strace output for fsync

pg_basebackup, walreceiver and wal_sender_timeout

2019-01-25 Thread Nick B
Hi, hackers. When running pg_basebackup with -X s with network file system as target we would consistently get "could not receive data from WAL stream: server closed the connection unexpectedly". On server we see this error firing: "terminating walsender process due to replication timeout" The