Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-09 Thread Fujii Masao
Hi, On Thu, Jul 9, 2009 at 11:13 PM, Kevin Grittner wrote: > Fujii Masao wrote: >> Kevin Grittner wrote: > >>> I think the interesting bit is when you're at this point and the >>> connection between the master and slave goes down for a couple >>> days.  How do you handle that? >> >> In the curren

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-09 Thread Kevin Grittner
Fujii Masao wrote: > Kevin Grittner wrote: >> I think the interesting bit is when you're at this point and the >> connection between the master and slave goes down for a couple >> days. How do you handle that? > > In the current design of synch rep, you have only to restart the > standby afte

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-09 Thread Fujii Masao
Hi, On Tue, Jul 7, 2009 at 8:51 PM, Fujii Masao wrote: > http://archives.postgresql.org/message-id/4951108a.5040...@enterprisedb.com >> I don't think we need or should >> allow running regular queries before entering "replication mode". the >> backend should become a walsender process directly aft

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Fujii Masao
Hi, On Wed, Jul 8, 2009 at 10:59 PM, Kevin Grittner wrote: > Dimitri Fontaine wrote: > >>  4. sync: slave is no more lagging, it's applying the stream as it >>     gets it, either as part of the master transaction or not >>     depending on the GUC settings > > I think the interesting bit is when

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Heikki Linnakangas
Fujii Masao wrote: > On Wed, Jul 8, 2009 at 4:00 AM, Heikki > Linnakangas wrote: >>> I would envision the slaves >>> connecting to the master's replication port and asking "feed me WAL >>> beginning at LSN position thus-and-so", with no notion of WAL file >>> boundaries exposed anyplace. >> Yep, t

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Heikki Linnakangas
Mark Mielke wrote: > On 07/08/2009 09:59 AM, Kevin Grittner wrote: >> I think the interesting bit is when you're at this point and the >> connection between the master and slave goes down for a couple days. >> How do you handle that? > > Been following with great interest... > > If the updates ar

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Mark Mielke
On 07/08/2009 09:59 AM, Kevin Grittner wrote: Dimitri Fontaine wrote: 4. sync: slave is no more lagging, it's applying the stream as it gets it, either as part of the master transaction or not depending on the GUC settings I think the interesting bit is when you're at t

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Dimitri Fontaine
"Kevin Grittner" writes: > Dimitri Fontaine wrote: > >> 4. sync: slave is no more lagging, it's applying the stream as it >> gets it, either as part of the master transaction or not >> depending on the GUC settings > > I think the interesting bit is when you're at this point and th

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Kevin Grittner
Dimitri Fontaine wrote: > 4. sync: slave is no more lagging, it's applying the stream as it > gets it, either as part of the master transaction or not > depending on the GUC settings I think the interesting bit is when you're at this point and the connection between the master and sl

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-08 Thread Dimitri Fontaine
Hi, Tom Lane writes: > I think this fails the basic sanity check: do you need it to still work > when the master is dead. I don't get it. Why would we want to setup a slave against a dead master? The way I understand the current design of Synch Rep, when you start a new slave the following ha

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Fujii Masao
Hi, Thanks for the brilliant comments! On Wed, Jul 8, 2009 at 4:00 AM, Heikki Linnakangas wrote: >> There are still some interesting questions in this about exactly how you >> switch over from "catchup mode" to following the live WAL broadcast. >> With the above design it would be the master's re

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Fujii Masao
Hi, On Wed, Jul 8, 2009 at 4:00 AM, Heikki Linnakangas wrote: >>  I would envision the slaves >> connecting to the master's replication port and asking "feed me WAL >> beginning at LSN position thus-and-so", with no notion of WAL file >> boundaries exposed anyplace. > > Yep, that's the way I envis

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Fujii Masao
Hi, On Wed, Jul 8, 2009 at 12:49 AM, Tom Lane wrote: > This design seems totally wrong to me.  It's confusing the master's > pg_xlog directory with the archive.  We should *not* use pg_xlog as > a long-term archive area; that's terrible from both a performance > and a reliability perspective.  Per

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Greg Stark
On Tue, Jul 7, 2009 at 8:12 PM, Tom Lane wrote: > (If nothing else, there is no point in keeping so much WAL that catching > up by scanning it would take longer than taking a fresh base backup. > My impression from recent complaints about our WAL-reading speed is that > that might be a pretty tight

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Tom Lane
Dimitri Fontaine writes: > Could we add yet another postmaster specialized child to handle the > archive, which would be like a default archive_command implemented in > core. I think this fails the basic sanity check: do you need it to still work when the master is dead. It's reasonable to a

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Dimitri Fontaine
Le 7 juil. 09 à 21:12, Tom Lane a écrit : Heikki Linnakangas writes: And I'm sure people will want the option to retain WAL longer in the master, to avoid an expensive resync if the slave falls behind. It would be simple to provide a GUC option for "always retain X GB of old WAL in pg_xlog

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Tom Lane
Heikki Linnakangas writes: > And I'm sure people will want the option to retain WAL longer in the > master, to avoid an expensive resync if the slave falls behind. It would > be simple to provide a GUC option for "always retain X GB of old WAL in > pg_xlog". Right, we would want to provide some m

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Heikki Linnakangas
Tom Lane wrote: > Greg Stark writes: >> On Tue, Jul 7, 2009 at 4:49 PM, Tom Lane wrote: >>> This design seems totally wrong to me. >>> ... > >> But this conflicts with earlier discussions where we were concerned >> about the length of the path wal has to travel between the master and >> the slave

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Tom Lane
Greg Stark writes: > On Tue, Jul 7, 2009 at 4:49 PM, Tom Lane wrote: >> This design seems totally wrong to me. >> ... > But this conflicts with earlier discussions where we were concerned > about the length of the path wal has to travel between the master and > the slaves. We want slaves to be ab

[HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Greg Stark
On Tue, Jul 7, 2009 at 4:49 PM, Tom Lane wrote: > This design seems totally wrong to me.  It's confusing the master's > pg_xlog directory with the archive.  We should *not* use pg_xlog as > a long-term archive area; that's terrible from both a performance > and a reliability perspective.  Performan

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Tom Lane
Fujii Masao writes: > On Tue, Jul 7, 2009 at 12:16 AM, Tom Lane wrote: >> I confess to not having paid much attention to this thread so far, but ... >> what is the rationale for having such a capability at all? > If the XLOG files which are required for recovery exist only in the > primary server

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Fujii Masao
Hi, Thanks for the comment! On Tue, Jul 7, 2009 at 5:07 PM, Heikki Linnakangas wrote: > pg_read_xlogfile() feels like a quite hacky way to implement that. Do we > require the master to always have read access to the PITR archive? And > indeed, to have a PITR archive configured to begin with. If y

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Andrew Dunstan
Heikki Linnakangas wrote: While supporting multiple slaves isn't a priority, Really? I should have thought it was a basic requirement. At the very least we need to design with it in mind. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-07 Thread Heikki Linnakangas
Fujii Masao wrote: > On Tue, Jul 7, 2009 at 12:16 AM, Tom Lane wrote: >> Fujii Masao writes: >>> In order for the primary server (ie. a normal backend) to read an archived >>> file, >>> restore_command needs to be specified in also postgresql.conf. In this case, >>> how should we handle restore_c

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-06 Thread Fujii Masao
Hi, Thanks for the comment! On Tue, Jul 7, 2009 at 12:16 AM, Tom Lane wrote: > Fujii Masao writes: >> In order for the primary server (ie. a normal backend) to read an archived >> file, >> restore_command needs to be specified in also postgresql.conf. In this case, >> how should we handle resto

Re: [HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-06 Thread Tom Lane
Fujii Masao writes: > In order for the primary server (ie. a normal backend) to read an archived > file, > restore_command needs to be specified in also postgresql.conf. In this case, > how should we handle restore_command in recovery.conf? I confess to not having paid much attention to this thr

[HACKERS] Re: Synch Rep: direct transfer of WAL file from the primary to the standby

2009-07-05 Thread Fujii Masao
Hi, On Tue, Jun 16, 2009 at 3:13 PM, Fujii Masao wrote: > The main part of this capability is the new function to read the specified > WAL file. The following is the definition of it. > >    pg_read_xlogfile (filename text [, restore bool]) returns setof bytea > >    - filename: name of file to re