Re: [HACKERS] Replication status in logical replication

[Masahiko Sawada]

On Tue, Sep 26, 2017 at 10:36 AM, Vaishnavi Prabakaran
 wrote:
> Hi,
>
> On Wed, Sep 13, 2017 at 9:59 AM, Daniel Gustafsson  wrote:
>>
>>
>> I’m not entirely sure why this was flagged as "Waiting for Author” by the
>> automatic run, the patch applies for me and builds so resetting back to
>> “Needs
>> review”.
>>
>
> This patch applies and build cleanly and I did a testing with one publisher
> and one subscriber, and confirm that the replication state after restarting
> the server now is "streaming" and not "Catchup".
>
> And, I don't find any issues with code and patch to me is ready for
> committer, marked the same in cf entry.
>

Thank you for the reviewing the patch!

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Replication status in logical replication

[Vaishnavi Prabakaran]

Hi,

On Wed, Sep 13, 2017 at 9:59 AM, Daniel Gustafsson  wrote:

>
> I’m not entirely sure why this was flagged as "Waiting for Author” by the
> automatic run, the patch applies for me and builds so resetting back to
> “Needs
> review”.
>
>
This patch applies and build cleanly and I did a testing with one publisher
and one subscriber, and confirm that the replication state after restarting
the server now is "streaming" and not "Catchup".

And, I don't find any issues with code and patch to me is ready for
committer, marked the same in cf entry.

Thanks & Regards,
Vaishnavi,
Fujitsu Australia.

Re: [HACKERS] Replication status in logical replication

[Daniel Gustafsson]

> On 30 May 2017, at 19:55, Peter Eisentraut  
> wrote:
> 
> On 5/29/17 22:56, Noah Misch wrote:
>> On Fri, May 19, 2017 at 11:33:48AM +0900, Masahiko Sawada wrote:
>>> On Wed, Apr 12, 2017 at 5:31 AM, Simon Riggs  wrote:
 Looks like a bug that we should fix in PG10, with backpatch to 9.4 (or
 as far as it goes).
 
 Objections to commit?
 
>>> 
>>> Seems we still have this issue. Any update or comment on this? Barring
>>> any objections, I'll add this to the open item so it doesn't get
>>> missed.
>> 
>> [Action required within three days.  This is a generic notification.]
>> 
>> The above-described topic is currently a PostgreSQL 10 open item.  Peter,
>> since you committed the patch believed to have created it, you own this open
>> item.  If some other commit is more relevant or if this does not belong as a
>> v10 open item, please let us know.
> 
> I would ask Simon to go ahead with this patch if he feels comfortable
> with it.
> 
> I'm disclaiming this open item, since it's an existing bug from previous
> releases (and I have other open items to focus on).

I’m not entirely sure why this was flagged as "Waiting for Author” by the
automatic run, the patch applies for me and builds so resetting back to “Needs
review”.

Simon: do you think you will have time to look at this patch in this CF?

cheers ./daniel

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Replication status in logical replication

[Peter Eisentraut]

On 5/29/17 22:56, Noah Misch wrote:
> On Fri, May 19, 2017 at 11:33:48AM +0900, Masahiko Sawada wrote:
>> On Wed, Apr 12, 2017 at 5:31 AM, Simon Riggs  wrote:
>>> Looks like a bug that we should fix in PG10, with backpatch to 9.4 (or
>>> as far as it goes).
>>>
>>> Objections to commit?
>>>
>>
>> Seems we still have this issue. Any update or comment on this? Barring
>> any objections, I'll add this to the open item so it doesn't get
>> missed.
> 
> [Action required within three days.  This is a generic notification.]
> 
> The above-described topic is currently a PostgreSQL 10 open item.  Peter,
> since you committed the patch believed to have created it, you own this open
> item.  If some other commit is more relevant or if this does not belong as a
> v10 open item, please let us know.

I would ask Simon to go ahead with this patch if he feels comfortable
with it.

I'm disclaiming this open item, since it's an existing bug from previous
releases (and I have other open items to focus on).

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Replication status in logical replication

[Noah Misch]

On Fri, May 19, 2017 at 11:33:48AM +0900, Masahiko Sawada wrote:
> On Wed, Apr 12, 2017 at 5:31 AM, Simon Riggs  wrote:
> > On 22 March 2017 at 02:50, Masahiko Sawada  wrote:
> >
> >> When using logical replication, I ran into a situation where the
> >> pg_stat_replication.state is not updated until any wal record is sent
> >> after started up. For example, I set up logical replication with 2
> >> subscriber and restart the publisher server, but I see the following
> >> status for a while (maybe until autovacuum run).
> > ...
> >
> >> Attached patch fixes this behavior by updating WalSndCaughtUp before
> >> trying to read next WAL if already caught up.
> >
> > Looks like a bug that we should fix in PG10, with backpatch to 9.4 (or
> > as far as it goes).
> >
> > Objections to commit?
> >
> 
> Seems we still have this issue. Any update or comment on this? Barring
> any objections, I'll add this to the open item so it doesn't get
> missed.

[Action required within three days.  This is a generic notification.]

The above-described topic is currently a PostgreSQL 10 open item.  Peter,
since you committed the patch believed to have created it, you own this open
item.  If some other commit is more relevant or if this does not belong as a
v10 open item, please let us know.  Otherwise, please observe the policy on
open item ownership[1] and send a status update within three calendar days of
this message.  Include a date for your subsequent status update.  Testers may
discover new open items at any time, and I want to plan to get them all fixed
well in advance of shipping v10.  Consequently, I will appreciate your efforts
toward speedy resolution.  Thanks.

[1] 
https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Replication status in logical replication

[Masahiko Sawada]

On Wed, Apr 12, 2017 at 5:31 AM, Simon Riggs  wrote:
> On 22 March 2017 at 02:50, Masahiko Sawada  wrote:
>
>> When using logical replication, I ran into a situation where the
>> pg_stat_replication.state is not updated until any wal record is sent
>> after started up. For example, I set up logical replication with 2
>> subscriber and restart the publisher server, but I see the following
>> status for a while (maybe until autovacuum run).
> ...
>
>> Attached patch fixes this behavior by updating WalSndCaughtUp before
>> trying to read next WAL if already caught up.
>
> Looks like a bug that we should fix in PG10, with backpatch to 9.4 (or
> as far as it goes).
>
> Objections to commit?
>

Seems we still have this issue. Any update or comment on this? Barring
any objections, I'll add this to the open item so it doesn't get
missed.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Replication status in logical replication

[Simon Riggs]

On 22 March 2017 at 02:50, Masahiko Sawada  wrote:

> When using logical replication, I ran into a situation where the
> pg_stat_replication.state is not updated until any wal record is sent
> after started up. For example, I set up logical replication with 2
> subscriber and restart the publisher server, but I see the following
> status for a while (maybe until autovacuum run).
...

> Attached patch fixes this behavior by updating WalSndCaughtUp before
> trying to read next WAL if already caught up.

Looks like a bug that we should fix in PG10, with backpatch to 9.4 (or
as far as it goes).

Objections to commit?

-- 
Simon Riggshttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Replication status in logical replication

[Masahiko Sawada]

Hi all,

When using logical replication, I ran into a situation where the
pg_stat_replication.state is not updated until any wal record is sent
after started up. For example, I set up logical replication with 2
subscriber and restart the publisher server, but I see the following
status for a while (maybe until autovacuum run).

=# select application_name, state, sent_location, write_location,
flush_location, replay_location, sync_state from pg_stat_replication ;
 application_name |  state  | sent_location | write_location |
flush_location | replay_location | sync_state
--+-+---+++-+
 node1| catchup | 0/16329F8 | 0/16329F8  |
0/16329F8  | 0/16329F8   | potential
 node2| catchup | 0/16329F8 | 0/16329F8  |
0/16329F8  | 0/16329F8   | async
(2 rows)

It seems that all wal senders have caught up but
pg_stat_replication.state is still "catchup". The reason of this
behavior is that WalSndCaughtUp is updated only in WalSndWaitForWal in
logical replication during running, and in logical_read_xlog_page
always try to read next wal record (i.g. it calls
WalSndWaitForWal(targetPagePtr + reqLen)). So WalSndWaitForWal cannot
update WalSndCaughtUp until any new wal record is created after
started up and wal sender read it.

Attached patch fixes this behavior by updating WalSndCaughtUp before
trying to read next WAL if already caught up.

Regards,

--
Masahiko Sawada
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


logical_repl_caught_up.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Replication status

[Andrew Sullivan]

On Tue, May 28, 2002 at 06:08:21PM -0700, Thomas Lockhart wrote:

 clients. We have been very low-key (imho) in representing this solution
 to the developer community, but it should be considered for applications
 matching its capabilities. 

I should like to emphasise that I have no desire to run down rserv --
I think it's pretty good, and I'm more than happy with its
performance.  That I'm now facing a feature-lust argument for ORAC is
a political, and not technical problem.   

 Full transactional integrity across primary
 and secondary servers is not easy to come by and not offered by most
 other solutions. 

Exactly, plus there appears to be a big price to be paid for that
full integrity.

A

-- 

Andrew Sullivan   87 Mowat Avenue 
Liberty RMS   Toronto, Ontario Canada
[EMAIL PROTECTED]  M6K 3E3
 +1 416 646 3304 x110


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Replication status

[Karel Zak]

On Mon, May 27, 2002 at 11:40:20AM -0400, Michael Meskes wrote:

 could anyone please enlighten me about the status of replication? I do
 expect lots of questions about this, and I'm not really sure if I can
 promise it for 7.3. :-)

 8.0 ;-) (?)

 I add the other quesion: how is current status of on-line backup log
 based on WAL? The enterprise usage require it maybe more than
 replication.

Karel

-- 
 Karel Zak  [EMAIL PROTECTED]
 http://home.zf.jcu.cz/~zakkr/
 
 C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Replication status

[Steven Singer]

On Tue, 28 May 2002, Michael Meskes wrote:

 
 This is about pgreplication I think. Is the the replication project of
 choice for pgsql? IIRC there quite some projects for this topic:
 
 PostgreSQL replicator
 Rserver
 Usogres
 dbbalancer


There's also DBMirror which I submitted to the contrib directory just 
after the 7.2 release. I got an email last month saying that it had been 
applied against the 7.3 tree but I don't see it there.

Its a trigger based lazy replication system and has all the associated 
drawbacks but works for master-slave.  I've been working on adding 
selective replication to it and hope to be able to release another version 
of that in June.


 
 Michael
 
 

-- 
Steven Singer   [EMAIL PROTECTED]
Aircraft Performance SystemsPhone:  519-747-1170 ext 282
Navtech Systems Support Inc.AFTN:   CYYZXNSX SITA: YYZNSCR
Waterloo, Ontario   ARINC:  YKFNSCR



---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly

Re: [HACKERS] Replication status

[Bruce Momjian]

Karel Zak wrote:
 On Mon, May 27, 2002 at 11:40:20AM -0400, Michael Meskes wrote:
 
  could anyone please enlighten me about the status of replication? I do
  expect lots of questions about this, and I'm not really sure if I can
  promise it for 7.3. :-)
 
  8.0 ;-) (?)
 
  I add the other quesion: how is current status of on-line backup log
  based on WAL? The enterprise usage require it maybe more than
  replication.

Yes!  Point-in-time recovery and replication are our only to urgent
items on the TODO list.

Jan's idea of implementing point-in-time recovery as a playback of the
replication logs seems like a great idea, so I think replication may
solve both issues.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 853-3000
  +  If your life is a hard drive, |  830 Blythe Avenue
  +  Christ can be your backup.|  Drexel Hill, Pennsylvania 19026

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Replication status

[Bruce Momjian]

Michael Meskes wrote:
 On Mon, May 27, 2002 at 05:12:33PM -0400, Bruce Momjian wrote:
  Last I talked to Darren, the replication code was modified to merge into
  our 7.2 tree.  There are still pieces missing so it will not be
  functional when applied.  It is remotely possible there could be
  master-slave in 7.3, but I doubt it.
 
 This is about pgreplication I think. Is the the replication project of
 choice for pgsql? IIRC there quite some projects for this topic:
 
 PostgreSQL replicator
 Rserver
 Usogres
 dbbalancer
 
 What about these? We seem to have some proof-of-concept code of rserver
 in contrib. Dbbalancer seems to be more focussed on balancing access and
 not replication, but can do this too.

rserver only does single-master, while most people want multi-master. 
Usogres is more of a load balancer/replication, where the query is sent
to both servers.  Not sure about the others.

The only multi-master solution proposed is pgreplication.  I think there
is a PDF on that web site that describes the various replication
options. I should probably write up a little replication FAQ.

Jan is doing a replication talk at O'Reilly in July and hopefully we can
get a PDF of that.

pgreplication is not good for nodes over slow links or nodes that are
intermittently connected, so it is not going to solve all cases either.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 853-3000
  +  If your life is a hard drive, |  830 Blythe Avenue
  +  Christ can be your backup.|  Drexel Hill, Pennsylvania 19026

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Replication status

[Thomas Lockhart]

...
 rserver only does single-master, while most people want multi-master.

As you probably know, rserv is not limited to only a single instance of
a single master. Many replication problems can be described as a single
source problem (or should be described as such; moving to a fully
distributed database brings a host of other issues). So any problem
which can be decomposed to having single sources of subsets of
information can be handled with this system.

The contrib/rserv code has received no contributions from the community
beyond our original submission, which of course pushes all of the
development and recurring costs back onto PostgreSQL Inc and their
clients. We have been very low-key (imho) in representing this solution
to the developer community, but it should be considered for applications
matching its capabilities. Full transactional integrity across primary
and secondary servers is not easy to come by and not offered by most
other solutions. fwiw we have demonstrated well over 2000 updates per
second flowing through rserv systems.

  - Thomas

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Re: [HACKERS] Replication status

[Bruce Momjian]

Agreed.  It would be nice to see both a single-master and multi-master
server included in our main tree and a clear description of when to use
each.  The confusion over the various replication solutions and their
strengths/weaknesses is a major problem.

I always felt a clearer README for rserv would help greatly.  We do get
lots of questions about how to get it working.  README.rserv goes over
the major 'toolset' items and describes a demo, but that is it.  (I
don't even know what the 'toolset' items are or how to access them, at
least from reading the README.)  I thought of doing the README
improvements myself, but because I didn't write it, I left it alone.

---

Thomas Lockhart wrote:
 ...
  rserver only does single-master, while most people want multi-master.
 
 As you probably know, rserv is not limited to only a single instance of
 a single master. Many replication problems can be described as a single
 source problem (or should be described as such; moving to a fully
 distributed database brings a host of other issues). So any problem
 which can be decomposed to having single sources of subsets of
 information can be handled with this system.
 
 The contrib/rserv code has received no contributions from the community
 beyond our original submission, which of course pushes all of the
 development and recurring costs back onto PostgreSQL Inc and their
 clients. We have been very low-key (imho) in representing this solution
 to the developer community, but it should be considered for applications
 matching its capabilities. Full transactional integrity across primary
 and secondary servers is not easy to come by and not offered by most
 other solutions. fwiw we have demonstrated well over 2000 updates per
 second flowing through rserv systems.
 
   - Thomas
 

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 853-3000
  +  If your life is a hard drive, |  830 Blythe Avenue
  +  Christ can be your backup.|  Drexel Hill, Pennsylvania 19026

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

[HACKERS] Replication status

[Michael Meskes]

Hi,

could anyone please enlighten me about the status of replication? I do
expect lots of questions about this, and I'm not really sure if I can
promise it for 7.3. :-)

Yes, I know it#s marked urgent in the TODO list, but no one seems to be
listed as tackling this topic.

Thanks a lot.

Michael
--
Michael Meskes
[EMAIL PROTECTED]
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[HACKERS] Replication status

[Michael Meskes]

Hi,

could anyone please enlighten me about the status of replication? I do
expect lots of questions about this, and I'm not really sure if I can
promise it for 7.3. :-)

Yes, I know it's marked urgent in the TODO list, but no one seems to be
listed as tackling this topic.

Thanks a lot.

Michael
-- 
Michael Meskes
[EMAIL PROTECTED]
Go SF 49ers! Go Rhein Fire!
Use Debian GNU/Linux! Use PostgreSQL!

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Re: [HACKERS] Replication status

[Tom Lane]

Michael Meskes [EMAIL PROTECTED] writes:
 could anyone please enlighten me about the status of replication? I do
 expect lots of questions about this, and I'm not really sure if I can
 promise it for 7.3. :-)

Unless 7.3 slips drastically from our current intended schedule
(beta in late August), I think it's pretty safe to say there will
be no replication in 7.3, beyond what's already available (rserv
and so forth).

regards, tom lane

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly

Re: [HACKERS] Replication status

[Darren Johnson]

Unless 7.3 slips drastically from our current intended schedule
(beta in late August), I think it's pretty safe to say there will
be no replication in 7.3, beyond what's already available (rserv
and so forth).


I can't speak for any of the other replication projects, but 
pgreplication won't be
ready for 7.3.  If all goes according to plan, I should have some free 
time over
the summer months to put a good dent in the first phase, but at best it 
would
be a very limited experimental patch.

More information on pgreplication can be found @

http://gborg.postgresql.org/project/pgreplication/projdisplay.php


Darren



---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] Replication status

[Bruce Momjian]

Tom Lane wrote:
 Michael Meskes [EMAIL PROTECTED] writes:
  could anyone please enlighten me about the status of replication? I do
  expect lots of questions about this, and I'm not really sure if I can
  promise it for 7.3. :-)
 
 Unless 7.3 slips drastically from our current intended schedule
 (beta in late August), I think it's pretty safe to say there will
 be no replication in 7.3, beyond what's already available (rserv
 and so forth).

Last I talked to Darren, the replication code was modified to merge into
our 7.2 tree.  There are still pieces missing so it will not be
functional when applied.  It is remotely possible there could be
master-slave in 7.3, but I doubt it.

I was hoping to spend major time on it myself (and SRA/Japan has
encouraged me to get involved), but have been too busy to dive in.  I
think once it is in CVS, it will be easier to grasp what is going on,
and perhaps to move it forward.

I saw a message (I think for Darrren) saying he hoped to restart on it
in two weeks.

-- 
  Bruce Momjian|  http://candle.pha.pa.us
  [EMAIL PROTECTED]   |  (610) 853-3000
  +  If your life is a hard drive, |  830 Blythe Avenue
  +  Christ can be your backup.|  Drexel Hill, Pennsylvania 19026

---(end of broadcast)---
TIP 6: Have you searched our list archives?

http://archives.postgresql.org