Streaming Logical Decoding Events

[Matt Dee]

Hi there,

I am trying to build an application which reads messages off of a
replication slot and does some processing with them.  I want to guarantee
that every event gets processed exactly once.  Unfortunately, I've been
unable to find a guide to the correct way to read this data.  I've been
experimenting and it seems to work, but I wanted to confirm a few things.
Apologies if this is the wrong place for this kind of question, or if
reading off a slot is described elsewhere (if so, I would appreciate being
pointed to the right place).

I'm trying to use terminology from
https://www.postgresql.org/docs/9.5/static/protocol-replication.html

My basic idea is:
* On startup, create a replication connection and do START CONNECTION with
LSN 0/0.  This seems to start off from the last "committed" LSN (last LSN
sent back in a standby status update).

* Keep track of a "last committed" LSN, and periodically send back a
Standby Status Update with the "written", "flushed", and "applied" LSNn all
set to that value (I could not find documentation on the difference between
the three, but setting all of them equal seems to work).

* When I see an XLogData message, process the data and then set the "last
committed" LSN to the "starting point of the WAL data" LSN.

* When I see a Primary Keepalive Message, set "last committed" LSN to the
"current end WAL" field, assuming all prior XLogDatas have been processed.
If the "should reply" byte is set, send back a Standby Status Update
immediately

Is this the way that this feature was intended to be used?

In particular, I have two concerns with this approach:

1) On the XLogData, should I be using "starting point of WAL data" or
"current end of WAL"?  In my testing, they seem to always be the same, so
I'm worried there's some case I haven't handled.

2) I'm not sure if committing the end WAL LSNs on the Keepalive messages is
safe.  Is it possible to receive a Keepalive message, followed by an
XLogData message, such that the Keepalive's LSN is higher than the
XLogData's?  This could cause problems, because after committing the
Keepalive's LSN, if my application needed to restart, when it comes back
up, we might miss the XLogData, since we committed a higher LSN.

Thanks, and I appreciate any pointers you can provide!
-Matt

Specifying WAL Location in Streaming Replication

[Matt Dee]

Hi,

I am trying to use the streaming replication protocol described in
https://www.postgresql.org/docs/10/static/protocol-replication.html to read
logical decoding events from a replication slot.

I'm doing this by starting replication with START_REPLICATION, and sending
down the most recent position consumed in a standby status update.  When
starting replication, I want to begin reading from the last position
"committed" by the standby status update.

In the documentation for START_REPLICATION, a required argument is the WAL
location to begin streaming at, and I'm not sure what to use here.  I have
been using 0, and it seems to work fine.  Additionally, it seems that when
--startpos is not provided to pg_recvlogical, it defaults to just sending a
wal location of 0.

While this seems to work, I feel a bit uneasy about it since I can't find
any documentation explaining why this works or what this argument does when
using a replication slot.  Any clarification would be appreciated.

Thanks,
-Matt