On 23/09/17 18:42, Euler Taveira wrote:
2017-09-22 19:28 GMT-03:00 Gregory Brail <gregbr...@google.com>:
We have been working on a project that makes extensive use of logical
replication for use inside Apigee (which is a very small part of Google):
https://github.com/apigee-labs/transicator
In order to do this, we had to write our own logical replication plugin
because the supplied "test_decoding" plugin from the "contrib" directory was
hard for us to work with. Primarily:
test_decoding is a proof of concept to illustrate the logical decoding
potential. It is not intended for production.
However, AFAIK, AWS's DMS uses it for production purposes (see
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html).
I developed wal2json [1]
for general use. It outputs changes in JSON. It was one of the first
logical decoding plugins.
1) It doesn't include all the fields that we need for Transicator (most
importantly we need the LSN and the 64-bit transaction ID).
wal2json includes both.
2) It outputs a text format that is hard to parse.
There are a lot of JSON parsers.
I imagine that other users of logical decoding are facing similar problems.
Would the community support the development of another plugin that is
distributed as part of "contrib" that addresses these issues? I'd be happy
to submit a patch, or GitHub repo, or whatever works best as an example.
(Also, although Transicator uses protobuf, I'm happy to have it output a
simple binary format as well.)
There was a prior discussion and it was suggestted that we have a
ready-for-production plugin in core (besides pgoutput). It was
suggested [1] that I submit wal2json for 11. I'm in process to clean
up the code and hope to submit it to CF2.
I would be happy to see another logical decoding plugin into core
starting on 11. However, this also poses a bit of a challenge for
middleware implementors: you need to support one for 9.4-9.5
(test_decoding), another for 10 (pgoutput) and maybe another for 11
onwards. The idea of asking users to install a binary plugin is very
unsexy, so these are the options available.
However, having said that, and while json is a great output format
for interoperability, if there's a discussion on which plugin to include
next, I'd also favor one that has some more compact representation
format (or that supports several formats, not only json).
Regards,
Álvaro
--
Alvaro Hernandez
-----------
OnGres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
