Anyone who has faced similar issues with cdc with Postgres. I see the restart_lsn and confirmed_flush_lsn constant since the snapshot replication records were streamed even though I have tried inserting a record in the whitelisted table.
select * from pg_replication_slots; slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn -------------+----------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+--------------------- stream_cdc3 | pgoutput | logical | 16411 | test_cdc | f | t | 1146 | | 6872 | 62/34000828 | 62/34000860 I have passed the heartbeat.interval.ms = 1000 and could see the heartbeat events streamed to flink however the transaction log disk usage and oldest replication slot lag consistently increasing. From [1] I have also tried this - For other decoder plug-ins, it is recommended to create a supplementary table that is not monitored by Debezium. A separate process would then periodically update the table (either inserting a new event or updating the same row all over). PostgreSQL then will invoke Debezium which will confirm the latest LSN and allow the database to reclaim the WAL space. [image: Screenshot 2021-04-08 at 2.07.18 PM.png] [image: Screenshot 2021-04-08 at 2.07.52 PM.png] [1] - https://debezium.io/documentation/reference/1.0/connectors/postgresql.html#wal-disk-space Thanks. On Wed, Apr 7, 2021 at 12:51 PM bat man <tintin0...@gmail.com> wrote: > Hi there, > > I am using flink 1.11 and cdc connector 1.1 to stream changes from a > postgres table. I see the WAL consumption is increasing gradually even > though the writes to tables are very less. > > I am using AWS RDS, from [1] I understand that setting the parameter > heartbeat.interval.ms solves this WAL consumption issue. However, I tried > setting this with no success. > > I found a bug [2] which seems to be taking care of committing the lsn for > the db to release the wal. however this seems to be only fixed in 1.3 which > is compatible with flink 1.12.1. Is there any way this can be fixed in > 1.11.1. Since I am using EMR and the latest flink version available is 1.11. > > > [1] - > https://debezium.io/documentation/reference/connectors/postgresql.html > [2] - https://github.com/ververica/flink-cdc-connectors/issues/97 > > Thanks. > Hemant >
