Postgres-CDC start replication fails after stop/start on flink stream

2024-07-02 Thread David Bryson
I have a flink stream using Postgres-CDC as a source. It's been operating mostly fine, but I recently had to stop, and then start the stream again. The stream is never able to start again as the replication never completes and Flink enters a restart loop. Upon starting the cdc reader task issues

Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread David Bryson
you want each sink to have its own > source, you can set a different slot name for each source (the option name > is slot.name[1]). > > [1] > https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/flink-sources/postgres-cdc/#connector-options > > On Sat, Jun 15,

Problems with multiple sinks using postgres-cdc connector

2024-06-14 Thread David Bryson
Hi, I have a stream reading from postgres-cdc connector version 3.1.0. I read from two tables: flink.cleaned_migrations public.cleaned I convert the tables into a datastream, do some processing, then write it to a sink at the end of my stream: joined_table_result =