Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread Hongshun Wang
Hi David, In your modified pipeline, just one source from table1 is sufficient, with both sink1 and process2 sharing a single source from process1. However, based on your log, it appears that two sources have been generated. Do you have the execution graph available in the Flink UI? Best, Hongshun

Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread David Bryson
These sink share the same source. The pipeline that works looks something like this: table1 -> process1 -> process2 -> sink2 When I change it to this: table1 -> process1 -> process2 -> sink2 `--> sink1 I get the errors described, where it appears that a second proc

Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread Hongshun Wang
Hi David, > When I add this second sink, the postgres-cdc connector appears to add a second reader from the replication log, but with the same slot name. I don't understand what you mean by adding a second sink. Do they share the same source, or does each have a separate pipeline? If the former on

Problems with multiple sinks using postgres-cdc connector

2024-06-14 Thread David Bryson
Hi, I have a stream reading from postgres-cdc connector version 3.1.0. I read from two tables: flink.cleaned_migrations public.cleaned I convert the tables into a datastream, do some processing, then write it to a sink at the end of my stream: joined_table_result = joined_with_metadata.exec