Re: Flink Stateful Functions 3.4

2024-06-17 Thread Zhanghao Chen
There's no active maintenance of the StateFun project since the release of v3.3 in last September by GitHub commit history [1]. So currently, there's no estimate on when v3.4 could be released and which Flink version would be supported. [1]

Re: Problem reading a CSV file with pyflink datastream in k8s with Flink operator

2024-06-17 Thread Robert Young
Hi Gwenael, >From the logs I thought it was a JVM module opens/exports issue, but I found it had a similar issue using a java8 base image too. I think the issue is it's not permitted for PythonCsvUtils to call the package-private constructor of CsvReaderFormat across class loaders. One

Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread Hongshun Wang
Hi David, In your modified pipeline, just one source from table1 is sufficient, with both sink1 and process2 sharing a single source from process1. However, based on your log, it appears that two sources have been generated. Do you have the execution graph available in the Flink UI? Best,

Flink Stateful Functions 3.4

2024-06-17 Thread L. Jiang
Hi there, Anyone knows which Flink version that Flink Stateful Functions 3.4 is compatible with? https://nightlies.apache.org/flink/flink-statefun-docs-master/docs/deployment/state-bootstrap/ I know Stateful Functions 3.3 is compatible with Flink 1.16.2, and Stateful Functions 3.2 is good with

Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread David Bryson
These sink share the same source. The pipeline that works looks something like this: table1 -> process1 -> process2 -> sink2 When I change it to this: table1 -> process1 -> process2 -> sink2 `--> sink1 I get the errors described, where it appears that a second

A way to meter number of deserialization errors

2024-06-17 Thread Ilya Karpov
Hi all, we are planning to use flink as a connector between kafka and external systems. We use protobuf as a message format in kafka. If non-backward compatible changes occur we want to skip those messages ('protobuf.ignore-parse-errors' = 'true') but record an error and raise an alert. I didn't

RE: Problem reading a CSV file with pyflink datastream in k8s with Flink operator

2024-06-17 Thread gwenael . lebarzic
Hello everyone. Does someone know how to solve this please ? Cdt. [Logo Orange] Gwenael Le Barzic Ingénieur technique techno BigData Orange/OF/DTSI/SI/DATA-IA/SOLID/CXP Mobile : +33 6 48 70 85 75

Re: Problems with multiple sinks using postgres-cdc connector

2024-06-17 Thread Hongshun Wang
Hi David, > When I add this second sink, the postgres-cdc connector appears to add a second reader from the replication log, but with the same slot name. I don't understand what you mean by adding a second sink. Do they share the same source, or does each have a separate pipeline? If the former