Metric to capture decoding failure in flink sources

2023-10-09 Thread Prateek Kohli
Hi, I need to get the difference between records which are collected by the source and the records which are emitted. For eg - If deserialization fails while reading records from kafka, in that case I want to expose the difference between records collected from Kafka Broker and records emitted fro

RE: confirm subscribe to user@flink.apache.org

2023-10-09 Thread Kumari, Anita
Hello Team, I am beginner to Apache Flink and developing Flink Java Application where I have requirement to consume events form kafkaSource and do some processing then Sink to MongoDb and from same MongoDB source/Fetch the same collection and display in console fetch data, how can I create thi

Re: observedGeneration field in FlinkDeployment

2023-10-09 Thread Tony Chen
I think that a FLIP JIRA should be created to add an `observedGeneration` field to the spec. When I look at other kubernetes APIs, I see the `observedGeneration` field in many of them: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/ On Mon, Sep 11, 2023 at 11:51 AM Tony Chen

flink exactly once semantics implementation internals

2023-10-09 Thread Enric Ott
Hi,Team: Does Flink really discard replayed duplicated records via record timestamp as the paper Lightweight Asynchronous Snapshots for Distributed Dataflows states "To achieve this we can follow a similar scheme to SDGs [5] and mark records with sequence numbers from the sources, thus, every do