Flink Job Failed With Kafka Exception

2023-10-10 Thread Madan D via user
Hello Team, We are running the Flink pipeline by consuming data from multiple topics, but we recently encountered that if there's one topic having issues with participation, etc., the whole Flink pipeline is failing, which is affecting topics. Is there a way we can make Flink Piplein keep

Re: Metric to capture decoding failure in flink sources

2023-10-10 Thread Mason Chen
Hi Prateek, I agree, the reader should ideally expose the context to record metrics about deserialization. One option is to defer deserialization to another operator, say a RichMapFunction that has access to the RuntimeContext. Best, Mason On Mon, Oct 9, 2023 at 12:42 PM Prateek Kohli wrote:

Running FlinkSQL locally

2023-10-10 Thread Ralph Matthias Debusmann
Hi, I am trying to run FlinkSQL (through sql-client.sh) and read messages from Kafka topics. I downloaded 1.17.1 and extracted it. For the Kafka connectivity, I've added flink-sql-connector-kafka-1.1.7.jar, flink-connector-kafka-1.1.7.jar and kafka-clients-2.8.1.jar and now start the

Announcing public preview of Flink with Azure HDInsight on AKS

2023-10-10 Thread ramkrishna vasudevan
We are super excited to announce the public preview of Flink on Azure, as a managed platform as a service. You can read more information on this blog and reach out to me!

File Source Watermark Issue

2023-10-10 Thread Kirti Dhar Upadhyay K via user
Hi Team, I am using Flink File Source with window aggregator as process function, and stuck with a weird issues. File source doesn't seem emitting/progressing the watermarks, whereas if I put a delay (say 100ms) while extracting timestamp from event, it is working fine. A bit same thing I