Re:Re: Unable to do event time window aggregation with Kafka source

Hi Yuxia, Thanks for your reply! I expected the program to do reduce by key. It should count the number of data having the same username field. The program threw no exception, but it did not produce the expected output. Best regards, Lucas

Re: Unable to do event time window aggregation with Kafka source

Hi, Lucas. What do you mean by saying "unable to do event time window aggregation with watermarkedStream"? What exception it will throw? Best regards, Yuxia 发件人: "wei_yuze" 收件人: "User" 发送时间: 星期二, 2023年 2 月 07日 下午 1:43:59 主题: Unable to do event time window aggregation with Kafka source

Unable to do event time window aggregation with Kafka source

Hello！ I was unable to do event time window aggregation with Kafka source, but had no problem with "fromElement" source. The code is attached as follow. The code has two data sources, named "streamSource" and "kafkaSource" respectively. The program works well with "streamSource", but not wit

Re: Task Manger is getting failed

Hi, Maybe you should add the jvm.overhead memory. And you can do heap-dump to find which object has occupied a large number of memory. Best regards, Weijie P Singh 于2023年2月6日周一 11:24写道： > Hi weijie, > > I have 8gb memory still TM going oom though I have assigned all. Moreover > I am just pri

Re: Kafka Sink Kafka Producer metrics?

Hi Andrew, I misread the docs: `register.producer.metrics` is mentioned here, but it is not on by default. https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#kafka-connector-metrics Best, Mason On Mon, Feb 6, 2023 at 6:19 PM Mason Chen wrote: > Hi Andrew, >

Re: Kafka Sink Kafka Producer metrics?

Hi Andrew, Unfortunately, the functionality is undocumented, but you can set the property `register.producer.metrics` to true in your Kafka client properties map. This is a JIRA to document the feature: https://issues.apache.org/jira/browse/FLINK-30932 Best, Mason On Mon, Feb 6, 2023 at 11:49 AM

Re: Incompatible KafkaProducer version

Thanks for your suggestion Martjin. I am in the process of upgrading but this is kind of a blocker for coming to a sane state before proceeding. Is there any generic guide for version upgrading ? On Mon, Feb 6, 2023 at 11:38 AM Martijn Visser wrote: > Hi Milind Vaidya, > > I would highly re

Kafka Sink Kafka Producer metrics?

Hi! Kafka Source will emit KafkaConsumer metrics . It looks like Kafka Sink does no

Re: Incompatible KafkaProducer version

Hi Milind Vaidya, I would highly recommend upgrading your Flink cluster and applications. Flink 1.9 was released in August 2019 and is no longer supported by the community. Newer Kafka versions are supported on newer Flink versions. Best regards, Martijn Op ma 6 feb. 2023 om 20:19 schreef Milin

Incompatible KafkaProducer version

Hi Getting the following exception while trying to upgrade Kafka Libraries Flink version : 1.9.3 Old Kafka version : 2.2.1.8 New Kafka version : 2.4.1.4 Found a ticket : https://issues.apache.org/jira/browse/FLINK-9690 But that seems to be between 0.11 to 2.0.0. Cause : Caused by: java.lang.No

GCS setup

Im trying to set up my Flink cluster to store checkpoints on GCS - just a basic docker setup. per the instructions I add the gcs jar file and also export as env variable my credentials file path. *GOOGLE_APPLICATION_CREDENTIALS=/opt/flink/my-keys-sa.json* I also specify the checkpoints dir inside *