Re: OOM errors cause by the new KafkaSink API

Hi Hua Wei, Thanks for the investigation. Could you provide a heap dump before the crash? The OOM stacktrace that you are showing is rather random (at RPC message exchange). We need to see where the heap is growing. Alternatively, you can take heap dumps at different points in time and compare

Re: OOM errors cause by the new KafkaSink API

Hi Martijn, > Have you built your own Flink version? Since Flink doesn't support Scala 2.12.12, the latest Scala version that Flink 1.14 supports is Scala 2.12.7 No, we use the public artifact from here to build our application. How can I

Re: OOM errors cause by the new KafkaSink API

Hi Hua Wei, Have you built your own Flink version? Since Flink doesn't support Scala 2.12.12, the latest Scala version that Flink 1.14 supports is Scala 2.12.7. > Because the new Kafka API needs the new serializer (KafkaRecordSerializationSchema) and seems like cannot use the old one

Re: OOM errors cause by the new KafkaSink API

Hi Martijn, Thanks for your response. > What's the Flink version that you're using? Our Flink version is 1.14.4 and the scala version is 2.12.12. > Could you also separate the two steps (switching from the old Kafka interfaces to the new ones + modifying serializers) to determine which of the

Re: OOM errors cause by the new KafkaSink API

Hi, What's the Flink version that you're using? Could you also separate the two steps (switching from the old Kafka interfaces to the new ones + modifying serializers) to determine which of the two steps cause the problem? Best regards, Martijn Visser https://twitter.com/MartijnVisser82

Re: OOM errors cause by the new KafkaSink API

Hi Huweihua, Thanks for the reply. Yes, we increased memory first. But we are still curious about the memory increasing with the new Kafka APIs/Serilizers. On Mon, Apr 25, 2022 at 8:38 PM huweihua wrote: > Hi, > > You can try to increase the memory of TaskManager. > If there is persistent

Re: OOM errors cause by the new KafkaSink API

Hi, You can try to increase the memory of TaskManager. If there is persistent OOM, you can dump the memory and check which part is taking up memory. > 2022年4月25日 上午11:44，Hua Wei Chen 写道： > > Hi all, > > Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at Flink >

OOM errors cause by the new KafkaSink API

Hi all, Due to FlinkKafkaConsumer and FlinkKafkaProducer will be depreciated at Flink 1.15*[1]*, we are trying to migrate the APIs to KafkaSource and KafkaSink*[2]*. At the same time, we also modified the serilizers*[3]*. Our Kafka settings are not changed*[4]*. The services are very stable