Flink's Kafka Offset Management

I use FlinkKafkaConsumer to consume Kafka and enable checkpoints. Now I'm a little confused about the offset management and checkpoint mechanism. What is the behavior if I stop the application by executing the yarn application -kill appId and run the start command like ./bin flink run ...? Will

JobManager Resident memory Always Increasing

Hi, We are running the JobManager container with 1024GB out of which 512MB is allocated to the heap. We observe that the JobManager container's resident memory is always increasing and it never comes down. Heap usage remains to be constant and not rising abruptly. Can anyone help here where els

Getting "NaN" in KafkaConsumer.records-lag-max in flink

Hi, I am not getting anything in the metric "flink.operator.KafkaConsumer.records-lag-max". Tried checking the value of Mbean using JMX for "org.apache.flink.taskmanager.job.task.operator.KafkaConsumer.records-lag-max" but I am always getting either NaN or 0. I have checked other Mbeans su

Lot of java.util.zip.ZipFile$Source in JobManager's Heap

Hi, On analyzing the heap dump of the JobManager process, I am seeing a lot of instances of java.util.zip.ZipFile$Source which includes open file handles to jar files. These instances are never garbage collected. I am also observing that the Resident Memory of the process never goes down. It co

Metric for JVM Overhaed

Hi, Is there any metric to track the task manager JVM overhead? Or is it the case that it is already included in the metric Status.JVM.Memory.NonHeap? Thanks, Pranjul