Hi, When running a streaming job that uses a kafka source, is it possible (without reactive mode) for flink to dynamically detect high load (high consumers lag, high cpu usage...) and increase the job parallelism automatically?
I am running flink streaming job on an application mode cluster using native k8s. My streaming job is consuming messages from a kafka topic with 16 partitions, parallelism.default is set to 2, no parallelism is set specifically on the operators/sources/sinks. I tried to send multiple message to the kafka topic at high rate, faster than the job can consume, and I saw that the consumer lag was increasing. I also saw in the flink UI that the source task was turning red, indicating a high usage of this task. Even though I created a high load on the job, I didn't see that flink automatically changes the parallelism of the job to handle the high load. Is possible for Flink to increase the parallelism of my job (or of my source) dynamically based on the current load (and add task managers automatically)? Or is this behavior only available by using reactive mode? For reactive mode, my understanding based on the documentation is that it is in MVP state and is only supported in standalone mode, and is not ready yet for production use. Thanks, Erez Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
