Hello, Thank you for your response. I have few more questions in following: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/
*Reactive Mode configures a job so that it always uses all resources available in the cluster. Adding a TaskManager will scale up your job, removing resources will scale it down. Flink will manage the parallelism of the job, always setting it to the highest possible values.* => Does this mean when I add/remove TaskManager in 'non-reactive' mode, resource(CPU/Memory/Etc.) of the cluster is not being changed? *Reactive Mode restarts a job on a rescaling event, restoring it from the latest completed checkpoint. This means that there is no overhead of creating a savepoint (which is needed for manually rescaling a job). Also, the amount of data that is reprocessed after rescaling depends on the checkpointing interval, and the restore time depends on the state size.* => As I know 'rescaling' also works in non-reactive mode, with restoring checkpoint. What is the difference of using 'reactive' here? *The Reactive Mode allows Flink users to implement a powerful autoscaling mechanism, by having an external service monitor certain metrics, such as consumer lag, aggregate CPU utilization, throughput or latency. As soon as these metrics are above or below a certain threshold, additional TaskManagers can be added or removed from the Flink cluster.* => Why is this only possible in 'reactive' mode? Seems this is more related to 'autoscaler'. Are there some specific features/API which can control TaskManager/Parallelism only in 'reactive' mode? Thank you. 2023년 9월 1일 (금) 오후 3:30, Gyula Fóra <gyula.f...@gmail.com>님이 작성: > The reactive mode reacts to available resources. The autoscaler reacts to > changing load and processing capacity and adjusts resources. > > Completely different concepts and applicability. > Most people want the autoscaler , but this is a recent feature and is > specific to the k8s operator at the moment. > > Gyula > > On Fri, 1 Sep 2023 at 04:50, Dennis Jung <inylov...@gmail.com> wrote: > >> Hello, >> Thanks for your notice. >> >> Than what is the purpose of using 'reactive', if this doesn't do anything >> itself? >> What is the difference if I use auto-scaler without 'reactive' mode? >> >> Regards, >> Jung >> >> >> >> 2023년 8월 18일 (금) 오후 7:51, Gyula Fóra <gyula.f...@gmail.com>님이 작성: >> >>> Hi! >>> >>> I think what you need is probably not the reactive mode but a proper >>> autoscaler. The reactive mode as you say doesn't do anything in itself, you >>> need to build a lot of logic around it. >>> >>> Check this instead: >>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ >>> >>> The Kubernetes Operator has a built in autoscaler that can scale jobs >>> based on kafka data rate / processing throughput. It also doesn't rely on >>> the reactive mode. >>> >>> Cheers, >>> Gyula >>> >>> On Fri, Aug 18, 2023 at 12:43 PM Dennis Jung <inylov...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> Sorry for frequent questions. This is a question about 'reactive' mode. >>>> >>>> 1. As far as I understand, though I've setup `scheduler-mode: >>>> reactive`, it will not change parallelism automatically by itself, by CPU >>>> usage or Kafka consumer rate. It needs additional resource monitor features >>>> (such as Horizontal Pod Autoscaler, or else). Is this correct? >>>> 2. Is it possible to create a custom resource monitor provider >>>> application? For example, if I want to increase/decrease parallelism by >>>> Kafka consumer rate, do I need to send specific API from outside, to order >>>> rescaling? >>>> 3. If 2 is correct, what is the difference when using 'reactive' mode? >>>> Because as far as I think, calling a specific API will rescale either using >>>> 'reactive' mode or not...(or is the API just working based on this mode)? >>>> >>>> Thanks. >>>> >>>> Regards >>>> >>>>
