Re: [EXTERNAL] Re: Query on flink-operator autoscale support

[Swathi Chandrashekar via user]

Sure, thanks Gyula.
Is there a roadmap to support standalone session clusters to scale based on the 
jobs added/deleted and change in parallelism ?

Regards,
Swathi C


From: Gyula Fóra 
Sent: Wednesday, February 1, 2023 8:54 PM
To: Swathi Chandrashekar 
Cc: user@flink.apache.org 
Subject: [EXTERNAL] Re: Query on flink-operator autoscale support

The autoscaler currently only works with Native App clusters.
Native session clusters may be supported in the future but standalone is not on 
our roadmap due to a different resource/scheduling model used.

Gyula

On Wed, Feb 1, 2023 at 4:22 PM Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Hi,

I'm was testing OSS flink operator with flink 1.17 for autoscale feature. I was 
able to scale the cluster based on load in application cluster in native mode, 
but the same did not work in standalone mode as the operator gave the following 
error as below [ both for app and session mode ].

Is the autoscale supported for the following :

  1.  Session cluster in standalone
  2.  Session cluster in native
  3.  App cluster in standalone

Exception in thread "pool-4-thread-3" java.lang.NoSuchMethodError: 
org.apache.flink.kubernetes.operator.kubeclient.Fabric8FlinkStandaloneKubeClient.create(Lorg/apache/flink/configuration/Configuration;Ljava/util/concurrent/ExecutorService;)Lorg/apache/flink/kubernetes/operator/kubeclient/Fabric8FlinkStandaloneKubeClient;
  at 
org.apache.flink.kubernetes.operator.service.StandaloneFlinkService.createNamespacedKubeClient(StandaloneFlinkService.java:105)
  at 
org.apache.flink.kubernetes.operator.service.StandaloneFlinkService.submitClusterInternal(StandaloneFlinkService.java:110)
  at 
org.apache.flink.kubernetes.operator.service.StandaloneFlinkService.deployApplicationCluster(StandaloneFlinkService.java:69)
  at 
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.submitApplicationCluster(AbstractFlinkService.java:180)
  at 
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:175)
  at 
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:58)
  at 
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:115)
  at 
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:130)
  at 
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)
  at 
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
  at 
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
  at 
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
  at 
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
  at 
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)


Regards,
Swathi C

Query on flink-operator autoscale support

[Swathi Chandrashekar via user]

Hi,

I'm was testing OSS flink operator with flink 1.17 for autoscale feature. I was 
able to scale the cluster based on load in application cluster in native mode, 
but the same did not work in standalone mode as the operator gave the following 
error as below [ both for app and session mode ].

Is the autoscale supported for the following :

  1.  Session cluster in standalone
  2.  Session cluster in native
  3.  App cluster in standalone

Exception in thread "pool-4-thread-3" java.lang.NoSuchMethodError: 
org.apache.flink.kubernetes.operator.kubeclient.Fabric8FlinkStandaloneKubeClient.create(Lorg/apache/flink/configuration/Configuration;Ljava/util/concurrent/ExecutorService;)Lorg/apache/flink/kubernetes/operator/kubeclient/Fabric8FlinkStandaloneKubeClient;
  at 
org.apache.flink.kubernetes.operator.service.StandaloneFlinkService.createNamespacedKubeClient(StandaloneFlinkService.java:105)
  at 
org.apache.flink.kubernetes.operator.service.StandaloneFlinkService.submitClusterInternal(StandaloneFlinkService.java:110)
  at 
org.apache.flink.kubernetes.operator.service.StandaloneFlinkService.deployApplicationCluster(StandaloneFlinkService.java:69)
  at 
org.apache.flink.kubernetes.operator.service.AbstractFlinkService.submitApplicationCluster(AbstractFlinkService.java:180)
  at 
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:175)
  at 
org.apache.flink.kubernetes.operator.reconciler.deployment.ApplicationReconciler.deploy(ApplicationReconciler.java:58)
  at 
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:115)
  at 
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:130)
  at 
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)
  at 
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
  at 
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
  at 
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
  at 
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
  at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
  at 
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)


Regards,
Swathi C

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

[Swathi Chandrashekar via user]

Got it, thanks @Gyula Fóra 

From: Gyula Fóra 
Sent: Friday, January 13, 2023 12:09 PM
To: Swathi Chandrashekar 
Cc: Chesnay Schepler ; Tamir Sagi 
; user@flink.apache.org
Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS 
EKS

What I am trying to say is use the Kubernetes operator with Native (default) 
mode and forget about reactive .

The operator does everything you wwant plus has an actual autoscaler.

Gyula

On Fri, 13 Jan 2023 at 07:24, Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Got it, so this means, we should have standalone app mode cluster which is 
managed by a flink Kubernetes operator and the operator would update the 
replicas based on the metrics ( autoscale ) which in-tern changes the 
parallelism as reactivemode is enabled.

Regards,
Swathi C

From: Gyula Fóra mailto:gyula.f...@gmail.com>>
Sent: Friday, January 13, 2023 11:31 AM
To: Swathi Chandrashekar mailto:cswa...@microsoft.com>>
Cc: Chesnay Schepler mailto:ches...@apache.org>>; Tamir 
Sagi mailto:tamir.s...@niceactimize.com>>; 
user@flink.apache.org
Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS 
EKS

No but the Kubernetes operator itself already provides similar feature set.

Not sure why you want the reactive mode in the first place . If it's because 
you want to implement auto scaling on top of it, then I think the operator is a 
better alternative.

I think you should try to understand what exactly the reactive mode provides vs 
what the operator does. Reactive mode alone doesn’t do too much.

Gyula

On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Hi @Gyula Fóra,

Does this mean, with Kubernetes operator, we can have reactive mode in native 
flink which is in app mode ? [ Not just standalone app mode ]

Regards,
Swathi C

From: Gyula Fóra mailto:gyula.f...@gmail.com>>
Sent: Thursday, January 12, 2023 11:14 PM
To: Tamir Sagi mailto:tamir.s...@niceactimize.com>>
Cc: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org
Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only works 
in standalone deployments (not in Kubernetes native for instace) and it doesn't 
actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes HPA 
or something similar)

I suggest looking at the Flink Kubernetes Operator 
(https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
 that will provide actual autoscaler capability for native Kubernetes 
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
mailto:tamir.s...@niceactimize.com>> wrote:
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:image001.png@01D9274A.1DEF1050]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler mailto:ches...@apache.org>>
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi 

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

[Swathi Chandrashekar via user]

Got it, so this means, we should have standalone app mode cluster which is 
managed by a flink Kubernetes operator and the operator would update the 
replicas based on the metrics ( autoscale ) which in-tern changes the 
parallelism as reactivemode is enabled.

Regards,
Swathi C

From: Gyula Fóra 
Sent: Friday, January 13, 2023 11:31 AM
To: Swathi Chandrashekar 
Cc: Chesnay Schepler ; Tamir Sagi 
; user@flink.apache.org
Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS 
EKS

No but the Kubernetes operator itself already provides similar feature set.

Not sure why you want the reactive mode in the first place . If it's because 
you want to implement auto scaling on top of it, then I think the operator is a 
better alternative.

I think you should try to understand what exactly the reactive mode provides vs 
what the operator does. Reactive mode alone doesn't do too much.

Gyula

On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Hi @Gyula Fóra,

Does this mean, with Kubernetes operator, we can have reactive mode in native 
flink which is in app mode ? [ Not just standalone app mode ]

Regards,
Swathi C

From: Gyula Fóra mailto:gyula.f...@gmail.com>>
Sent: Thursday, January 12, 2023 11:14 PM
To: Tamir Sagi mailto:tamir.s...@niceactimize.com>>
Cc: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org
Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only works 
in standalone deployments (not in Kubernetes native for instace) and it doesn't 
actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes HPA 
or something similar)

I suggest looking at the Flink Kubernetes Operator 
(https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
 that will provide actual autoscaler capability for native Kubernetes 
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
mailto:tamir.s...@niceactimize.com>> wrote:
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:image001.png@01D92745.AD5EACF0]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler mailto:ches...@apache.org>>
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS


EXTERNAL EMAIL

The adaptive scheduler and reactive mode both already support application 
clusters since 1.13.

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

[Swathi Chandrashekar via user]

Hi @Gyula Fóra,

Does this mean, with Kubernetes operator, we can have reactive mode in native 
flink which is in app mode ? [ Not just standalone app mode ]

Regards,
Swathi C

From: Gyula Fóra 
Sent: Thursday, January 12, 2023 11:14 PM
To: Tamir Sagi 
Cc: Chesnay Schepler ; user@flink.apache.org
Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only works 
in standalone deployments (not in Kubernetes native for instace) and it doesn't 
actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes HPA 
or something similar)

I suggest looking at the Flink Kubernetes Operator 
(https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
 that will provide actual autoscaler capability for native Kubernetes 
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
mailto:tamir.s...@niceactimize.com>> wrote:
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:image001.png@01D9273E.B14ADF70]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler mailto:ches...@apache.org>>
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS


EXTERNAL EMAIL

The adaptive scheduler and reactive mode both already support application 
clusters since 1.13.

https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/

On 19/12/2022 10:17, Tamir Sagi wrote:
Hey,

We are running stream jobs on application clusters (v1.15.2) on AWS EKS.

I was reviewing the following pages on Flink confluence

  *   Reactive mode [1]
  *   Adaptive Scheduler [2]
I also encountered the following POC conducted by Robert Metzger 
(@rmetzger_)
 on 06 May 2021. [3]

my question is whether that feature will be supported in the future for 
application clusters or not.

[1]