Hi @Gyula Fóra<mailto:gyula.f...@gmail.com>,
Does this mean, with Kubernetes operator, we can have reactive mode in native flink which is in app mode ? [ Not just standalone app mode ] Regards, Swathi C From: Gyula Fóra <gyula.f...@gmail.com> Sent: Thursday, January 12, 2023 11:14 PM To: Tamir Sagi <tamir.s...@niceactimize.com> Cc: Chesnay Schepler <ches...@apache.org>; user@flink.apache.org Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS Hey! I think the reactive scaling is a somewhat misunderstood feature. It only works in standalone deployments (not in Kubernetes native for instace) and it doesn't actually provide any autoscaling functionality on its own. You would have to implement your scaling logic yourself somehow (Kubernetes HPA or something similar) I suggest looking at the Flink Kubernetes Operator (https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-kubernetes-operator-docs-main%2F&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pu6i2YmIsU9WtvaXRIZnmEJraW9c8X5NT81F7%2BZLyVI%3D&reserved=0>) that will provide actual autoscaler capability for native Kubernetes deployments. Cheers, Gyula On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi <tamir.s...@niceactimize.com<mailto:tamir.s...@niceactimize.com>> wrote: Hey Chesnay, Just to be more clear, I'm talking about plans to support reactive mode for application clusters in Native Kubernetes. https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fdeployment%2Fresource-providers%2Fnative_kubernetes%2F%23application-mode&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8Pg0eyyEZc5SC0cBnjh14nLY2EWXOhRI3tLIK9sEGFM%3D&reserved=0> Thanks, Tamir. ________________________________ From: Tamir Sagi <tamir.s...@niceactimize.com<mailto:tamir.s...@niceactimize.com>> Sent: Thursday, January 12, 2023 6:17 PM To: Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>>; user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Flink reactive mode for application clusters on AWS EKS Hey Chesnay, Thank you for your response. Since we are running our Flink jobs on EKS (Elastic Kubernetes Service) I was asking regarding Application cluster on Kubernetes. The documentations I referred to clearly state that it is not supported, the same as shown on Flink website. [cid:image001.png@01D9273E.B14ADF70] Is there any plan to support that anytime soon? Thanks Tamir. ________________________________ From: Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>> Sent: Thursday, January 12, 2023 4:30 PM To: Tamir Sagi <tamir.s...@niceactimize.com<mailto:tamir.s...@niceactimize.com>>; user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Flink reactive mode for application clusters on AWS EKS EXTERNAL EMAIL The adaptive scheduler and reactive mode both already support application clusters since 1.13. https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-release-1.16%2Fdocs%2Fdeployment%2Felastic_scaling%2F&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hz3WIHO4qE8kc9l5RwzWVdaZ1eXIPy329hozq%2BNM0Vs%3D&reserved=0> On 19/12/2022 10:17, Tamir Sagi wrote: Hey, We are running stream jobs on application clusters (v1.15.2) on AWS EKS. I was reviewing the following pages on Flink confluence * Reactive mode [1] * Adaptive Scheduler [2] I also encountered the following POC conducted by Robert Metzger (@rmetzger_<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frmetzger_&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UbfHEKsidqafYPLYzMY%2B2e17bARKAnp2YtywVVk5st0%3D&reserved=0>) on 06 May 2021. [3] my question is whether that feature will be supported in the future for application clusters or not. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-159%253A%2BReactive%2BMode&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DuEpGmH7aTXRFACDJxM%2FvhVEcQcKghjrayDA6fhDfsY%3D&reserved=0> [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-160%253A%2BAdaptive%2BScheduler&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XEL4Ezv%2F0FiRx7FkEeJgDQ7cJixxfhmjNycJ9P5l6PQ%3D&reserved=0> [3] https://flink.apache.org/2021/05/06/reactive-mode.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fflink.apache.org%2F2021%2F05%2F06%2Freactive-mode.html&data=05%7C01%7Ccswathi%40microsoft.com%7C86c2c646cd6e4d9d221e08daf4c49f6d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638091422654166711%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bF8oXBdbdGbM%2BuGwh0F%2BPxTR7U%2Fny72wi91us9qh2MI%3D&reserved=0> Thanks, Tamir. Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free. Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free. Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
