HybridSource: how to specify different watermark strategies for underlying sources

2023-01-12 Thread David Christle via user 
Hello,

I'm trying to use the HybridSource to read from a bounded FLIP-27 Iceberg
source and an unbounded FLIP-27 PubSub source. Based on my searches, the
most common patterns for assigning timestamps and watermarks with FLIP-27
sources are either via the fromSource method in
StreamingExecutionEnvironment, which returns a subclass of DataStream, or
using assignTimestampsAndWatermarks(WatermarkStrategy wms) on an already
existing DataStream.

But the only way to build a HybridSource, it seems, is via its addSource
method, which accepts only Source types. We cannot call an analogue of
assignTimestampsAndWatermarks on a Source.

We could perhaps call that method on the DataStream produced by
env.fromSource(hybridSource), but I believe that would use the same
timestamp & watermark assignment strategy for both. We need the watermarks
to be assigned differently based on the source. Our Iceberg tables are
partitioned by day, so the watermark lags by about 1 day when we use
event-time aligned assignment until it finishes. Those watermarks originate
within the Source. But for PubSub, we'd like to use the generic bounded
out-of-orderness watermark strategy for just a few seconds of
out-of-orderness. The PubSub source doesn't generate watermarks within the
source -- there is no per-split information exposed by PubSub that would
make a better watermark, like in the Kafka connector. The only way to
assign it a watermark strategy is via the DataStream-level assignment
methods.

Is the only way to use different watermark strategies to modify the PubSub
Source to do the assignment within the Source?

I wonder if, given that it seems much more common to assign watermarks at
the DataStream level (e.g. FLIP-182's examples), if HybridSource could be
modified to accept an optional WatermarkStrategy for each underlying
source, similar to how fromSource works.

Kind regards,
David

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Swathi Chandrashekar via user 
Got it, thanks @Gyula Fóra 

From: Gyula Fóra 
Sent: Friday, January 13, 2023 12:09 PM
To: Swathi Chandrashekar 
Cc: Chesnay Schepler ; Tamir Sagi 
; user@flink.apache.org
Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS 
EKS

What I am trying to say is use the Kubernetes operator with Native (default) 
mode and forget about reactive .

The operator does everything you wwant plus has an actual autoscaler.

Gyula

On Fri, 13 Jan 2023 at 07:24, Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Got it, so this means, we should have standalone app mode cluster which is 
managed by a flink Kubernetes operator and the operator would update the 
replicas based on the metrics ( autoscale ) which in-tern changes the 
parallelism as reactivemode is enabled.

Regards,
Swathi C

From: Gyula Fóra mailto:gyula.f...@gmail.com>>
Sent: Friday, January 13, 2023 11:31 AM
To: Swathi Chandrashekar mailto:cswa...@microsoft.com>>
Cc: Chesnay Schepler mailto:ches...@apache.org>>; Tamir 
Sagi mailto:tamir.s...@niceactimize.com>>; 
user@flink.apache.org
Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS 
EKS

No but the Kubernetes operator itself already provides similar feature set.

Not sure why you want the reactive mode in the first place . If it's because 
you want to implement auto scaling on top of it, then I think the operator is a 
better alternative.

I think you should try to understand what exactly the reactive mode provides vs 
what the operator does. Reactive mode alone doesn’t do too much.

Gyula

On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Hi @Gyula Fóra,

Does this mean, with Kubernetes operator, we can have reactive mode in native 
flink which is in app mode ? [ Not just standalone app mode ]

Regards,
Swathi C

From: Gyula Fóra mailto:gyula.f...@gmail.com>>
Sent: Thursday, January 12, 2023 11:14 PM
To: Tamir Sagi mailto:tamir.s...@niceactimize.com>>
Cc: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org
Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only works 
in standalone deployments (not in Kubernetes native for instace) and it doesn't 
actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes HPA 
or something similar)

I suggest looking at the Flink Kubernetes Operator 
(https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
 that will provide actual autoscaler capability for native Kubernetes 
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
mailto:tamir.s...@niceactimize.com>> wrote:
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:image001.png@01D9274A.1DEF1050]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler mailto:ches...@apache.org>>
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Gyula Fóra 
What I am trying to say is use the Kubernetes operator with Native
(default) mode and forget about reactive .

The operator does everything you wwant plus has an actual autoscaler.

Gyula

On Fri, 13 Jan 2023 at 07:24, Swathi Chandrashekar 
wrote:

> Got it, so this means, we should have standalone app mode cluster which is
> managed by a flink Kubernetes operator and the operator would update the
> replicas based on the metrics ( autoscale ) which in-tern changes the
> parallelism as reactivemode is enabled.
>
>
>
> Regards,
>
> Swathi C
>
>
>
> *From:* Gyula Fóra 
> *Sent:* Friday, January 13, 2023 11:31 AM
> *To:* Swathi Chandrashekar 
> *Cc:* Chesnay Schepler ; Tamir Sagi <
> tamir.s...@niceactimize.com>; user@flink.apache.org
> *Subject:* Re: [EXTERNAL] Re: Flink reactive mode for application
> clusters on AWS EKS
>
>
>
> No but the Kubernetes operator itself already provides similar feature set.
>
>
>
> Not sure why you want the reactive mode in the first place . If it's
> because you want to implement auto scaling on top of it, then I think the
> operator is a better alternative.
>
>
>
> I think you should try to understand what exactly the reactive mode
> provides vs what the operator does. Reactive mode alone doesn’t do too much.
>
>
>
> Gyula
>
>
>
> On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar 
> wrote:
>
> Hi @Gyula Fóra ,
>
>
>
> Does this mean, with Kubernetes operator, we can have reactive mode in
> native flink which is in app mode ? [ Not just standalone app mode ]
>
>
>
> Regards,
>
> Swathi C
>
>
>
> *From:* Gyula Fóra 
> *Sent:* Thursday, January 12, 2023 11:14 PM
> *To:* Tamir Sagi 
> *Cc:* Chesnay Schepler ; user@flink.apache.org
> *Subject:* [EXTERNAL] Re: Flink reactive mode for application clusters on
> AWS EKS
>
>
>
> Hey!
>
> I think the reactive scaling is a somewhat misunderstood feature. It only
> works in standalone deployments (not in Kubernetes native for instace) and
> it doesn't actually provide any autoscaling functionality on its own.
> You would have to implement your scaling logic yourself somehow
> (Kubernetes HPA or something similar)
>
> I suggest looking at the Flink Kubernetes Operator (
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
> )
> that will provide actual autoscaler capability for native Kubernetes
> deployments.
>
> Cheers,
> Gyula
>
>
>
> On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
> wrote:
>
> Hey Chesnay,
>
>
>
> Just to be more clear,
>
> I'm talking about plans to support reactive mode for application clusters
> in Native Kubernetes.
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode
> 
>
>
>
> Thanks,
>
> Tamir.
>
>
> --
>
> *From:* Tamir Sagi 
> *Sent:* Thursday, January 12, 2023 6:17 PM
> *To:* Chesnay Schepler ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Flink reactive mode for application clusters on AWS EKS
>
>
>
> Hey Chesnay,
>
>
>
> Thank you for your response.
>
>
>
> Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I
> was asking regarding Application cluster on Kubernetes.
>
>
>
> The documentations I referred to clearly state that it is not
> supported, the same as shown on Flink website.
>
>
>
> Is there any plan to support that anytime soon?
>
>
>
> Thanks
>
>
>
> Tamir.
> --
>
> *From:* Chesnay Schepler 
> *Sent:* Thursday, January 12, 2023 4:30 PM
> *To:* Tamir Sagi ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Flink reactive mode for application clusters on AWS EKS
>
>
>
> *EXTERNAL EMAIL*
>
>
>
> The adaptive scheduler and reactive mode both already support application
> clusters since 1.13.
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/
>

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Swathi Chandrashekar via user 
Got it, so this means, we should have standalone app mode cluster which is 
managed by a flink Kubernetes operator and the operator would update the 
replicas based on the metrics ( autoscale ) which in-tern changes the 
parallelism as reactivemode is enabled.

Regards,
Swathi C

From: Gyula Fóra 
Sent: Friday, January 13, 2023 11:31 AM
To: Swathi Chandrashekar 
Cc: Chesnay Schepler ; Tamir Sagi 
; user@flink.apache.org
Subject: Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS 
EKS

No but the Kubernetes operator itself already provides similar feature set.

Not sure why you want the reactive mode in the first place . If it's because 
you want to implement auto scaling on top of it, then I think the operator is a 
better alternative.

I think you should try to understand what exactly the reactive mode provides vs 
what the operator does. Reactive mode alone doesn't do too much.

Gyula

On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar 
mailto:cswa...@microsoft.com>> wrote:
Hi @Gyula Fóra,

Does this mean, with Kubernetes operator, we can have reactive mode in native 
flink which is in app mode ? [ Not just standalone app mode ]

Regards,
Swathi C

From: Gyula Fóra mailto:gyula.f...@gmail.com>>
Sent: Thursday, January 12, 2023 11:14 PM
To: Tamir Sagi mailto:tamir.s...@niceactimize.com>>
Cc: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org
Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only works 
in standalone deployments (not in Kubernetes native for instace) and it doesn't 
actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes HPA 
or something similar)

I suggest looking at the Flink Kubernetes Operator 
(https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
 that will provide actual autoscaler capability for native Kubernetes 
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
mailto:tamir.s...@niceactimize.com>> wrote:
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:image001.png@01D92745.AD5EACF0]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler mailto:ches...@apache.org>>
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS


EXTERNAL EMAIL

The adaptive scheduler and reactive mode both already support application 
clusters since 1.13.

Re: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Gyula Fóra 
No but the Kubernetes operator itself already provides similar feature set.

Not sure why you want the reactive mode in the first place . If it's
because you want to implement auto scaling on top of it, then I think the
operator is a better alternative.

I think you should try to understand what exactly the reactive mode
provides vs what the operator does. Reactive mode alone doesn’t do too much.

Gyula

On Fri, 13 Jan 2023 at 06:33, Swathi Chandrashekar 
wrote:

> Hi @Gyula Fóra ,
>
>
>
> Does this mean, with Kubernetes operator, we can have reactive mode in
> native flink which is in app mode ? [ Not just standalone app mode ]
>
>
>
> Regards,
>
> Swathi C
>
>
>
> *From:* Gyula Fóra 
> *Sent:* Thursday, January 12, 2023 11:14 PM
> *To:* Tamir Sagi 
> *Cc:* Chesnay Schepler ; user@flink.apache.org
> *Subject:* [EXTERNAL] Re: Flink reactive mode for application clusters on
> AWS EKS
>
>
>
> Hey!
>
> I think the reactive scaling is a somewhat misunderstood feature. It only
> works in standalone deployments (not in Kubernetes native for instace) and
> it doesn't actually provide any autoscaling functionality on its own.
> You would have to implement your scaling logic yourself somehow
> (Kubernetes HPA or something similar)
>
> I suggest looking at the Flink Kubernetes Operator (
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
> )
> that will provide actual autoscaler capability for native Kubernetes
> deployments.
>
> Cheers,
> Gyula
>
>
>
> On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
> wrote:
>
> Hey Chesnay,
>
>
>
> Just to be more clear,
>
> I'm talking about plans to support reactive mode for application clusters
> in Native Kubernetes.
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode
> 
>
>
>
> Thanks,
>
> Tamir.
>
>
> --
>
> *From:* Tamir Sagi 
> *Sent:* Thursday, January 12, 2023 6:17 PM
> *To:* Chesnay Schepler ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Flink reactive mode for application clusters on AWS EKS
>
>
>
> Hey Chesnay,
>
>
>
> Thank you for your response.
>
>
>
> Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I
> was asking regarding Application cluster on Kubernetes.
>
>
>
> The documentations I referred to clearly state that it is not
> supported, the same as shown on Flink website.
>
>
>
> Is there any plan to support that anytime soon?
>
>
>
> Thanks
>
>
>
> Tamir.
> --
>
> *From:* Chesnay Schepler 
> *Sent:* Thursday, January 12, 2023 4:30 PM
> *To:* Tamir Sagi ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Flink reactive mode for application clusters on AWS EKS
>
>
>
> *EXTERNAL EMAIL*
>
>
>
> The adaptive scheduler and reactive mode both already support application
> clusters since 1.13.
>
>
>
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/
> 
>
>
>
> On 19/12/2022 10:17, Tamir Sagi wrote:
>
> Hey,
>
>
>
> We are running stream jobs on application clusters (v1.15.2) on AWS EKS.
>
>
>
> I was reviewing the following pages on Flink confluence
>
>- Reactive mode [1]
>- Adaptive Scheduler [2]
>
> I also encountered the following POC conducted by Robert Metzger (
> @rmetzger_
>

RE: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Swathi Chandrashekar via user 
Hi @Gyula Fóra,

Does this mean, with Kubernetes operator, we can have reactive mode in native 
flink which is in app mode ? [ Not just standalone app mode ]

Regards,
Swathi C

From: Gyula Fóra 
Sent: Thursday, January 12, 2023 11:14 PM
To: Tamir Sagi 
Cc: Chesnay Schepler ; user@flink.apache.org
Subject: [EXTERNAL] Re: Flink reactive mode for application clusters on AWS EKS

Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only works 
in standalone deployments (not in Kubernetes native for instace) and it doesn't 
actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes HPA 
or something similar)

I suggest looking at the Flink Kubernetes Operator 
(https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
 that will provide actual autoscaler capability for native Kubernetes 
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
mailto:tamir.s...@niceactimize.com>> wrote:
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler mailto:ches...@apache.org>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:image001.png@01D9273E.B14ADF70]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler mailto:ches...@apache.org>>
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi 
mailto:tamir.s...@niceactimize.com>>; 
user@flink.apache.org 
mailto:user@flink.apache.org>>
Subject: Re: Flink reactive mode for application clusters on AWS EKS


EXTERNAL EMAIL

The adaptive scheduler and reactive mode both already support application 
clusters since 1.13.

https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/

On 19/12/2022 10:17, Tamir Sagi wrote:
Hey,

We are running stream jobs on application clusters (v1.15.2) on AWS EKS.

I was reviewing the following pages on Flink confluence

  *   Reactive mode [1]
  *   Adaptive Scheduler [2]
I also encountered the following POC conducted by Robert Metzger 
(@rmetzger_)
 on 06 May 2021. [3]

my question is whether that feature will be supported in the future for 
application clusters or not.

[1]

退订

2023-01-12 Thread willluzheng 
退订

退订

2023-01-12 Thread DannyLau 
退订

Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Gyula Fóra 
Hey!

I think the reactive scaling is a somewhat misunderstood feature. It only
works in standalone deployments (not in Kubernetes native for instace) and
it doesn't actually provide any autoscaling functionality on its own.
You would have to implement your scaling logic yourself somehow (Kubernetes
HPA or something similar)

I suggest looking at the Flink Kubernetes Operator (
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/)
that will provide actual autoscaler capability for native Kubernetes
deployments.

Cheers,
Gyula

On Thu, Jan 12, 2023 at 6:23 PM Tamir Sagi 
wrote:

> Hey Chesnay,
>
> Just to be more clear,
> I'm talking about plans to support reactive mode for application clusters
> in Native Kubernetes.
>
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode
>
> Thanks,
> Tamir.
>
> --
> *From:* Tamir Sagi 
> *Sent:* Thursday, January 12, 2023 6:17 PM
> *To:* Chesnay Schepler ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Flink reactive mode for application clusters on AWS EKS
>
> Hey Chesnay,
>
> Thank you for your response.
>
> Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I
> was asking regarding Application cluster on Kubernetes.
>
> The documentations I referred to clearly state that it is not supported, the
> same as shown on Flink website.
>
>
> Is there any plan to support that anytime soon?
>
> Thanks
>
> Tamir.
> --
> *From:* Chesnay Schepler 
> *Sent:* Thursday, January 12, 2023 4:30 PM
> *To:* Tamir Sagi ; user@flink.apache.org <
> user@flink.apache.org>
> *Subject:* Re: Flink reactive mode for application clusters on AWS EKS
>
>
> *EXTERNAL EMAIL*
>
>
> The adaptive scheduler and reactive mode both already support application
> clusters since 1.13.
>
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/
>
> On 19/12/2022 10:17, Tamir Sagi wrote:
>
> Hey,
>
> We are running stream jobs on application clusters (v1.15.2) on AWS EKS.
>
> I was reviewing the following pages on Flink confluence
>
>- Reactive mode [1]
>- Adaptive Scheduler [2]
>
> I also encountered the following POC conducted by Robert Metzger (
> @rmetzger_ ) on 06 May 2021. [3]
>
> my question is whether that feature will be supported in the future for
> application clusters or not.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler
> [3] https://flink.apache.org/2021/05/06/reactive-mode.html
>
>
> Thanks,
> Tamir.
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually

Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Tamir Sagi 
Hey Chesnay,

Just to be more clear,
I'm talking about plans to support reactive mode for application clusters in 
Native Kubernetes.

https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#application-mode

Thanks,
Tamir.


From: Tamir Sagi 
Sent: Thursday, January 12, 2023 6:17 PM
To: Chesnay Schepler ; user@flink.apache.org 

Subject: Re: Flink reactive mode for application clusters on AWS EKS

Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:29a0549c-c874-41ad-bd24-b58a2b8ae3a4]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler 
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi ; user@flink.apache.org 

Subject: Re: Flink reactive mode for application clusters on AWS EKS


EXTERNAL EMAIL


The adaptive scheduler and reactive mode both already support application 
clusters since 1.13.

https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/

On 19/12/2022 10:17, Tamir Sagi wrote:
Hey,

We are running stream jobs on application clusters (v1.15.2) on AWS EKS.

I was reviewing the following pages on Flink confluence

  *   Reactive mode [1]
  *   Adaptive Scheduler [2]

I also encountered the following POC conducted by Robert Metzger 
(@rmetzger_) on 06 May 2021. [3]

my question is whether that feature will be supported in the future for 
application clusters or not.

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler
[3] https://flink.apache.org/2021/05/06/reactive-mode.html


Thanks,
Tamir.


Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.


Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Tamir Sagi 
Hey Chesnay,

Thank you for your response.

Since we are running our Flink jobs on EKS (Elastic Kubernetes Service)  I was 
asking regarding Application cluster on Kubernetes.

The documentations I referred to clearly state that it is not supported, the 
same as shown on Flink website.

[cid:29a0549c-c874-41ad-bd24-b58a2b8ae3a4]
Is there any plan to support that anytime soon?

Thanks

Tamir.

From: Chesnay Schepler 
Sent: Thursday, January 12, 2023 4:30 PM
To: Tamir Sagi ; user@flink.apache.org 

Subject: Re: Flink reactive mode for application clusters on AWS EKS


EXTERNAL EMAIL


The adaptive scheduler and reactive mode both already support application 
clusters since 1.13.

https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/

On 19/12/2022 10:17, Tamir Sagi wrote:
Hey,

We are running stream jobs on application clusters (v1.15.2) on AWS EKS.

I was reviewing the following pages on Flink confluence

  *   Reactive mode [1]
  *   Adaptive Scheduler [2]

I also encountered the following POC conducted by Robert Metzger 
(@rmetzger_) on 06 May 2021. [3]

my question is whether that feature will be supported in the future for 
application clusters or not.

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler
[3] https://flink.apache.org/2021/05/06/reactive-mode.html


Thanks,
Tamir.


Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.


Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

Re: Metrics exposed by flink containing long label values

2023-01-12 Thread Chesnay Schepler 
See 
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/#scope-variables-excludes


On 06/01/2023 12:09, Surendra Lalwani via user wrote:

Hi Team,

We are exposing metrics by Flink to prometheus but we are seeing it 
contains various labels even including operator details. and 
task_name, do we have any way from which we can remove these extra / 
unnecessary labels?


Thanks and Regards ,
Surendra Lalwani



IMPORTANT NOTICE: This e-mail, including any attachments, may contain 
confidential information and is intended only for the addressee(s) 
named above. If you are not the intended recipient(s), you should not 
disseminate, distribute, or copy this e-mail. Please notify the sender 
by reply e-mail immediately if you have received this e-mail in error 
and permanently delete all copies of the original message from your 
system. E-mail transmission cannot be guaranteed to be secure as it 
could be intercepted, corrupted, lost, destroyed, arrive late or 
incomplete, or contain viruses. Company accepts no liability for any 
damage or loss of confidential information caused by this email or due 
to any virus transmitted by this email or otherwise.

Re: Async IO & Retry: how to get error details

2023-01-12 Thread Chesnay Schepler 
Retry logic and per-request timeouts should be setup within 
asyncInvoke() (with all error-handling being done via plain 
CompletableFuture logic), with timeout() sort of acting as a global 
timeout after which you want the job to fail (e.g., to guard against 
mistakes in the asyncInvoke() logic).


The reason you shouldn't use timeout() for retries is that it is only 
ever called once for an input element.


On 04/01/2023 13:40, Yoni Gibbs wrote:
I'm following the documentation here 
, 
getting async IO to retry on failure. What I want is for the async 
action to be attempted, say, three times, and then give up and 
continue processing further records. If it fails after three times, I 
want to sink the record to a DLQ. I believe the way I should do that 
is by overriding |timeout|​, and in there outputting the record to a 
side output, which I then sink to a DLQ of some sort. (Correct me if 
I'm wrong and there's a better way of doing this.)


The record in the DLQ should contain error information about what went 
wrong (e.g. the exceptions that occurred on the three failed 
attempts). How can I get access to this in the |timeout|​ function?


Thanks!

Re: Using filesystem plugin with MiniCluster

2023-01-12 Thread Chesnay Schepler 

There is no good way in 1.15 IIRC.

Adding a dependency on flink-s3-fs-hadoop _can_ work, if you dont run 
into dependency conflicts.


Otherwise you have to create a plugin manager yourself, point it to some 
local directory via a system property (I think?), and then eagerly call 
FileSystem#initialize with the plugin manager.


On 04/01/2023 02:11, Yaroslav Tkachenko wrote:

Hey Ken,

I have flink-s3-fs-hadoop as a provided dependency in my project, and 
I've configured my IDE to include provided dependencies when starting 
applications. Works just fine.


On Tue, Jan 3, 2023 at 5:06 PM Ken Krugler 
 wrote:


Hi all,

With Flink 1.15.x, is there a way to use the S3 Presto plugin when
running code on the MiniCluster?

I can’t just add that jar as a dependency when testing, as I get:

java.lang.NoClassDefFoundError: Could not initialize class
com.facebook.presto.hive.s3.PrestoS3FileSystem
at

org.apache.flink.fs.s3presto.S3FileSystemFactory.createHadoopFileSystem(S3FileSystemFactory.java:88)
~[flink-s3-fs-presto-1.15.1.jar:1.15.1]
at

org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:126)
~[flink-s3-fs-presto-1.15.1.jar:1.15.1]

I assume that’s because of this warning in the Flink docs:


The s3 file systems (flink-s3-fs-presto and flink-s3-fs-hadoop)
can only be used as plugins as we already removed the
relocations. Placing them in libs/ will result in system failures.


In the latest Flink JavaDocs, there’s a way to specify the
PluginManager for the MiniClusterConfiguration, but I don’t see
that in Flink 1.15.x

So is there a workaround to allow me to run a test from inside of
my IDE, using the MiniCluster, that reads from S3?

Thanks,

— Ken

--
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch

Re: Flink reactive mode for application clusters on AWS EKS

2023-01-12 Thread Chesnay Schepler 
The adaptive scheduler and reactive mode both already support 
application clusters since 1.13.


https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/

On 19/12/2022 10:17, Tamir Sagi wrote:

Hey,

We are running stream jobs on application clusters (v1.15.2) on AWS EKS.

I was reviewing the following pages on Flink confluence

  * Reactive mode [1]
  * Adaptive Scheduler [2]

I also encountered the following POC conducted by Robert Metzger 
(@rmetzger_ ) on 06 May 2021. [3]


my question is whether that feature will be supported in the future 
for application clusters or not.


[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-159%3A+Reactive+Mode
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-160%3A+Adaptive+Scheduler

[3] https://flink.apache.org/2021/05/06/reactive-mode.html


Thanks,
Tamir.


Confidentiality: This communication and any attachments are intended 
for the above-named persons only and may be confidential and/or 
legally privileged. Any opinions expressed in this communication are 
not necessarily those of NICE Actimize. If this communication has come 
to you in error you must take no action based on it, nor must you copy 
or show it to anyone; please delete/destroy and inform the sender by 
e-mail immediately.

Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail 
and attachments are free from any virus, we advise that in keeping 
with good computing practice the recipient should ensure they are 
actually virus free.

sql-client.sh 终止后, 进程不会被 kill

2023-01-12 Thread 704669594 
Hi 各位, 

 

碰到sql-client进程不会被kill, 如下所示

---

Flink SQL> 

[4]+  Stopped ./bin/sql-client.sh

[l3@node1 flink-1.16.0]$ jps

35040 SqlClient

20148 SqlClient

19284 NameNode

19540 GetConf

19733 RunJar

19544 Jps

19497 DataNode

9644 SqlClient

36063 SqlClient

 

---

 

Best regards

 

AboutMe

*    Hi, I’m @lllong33
*    I’m interested in Flink & dbt & StarRocks

Re: Unable to list jobs in flink cluster with multiple jobManagers

2023-01-12 Thread yidan zhao 
I think it is a bug: https://issues.apache.org/jira/browse/FLINK-25732

Yael Adsl  于2022年12月12日周一 23:56写道：
>
> Hi,
>
> We are running a flink cluster (Flink version 1.14.3) on kubernetes with 
> high-availablity.type: kubernetes. We have 3 jobmanagers. When we send jobs 
> to the flink cluster, we run a "flink list --jobmanager 
> flink-jobmanager:8081" command as part of the process".
>
> At first, we succeeded to run this command from within any of the jobmanager 
> CLIs.
> But after the elected leader is deleted (For whatever reason. For example, 
> server failure), the configmaps with the following format are updated with 
> the new leader IP address:
> flink-cluster-ecb24e88d60bb06917da1c933785811a-jobmanager-leader
> flink-cluster-b4bef19e6481a6c42340e51b69e30923-jobmanager-leader
> ...
>
> But the following configmaps are not always updated with the same IP address 
> as the others:
> flink-cluster-restserver-leader
> flink-cluster-resourcemanager-leader
> flink-cluster-dispatcher-leader
>
> Then, when we run the flink list command, we receive the error attached at 
> the end of this mail.
> If we delete the jobmanager where the flink-cluster-restserver-leader 
> configmap is pointing, the configmap gets modified to the IP address of the 
> other configmaps, and the "flink list" command succeeds.
> Note: I can see in the log that the command attempts to connect to the IP 
> which is set in the configmap: flink-cluster-restserver-leader'
>
> How do we fix this issue without needing any manual intervention?
>
> Thanks,
> Yael
>
> Error from CLI when running flink list command:
>
> root@flink-jobmanager-68b5fb748d-wwmvt:/opt/flink# flink list --jobmanager 
> localhost:8081
> 2022-12-12 15:04:19,037 INFO  
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer
>  [] - Starting to watch for xi-env/flink-cluster-restserver-leader, watching 
> id:fe7b3bff-1d4f-4e3e-bcf8-26afd74e4
> 12c
> Waiting for response...
> 2022-12-12 15:04:21,231 INFO  
> org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver 
> [] - Stopping 
> KubernetesLeaderRetrievalDriver{configMapName='flink-cluster-restserver-leader'}.
> 2022-12-12 15:04:21,233 INFO  
> org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapSharedInformer
>  [] - Stopped to watch for xi-env/flink-cluster-restserver-leader, watching 
> id:fe7b3bff-1d4f-4e3e-bcf8-26afd74e41
> 2c
>
> 
>  The program finished with the following exception:
>
> org.apache.flink.util.FlinkException: Failed to retrieve job list.
> at 
> org.apache.flink.client.cli.CliFrontend.listJobs(CliFrontend.java:449)
> at 
> org.apache.flink.client.cli.CliFrontend.lambda$list$0(CliFrontend.java:430)
> at 
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
> at org.apache.flink.client.cli.CliFrontend.list(CliFrontend.java:427)
> at 
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1060)
> at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at 
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: org.apache.flink.runtime.rest.util.RestClientException: [Internal 
> server error.,  org.apache.flink.runtime.rpc.akka.exceptions.AkkaRpcException: Failed to 
> serialize the result for RPC call : requestMultipleJobDetails.
> at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.serializeRemoteResultAndVerifySize(AkkaRpcActor.java:417)
> at 
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$sendAsyncResponse$2(AkkaRpcActor.java:373)
> at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown 
> Source)
> at 
> java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown 
> Source)
> at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
> at java.base/java.util.concurrent.CompletableFuture.complete(Unknown 
> Source)
> at 
> org.apache.flink.util.concurrent.FutureUtils$ResultConjunctFuture.handleCompletedFuture(FutureUtils.java:858)
> at 
> org.apache.flink.util.concurrent.FutureUtils$ResultConjunctFuture.lambda$new$0(FutureUtils.java:876)
> at 
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown 
> Source)
> at 
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown
>  Source)
> at 
> java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
> at java.base/java.util.concurrent.CompletableFuture.complete(Unknown 
> Source)
> at 
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
> at 
>

Re: 请问cancel的任务能够恢复running状态吗？

2023-01-12 Thread yidan zhao 
话说我也有个问题，stop后基于savepoint恢复 不同于先savepoint，然后cancel后基于savepoint恢复？

Weihua Hu  于2023年1月5日周四 10:38写道：
>
> Hi，
>
> 简单来说是不能，已经 cancel 的 job 状态不能恢复到 running 状态。用 savepoint 恢复的任务是新的 job。
>
> 这个问题的背景是什么呢？什么情况下需要将已经 cancel 的 job 恢复呢？
>
>
> Best,
> Weihua
>
>
> On Fri, Dec 30, 2022 at 5:12 PM 陈佳豪  wrote:
>
> > hi
> > 我目前测试flink restapi
> > 指定savepointpath来恢复任务发现会重新触发创建一个新的任务原有的任务还是cancel状态，请问有办法恢复原有cancel状态的任务为running吗？

Re: 任务本地运行正常，提交到集群报错 - 图片挂掉，文字贴一下报错信息，非常抱歉打扰

2023-01-12 Thread yidan zhao 
看报错 Could not connect to BlobServer at address
localhost/127.0.0.1:33271，你本地的配置是不是不对。提交到什么模式部署的集群，配置是否配对了。

WD.Z  于2023年1月10日周二 10:56写道：
>
> 任务在webui点击submit时报错，看起来是从JM提交到TM时报错，服务器防火墙已关闭，资源足够，还没有安装hadoop，但以standalone模式启动，看了下文档是不需要hadoop?
>  报错中的Caused by列表如下：
>
>
> 2023-01-10 09:46:14,627 INFO  
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] 
> - Job e343bc906ea6889d34d9472d40d4f8ff is submitted.
> 2023-01-10 09:46:14,627 INFO  
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] 
> - Submitting Job with JobId=e343bc906ea6889d34d9472d40d4f8ff.
> 2023-01-10 09:46:14,629 WARN  
> org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - 
> Could not execute application:
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: org.apache.flink.util.FlinkException: Failed to execute job 
> 'Flink Streaming Job'.
>
>
> Caused by: java.lang.RuntimeException: org.apache.flink.util.FlinkException: 
> Failed to execute job 'Flink Streaming Job'.
>
>
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'Flink 
> Streaming Job'.
>
>
> Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'Flink 
> Streaming Job'.
>
>
> Caused by: org.apache.flink.util.FlinkException: Could not upload job files.
>
>
> Caused by: java.io.IOException: Could not connect to BlobServer at address 
> localhost/127.0.0.1:33271
>
>
> Caused by: java.net.ConnectException: 拒绝连接 (Connection refused)

回复: How to specify both the resource limit and resource request for JM/TM in flink-operator

2023-01-12 Thread Liting Liu (litiliu) 
Seems i can achieve this by specify the 
"kubernetes.jobmanager.cpu.limit-factor" and  
"kubernetes.taskmanager.cpu.limit-factor" in flink properties.  Those parameter 
are supported since flink 1.15

发件人: Liting Liu (litiliu) 
发送时间: 2023年1月12日 16:28
收件人: user 
主题: How to specify both the resource limit and resource request for JM/TM in 
flink-operator

  Hi, community.  I wonder how can i specify both the resource request and 
limit for JM/TM in the podTemplate using flink-operator?  We have the need to 
set the request resource and limit resource to different value.

For example:
jobManager:
  limits:
cpu: 500m
memory: 500Mi
  requests:
cpu: 1000m
memory: 1024Mi
taskManager:
  limits:
cpu: 500m
memory: 1024Mi
  requests:
cpu: 1000m
memory: 2048Mi

How to specify both the resource limit and resource request for JM/TM in flink-operator

2023-01-12 Thread Liting Liu (litiliu) 
  Hi, community.  I wonder how can i specify both the resource request and 
limit for JM/TM in the podTemplate using flink-operator?  We have the need to 
set the request resource and limit resource to different value.

For example:
jobManager:
  limits:
cpu: 500m
memory: 500Mi
  requests:
cpu: 1000m
memory: 1024Mi
taskManager:
  limits:
cpu: 500m
memory: 1024Mi
  requests:
cpu: 1000m
memory: 2048Mi