[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[
https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585114#comment-17585114
]
Hangxiang Yu commented on FLINK-29108:
--
Hi, [~rcroc...@newrelic.com] . I see "Querable state" has been marked as
"Reaching end-of-life"[1].
You want to support Querable based on it ?
[1] https://flink.apache.org/roadmap.html
> Kubernetes operator: Support queryable state
>
>
> Key: FLINK-29108
> URL: https://issues.apache.org/jira/browse/FLINK-29108
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
>Reporter: Ron Crocker
>Priority: Minor
>
> Enable the kubernetes operator to deploy jobs where queryable state is
> desired.
> When queryable state is desired, the operator should configure the deployed
> job with
> # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it.
> # Configure the Queryable State proxy and Queryable State server (via the
> {{queryable-state.proxy}} and {{queryable-state.server}} configuration
> sections respectively). If these sections aren't provided, then the default
> configuration is used.
> The operator will need to create a Kubernetes service fronting the Task
> Managers {{QueryableStateClientProxy}} port (as configured by the above).
> Tearing down the job also tears down the service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[
https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585531#comment-17585531
]
Ron Crocker commented on FLINK-29108:
-
[~masteryhx] In a word, yes.
There's two things in play:
# While its "reaching end of life," Queryable State remains a supported
capability in all of the Flink versions that the operator currently supports.
My company would like to use this Kubernetes operator to manage our Flink jobs,
and some of those jobs require Queryable State. We can't use this operator for
those jobs until it supports Queryable State for some of those jobs.
# I'm trying to rescue Queryable State from deprecation. In [my recent
presentation at Flink
Forward|[http://example.com|https://www.slideshare.net/FlinkForward/using-queryable-state-for-fun-and-profit]]
I made what I'd claim is a fairly strong argument for keeping queryable state
in the Flink feature set. ({_}TLDR: Using Flink Queryable State, I can save
>90% of the cost of the equivalent Redis-based solution{_})
I'm looking for allies in the fight to keep Queryable State alive.
> Kubernetes operator: Support queryable state
>
>
> Key: FLINK-29108
> URL: https://issues.apache.org/jira/browse/FLINK-29108
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
>Reporter: Ron Crocker
>Priority: Minor
>
> Enable the kubernetes operator to deploy jobs where queryable state is
> desired.
> When queryable state is desired, the operator should configure the deployed
> job with
> # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it.
> # Configure the Queryable State proxy and Queryable State server (via the
> {{queryable-state.proxy}} and {{queryable-state.server}} configuration
> sections respectively). If these sections aren't provided, then the default
> configuration is used.
> The operator will need to create a Kubernetes service fronting the Task
> Managers {{QueryableStateClientProxy}} port (as configured by the above).
> Tearing down the job also tears down the service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[
https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17596903#comment-17596903
]
Gyula Fora commented on FLINK-29108:
I understand that this is an important feature for you. But putting something
like this into the CRD and adding it to the operator logic is not necessarily
something we should do for a deprecated feature.
If you manage to convince the community to "undeprecate" it and support it
forward then it's a slightly different situation but even then it might be an
overkill to bake the support into the operator.
What I suggest as a first step and a middle-ground is to add some plugin
mechanism to be able to extend the core opertaor logic in a way that something
like this can be implemented. We can then have this as a reference
implementation / example in the repo.
> Kubernetes operator: Support queryable state
>
>
> Key: FLINK-29108
> URL: https://issues.apache.org/jira/browse/FLINK-29108
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
>Reporter: Ron Crocker
>Priority: Minor
>
> Enable the kubernetes operator to deploy jobs where queryable state is
> desired.
> When queryable state is desired, the operator should configure the deployed
> job with
> # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it.
> # Configure the Queryable State proxy and Queryable State server (via the
> {{queryable-state.proxy}} and {{queryable-state.server}} configuration
> sections respectively). If these sections aren't provided, then the default
> configuration is used.
> The operator will need to create a Kubernetes service fronting the Task
> Managers {{QueryableStateClientProxy}} port (as configured by the above).
> Tearing down the job also tears down the service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[
https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17596904#comment-17596904
]
Gyula Fora commented on FLINK-29108:
We already have some plugin mechanisms, such as validators, resource listeners
that allows users to extend the operator for their own purposes. This could be
something very similar :)
> Kubernetes operator: Support queryable state
>
>
> Key: FLINK-29108
> URL: https://issues.apache.org/jira/browse/FLINK-29108
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
>Reporter: Ron Crocker
>Priority: Minor
>
> Enable the kubernetes operator to deploy jobs where queryable state is
> desired.
> When queryable state is desired, the operator should configure the deployed
> job with
> # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it.
> # Configure the Queryable State proxy and Queryable State server (via the
> {{queryable-state.proxy}} and {{queryable-state.server}} configuration
> sections respectively). If these sections aren't provided, then the default
> configuration is used.
> The operator will need to create a Kubernetes service fronting the Task
> Managers {{QueryableStateClientProxy}} port (as configured by the above).
> Tearing down the job also tears down the service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[
https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17596924#comment-17596924
]
Ron Crocker commented on FLINK-29108:
-
[~gyfora] I can accept starting with "extend operator via plugin mechanism".
> Kubernetes operator: Support queryable state
>
>
> Key: FLINK-29108
> URL: https://issues.apache.org/jira/browse/FLINK-29108
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
>Reporter: Ron Crocker
>Priority: Minor
>
> Enable the kubernetes operator to deploy jobs where queryable state is
> desired.
> When queryable state is desired, the operator should configure the deployed
> job with
> # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it.
> # Configure the Queryable State proxy and Queryable State server (via the
> {{queryable-state.proxy}} and {{queryable-state.server}} configuration
> sections respectively). If these sections aren't provided, then the default
> configuration is used.
> The operator will need to create a Kubernetes service fronting the Task
> Managers {{QueryableStateClientProxy}} port (as configured by the above).
> Tearing down the job also tears down the service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[
https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713009#comment-17713009
]
Salva commented on FLINK-29108:
---
[~rcroc...@newrelic.com] There is no recording available for your presentation
right? Queryable state looks like a good feature to me too, I would be
interested in keeping it alive too. On a related note, some people are opting
for externalizing the local state instead, see, e.g.:
-
[https://engineering.contentsquare.com/2022/flink-external-rocksdb-state-to-aerospike/]
- [https://www.youtube.com/watch?v=ZWq_TzsXssM
(FlinkNDB)|https://www.youtube.com/watch?v=ZWq_TzsXssM]
How likely is to add an external state backend to Flink in the short term? In
the meantime, it would be nice to have some guidelines / recommendations for
when to rely on Queryable State vs an external/global DB...
> Kubernetes operator: Support queryable state
>
>
> Key: FLINK-29108
> URL: https://issues.apache.org/jira/browse/FLINK-29108
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
>Reporter: Ron Crocker
>Priority: Minor
>
> Enable the kubernetes operator to deploy jobs where queryable state is
> desired.
> When queryable state is desired, the operator should configure the deployed
> job with
> # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it.
> # Configure the Queryable State proxy and Queryable State server (via the
> {{queryable-state.proxy}} and {{queryable-state.server}} configuration
> sections respectively). If these sections aren't provided, then the default
> configuration is used.
> The operator will need to create a Kubernetes service fronting the Task
> Managers {{QueryableStateClientProxy}} port (as configured by the above).
> Tearing down the job also tears down the service.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
