[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[ https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585114#comment-17585114 ] Hangxiang Yu commented on FLINK-29108: -- Hi, [~rcroc...@newrelic.com] . I see "Querable state" has been marked as "Reaching end-of-life"[1]. You want to support Querable based on it ? [1] https://flink.apache.org/roadmap.html > Kubernetes operator: Support queryable state > > > Key: FLINK-29108 > URL: https://issues.apache.org/jira/browse/FLINK-29108 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Ron Crocker >Priority: Minor > > Enable the kubernetes operator to deploy jobs where queryable state is > desired. > When queryable state is desired, the operator should configure the deployed > job with > # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it. > # Configure the Queryable State proxy and Queryable State server (via the > {{queryable-state.proxy}} and {{queryable-state.server}} configuration > sections respectively). If these sections aren't provided, then the default > configuration is used. > The operator will need to create a Kubernetes service fronting the Task > Managers {{QueryableStateClientProxy}} port (as configured by the above). > Tearing down the job also tears down the service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[ https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585531#comment-17585531 ] Ron Crocker commented on FLINK-29108: - [~masteryhx] In a word, yes. There's two things in play: # While its "reaching end of life," Queryable State remains a supported capability in all of the Flink versions that the operator currently supports. My company would like to use this Kubernetes operator to manage our Flink jobs, and some of those jobs require Queryable State. We can't use this operator for those jobs until it supports Queryable State for some of those jobs. # I'm trying to rescue Queryable State from deprecation. In [my recent presentation at Flink Forward|[http://example.com|https://www.slideshare.net/FlinkForward/using-queryable-state-for-fun-and-profit]] I made what I'd claim is a fairly strong argument for keeping queryable state in the Flink feature set. ({_}TLDR: Using Flink Queryable State, I can save >90% of the cost of the equivalent Redis-based solution{_}) I'm looking for allies in the fight to keep Queryable State alive. > Kubernetes operator: Support queryable state > > > Key: FLINK-29108 > URL: https://issues.apache.org/jira/browse/FLINK-29108 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Ron Crocker >Priority: Minor > > Enable the kubernetes operator to deploy jobs where queryable state is > desired. > When queryable state is desired, the operator should configure the deployed > job with > # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it. > # Configure the Queryable State proxy and Queryable State server (via the > {{queryable-state.proxy}} and {{queryable-state.server}} configuration > sections respectively). If these sections aren't provided, then the default > configuration is used. > The operator will need to create a Kubernetes service fronting the Task > Managers {{QueryableStateClientProxy}} port (as configured by the above). > Tearing down the job also tears down the service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[ https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17596903#comment-17596903 ] Gyula Fora commented on FLINK-29108: I understand that this is an important feature for you. But putting something like this into the CRD and adding it to the operator logic is not necessarily something we should do for a deprecated feature. If you manage to convince the community to "undeprecate" it and support it forward then it's a slightly different situation but even then it might be an overkill to bake the support into the operator. What I suggest as a first step and a middle-ground is to add some plugin mechanism to be able to extend the core opertaor logic in a way that something like this can be implemented. We can then have this as a reference implementation / example in the repo. > Kubernetes operator: Support queryable state > > > Key: FLINK-29108 > URL: https://issues.apache.org/jira/browse/FLINK-29108 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Ron Crocker >Priority: Minor > > Enable the kubernetes operator to deploy jobs where queryable state is > desired. > When queryable state is desired, the operator should configure the deployed > job with > # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it. > # Configure the Queryable State proxy and Queryable State server (via the > {{queryable-state.proxy}} and {{queryable-state.server}} configuration > sections respectively). If these sections aren't provided, then the default > configuration is used. > The operator will need to create a Kubernetes service fronting the Task > Managers {{QueryableStateClientProxy}} port (as configured by the above). > Tearing down the job also tears down the service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[ https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17596904#comment-17596904 ] Gyula Fora commented on FLINK-29108: We already have some plugin mechanisms, such as validators, resource listeners that allows users to extend the operator for their own purposes. This could be something very similar :) > Kubernetes operator: Support queryable state > > > Key: FLINK-29108 > URL: https://issues.apache.org/jira/browse/FLINK-29108 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Ron Crocker >Priority: Minor > > Enable the kubernetes operator to deploy jobs where queryable state is > desired. > When queryable state is desired, the operator should configure the deployed > job with > # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it. > # Configure the Queryable State proxy and Queryable State server (via the > {{queryable-state.proxy}} and {{queryable-state.server}} configuration > sections respectively). If these sections aren't provided, then the default > configuration is used. > The operator will need to create a Kubernetes service fronting the Task > Managers {{QueryableStateClientProxy}} port (as configured by the above). > Tearing down the job also tears down the service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[ https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17596924#comment-17596924 ] Ron Crocker commented on FLINK-29108: - [~gyfora] I can accept starting with "extend operator via plugin mechanism". > Kubernetes operator: Support queryable state > > > Key: FLINK-29108 > URL: https://issues.apache.org/jira/browse/FLINK-29108 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Ron Crocker >Priority: Minor > > Enable the kubernetes operator to deploy jobs where queryable state is > desired. > When queryable state is desired, the operator should configure the deployed > job with > # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it. > # Configure the Queryable State proxy and Queryable State server (via the > {{queryable-state.proxy}} and {{queryable-state.server}} configuration > sections respectively). If these sections aren't provided, then the default > configuration is used. > The operator will need to create a Kubernetes service fronting the Task > Managers {{QueryableStateClientProxy}} port (as configured by the above). > Tearing down the job also tears down the service. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29108) Kubernetes operator: Support queryable state
[ https://issues.apache.org/jira/browse/FLINK-29108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713009#comment-17713009 ] Salva commented on FLINK-29108: --- [~rcroc...@newrelic.com] There is no recording available for your presentation right? Queryable state looks like a good feature to me too, I would be interested in keeping it alive too. On a related note, some people are opting for externalizing the local state instead, see, e.g.: - [https://engineering.contentsquare.com/2022/flink-external-rocksdb-state-to-aerospike/] - [https://www.youtube.com/watch?v=ZWq_TzsXssM (FlinkNDB)|https://www.youtube.com/watch?v=ZWq_TzsXssM] How likely is to add an external state backend to Flink in the short term? In the meantime, it would be nice to have some guidelines / recommendations for when to rely on Queryable State vs an external/global DB... > Kubernetes operator: Support queryable state > > > Key: FLINK-29108 > URL: https://issues.apache.org/jira/browse/FLINK-29108 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Ron Crocker >Priority: Minor > > Enable the kubernetes operator to deploy jobs where queryable state is > desired. > When queryable state is desired, the operator should configure the deployed > job with > # The deployed job has {{queryable-state.enabled:}} {{true}} applied to it. > # Configure the Queryable State proxy and Queryable State server (via the > {{queryable-state.proxy}} and {{queryable-state.server}} configuration > sections respectively). If these sections aren't provided, then the default > configuration is used. > The operator will need to create a Kubernetes service fronting the Task > Managers {{QueryableStateClientProxy}} port (as configured by the above). > Tearing down the job also tears down the service. -- This message was sent by Atlassian Jira (v8.20.10#820010)