[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873111999
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
Let's see what @gyfora and @wangyang0918 think/suggest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-27497) Track terminal job states in the observer
[ https://issues.apache.org/jira/browse/FLINK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537140#comment-17537140 ] Aitozi commented on FLINK-27497: working on this now > Track terminal job states in the observer > - > > Key: FLINK-27497 > URL: https://issues.apache.org/jira/browse/FLINK-27497 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.0.0 >Reporter: Gyula Fora >Assignee: Aitozi >Priority: Critical > > With the improvements in FLINK-27468 Flink 1.15 app clusters will not be shut > down in case of terminal job states (failed, finished) etc. > It is important to properly handle these states and let the user know about > it. > We should always trigger events, and for terminally failed jobs record the > error information in the FlinkDeployment status. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
Aitozi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873111327
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
As I know we have manually patch the status after
https://github.com/apache/flink-kubernetes-operator/pull/199 . So this change
do not bring the resourceVersion conflicts, I think. Do you mean keep it as now
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19725: [FLINK-17232][k8s] Disable the implicit behavior to use the Service e…
flinkbot commented on PR #19725: URL: https://github.com/apache/flink/pull/19725#issuecomment-1126862934 ## CI report: * d51162d8992114b1a0bc9da7211e0d78e76b2106 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17232) Rethink the implicit behavior to use the Service externalIP as the address of the Endpoint
[
https://issues.apache.org/jira/browse/FLINK-17232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537136#comment-17537136
]
Aitozi commented on FLINK-17232:
I have pushed a PR to remove the implicit behavior to fallback to externalIP,
Please take a look when you are free [~wangyang0918]
> Rethink the implicit behavior to use the Service externalIP as the address of
> the Endpoint
> --
>
> Key: FLINK-17232
> URL: https://issues.apache.org/jira/browse/FLINK-17232
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Canbin Zheng
>Assignee: Aitozi
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-assigned
>
> Currently, for the LB/NodePort type Service, if we found that the
> {{LoadBalancer}} in the {{Service}} is null, we would use the externalIPs
> configured in the external Service as the address of the Endpoint. Again,
> this is another implicit toleration and may confuse the users.
> This ticket proposes to rethink the implicit toleration behaviour.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#issuecomment-1126862712 +1 For the record we need this functionality due: [Enable arbitrary CRD field selectors](https://github.com/kubernetes/kubernetes/issues/53459) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17232) Rethink the implicit behavior to use the Service externalIP as the address of the Endpoint
[
https://issues.apache.org/jira/browse/FLINK-17232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-17232:
---
Labels: auto-unassigned pull-request-available stale-assigned (was:
auto-unassigned stale-assigned)
> Rethink the implicit behavior to use the Service externalIP as the address of
> the Endpoint
> --
>
> Key: FLINK-17232
> URL: https://issues.apache.org/jira/browse/FLINK-17232
> Project: Flink
> Issue Type: Sub-task
> Components: Deployment / Kubernetes
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Canbin Zheng
>Assignee: Aitozi
>Priority: Major
> Labels: auto-unassigned, pull-request-available, stale-assigned
>
> Currently, for the LB/NodePort type Service, if we found that the
> {{LoadBalancer}} in the {{Service}} is null, we would use the externalIPs
> configured in the external Service as the address of the Endpoint. Again,
> this is another implicit toleration and may confuse the users.
> This ticket proposes to rethink the implicit toleration behaviour.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink] Aitozi opened a new pull request, #19725: [FLINK-17232][k8s] Disable the implicit behavior to use the Service e…
Aitozi opened a new pull request, #19725: URL: https://github.com/apache/flink/pull/19725 …xternalIP as the address of the Endpoint ## What is the purpose of the change This PR is to disable the implicit behavior to use the Service externalIp as the endpoint. This will decrease the implicit behavior around generating the endpoint . Since the external ip can only be used by set explicit, we have not expose the way to set external ip. so I think it will not break the compatibility -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873110405
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
We hit some race conditions anyway with the status updates, so it maybe good
as is with the java-operator-sdk v2 and we can change it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r872986702 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java: ## @@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob, Context context) throws E OperatorUtils.getSecondaryResource( flinkSessionJob, context, configManager.getOperatorConfiguration()); +var labels = flinkSessionJob.getMetadata().getLabels(); Review Comment: I suggest to extract adding labels into a utility method to keep the reconcile method concise -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
Aitozi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873110013
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
Yes, agree with it. I'm ok to rework the interface to work out it. Also cc
@gyfora @wangyang0918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#issuecomment-1126859076 > > k get sessionjob --field-selector spec.deploymentName=basic-session-cluster > > I tried this way before, but the custom resource seems do not support field selector > > ``` > kubectl get sessionjob --field-selector spec.deploymentName=basic-session-cluster > Error from server (BadRequest): Unable to find "flink.apache.org/v1beta1, Resource=flinksessionjobs" that match label selector "", field selector "spec.deploymentName=basic-session-cluster": field label not supported: spec.deploymentName > > kubectl version --short > Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. > Client Version: v1.24.0 > Kustomize Version: v4.5.4 > Server Version: v1.23.3 > ``` > > After search the related issues, I think the custom resource can only support field selector by `metadata.namespace` and `metadata.name` > > [kubernetes/kubernetes#53345](https://github.com/kubernetes/kubernetes/pull/53345) > > k get sessionjob --field-selector spec.deploymentName=basic-session-cluster > > I tried this way before, but the custom resource seems do not support field selector > > ``` > kubectl get sessionjob --field-selector spec.deploymentName=basic-session-cluster > Error from server (BadRequest): Unable to find "flink.apache.org/v1beta1, Resource=flinksessionjobs" that match label selector "", field selector "spec.deploymentName=basic-session-cluster": field label not supported: spec.deploymentName > > kubectl version --short > Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. > Client Version: v1.24.0 > Kustomize Version: v4.5.4 > Server Version: v1.23.3 > ``` > > After search the related issues, I think the custom resource can only support field selector by `metadata.namespace` and `metadata.name` > > [kubernetes/kubernetes#53345](https://github.com/kubernetes/kubernetes/pull/53345) Yeah, I was almost sure you've tried it, but I thought it was worth asking :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873107836
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
Got it, but this is the first use case that we modify something else then
the status. We might need to reconsider this again. Updating the CR deep in the
stack sounds worse to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] deadwind4 commented on a diff in pull request #19682: [FLINK-25795][python][connector/pulsar] Add pulsar sink DataStream API
deadwind4 commented on code in PR #19682:
URL: https://github.com/apache/flink/pull/19682#discussion_r872081349
##
flink-python/pyflink/datastream/connectors.py:
##
@@ -1449,6 +1455,314 @@ def build(self) -> 'PulsarSource':
return PulsarSource(self._j_pulsar_source_builder.build())
+class DeliveryGuarantee(Enum):
+"""
+DeliverGuarantees that can be chosen. In general your pipeline can only
offer the lowest
+delivery guarantee which is supported by your sources and sinks.
+
+:data: `EXACTLY_ONCE`:
+Records are only delivered exactly-once also under failover scenarios. To
build a complete
+exactly-once pipeline is required that the source and sink support
exactly-once and are
+properly configured.
+
+:data: `AT_LEAST_ONCE`:
+Records are ensured to be delivered but it may happen that the same record
is delivered
+multiple times. Usually, this guarantee is faster than the exactly-once
delivery.
+
+:data: `NONE`:
+Records are delivered on a best effort basis. It is often the fastest way
to process records
+but it may happen that records are lost or duplicated.
+"""
+
+EXACTLY_ONCE = 0,
+AT_LEAST_ONCE = 1,
+NONE = 2
+
+def _to_j_delivery_guarantee(self):
+JDeliveryGuarantee = get_gateway().jvm \
+.org.apache.flink.connector.base.DeliveryGuarantee
+return getattr(JDeliveryGuarantee, self.name)
+
+
+class PulsarSerializationSchema(object):
+"""
+The serialization schema for how to serialize records into Pulsar.
+"""
+
+def __init__(self, _j_pulsar_serialization_schema):
+self._j_pulsar_serialization_schema = _j_pulsar_serialization_schema
+
+@staticmethod
+def flink_schema(serialization_schema: SerializationSchema) \
+-> 'PulsarSerializationSchema':
+"""
+Create a PulsarSerializationSchema by using the flink's
SerializationSchema. It would
+serialize the message into byte array and send it to Pulsar with
Schema#BYTES.
+"""
+JPulsarSerializationSchema = get_gateway().jvm.org.apache.flink \
+.connector.pulsar.sink.writer.serializer.PulsarSerializationSchema
+_j_pulsar_serialization_schema =
JPulsarSerializationSchema.flinkSchema(
+serialization_schema._j_serialization_schema)
+return PulsarSerializationSchema(_j_pulsar_serialization_schema)
+
+
+class TopicRoutingMode(Enum):
+"""
+The routing policy for choosing the desired topic by the given message.
+
+:data: `ROUND_ROBIN`:
+The producer will publish messages across all partitions in a round-robin
fashion to achieve
+maximum throughput. Please note that round-robin is not done per
individual message but
+rather it's set to the same boundary of batching delay, to ensure batching
is effective.
+
+:data: `MESSAGE_KEY_HASH`:
+If no key is provided, The partitioned producer will randomly pick one
single topic partition
+and publish all the messages into that partition. If a key is provided on
the message, the
+partitioned producer will hash the key and assign the message to a
particular partition.
+
+:data: `CUSTOM`:
+Use custom topic router implementation that will be called to determine
the partition for a
+particular message.
+"""
+
+ROUND_ROBIN = 0
+MESSAGE_KEY_HASH = 1
+CUSTOM = 2
+
+def _to_j_topic_routing_mode(self):
+JTopicRoutingMode = get_gateway().jvm \
+
.org.apache.flink.connector.pulsar.sink.writer.router.TopicRoutingMode
+return getattr(JTopicRoutingMode, self.name)
+
+
+class MessageDelayer(object):
+"""
+A delayer for Pulsar broker passing the sent message to the downstream
consumer. This is only
+works in {@link SubscriptionType#Shared} subscription.
+
+Read delayed message delivery
+
https://pulsar.apache.org/docs/en/next/concepts-messaging/#delayed-message-delivery
for better
+understanding this feature.
+"""
+def __init__(self, _j_message_delayer):
+self._j_message_delayer = _j_message_delayer
+
+@staticmethod
+def never() -> 'MessageDelayer':
+"""
+All the messages should be consumed immediately.
+"""
+JMessageDelayer = get_gateway().jvm \
+
.org.apache.flink.connector.pulsar.sink.writer.delayer.MessageDelayer
+return MessageDelayer(JMessageDelayer.never())
+
+@staticmethod
+def fixed(duration: Duration) -> 'MessageDelayer':
+"""
+All the messages should be consumed in a fixed duration.
+"""
+JMessageDelayer = get_gateway().jvm \
+
.org.apache.flink.connector.pulsar.sink.writer.delayer.MessageDelayer
+return MessageDelayer(JMessageDelayer.fixed(duration._j_duration))
+
+
+class PulsarSink(Sink):
+"""
+The Sink implementation of Pulsar. Please use a PulsarSinkBuilder to
construct a
+PulsarSink. The following ex
[jira] [Updated] (FLINK-27504) State compaction not happening with sliding window and incremental RocksDB backend
[ https://issues.apache.org/jira/browse/FLINK-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexis Sarda-Espinosa updated FLINK-27504: -- Attachment: duration_trend_c5dd5d2.png size_growth_c5dd5d2.png > State compaction not happening with sliding window and incremental RocksDB > backend > -- > > Key: FLINK-27504 > URL: https://issues.apache.org/jira/browse/FLINK-27504 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.14.4 > Environment: Local Flink cluster on Arch Linux. >Reporter: Alexis Sarda-Espinosa >Priority: Major > Attachments: duration_trend_52ca77c.png, duration_trend_67c76bb.png, > duration_trend_c5dd5d2.png, image-2022-05-06-10-34-35-007.png, > size_growth_52ca77c.png, size_growth_67c76bb.png, size_growth_c5dd5d2.png > > > Hello, > I'm trying to estimate an upper bound for RocksDB's state size in my > application. For that purpose, I have created a small job with faster timings > whose code you can find on GitHub: > [https://github.com/asardaes/flink-rocksdb-ttl-test]. You can see some of the > results there, but I summarize here as well: > * Approximately 20 events per second, 10 unique keys for partitioning are > pre-specified. > * Sliding window of 11 seconds with a 1-second slide. > * Allowed lateness of 11 seconds. > * State TTL configured to 1 minute and compaction after 1000 entries. > * Both window-specific and window-global state used. > * Checkpoints every 2 seconds. > * Parallelism of 4 in stateful tasks. > The goal is to let the job run and analyze state compaction behavior with > RocksDB. I should note that global state is cleaned manually inside the > functions, TTL for those is in case some keys are no longer seen in the > actual production environment. > I have been running the job on a local cluster (outside IDE), the > configuration YAML is also available in the repository. After running for > approximately 1.6 days, state size is currently 2.3 GiB (see attachments). I > understand state can retain expired data for a while, but since TTL is 1 > minute, this seems excessive to me. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27504) State compaction not happening with sliding window and incremental RocksDB backend
[ https://issues.apache.org/jira/browse/FLINK-27504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537129#comment-17537129 ] Alexis Sarda-Espinosa commented on FLINK-27504: --- I suppose I'm still missing something, with a 4mb memtable, growth is much slower, but after approximately 3.5 days it reached 800MiB with no sign of slowing down. > State compaction not happening with sliding window and incremental RocksDB > backend > -- > > Key: FLINK-27504 > URL: https://issues.apache.org/jira/browse/FLINK-27504 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.14.4 > Environment: Local Flink cluster on Arch Linux. >Reporter: Alexis Sarda-Espinosa >Priority: Major > Attachments: duration_trend_52ca77c.png, duration_trend_67c76bb.png, > image-2022-05-06-10-34-35-007.png, size_growth_52ca77c.png, > size_growth_67c76bb.png > > > Hello, > I'm trying to estimate an upper bound for RocksDB's state size in my > application. For that purpose, I have created a small job with faster timings > whose code you can find on GitHub: > [https://github.com/asardaes/flink-rocksdb-ttl-test]. You can see some of the > results there, but I summarize here as well: > * Approximately 20 events per second, 10 unique keys for partitioning are > pre-specified. > * Sliding window of 11 seconds with a 1-second slide. > * Allowed lateness of 11 seconds. > * State TTL configured to 1 minute and compaction after 1000 entries. > * Both window-specific and window-global state used. > * Checkpoints every 2 seconds. > * Parallelism of 4 in stateful tasks. > The goal is to let the job run and analyze state compaction behavior with > RocksDB. I should note that global state is cleaned manually inside the > functions, TTL for those is in case some keys are no longer seen in the > actual production environment. > I have been running the job on a local cluster (outside IDE), the > configuration YAML is also available in the repository. After running for > approximately 1.6 days, state size is currently 2.3 GiB (see attachments). I > understand state can retain expired data for a while, but since TTL is 1 > minute, this seems excessive to me. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
Aitozi commented on code in PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873102772 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java: ## @@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob, Context context) throws E OperatorUtils.getSecondaryResource( flinkSessionJob, context, configManager.getOperatorConfiguration()); +var labels = flinkSessionJob.getMetadata().getLabels(); Review Comment: Moved to a dedicated method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
Aitozi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r873102697
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
The reconciler do not return `UpdateControl` now, If we want to use the
update control for it, we should adjust the interface first to return the
update control :/ . We have just decide not to do
[this](https://issues.apache.org/jira/browse/FLINK-26915) now. I'm not sure
whether we need to break it for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wolfboys closed pull request #19511: [FLINK-27299][Runtime/Configuration] flink parsing parameter bug fixed.
wolfboys closed pull request #19511: [FLINK-27299][Runtime/Configuration] flink parsing parameter bug fixed. URL: https://github.com/apache/flink/pull/19511 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27354) JobMaster still processes requests while terminating
[
https://issues.apache.org/jira/browse/FLINK-27354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537125#comment-17537125
]
yanpengshi commented on FLINK-27354:
When the JobMaster receives disconnectResourceManager RPC call from the
resourceManager, it will try to call
reconnectToResourceManage and cause the problem.
If we set the resourceManagerAddress as null when JobMaster shuts down, then
JobMaster::
isConnectingToResourceManager will return false and JobMaster will not try to
reconncet to resourceManager.
. What do you think? @[~mapohl]
> JobMaster still processes requests while terminating
>
>
> Key: FLINK-27354
> URL: https://issues.apache.org/jira/browse/FLINK-27354
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Matthias Pohl
>Priority: Major
> Attachments: flink-logs.zip
>
>
> An issue was reported in the [user
> ML|https://lists.apache.org/thread/5pm3crntmb1hl17h4txnlhjz34clghrg] about
> the JobMaster trying to reconnect to the ResourceManager during shutdown.
> The JobMaster is disconnecting from the ResourceManager during shutdown (see
> [JobMaster:1182|https://github.com/apache/flink/blob/da532423487e0534b5fe61f5a02366833f76193a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L1182]).
> This triggers the deregistration of the job in the {{ResourceManager}}. The
> RM responses asynchronously at the end of this deregistration through
> {{disconnectResourceManager}} (see
> [ResourceManager:993|https://github.com/apache/flink/blob/da532423487e0534b5fe61f5a02366833f76193a/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L993])
> which will trigger a reconnect on the {{JobMaster}}'s side (see
> [JobMaster::disconnectResourceManager|https://github.com/apache/flink/blob/da532423487e0534b5fe61f5a02366833f76193a/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L789])
> if it's still around because the {{resourceManagerAddress}} (used in
> {{isConnectingToResourceManager}}) is not cleared. This would only happen
> during a RM leader change.
> The {{disconnectResourceManager}} will be ignored if the {{JobMaster}} is
> gone already.
> We should add a guard in some way to {{JobMaster}} to avoid reconnecting to
> other components during shutdown. This might not only include the
> ResourceManager connection but might also affect other parts of the
> {{JobMaster}} API.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] Aitozi commented on pull request #215: [FLINK-27163] Add label for the session job to help list the session …
Aitozi commented on PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#issuecomment-1126848686 > LGTM overall. nit: Shouldn't we use a shorter label, e.g: > > ``` > k get sessionjob -l target=basic-session-cluster > ``` > > It's easier to remember :) Good idea, but the `target` is a bit too generic, I lean to rename it to `target.session` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on pull request #215: [FLINK-27163] Add label for the session job to help list the session …
Aitozi commented on PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#issuecomment-1126848340 > k get sessionjob --field-selector spec.deploymentName=basic-session-cluster I tried this way before, but the custom resource seems do not support field selector ``` kubectl get sessionjob --field-selector spec.deploymentName=basic-session-cluster Error from server (BadRequest): Unable to find "flink.apache.org/v1beta1, Resource=flinksessionjobs" that match label selector "", field selector "spec.deploymentName=basic-session-cluster": field label not supported: spec.deploymentName kubectl version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.24.0 Kustomize Version: v4.5.4 Server Version: v1.23.3 ``` After search the related issues, I think the custom resource can only support field selector by `metadata.namespace` and `metadata.name` https://github.com/kubernetes/kubernetes/pull/53345 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise commented on pull request #19704: [FLINK-27465] Handle conversion of negative long to timestamp in Avro…
tweise commented on PR #19704: URL: https://github.com/apache/flink/pull/19704#issuecomment-1126842552 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tweise closed pull request #19704: [FLINK-27465] Handle conversion of negative long to timestamp in Avro…
tweise closed pull request #19704: [FLINK-27465] Handle conversion of negative long to timestamp in Avro… URL: https://github.com/apache/flink/pull/19704 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19724: [hotfix][connector][pulsar][docs] Corrected typo in topic partition subscription section
flinkbot commented on PR #19724: URL: https://github.com/apache/flink/pull/19724#issuecomment-1126820394 ## CI report: * 7eec96eb0803d2b998279d33424c24ea548a66ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] mans2singh opened a new pull request, #19724: [hotfix][connector][pulsar][docs] Corrected typo in topic partition subscription section
mans2singh opened a new pull request, #19724: URL: https://github.com/apache/flink/pull/19724 ## What is the purpose of the change * Corrected typo in topic partition subscription description ## Brief change log * Updated Pulsar source **provide** two ways to Pulsar source **provides** two ways ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19723: [FLINK-27251][checkpoint] Timeout aligned to unaligned checkpoint barrier in the output buffers
flinkbot commented on PR #19723: URL: https://github.com/apache/flink/pull/19723#issuecomment-1126741736 ## CI report: * cca12d1f2290fa47ad556f9b4103ee6124e7ffc4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27251) Timeout aligned to unaligned checkpoint barrier in the output buffers of an upstream subtask
[
https://issues.apache.org/jira/browse/FLINK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-27251:
---
Labels: pull-request-available (was: )
> Timeout aligned to unaligned checkpoint barrier in the output buffers of an
> upstream subtask
>
>
> Key: FLINK-27251
> URL: https://issues.apache.org/jira/browse/FLINK-27251
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.15.0
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> After FLINK-23041, the downstream task can be switched UC when {_}currentTime
> - triggerTime > timeout{_}. But the downstream task still needs wait for all
> barriers of upstream.
> If the back pressure is serve, the downstream task cannot receive all barrier
> within CP timeout, causes CP to fail.
>
> Can we support upstream Task switching from Aligned to UC? It means that when
> the barrier cannot be sent from the output buffer to the downstream task
> within the
> [execution.checkpointing.aligned-checkpoint-timeout|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#execution-checkpointing-aligned-checkpoint-timeout],
> the upstream task switches to UC and takes a snapshot of the data before the
> barrier in the output buffer.
>
> Hi [~akalashnikov] , please help take a look in your free time, thanks a lot.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink] 1996fanrui opened a new pull request, #19723: [FLINK-27251][checkpoint] Timeout aligned to unaligned checkpoint barrier in the output buffers
1996fanrui opened a new pull request, #19723: URL: https://github.com/apache/flink/pull/19723 ## What is the purpose of the change Support upstream Task switching from Aligned Checkpoint to Unaligned Checkpoint to improve the UC when the back pressure is severe. When the barrier cannot be sent from the output buffer to the downstream task within the [execution.checkpointing.aligned-checkpoint-timeout](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#execution-checkpointing-aligned-checkpoint-timeout), the upstream task switches to UC and takes a snapshot of the data before the barrier in the output buffer. ## Brief change log - *When broadcasting Barrier, if checkpointOptions.isTimeoutable(), register timer.* - *The time when the timer is triggered is: CP triggerTime + alignmentTimeout* - *Behavior when the timer is triggered: snapshot all buffers that have not been sent to the downstream before the Barrier* - *If the UC is enabled and the Aligned Barrier is received, the CP cannot succeed immediately and needs to wait for the outputBufferFuture:* - *When all barriers are sent downstream quickly, and an empty output buffer is returned, the CP ends.(when the back pressure isn't severe)* - *Or wait for the timer to trigger, snapshot these output buffers before the barrier, and the CP ends.(when the back pressure is severe)* - *If the CP is aborted, flink will execute the`channelStateFuture.completeExceptionally(cause)`* ## Verifying this change This change added tests and can be verified as follows: - Finish unit test later. - This PR will affect the Unaligned checkpoint benchmark. We can view the benefit after merge [here](http://codespeed.dak8s.net:8000/timeline/#/?exe=1,6&ben=checkpointSingleInput.UNALIGNED_1&env=2&revs=200&equid=off&quarts=on&extr=on). ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? docs, it need to update some doc of UC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] ajian2002 commented on a diff in pull request #121: Introduce AggregatuibMergeFunction
ajian2002 commented on code in PR #121:
URL: https://github.com/apache/flink-table-store/pull/121#discussion_r872990828
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/mergetree/compact/AggregationMergeFunction.java:
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.file.mergetree.compact;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+
+import javax.annotation.Nullable;
+
+/**
+ * A {@link MergeFunction} where key is primary key (unique) and value is the
partial record, update
+ * non-null fields on merge.
+ */
+public class AggregationMergeFunction implements MergeFunction {
+
+private static final long serialVersionUID = 1L;
+
+private final RowData.FieldGetter[] getters;
+
+private transient GenericRowData row;
+
+public AggregationMergeFunction(RowData.FieldGetter[] getters) {
+this.getters = getters;
+}
+
+@Override
+public void reset() {
+this.row = new GenericRowData(getters.length);
+}
+
+@Override
+public void add(RowData value) {
+for (int i = 0; i < getters.length; i++)
+{
+Object currentField = getters[i].getFieldOrNull(value);
+Object oldValue = row.getField(i);
+Object result = sum(oldValue, currentField);
+if (result != null)
+{
+row.setField(i, result);
+}
+}
+}
+
+private Object sum(Object oldValue, Object currentField) {
+if (currentField == null)
+{
+return null;
+}
+if (oldValue == null)
+{
+return currentField;
+}
+if (oldValue instanceof Integer && currentField instanceof Integer)
+{
+return Integer.sum((Integer) oldValue, (Integer) currentField);
+}
+else if (oldValue instanceof Long && currentField instanceof Long)
+{
+return Long.sum((Long) oldValue, (Long) currentField);
+}
Review Comment:
should it be used
org.apache.flink.api.java.aggregation.AggregationFunction to handle
aggregation? This is a convenient way to deal with different field types, but
it also brings some problems. It is difficult for me to deal with delete/update
RowData#getRowKind. How to solve it?
1. Modify the original interface
org.apache.flink.api.java.aggregation.AggregationFunction to add delete and
update functions
2. Implement the Aggregator class by yourself, manually switch the class of
different fields and implement it?
Is there any other solution
For the type processing of object/class, I guess flink/flink-table-store
already has useful tool classes to reduce development workload (such as
org.apache.flink.api.java.aggregation.AggregationFunction, etc.), you can
provide guide?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r872986702 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java: ## @@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob, Context context) throws E OperatorUtils.getSecondaryResource( flinkSessionJob, context, configManager.getOperatorConfiguration()); +var labels = flinkSessionJob.getMetadata().getLabels(); Review Comment: I suggest to extract adding labels into a utility method to keep the reconcile method concise (we could use computeIfAbsent in the helper) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r872986702 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java: ## @@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob, Context context) throws E OperatorUtils.getSecondaryResource( flinkSessionJob, context, configManager.getOperatorConfiguration()); +var labels = flinkSessionJob.getMetadata().getLabels(); Review Comment: I suggest to extract adding labels into a utility method (we could use computeIfAbsent) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on a diff in pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on code in PR #215:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/215#discussion_r872986702
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
Review Comment:
I suggest to extract this into a utility method (we could use
computeIfAbsent)
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/sessionjob/FlinkSessionJobReconciler.java:
##
@@ -76,6 +78,24 @@ public void reconcile(FlinkSessionJob flinkSessionJob,
Context context) throws E
OperatorUtils.getSecondaryResource(
flinkSessionJob, context,
configManager.getOperatorConfiguration());
+var labels = flinkSessionJob.getMetadata().getLabels();
+if (labels == null) {
+labels = new HashMap<>();
+}
+if (!flinkSessionJob
+.getSpec()
+.getDeploymentName()
+.equals(labels.get(LABEL_SESSION_CLUSTER))) {
+labels.put(LABEL_SESSION_CLUSTER,
flinkSessionJob.getSpec().getDeploymentName());
+flinkSessionJob.getMetadata().setLabels(labels);
+kubernetesClient
Review Comment:
Can't we use an update control for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#issuecomment-1126726044 On second thought we could use something like this for filtering, aren't we? ``` k get --field-selector spec.deploymentName=basic-session-cluster ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #215: [FLINK-27163] Add label for the session job to help list the session …
morhidi commented on PR #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215#issuecomment-1126724095 LGTM overall. nit: Shouldn't we use a shorter label, e.g: ``` k get sessionjob -l target=basic-session-cluster ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #19372: [FLINK-26043][runtime][security] Add periodic kerberos relogin to KerberosDelegationTokenManager
gaborgsomogyi commented on PR #19372: URL: https://github.com/apache/flink/pull/19372#issuecomment-1126717741 Finally w/ all the modifications tests passed :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on pull request #121: Introduce AggregatuibMergeFunction
tsreaper commented on PR #121: URL: https://github.com/apache/flink-table-store/pull/121#issuecomment-1126717355 > If I want to test the same MergeFunction on tables with different structures, what should I do? (Is it possible to create a new Test file again?) In that case just create multiple tables with different names in `ddl` method. Table names should be different within the same test class as they'll be created all at once (see `FileStoreTableITCase`). However different tests run in different temporary directories so it is OK to have the same table name across test classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-27616) Modify the spelling mistakes of the comment word
[ https://issues.apache.org/jira/browse/FLINK-27616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] codingcaproni closed FLINK-27616. - Resolution: Abandoned > Modify the spelling mistakes of the comment word > > > Key: FLINK-27616 > URL: https://issues.apache.org/jira/browse/FLINK-27616 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.14.3 >Reporter: codingcaproni >Priority: Major > > Modify the spelling mistakes of the comment word. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27616) Modify the spelling mistakes of the comment word
codingcaproni created FLINK-27616: - Summary: Modify the spelling mistakes of the comment word Key: FLINK-27616 URL: https://issues.apache.org/jira/browse/FLINK-27616 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Affects Versions: 1.14.3 Reporter: codingcaproni Modify the spelling mistakes of the comment word. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-26784) FLIP-215: Introduce FlinkSessionJob CRD in the kubernetes operator
[ https://issues.apache.org/jira/browse/FLINK-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26784: --- Labels: pull-request-available (was: ) > FLIP-215: Introduce FlinkSessionJob CRD in the kubernetes operator > -- > > Key: FLINK-26784 > URL: https://issues.apache.org/jira/browse/FLINK-26784 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Aitozi >Assignee: Aitozi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.0.0 > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-215%3A+Introduce+FlinkSessionJob+CRD+in+the+kubernetes+operator -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] Aitozi opened a new pull request, #215: [FLINK-26784] Add label for the session job to help list the session …
Aitozi opened a new pull request, #215: URL: https://github.com/apache/flink-kubernetes-operator/pull/215 …jobs in the same session cluster This PR is meant to add label for the session job to help to list session jobs under the same session.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27615) Document how to define namespaceSelector for k8s operator's webhook for different k8s versions
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Geng updated FLINK-27615:
--
Summary: Document how to define namespaceSelector for k8s operator's
webhook for different k8s versions (was: Document the how to define
namespaceSelector for k8s operator's webhook)
> Document how to define namespaceSelector for k8s operator's webhook for
> different k8s versions
> --
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
>
> Due to aitozi's advice, it may be better to add document of how to support
> the nameSelector based validation instead of limiting the k8s version to be
> >= 1.21.1 to support more users.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[GitHub] [flink] gaborgsomogyi commented on pull request #19372: [FLINK-26043][runtime][security] Add periodic kerberos relogin to KerberosDelegationTokenManager
gaborgsomogyi commented on PR #19372: URL: https://github.com/apache/flink/pull/19372#issuecomment-1126687852 I've rebased the whole change to the top of the latest master. Maybe this helps since other PRs are not suffering from this: ``` --2022-05-14 08:05:31-- http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.8_amd64.deb Resolving security.ubuntu.com (security.ubuntu.com)... 185.125.190.36, 91.189.91.38, 185.125.190.39, ... Connecting to security.ubuntu.com (security.ubuntu.com)|185.125.190.36|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2022-05-14 08:05:31 ERROR 404: Not Found. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-27615) Document the how to define namespaceSelector for k8s operator's webhook
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537012#comment-17537012
]
Biao Geng edited comment on FLINK-27615 at 5/14/22 10:29 AM:
-
[~aitozi] I think your proposal makes sense.
I checked the release notes of k8s and just noticed that the 1.21.1 was just
released 1 year ago. It looks like k8s iterates much more quickly than flink.
IIUC, users, especially those use k8s in production, will not update k8s to new
versions frequently.
I was doing my tests on k8s 1.20.4. Currently, besides the nameSelector
feature, I have not met other issues. To support more k8s versions, I agree
with you on makeing this feature non-default with better documents
was (Author: bgeng777):
[~aitozi] I think your proposal makes a lot of sense.
I checked the release notes of k8s and just noticed that the 1.21.1 was just
released 1 year ago. K8s iterates much more quickly than flink. IIUC, users,
especially those use k8s in production, will not update k8s to new versions
frequently.
I was doing my tests on k8s 1.20.4. Currently, besides the nameSelector
feature, I have not met other issues. To support more k8s versions, I agree
with you on makeing this feature non-default with better documents
> Document the how to define namespaceSelector for k8s operator's webhook
> ---
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
>
> Due to aitozi's advice, it may be better to add document of how to support
> the nameSelector based validation instead of limiting the k8s version to be
> >= 1.21.1 to support more users.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27615) Document the how to define namespaceSelector for k8s operator's webhook
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Geng updated FLINK-27615:
--
Summary: Document the how to define namespaceSelector for k8s operator's
webhook (was: Document the how to define )
> Document the how to define namespaceSelector for k8s operator's webhook
> ---
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
>
> Due to aitozi's advice, it may be better to add document of how to support
> the nameSelector based validation instead of limiting the k8s version to be
> >= 1.21.1 to support more users.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27615) Document the how to define
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Geng updated FLINK-27615:
--
Summary: Document the how to define (was: Document the minimum supported
version of k8s for flink k8s operator)
> Document the how to define
> ---
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
>
> Due to aitozi's advice, it may be better to add document of how to support
> the nameSelector based validation instead of limiting the k8s version to be
> >= 1.21.1 to support more users.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Geng updated FLINK-27615:
--
Description:
In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
{{kubernetes.io/metadata.name}} to filter the validation requests. However,
this label will be automatically added to a namespace only since k8s 1.21.1 due
to k8s' [release
notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
. If users run the flink k8s operator on older k8s versions, they have to add
such label by themselevs to support the feature of namespaceSelector in our
webhook.
As a result, if we want to support the feature defaultly, we may need to
emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
Due to aitozi's advice, it may be better to add document of how to support the
nameSelector based validation instead of limiting the k8s version to be >=
1.21.1 to support more users.
was:
In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
{{kubernetes.io/metadata.name}} to filter the validation requests. However,
this label will be automatically added to a namespace only since k8s 1.21.1 due
to k8s' [release
notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
. If users run the flink k8s operator on older k8s versions, they have to add
such label by themselevs to support the feature of namespaceSelector in our
webhook.
As a result, if we want to support the feature defaultly, we may need to
emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
> Document the minimum supported version of k8s for flink k8s operator
>
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
>
> Due to aitozi's advice, it may be better to add document of how to support
> the nameSelector based validation instead of limiting the k8s version to be
> >= 1.21.1 to support more users.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537012#comment-17537012
]
Biao Geng commented on FLINK-27615:
---
[~aitozi] I think your proposal makes a lot of sense.
I checked the release notes of k8s and just noticed that the 1.21.1 was just
released 1 year ago. K8s iterates much more quickly than flink. IIUC, users,
especially those use k8s in production, will not update k8s to new versions
frequently.
I was doing my tests on k8s 1.20.4. Currently, besides the nameSelector
feature, I have not met other issues. To support more k8s versions, I agree
with you on makeing this feature non-default with better documents
> Document the minimum supported version of k8s for flink k8s operator
>
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537011#comment-17537011
]
Aitozi edited comment on FLINK-27615 at 5/14/22 10:12 AM:
--
what about supporting user to define their own namespace selector label, this
will let more k8s version can be supported. Certainly, we also have to document
it.
ps: thanks for your link, I also try to find this feature documentation before
:)
was (Author: aitozi):
what about supporting user to define their own namespace selector label, this
will let more k8s version can be supported. Certainly, we also have to document
it.
> Document the minimum supported version of k8s for flink k8s operator
>
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537011#comment-17537011
]
Aitozi commented on FLINK-27615:
what about supporting user to define their own namespace selector label, this
will let more k8s version can be supported. Certainly, we also have to document
it.
> Document the minimum supported version of k8s for flink k8s operator
>
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Geng updated FLINK-27615:
--
Description:
In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
{{kubernetes.io/metadata.name}} to filter the validation requests. However,
this label will be automatically added to a namespace only since k8s 1.21.1 due
to k8s' [release
notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
. If users run the flink k8s operator on older k8s versions, they have to add
such label by themselevs to support the feature of namespaceSelector in our
webhook.
As a result, if we want to support the feature defaultly, we may need to
emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
was:
In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
{{kubernetes.io/metadata.name}} to filter the validation requests. However,
this label will be automatically added to a namespace only since k8s 1.21.1 due
to k8s' [release
notes|[https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
. If users run the flink k8s operator on older k8s versions, they have add
such label by themselevs to support the feature of namespaceSelector in our
webhook.
As a result, if we want to support the feature defaultly, we may need to
emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
> Document the minimum supported version of k8s for flink k8s operator
>
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have to
> add such label by themselevs to support the feature of namespaceSelector in
> our webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
Biao Geng created FLINK-27615:
-
Summary: Document the minimum supported version of k8s for flink
k8s operator
Key: FLINK-27615
URL: https://issues.apache.org/jira/browse/FLINK-27615
Project: Flink
Issue Type: Improvement
Reporter: Biao Geng
In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
{{kubernetes.io/metadata.name}} to filter the validation requests. However,
this label will be automatically added to a namespace only since k8s 1.21.1 due
to k8s' [release
notes|[https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
. If users run the flink k8s operator on older k8s versions, they have add
such label by themselevs to support the feature of namespaceSelector in our
webhook.
As a result, if we want to support the feature defaultly, we may need to
emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27615) Document the minimum supported version of k8s for flink k8s operator
[
https://issues.apache.org/jira/browse/FLINK-27615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biao Geng updated FLINK-27615:
--
Component/s: Kubernetes Operator
> Document the minimum supported version of k8s for flink k8s operator
>
>
> Key: FLINK-27615
> URL: https://issues.apache.org/jira/browse/FLINK-27615
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Biao Geng
>Priority: Major
>
> In our webhook, to support {{{}watchNamespaces{}}}, we rely on the
> {{kubernetes.io/metadata.name}} to filter the validation requests. However,
> this label will be automatically added to a namespace only since k8s 1.21.1
> due to k8s' [release
> notes|[https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#v1211]
> . If users run the flink k8s operator on older k8s versions, they have add
> such label by themselevs to support the feature of namespaceSelector in our
> webhook.
> As a result, if we want to support the feature defaultly, we may need to
> emphasize that users should use k8s >= v1.21.1 to run the flink k8s operator.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27613) Add label for the session job to help list the session jobs in the same session cluster
[ https://issues.apache.org/jira/browse/FLINK-27613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aitozi updated FLINK-27613: --- Description: The custorm resource do not supported list by field selector, So I think add a label to get the session jobs under the same session cluster will be more convenient > Add label for the session job to help list the session jobs in the same > session cluster > --- > > Key: FLINK-27613 > URL: https://issues.apache.org/jira/browse/FLINK-27613 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Aitozi >Priority: Major > Fix For: kubernetes-operator-1.0.0 > > > The custorm resource do not supported list by field selector, So I think add > a label to get the session jobs under the same session cluster will be more > convenient -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27614) Use informer in webhook to avoid query apiserver at each update
[ https://issues.apache.org/jira/browse/FLINK-27614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27614: --- Labels: pull-request-available (was: ) > Use informer in webhook to avoid query apiserver at each update > --- > > Key: FLINK-27614 > URL: https://issues.apache.org/jira/browse/FLINK-27614 > Project: Flink > Issue Type: Sub-task > Components: Kubernetes Operator >Reporter: Aitozi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink-kubernetes-operator] Aitozi opened a new pull request, #214: [FLINK-27614] Use informer in webhook to avoid query apiserver at eac…
Aitozi opened a new pull request, #214: URL: https://github.com/apache/flink-kubernetes-operator/pull/214 …h update This PR is meant to enhance the webhook logic to use the informer to avoid query apiserver during each update event -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-27614) Use informer in webhook to avoid query apiserver at each update
Aitozi created FLINK-27614: -- Summary: Use informer in webhook to avoid query apiserver at each update Key: FLINK-27614 URL: https://issues.apache.org/jira/browse/FLINK-27614 Project: Flink Issue Type: Sub-task Components: Kubernetes Operator Reporter: Aitozi Fix For: kubernetes-operator-1.0.0 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-26784) FLIP-215: Introduce FlinkSessionJob CRD in the kubernetes operator
[
https://issues.apache.org/jira/browse/FLINK-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536992#comment-17536992
]
Aitozi commented on FLINK-26784:
The {{{}FlinkSessionJob{}}}'s functionality are basically completed. Please
help review are there something missed here. IMO, there are two abilities still
need to improve here:
- Currently, the {{LAST_STATE}} upgrade mode is not supported now, I will try
to take a look on this
- The session cluster upgrade directly delete the session cluster now, which
still need to be improved
> FLIP-215: Introduce FlinkSessionJob CRD in the kubernetes operator
> --
>
> Key: FLINK-26784
> URL: https://issues.apache.org/jira/browse/FLINK-26784
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Aitozi
>Assignee: Aitozi
>Priority: Major
> Fix For: kubernetes-operator-1.0.0
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-215%3A+Introduce+FlinkSessionJob+CRD+in+the+kubernetes+operator
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (FLINK-26784) FLIP-215: Introduce FlinkSessionJob CRD in the kubernetes operator
[
https://issues.apache.org/jira/browse/FLINK-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536992#comment-17536992
]
Aitozi edited comment on FLINK-26784 at 5/14/22 8:21 AM:
-
The {{{}FlinkSessionJob{}}}'s functionality are basically completed. Please
help review are there something missed here. IMO, there are two abilities still
need to improve here:
- Currently, the {{LAST_STATE}} upgrade mode is not supported now, I will try
to take a look on this
- The session cluster upgrade directly delete the session cluster now, which
still need to be improved
cc [~gyfora] [~wangyang0918]
was (Author: aitozi):
The {{{}FlinkSessionJob{}}}'s functionality are basically completed. Please
help review are there something missed here. IMO, there are two abilities still
need to improve here:
- Currently, the {{LAST_STATE}} upgrade mode is not supported now, I will try
to take a look on this
- The session cluster upgrade directly delete the session cluster now, which
still need to be improved
> FLIP-215: Introduce FlinkSessionJob CRD in the kubernetes operator
> --
>
> Key: FLINK-26784
> URL: https://issues.apache.org/jira/browse/FLINK-26784
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Reporter: Aitozi
>Assignee: Aitozi
>Priority: Major
> Fix For: kubernetes-operator-1.0.0
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-215%3A+Introduce+FlinkSessionJob+CRD+in+the+kubernetes+operator
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (FLINK-27613) Add label for the session job to help list the session jobs in the same session cluster
Aitozi created FLINK-27613: -- Summary: Add label for the session job to help list the session jobs in the same session cluster Key: FLINK-27613 URL: https://issues.apache.org/jira/browse/FLINK-27613 Project: Flink Issue Type: Sub-task Components: Kubernetes Operator Reporter: Aitozi Fix For: kubernetes-operator-1.0.0 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[GitHub] [flink-table-store] ajian2002 commented on a diff in pull request #121: Introduce AggregatuibMergeFunction
ajian2002 commented on code in PR #121:
URL: https://github.com/apache/flink-table-store/pull/121#discussion_r872947138
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/mergetree/compact/AggregationMergeFunction.java:
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.file.mergetree.compact;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+
+import javax.annotation.Nullable;
+
+/**
+ * A {@link MergeFunction} where key is primary key (unique) and value is the
partial record, update
+ * non-null fields on merge.
+ */
+public class AggregationMergeFunction implements MergeFunction {
+
+private static final long serialVersionUID = 1L;
+
+private final RowData.FieldGetter[] getters;
+
+private transient GenericRowData row;
+
+public AggregationMergeFunction(RowData.FieldGetter[] getters) {
+this.getters = getters;
+}
+
+@Override
+public void reset() {
+this.row = new GenericRowData(getters.length);
+}
+
+@Override
+public void add(RowData value) {
+for (int i = 0; i < getters.length; i++)
+{
+Object currentField = getters[i].getFieldOrNull(value);
+Object oldValue = row.getField(i);
+Object result = sum(oldValue, currentField);
+if (result != null)
+{
+row.setField(i, result);
+}
+}
+}
+
+private Object sum(Object oldValue, Object currentField) {
+if (currentField == null)
+{
+return null;
+}
+if (oldValue == null)
+{
+return currentField;
+}
+if (oldValue instanceof Integer && currentField instanceof Integer)
+{
+return Integer.sum((Integer) oldValue, (Integer) currentField);
+}
+else if (oldValue instanceof Long && currentField instanceof Long)
+{
+return Long.sum((Long) oldValue, (Long) currentField);
+}
+else if (oldValue instanceof Double && currentField instanceof Double)
+{
+return Double.sum((Double) oldValue, (Double) currentField);
+}
+else if (oldValue instanceof Float && currentField instanceof Float)
+{
+return Float.sum((Float) oldValue, (Float) currentField);
+}
+else if (oldValue instanceof String && currentField instanceof String)
+{
+return "null";
+}
+return null;
Review Comment:
Is there a more explicit exception than RuntimeException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] ajian2002 commented on a diff in pull request #121: Introduce AggregatuibMergeFunction
ajian2002 commented on code in PR #121:
URL: https://github.com/apache/flink-table-store/pull/121#discussion_r872947013
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/mergetree/compact/AggregationMergeFunction.java:
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.file.mergetree.compact;
+
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+
+import javax.annotation.Nullable;
+
+/**
+ * A {@link MergeFunction} where key is primary key (unique) and value is the
partial record, update
+ * non-null fields on merge.
+ */
+public class AggregationMergeFunction implements MergeFunction {
+
+private static final long serialVersionUID = 1L;
+
+private final RowData.FieldGetter[] getters;
+
+private transient GenericRowData row;
+
+public AggregationMergeFunction(RowData.FieldGetter[] getters) {
+this.getters = getters;
+}
+
+@Override
+public void reset() {
+this.row = new GenericRowData(getters.length);
+}
+
+@Override
+public void add(RowData value) {
Review Comment:
Should update_before and update_after be supported for Aggregation? I don't
know how to handle this situation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] mbalassi commented on pull request #19372: [FLINK-26043][runtime][security] Add periodic kerberos relogin to KerberosDelegationTokenManager
mbalassi commented on PR #19372: URL: https://github.com/apache/flink/pull/19372#issuecomment-1126658703 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
