[GitHub] [flink] MartijnVisser commented on pull request #20090: [FLINK-28269][Kubernetes] Install cri-dockerd when running Kubernetes tests
MartijnVisser commented on PR #20090: URL: https://github.com/apache/flink/pull/20090#issuecomment-1169606056 > Maybe we could try to remove the driver `none` and let the minikube select the driver automatically. We have removed the `none` driver in the flink-kubernetes-operator project. That was the reason why I asked Robert and yourself in https://issues.apache.org/jira/browse/FLINK-28269 if we had made a deliberate decision to go for driver `none`. Should it be sufficient to remove `--vm-driver=none` from https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_kubernetes.sh#L81 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #20090: [FLINK-28269][Kubernetes] Install cri-dockerd when running Kubernetes tests
wangyang0918 commented on PR #20090: URL: https://github.com/apache/flink/pull/20090#issuecomment-1169601350 Maybe we could try to remove the driver `none` and let the minikube select the driver automatically. We have removed the `none` driver in the flink-kubernetes-operator project. ``` start_minikube shell: /usr/bin/bash -e {0} env: JAVA_HOME: /opt/hostedtoolcache/Java_Adopt_jdk/11.0.11-9/x6[4](https://github.com/apache/flink-kubernetes-operator/runs/7103609311?check_suite_focus=true#step:5:4) * Profile "minikube" not found. Run "minikube profile list" to view all profiles. To start a cluster, run: "minikube start" Starting minikube ... * minikube v1.26.0 on Ubuntu 20.04 * Automatically selected the docker driver. Other choices: podman, ssh, none * Using Docker driver with root privileges * Starting control plane node minikube in cluster minikube * Pulling base image ... * Downloading Kubernetes v1.24.1 preload ... * Creating docker container (CPUs=2, Memory=2200MB) ... * Preparing Kubernetes v1.24.1 on Docker 20.10.17 ... - kubelet.image-gc-high-threshold=99 - kubelet.image-gc-low-threshold=98 - kubelet.minimum-container-ttl-duration=120m - kubelet.eviction-hard=memory.available<[5](https://github.com/apache/flink-kubernetes-operator/runs/7103609311?check_suite_focus=true#step:5:5)Mi,nodefs.available<1Mi,imagefs.available<1Mi - kubelet.eviction-soft=memory.available<5Mi,nodefs.available<2Mi,imagefs.available<2Mi - kubelet.eviction-soft-grace-period=memory.available=2h,nodefs.available=2h,imagefs.available=2h - Generating certificates and keys ... - Booting up control plane ... - Configuring RBAC rules ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28294) Some metric reporters don't allow register same metric with different names
Title: Message Title Xuannan Su created an issue Flink / FLINK-28294 Some metric reporters don't allow register same metric with different names Issue Type: Bug Affects Versions: 1.15.0 Assignee: Unassigned Components: Runtime / Metrics Created: 29/Jun/22 06:50 Priority: Major Reporter: Xuannan Su Currently, some metric reporters keep an internal Map keyed by the metric object to keep track of the registered metric. The problem with the map with metric object as the key is that when the same metric object is registered with different names or metric groups, only the last registered metric will be reported. If I understand correctly, we do not forbid registering the same metric with different names or metric groups. For example, in `SchedulerBase#registerJobMetrics`, we register `numberOfRestarts` with two names, "numRestarts" and "fullRestarts". Unfortunately, in this case, the metric reporter will only report the fullRestarts metric, which is deprecated. I found that the following metric reporters have the problem, Influxdb metric reporter, Datadog metric reporter, and Dropwizard metric reporter. One possible fix is to swap the key and value of the internal map in the metric reporter.
[GitHub] [flink] lsyldliu commented on pull request #20057: [FLINK-28213][runtime] StreamExecutionEnvironment configure method support override pipeline.jars option
lsyldliu commented on PR #20057: URL: https://github.com/apache/flink/pull/20057#issuecomment-1169588098 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26402) MinioTestContainerTest.testS3EndpointNeedsToBeSpecifiedBeforeInitializingFileSyste failed due to Container startup failed
Title: Message Title Huang Xingbo updated an issue Flink / FLINK-26402 MinioTestContainerTest.testS3EndpointNeedsToBeSpecifiedBeforeInitializingFileSyste failed due to Container startup failed Change By: Huang Xingbo Priority: Major Critical Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-26402) MinioTestContainerTest.testS3EndpointNeedsToBeSpecifiedBeforeInitializingFileSyste failed due to Container startup failed
Title: Message Title Huang Xingbo commented on FLINK-26402 Re: MinioTestContainerTest.testS3EndpointNeedsToBeSpecifiedBeforeInitializingFileSyste failed due to Container startup failed https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37335&view=logs&j=4eda0b4a-bd0d-521a-0916-8285b9be9bb5&t=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37334&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-26721) PulsarSourceITCase.testSavepoint failed on azure pipeline
Title: Message Title Huang Xingbo commented on FLINK-26721 Re: PulsarSourceITCase.testSavepoint failed on azure pipeline https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37332&view=logs&j=9c5a5fe6-2f39-545e-1630-feb3d8d0a1ba&t=99b23320-1d05-5741-d63f-9e78473da39e Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-26600) PerJobMiniClusterFactoryTest.testJobClientSavepoint failed on azure
Title: Message Title Huang Xingbo commented on FLINK-26600 Re: PerJobMiniClusterFactoryTest.testJobClientSavepoint failed on azure https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37332&view=logs&j=0e7be18f-84f2-53f0-a32d-4a5e4a174679&t=7c1d86e3-35bd-5fd5-3b7c-30c126a78702 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Closed] (FLINK-28293) SqlGatewayServiceITCase.testCloseOperationAndFetchResultInParallel failed
Title: Message Title Huang Xingbo closed an issue as Duplicate Flink / FLINK-28293 SqlGatewayServiceITCase.testCloseOperationAndFetchResultInParallel failed Change By: Huang Xingbo Resolution: Duplicate Status: Open Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] zhuzhurk commented on pull request #20087: [FLINK-28271][runtime] Add config options for speculative scheduler
zhuzhurk commented on PR #20087: URL: https://github.com/apache/flink/pull/20087#issuecomment-1169577879 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28293) SqlGatewayServiceITCase.testCloseOperationAndFetchResultInParallel failed
Title: Message Title Huang Xingbo commented on FLINK-28293 Re: SqlGatewayServiceITCase.testCloseOperationAndFetchResultInParallel failed cc Shengkai Fang Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28293) SqlGatewayServiceITCase.testCloseOperationAndFetchResultInParallel failed
Title: Message Title Huang Xingbo created an issue Flink / FLINK-28293 SqlGatewayServiceITCase.testCloseOperationAndFetchResultInParallel failed Issue Type: Bug Affects Versions: 1.16.0 Assignee: Unassigned Components: Table SQL / Gateway Created: 29/Jun/22 06:14 Labels: test-stability Priority: Major Reporter: Huang Xingbo 2022-06-29T03:17:48.4575178Z Jun 29 03:17:48 [INFO] Running org.apache.flink.table.gateway.service.SqlGatewayServiceITCase 2022-06-29T03:17:49.1369728Z Exception in thread "Thread-137" org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to cancelOperation. 2022-06-29T03:17:49.1370535Z at org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.cancelOperation(SqlGatewayServiceImpl.java:99) 2022-06-29T03:17:49.1371467Z at org.apache.flink.table.gateway.service.SqlGatewayServiceITCase.lambda$testCancelAndCloseOperationInParallel$12(SqlGatewayServiceITCase.java:303) 2022-06-29T03:17:49.1372442Z at java.lang.Thread.run(Thread.java:748) 2022-06-29T03:17:49.1373501Z Caused by: org.apache.flink.table.gateway.api.utils.SqlGatewayException: Can not find the submitted operation in the OperationManager with the 19fe6b8c-ec82-48ec-8bce-251b71d62fcd. 2022-06-29T03:17:49.1378602Z at org.apache.flink.table.gateway.service.operation.OperationManager.lambda$getOperation$3(OperationManager.java:168) 2022-06-29T03:17:49.1379559Z at org.apache.flink.table.
[jira] [Commented] (FLINK-26621) flink-tests failed on azure due to Error occurred in starting fork
Title: Message Title Huang Xingbo commented on FLINK-26621 Re: flink-tests failed on azure due to Error occurred in starting fork https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37332&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37332&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] lsyldliu commented on a diff in pull request #20063: [FLINK-28238][sql-gateway] Fix unstable testCancelOperationAndFetchResultInParallel
lsyldliu commented on code in PR #20063: URL: https://github.com/apache/flink/pull/20063#discussion_r909228723 ## flink-table/flink-sql-gateway/src/test/java/org/apache/flink/table/gateway/service/utils/IgnoreErrorThreadFactory.java: ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.gateway.service.utils; + +import org.jetbrains.annotations.NotNull; + +import java.util.concurrent.ThreadFactory; + +/** Factory to create the thread that ignores error message. */ +public class IgnoreErrorThreadFactory implements ThreadFactory { + +public static final IgnoreErrorThreadFactory INSTANCE = new IgnoreErrorThreadFactory(); + +private IgnoreErrorThreadFactory() {} + +@Override +public Thread newThread(@NotNull Runnable r) { +Thread thread = new Thread(r); +thread.setUncaughtExceptionHandler( +new Thread.UncaughtExceptionHandler() { +@Override +public void uncaughtException(Thread t, Throwable e) { +// ignore error Review Comment: Would better if here log the exception message, please refer the `FatalExitExceptionHandler` ## flink-table/flink-sql-gateway/src/test/java/org/apache/flink/table/gateway/service/utils/IgnoreErrorThreadFactory.java: ## @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.table.gateway.service.utils; + +import org.jetbrains.annotations.NotNull; + +import java.util.concurrent.ThreadFactory; + +/** Factory to create the thread that ignores error message. */ +public class IgnoreErrorThreadFactory implements ThreadFactory { + +public static final IgnoreErrorThreadFactory INSTANCE = new IgnoreErrorThreadFactory(); + +private IgnoreErrorThreadFactory() {} + +@Override +public Thread newThread(@NotNull Runnable r) { +Thread thread = new Thread(r); Review Comment: Would be better if can refer to the `ExecutorThreadFactory` add the thread group and thread name for your purpose? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ commented on pull request #20042: [FLINK-19358][runtime] Make the job id distinct in application mode w…
KarmaGYZ commented on PR #20042: URL: https://github.com/apache/flink/pull/20042#issuecomment-1169561842 Thanks for the comments. PR updated. @xintongsong @wangyang0918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28282) Planner free in flink-table-store-codegen
Title: Message Title Jingsong Lee updated an issue Flink / FLINK-28282 Planner free in flink-table-store-codegen Change By: Jingsong Lee Parent: FLINK-28073 Issue Type: Improvement Sub-task Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28292) Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28292 Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory Change By: ASF GitHub Bot Labels: pull-request-available Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-table-store] JingsongLi opened a new pull request, #180: [FLINK-28292] Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory
JingsongLi opened a new pull request, #180: URL: https://github.com/apache/flink-table-store/pull/180 These APIs were changed significantly in 1.14 and 1.15 and are not compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28292) Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory
Title: Message Title Jingsong Lee created an issue Flink / FLINK-28292 Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory Issue Type: Sub-task Assignee: Jingsong Lee Components: Table Store Created: 29/Jun/22 05:48 Fix Versions: table-store-0.2.0 Priority: Major Reporter: Jingsong Lee These APIs were changed significantly in 1.14 and 1.15 and are not compatible Add Comment
[jira] [Updated] (FLINK-28073) Table Store supports Flink 1.14
Title: Message Title Jingsong Lee updated an issue Flink / FLINK-28073 Table Store supports Flink 1.14 Change By: Jingsong Lee Priority: Minor Critical Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] 6591812 commented on pull request #20072: [FLINK-28136][runtime] Implement ExecutionTimeBasedSlowTaskDetector
6591812 commented on PR #20072: URL: https://github.com/apache/flink/pull/20072#issuecomment-1169530980 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on a diff in pull request #20075: [FLINK-27789][network] Disable overdraft buffer for LegacySource
1996fanrui commented on code in PR #20075: URL: https://github.com/apache/flink/pull/20075#discussion_r909185339 ## flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java: ## @@ -385,11 +385,16 @@ public Task( shuffleEnvironment.createShuffleIOOwnerContext( taskNameWithSubtaskAndId, executionId, metrics.getIOMetricGroup()); +boolean overdraftBufferEnabled = +!nameOfInvokableClass.equals( + "org.apache.flink.streaming.runtime.tasks.SourceStreamTask"); Review Comment: Updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile
JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile URL: https://github.com/apache/flink-table-store/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a diff in pull request #20056: [FLINK-28142][runtime] Enrich TaskManagerLocation with node information
wangyang0918 commented on code in PR #20056: URL: https://github.com/apache/flink/pull/20056#discussion_r909164865 ## flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitTaskManagerDecorator.java: ## @@ -149,7 +153,16 @@ private Container decorateMainContainer(Container container) { .withResources(resourceRequirements); // Merge fields - mainContainerBuilder.addAllToPorts(getContainerPorts()).addAllToEnv(getCustomizedEnvs()); +mainContainerBuilder +.addAllToPorts(getContainerPorts()) +.addAllToEnv(getCustomizedEnvs()) +.addNewEnv() +.withName(ENV_FLINK_POD_NODE_ID) +.withValueFrom( Review Comment: I would like to suggest also setting the `ENV_FLINK_POD_NODE_ID` in standalone on K8s. https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/kubernetes/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28286) move “enablechangelog” constant out of flink-streaming-java module
Title: Message Title Yanfei Lei updated an issue Flink / FLINK-28286 move “enablechangelog” constant out of flink-streaming-java module Change By: Yanfei Lei Some methods in the flink-runtime module want to use {{StreamConfig.ENABLE_CHANGE_LOG_STATE_BACKEND}} constant(in flink-streaming-java module), but flink-runtime should not depend on flink-streaming-java. We should move {{ENABLE_CHANGE_LOG_STATE_BACKEND }} to a right place. {{{ } { } }} See this [discussion|https://github.com/apache/flink/pull/19907#discussion_r902485780] for more details. {{}} Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] Myasuka commented on pull request #19998: [FLINK-27822][docs-zh] Translate the doc of checkpoint/savepoint guar…
Myasuka commented on PR #19998: URL: https://github.com/apache/flink/pull/19998#issuecomment-1169492816 > Hi @Myasuka , It seems that test can't be triggered, have you ever come across this ? No, @zentol do you know what happened here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on pull request #19998: [FLINK-27822][docs-zh] Translate the doc of checkpoint/savepoint guar…
zoltar9264 commented on PR #19998: URL: https://github.com/apache/flink/pull/19998#issuecomment-1169491021 Hi @Myasuka , It seems that test can't be triggered, have you ever come across this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a diff in pull request #20057: [FLINK-28213][runtime] StreamExecutionEnvironment configure method support override pipeline.jars option
zhuzhurk commented on code in PR #20057: URL: https://github.com/apache/flink/pull/20057#discussion_r909160467 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java: ## @@ -1023,6 +1025,19 @@ public void configure(ReadableConfig configuration, ClassLoader classLoader) { .ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, flag)); +// merge PipelineOptions.JARS, user maybe set this option in high level such as table +// module, so here need to merge the jars from both configuration object +configuration +.getOptional(PipelineOptions.JARS) +.ifPresent( +jars -> +ConfigUtils.mergeCollectionsToConfig( +this.configuration, +PipelineOptions.JARS, +new HashSet<>(jars), Review Comment: ```suggestion Collections.unmodifiableCollection(jars), ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28206) EOFException on Checkpoint Recovery
Title: Message Title Yun Tang commented on FLINK-28206 Re: EOFException on Checkpoint Recovery uharaqo After you checked the _metadata, I think the problem is related with the serialized RetrievableStateHandle stored in high-availability.storageDir. Since the store path is named as random string, it might not be easy to find which one points to your checkpoint-58310. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
Title: Message Title Yun Tang commented on FLINK-24932 Re: Frocksdb cannot run on Apple M1 Sebastian Mattheis, from my knowledge, RocksDB community has a benchmark-tool, however, they seem do not have public procedure to run the benchmark with specific parameters before any release (maybe Facebook has some internal procedure to avoid performance regression, but they did not make it public). Since RocksDB-6.20.3 already has performance regression compared with previous RocksDB-5.17, I think another 10% regression is a blocker here. I think your pick of 541463 looks good. Did you ever consider to pick RocksDB's community change of supporting Apple Silicon to RocksDB-6.20.3 just as I suggested? Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-table-store] JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile
JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile URL: https://github.com/apache/flink-table-store/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
Title: Message Title jiulong.zhu updated an issue Flink / FLINK-28291 Add kerberos delegation token renewer feature instead of logged from keytab individually Change By: jiulong.zhu h2. 1. Design LifeCyclic LifeCycle of delegation token in RM: # Container starts with DT given by client. # Enable delegation token renewer by: ## set {{security.kerberos.token.renew.enabled}} true, default false. And ## specify {{security.kerberos.login.keytab}} and {{security.kerberos.login.principal}} # When enabled delegation token renewer, the renewer thread will re-obtain tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider now). Then the renewer thread will broadcast new tokens to RM locally, all JMs and all TMs by RPCGateway. # RM process adds new tokens in context by UserGroupInformation. LifeCyclic LifeCycle of delegation token in JM / TM: # TaskManager starts with keytab stored in remote hdfs. # When registered successfully, JM / TM get the current tokens of RM boxed by {{JobMasterRegistrationSuccess}} / {{{}TaskExecutorRegistrationSuccess{}}}. # JM / TM process add new tokens in context by UserGroupInformation.It’s too heavy and unnecessary to retrieval leader of ResourceManager by HAService, so DelegationTokenManager is instanced by ResourceManager. So DelegationToken can hold the reference of ResourceManager, instead of RM RPCGateway or self gateway.h2. 2. Test # No local junit test. It’s too heavy to build junit environments including KDC and local hadoop. # Cluster teststep 1: Specify krb5.conf with short token lifetime(ticket_lifetime, renew_lifetime) when submitting flink application.```{{flink run -yD security.kerberos.token.renew.enabled=true -yD security.kerberos.krb5-conf.path= /home/work/krb5.conf -yD security.kerberos.login.use-ticket-cache=false ...}}```step 2: Watch token identifier changelog and synchronizer between rm and worker.>> In RM / JM log, 2022-06-28 15:13:03,509 INFO org.apache.flink.runtime.util.HadoopUtils [] - New token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) created in KerberosDelegationToken, and next schedule delay is 64799880 ms. 2022-06-28 15:13:03,529 INFO org.apache.flink.runtime.util.HadoopUtils [] - Updating delegation tokens for current user. 2022-06-28 15:13:04,729 INFO org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) from RM.… 2022-06-29 09:13:03,732 INFO org.apache.flink.runtime.util.HadoopUtils [] - New token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy) created in KerberosDelegationToken, and next schedule delay is 64800045 ms.2022-06-29 09:13:03,805 INFO org.apache.flink.runtime.util.HadoopUtils [] - Updating delegation tokens for current user. 2022-06-29 09:13:03,806 INFO org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy) from RM.>> In TM log, 2022-06-28 15:13:17,983 INFO org.apache.flink.runtime.util.HadoopUtil
[jira] [Created] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
Title: Message Title jiulong.zhu created an issue Flink / FLINK-28291 Add kerberos delegation token renewer feature instead of logged from keytab individually Issue Type: New Feature Affects Versions: 1.13.5 Assignee: Unassigned Attachments: 0001-add-KerberosDelegationTokenManager-to-renew-token-pe.patch Components: Deployment / YARN Created: 29/Jun/22 02:52 Fix Versions: 1.13.5 Priority: Minor Reporter: jiulong.zhu 1. Design LifeCyclic of delegation token in RM: Container starts with DT given by client. Enable delegation token renewer by: set security.kerberos.token.renew.enabled true, default false. And specify security.kerberos.login.keytab and security.kerberos.login.principal When enabled delegation token renewer, the renewer thread will re-obtain tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider now). Then the renewer thread wil
[jira] [Commented] (FLINK-26979) JsonFunctionsITCase.test failed on azure
Title: Message Title Huang Xingbo commented on FLINK-26979 Re: JsonFunctionsITCase.test failed on azure https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37329&view=logs&j=0c940707-2659-5648-cbe6-a1ad63045f0a&t=075c2716-8010-5565-fe08-3c4bb45824a4 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo updated an issue Flink / FLINK-28269 Kubernetes test failed with permission denied Change By: Huang Xingbo Affects Version/s: 1.15.0 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37328&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37329&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37327&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=070ff179-953e-5bda-71fa-d6599415701c Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28198) CassandraConnectorITCase#testRaiseCassandraRequestsTimeouts fails with timeout
Title: Message Title Huang Xingbo commented on FLINK-28198 Re: CassandraConnectorITCase#testRaiseCassandraRequestsTimeouts fails with timeout https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37313&view=logs&j=4eda0b4a-bd0d-521a-0916-8285b9be9bb5&t=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37313&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] masteryhx commented on a diff in pull request #19679: [FLINK-23143][state/changelog] Support state migration for ChangelogS…
masteryhx commented on code in PR #19679: URL: https://github.com/apache/flink/pull/19679#discussion_r909148359 ## flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java: ## @@ -477,6 +478,50 @@ KeyGroupedInternalPriorityQueue create( } } +@Override +public S upgradeKeyedState( +TypeSerializer namespaceSerializer, StateDescriptor stateDescriptor) +throws Exception { +StateFactory stateFactory = getStateFactory(stateDescriptor); +Tuple2> registerResult = +tryRegisterKvStateInformation(stateDescriptor, namespaceSerializer, noTransform()); + Preconditions.checkState(kvStateInformation.containsKey(stateDescriptor.getName())); +kvStateInformation.computeIfPresent( +stateDescriptor.getName(), +(stateName, kvStateInfo) -> +new RocksDbKvStateInfo( +kvStateInfo.columnFamilyHandle, +new RegisteredKeyValueStateBackendMetaInfo<>( +kvStateInfo.metaInfo.snapshot(; +return stateFactory.createState( +stateDescriptor, registerResult, RocksDBKeyedStateBackend.this); Review Comment: Sure, I will also try to document the design and share it in the ticket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37306&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28290) SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError
Title: Message Title Huang Xingbo commented on FLINK-28290 Re: SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError cc Shengkai Fang Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28290) SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError
Title: Message Title Huang Xingbo created an issue Flink / FLINK-28290 SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError Issue Type: Bug Affects Versions: 1.16.0 Assignee: Unassigned Components: Table SQL / Gateway Created: 29/Jun/22 02:42 Labels: test-stability Priority: Major Reporter: Huang Xingbo 2022-06-28T11:40:17.0099766Z Jun 28 11:40:17 [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.192 s <<< FAILURE! - in org.apache.flink.table.gateway.service.session.SessionManagerTest 2022-06-28T11:40:17.0101180Z Jun 28 11:40:17 [ERROR] org.apache.flink.table.gateway.service.session.SessionManagerTest.testSessionNumberLimit Time elapsed: 0.115 s <<< FAILURE! 2022-06-28T11:40:17.0103132Z Jun 28 11:40:17 org.opentest4j.AssertionFailedError: Failed to create session, the count of active sessions exceeds the max count: 3 ==> Expected org.apache.flink.table.gateway.api.utils.SqlGatewayException to be thrown, but nothing was thrown. 2022-06-28T11:40:17.0104111Z Jun 28 11:40:17 at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:71) 2022-06-28T11:40:17.0105112Z Jun 28 11:40:17 at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:41) 2022-06-28T11:40:17.0105830Z Jun 28 11:40:17 at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3098) 2022-06-28T11:40:17.0106714Z Jun 28 11:40:17 at org.apache.
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37302&view=logs&j=bea52777-eaf8-5663-8482-18fbc3630e81&t=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] fsk119 commented on a diff in pull request #16108: [FLINK-22821][core] Stabilize NetUtils#getAvailablePort in order to avoid wrongly allocating any used ports
fsk119 commented on code in PR #16108: URL: https://github.com/apache/flink/pull/16108#discussion_r909145606 ## flink-core/src/main/java/org/apache/flink/util/FileLock.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.util; + +import org.apache.flink.annotation.Internal; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** A file lock used for avoiding race condition among multiple threads/processes. */ +@Internal +public class FileLock { +private static final String TEMP_DIR = System.getProperty("java.io.tmpdir"); +private final File file; +private FileOutputStream outputStream; +private java.nio.channels.FileLock lock; + +/** + * Initialize a FileLock using a file located at fullPath. + * + * @param fullPath The path of the locking file + */ +public FileLock(String fullPath) { +Preconditions.checkNotNull(fullPath, "fullPath should not be null"); +Path path = Paths.get(fullPath); +String normalizedFileName = normalizeFileName(path.getFileName().toString()); +if (normalizedFileName.isEmpty()) { +throw new IllegalArgumentException("There are no legal characters in the file name"); +} +this.file = +path.getParent() == null +? new File(normalizedFileName) Review Comment: Currently when the path.getParent() is null, the path of the File is a relative path. It means the FileLock will write a file into the current user directory according to the java doc of the `java.io.File`. If I use the `NetUtils#getAvaliablePort` in the test, the file will be created in the Flink source dir. For example, I add a test in the flink-connector-hive, the path of the `FileLock` is `/Users/shengkai/Work/flink/flink-connector-hive/orgapacheflinkutilNetUtils60481` . The idea will automatically add the file into the change list, which is very annoying. So I think why we don't create the file in the `TEMP_DIR` if the parent is null? I think it has two benefits: 1. It's much clean for the test that uses the `NetUtils#getAvaliablePort`. 2. It's better for different modules to check the port conflicts. For example, module A and module B run tests in parallel and both wants to use the same port. Currently the two module will create the file under their directoy rather than in the same directory. Any opinion are welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a diff in pull request #16108: [FLINK-22821][core] Stabilize NetUtils#getAvailablePort in order to avoid wrongly allocating any used ports
fsk119 commented on code in PR #16108: URL: https://github.com/apache/flink/pull/16108#discussion_r909145606 ## flink-core/src/main/java/org/apache/flink/util/FileLock.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.util; + +import org.apache.flink.annotation.Internal; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; + +/** A file lock used for avoiding race condition among multiple threads/processes. */ +@Internal +public class FileLock { +private static final String TEMP_DIR = System.getProperty("java.io.tmpdir"); +private final File file; +private FileOutputStream outputStream; +private java.nio.channels.FileLock lock; + +/** + * Initialize a FileLock using a file located at fullPath. + * + * @param fullPath The path of the locking file + */ +public FileLock(String fullPath) { +Preconditions.checkNotNull(fullPath, "fullPath should not be null"); +Path path = Paths.get(fullPath); +String normalizedFileName = normalizeFileName(path.getFileName().toString()); +if (normalizedFileName.isEmpty()) { +throw new IllegalArgumentException("There are no legal characters in the file name"); +} +this.file = +path.getParent() == null +? new File(normalizedFileName) Review Comment: Currently when the path.getParent() is null, the path of the File is a relative path. It means the FileLock will write a file into the current user directory according to the java doc of the `java.io.File`. If I use the `NetUtils#getAvaliablePort` in the test, the file will be created in the Flink source dir. For example, I add a test in the flink-connector-hive, the path of the `FileLock` is `/Users/shengkai/Work/flink/flink-connector-hive/orgapacheflinkutilNetUtils60481` . The idea will automatically add the file into the change list, which is very annoying. So I think think why we don't create the file in the `TEMP_DIR` if the parent is null? I think it has two benefits: 1. It's much clean for the test that uses the `NetUtils#getAvaliablePort`. 2. It's better for different modules to check the port conflicts. For example, module A and module B run tests in parallel and both wants to use the same port. Currently the two module will create the file under their directoy rather than in the same directory. Any opinion are welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 closed pull request #74: [FLINK-26828][example] add new examples sub module and create KMeans example
lindong28 closed pull request #74: [FLINK-26828][example] add new examples sub module and create KMeans example URL: https://github.com/apache/flink-ml/pull/74 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #74: [FLINK-26828][example] add new examples sub module and create KMeans example
lindong28 commented on PR #74: URL: https://github.com/apache/flink-ml/pull/74#issuecomment-1169469111 Thanks for the PR. We have added examples and the corresponding doc in https://github.com/apache/flink-ml/pull/119. I will close this PR as discussed offline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26721) PulsarSourceITCase.testSavepoint failed on azure pipeline
Title: Message Title Huang Xingbo commented on FLINK-26721 Re: PulsarSourceITCase.testSavepoint failed on azure pipeline https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37289&view=logs&j=aa18c3f6-13b8-5f58-86bb-c1cffb239496&t=502fb6c0-30a2-5e49-c5c2-a00fa3acb203 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-ml] lindong28 merged pull request #121: [FLINK-27715] Add pyflink examples
lindong28 merged PR #121: URL: https://github.com/apache/flink-ml/pull/121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #121: [FLINK-27715] Add pyflink examples
lindong28 commented on PR #121: URL: https://github.com/apache/flink-ml/pull/121#issuecomment-1169468374 Thanks for the update. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20098: [FLINK-28288][table][python] Support decode and encode in Table API
flinkbot commented on PR #20098: URL: https://github.com/apache/flink/pull/20098#issuecomment-1169465743 ## CI report: * aaf82b9952009cc3fc131cd86d0f0201a5986855 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Change By: ASF GitHub Bot Labels: pull-request-available Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] deadwind4 opened a new pull request, #20098: [FLINK-28288][table][python] Support decode and encode in Table API
deadwind4 opened a new pull request, #20098: URL: https://github.com/apache/flink/pull/20098 ## What is the purpose of the change Support decode and encode built-in functions in Table API. ## Brief change log - *Support decode and encode built-in functions in Table API.* - *Support decode and encode built-in functions in Python Table API.* ## Verifying this change This change added tests and can be verified as follows: *(example:)* - *Added test that validates that decode and encode in Table API work* - *Added test that validates that decode and encode in Python Table API work* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive)no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28289) Introduce Spark2 Reader for table store
Title: Message Title Jingsong Lee created an issue Flink / FLINK-28289 Introduce Spark2 Reader for table store Issue Type: Sub-task Assignee: Unassigned Components: Table Store Created: 29/Jun/22 02:20 Fix Versions: table-store-0.2.0 Priority: Major Reporter: Jingsong Lee Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-27907) Implement disk read and write logic for hybrid shuffle
Title: Message Title Weijie Guo updated an issue Flink / FLINK-27907 Implement disk read and write logic for hybrid shuffle Change By: Weijie Guo Implement disk read and write logic for hybrid shuffle , . In order to access the disk as sequentially as possible , Introduce HsMemoryDataSpiller, HsResultPartitionReadScheduler and HsSubpartitionFileReader. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-27907) Implement disk read and write logic for hybrid shuffle
Title: Message Title Weijie Guo updated an issue Flink / FLINK-27907 Implement disk read and write logic for hybrid shuffle Change By: Weijie Guo Implement disk read and write logic for hybrid shuffle, In order to access the disk as sequentially as possible Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-27907) Implement disk read and write logic for hybrid shuffle
Title: Message Title Weijie Guo updated an issue Flink / FLINK-27907 Implement disk read and write logic for hybrid shuffle Change By: Weijie Guo Summary: Abstract and implement Implement disk read and write logic for hybrid shuffle Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-27155) Reduce multiple reads to the same Changelog file in the same taskmanager during restore
Title: Message Title Feifan Wang commented on FLINK-27155 Re: Reduce multiple reads to the same Changelog file in the same taskmanager during restore Thanks Roman Khachatryan and Yuan Mei, I did have an implementation already, but it was based on 1.15.0. I noticed there is some change on master, and I also need modify the implementation as your comments. I will submit a pr after finished those. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title LuNing Wang updated an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Change By: LuNing Wang External issue ID: 28701 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title LuNing Wang updated an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Change By: LuNing Wang External issue ID: 28701 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title LuNing Wang created an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Issue Type: Improvement Affects Versions: 1.15.0 Assignee: Unassigned Components: API / Python, Table SQL / API Created: 29/Jun/22 02:12 Fix Versions: 1.16.0 Priority: Major Reporter: LuNing Wang Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] HuangXingBo commented on a diff in pull request #19878: [FLINK-27584][python] Support non-keyed co-broadcast processing
HuangXingBo commented on code in PR #19878: URL: https://github.com/apache/flink/pull/19878#discussion_r906952038 ## flink-python/src/main/java/org/apache/flink/streaming/api/transformations/python/PythonBroadcastStateTransformation.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.streaming.api.transformations.python; + +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.common.state.MapStateDescriptor; +import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.common.typeutils.TypeSerializer; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.streaming.api.functions.python.DataStreamPythonFunctionInfo; +import org.apache.flink.streaming.api.transformations.AbstractBroadcastStateTransformation; +import org.apache.flink.streaming.api.utils.ByteArrayWrapper; +import org.apache.flink.streaming.api.utils.ByteArrayWrapperSerializer; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; + +/** + * A {@link Transformation} representing a Python Co-Broadcast-Process operation, which will be + * translated into different operations by {@link + * org.apache.flink.streaming.runtime.translators.python.PythonBroadcastStateTransformationTranslator}. + */ +public class PythonBroadcastStateTransformation +extends AbstractBroadcastStateTransformation { + +private final Configuration configuration; +private final DataStreamPythonFunctionInfo dataStreamPythonFunctionInfo; + +public PythonBroadcastStateTransformation( +String name, +Configuration configuration, +DataStreamPythonFunctionInfo dataStreamPythonFunctionInfo, +Transformation regularInput, +Transformation broadcastInput, +List> broadcastStateDescriptors, +TypeInformation outTypeInfo, +int parallelism) { +super( +name, +regularInput, +broadcastInput, +broadcastStateDescriptors, +outTypeInfo, +parallelism); +this.configuration = configuration; +this.dataStreamPythonFunctionInfo = dataStreamPythonFunctionInfo; +updateManagedMemoryStateBackendUseCase(false); Review Comment: Could the step put in the PythonConfigUtil just like other python transformations? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #121: [FLINK-27715] Add pyflink examples
yunfengzhou-hub commented on code in PR #121: URL: https://github.com/apache/flink-ml/pull/121#discussion_r909134520 ## flink-ml-python/pyflink/examples/ml/__init__.py: ## @@ -15,3 +15,80 @@ # See the License for the specific language governing permissions and # limitations under the License. +from py4j.java_gateway import JavaClass, get_java_class, JavaObject +from pyflink.java_gateway import get_gateway +from pyflink.util import java_utils +from pyflink.util.java_utils import to_jarray, load_java_class + + +def add_jars_to_context_class_loader(jar_urls): +""" +Add jars to Python gateway server for local compilation and local execution (i.e. minicluster). +There are many component in Flink which won't be added to classpath by default. e.g. Kafka +connector, JDBC connector, CSV format etc. This utility function can be used to hot load the +jars. + +:param jar_urls: The list of jar urls. +""" +gateway = get_gateway() +# validate and normalize +jar_urls = [gateway.jvm.java.net.URL(url) for url in jar_urls] +context_classloader = gateway.jvm.Thread.currentThread().getContextClassLoader() +existing_urls = [] +class_loader_name = context_classloader.getClass().getName() +if class_loader_name == "java.net.URLClassLoader": +existing_urls = set([url.toString() for url in context_classloader.getURLs()]) +if all([url.toString() in existing_urls for url in jar_urls]): +# if urls all existed, no need to create new class loader. +return + +URLClassLoaderClass = load_java_class("java.net.URLClassLoader") +if is_instance_of(context_classloader, URLClassLoaderClass): +if class_loader_name == "org.apache.flink.runtime.execution.librarycache." \ + "FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader": +ensureInner = context_classloader.getClass().getDeclaredMethod("ensureInner", None) +ensureInner.setAccessible(True) +context_classloader = ensureInner.invoke(context_classloader, None) + +addURL = URLClassLoaderClass.getDeclaredMethod( +"addURL", +to_jarray( +gateway.jvm.Class, +[load_java_class("java.net.URL")])) +addURL.setAccessible(True) + +for url in jar_urls: +addURL.invoke(context_classloader, to_jarray(get_gateway().jvm.Object, [url])) + +else: +context_classloader = create_url_class_loader(jar_urls, context_classloader) + gateway.jvm.Thread.currentThread().setContextClassLoader(context_classloader) + + +def is_instance_of(java_object, java_class): +gateway = get_gateway() +if isinstance(java_class, str): +param = java_class +elif isinstance(java_class, JavaClass): +param = get_java_class(java_class) +elif isinstance(java_class, JavaObject): +if not is_instance_of(java_class, gateway.jvm.Class): +param = java_class.getClass() +else: +param = java_class +else: +raise TypeError( +"java_class must be a string, a JavaClass, or a JavaObject") + +return gateway.jvm.org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.isInstanceOf( +param, java_object) + + +def create_url_class_loader(urls, parent_class_loader): +gateway = get_gateway() +url_class_loader = gateway.jvm.java.net.URLClassLoader( +to_jarray(gateway.jvm.java.net.URL, urls), parent_class_loader) +return url_class_loader + + +java_utils.add_jars_to_context_class_loader = add_jars_to_context_class_loader Review Comment: Yes. It is a walk-around to a bug related to FLINK-15635 and FLINK-28002. I'll add a TODO to avoid overwriting `pyflink.util.java_utils` after this bug is fixed and released. And according to offline discussion, I'll try to reuse the `add_jars_to_context_class_loader` method across `pyflink/ml/__init__.py` and `pyflink/examples/ml/__init__.py`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #20003: [FLINK-28080][runtime] Introduce MutableURLClassLoader as parent class of FlinkUserClassLoader and SafetyNetWrapperClassLoader
lsyldliu commented on PR #20003: URL: https://github.com/apache/flink/pull/20003#issuecomment-1169449211 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #20057: [FLINK-28213][runtime] StreamExecutionEnvironment configure method support override pipeline.jars option
lsyldliu commented on PR #20057: URL: https://github.com/apache/flink/pull/20057#issuecomment-1169448529 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28172) Scatter dstl files into separate directories by job id
Title: Message Title Feifan Wang commented on FLINK-28172 Re: Scatter dstl files into separate directories by job id Thanks very much Roman Khachatryan. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] zoltar9264 commented on pull request #20093: [FLINK-28172][changelog] Scatter dstl files into separate directories…
zoltar9264 commented on PR #20093: URL: https://github.com/apache/flink/pull/20093#issuecomment-1169447489 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-9970) Add ASCII/CHR function for Table API
Title: Message Title Dian Fu closed an issue as Duplicate Closing this ticket as it's already supported in FLINK-28092. Flink / FLINK-9970 Add ASCII/CHR function for Table API Change By: Dian Fu Resolution: Duplicate Status: Open Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28092) Support ASCII and CHR built-in function in the Table API
Title: Message Title Dian Fu updated an issue Flink / FLINK-28092 Support ASCII and CHR built-in function in the Table API Change By: Dian Fu Affects Version/s: 1.15.0 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] xintongsong commented on pull request #20031: [FLINK-27906][runtime] Introduce HsDataIndex.
xintongsong commented on PR #20031: URL: https://github.com/apache/flink/pull/20031#issuecomment-1169445372 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28092) Support ASCII and CHR built-in function in the Table API
Title: Message Title Dian Fu closed an issue as Fixed Merged to master via e7e120acf9258ed8ef3c416fed6ed59cc0a06855 Flink / FLINK-28092 Support ASCII and CHR built-in function in the Table API Change By: Dian Fu Assignee: LuNing Wang Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] dianfu closed pull request #19988: [FLINK-28092][table] Support ASCII and CHR built-in function in the Table API
dianfu closed pull request #19988: [FLINK-28092][table] Support ASCII and CHR built-in function in the Table API URL: https://github.com/apache/flink/pull/19988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #19988: [FLINK-28092][table] Support ASCII and CHR built-in function in the Table API
dianfu commented on PR #19988: URL: https://github.com/apache/flink/pull/19988#issuecomment-1169444531 LGTM. Merging... > If we migrate all built-in functions to the new stack, we could create a new Jira ticket to target this. Agree. Let's do that in a separate ticket if we found it necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on pull request #19998: [FLINK-27822][docs-zh] Translate the doc of checkpoint/savepoint guar…
zoltar9264 commented on PR #19998: URL: https://github.com/apache/flink/pull/19998#issuecomment-1169443111 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283: URL: https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r909107158 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractDeploymentObserver.java: ## @@ -279,21 +279,35 @@ private void checkIfAlreadyUpgraded(FlinkDeployment flinkDep, Context context) { Long upgradeTargetGeneration = ReconciliationUtils.getUpgradeTargetGeneration(flinkDep); +Long currentSpecGeneration = flinkDep.getMetadata().getGeneration(); + if (deployedGeneration.equals(upgradeTargetGeneration)) { -logger.info( -"Last reconciled generation is already deployed, setting reconciliation status to " -+ ReconciliationState.DEPLOYED); +logger.info("Pending upgrade is already deployed, updating status."); var firstDeploy = status.getReconciliationStatus().getLastReconciledSpec() == null; -var conf = -firstDeploy -? configManager.getDeployConfig( -flinkDep.getMetadata(), flinkDep.getSpec()) -: configManager.getObserveConfig(flinkDep); +if (firstDeploy) { + ReconciliationUtils.updateForSpecReconciliationSuccess( +flinkDep, +JobState.RUNNING, +configManager.getDeployConfig( +flinkDep.getMetadata(), flinkDep.getSpec())); +} else { + ReconciliationUtils.updateStatusForAlreadyUpgraded(flinkDep); +} + +status.getJobStatus() + .setState(org.apache.flink.api.common.JobStatus.RECONCILING.name()); + status.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.DEPLOYING); +} else if (deployedGeneration.equals(currentSpecGeneration)) { Review Comment: What about also record the updateTargetSpec before execute the upgrade or submit. So that, we can always have the right base to compare. And it will not changed by user. In the observer, we compare it with the deployedGeneration. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] tweise commented on pull request #281: [FLINK-26891] Record important deployment events
tweise commented on PR #281: URL: https://github.com/apache/flink-kubernetes-operator/pull/281#issuecomment-1169419765 > The PR looks pretty good! > > My only question whether we should trigger events before or after executing the actions using the Flink service. > > For example now we trigger a suspended event before trying to cancel the job. So it might happen that the event is there but the job is not suspended. Of course we could have the reverse situation if we changed the order. Just wanted to discuss this explicitly. > > What do you think @tweise ? It should be OK to have the event logged first. If the subsequent action fails, we have a trace that it was attempted and any subsequent error would yield anther event so that there should not be any confusion. I pulled the reason codes into `EventRecorder`. (I did not add them to the listener as the listener does not actually reference them and it would also be a bit unusual to have them there from the operator code caller perspective.) @gyfora @morhidi PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nyingping commented on pull request #20028: [hotfix][docs] Remove redundant "." from sample code
nyingping commented on PR #20028: URL: https://github.com/apache/flink/pull/20028#issuecomment-1169414641 @Myasuka @MartijnVisser Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283: URL: https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r909093534 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractDeploymentObserver.java: ## @@ -279,21 +279,35 @@ private void checkIfAlreadyUpgraded(FlinkDeployment flinkDep, Context context) { Long upgradeTargetGeneration = ReconciliationUtils.getUpgradeTargetGeneration(flinkDep); +Long currentSpecGeneration = flinkDep.getMetadata().getGeneration(); + if (deployedGeneration.equals(upgradeTargetGeneration)) { -logger.info( -"Last reconciled generation is already deployed, setting reconciliation status to " -+ ReconciliationState.DEPLOYED); +logger.info("Pending upgrade is already deployed, updating status."); var firstDeploy = status.getReconciliationStatus().getLastReconciledSpec() == null; -var conf = -firstDeploy -? configManager.getDeployConfig( -flinkDep.getMetadata(), flinkDep.getSpec()) -: configManager.getObserveConfig(flinkDep); +if (firstDeploy) { + ReconciliationUtils.updateForSpecReconciliationSuccess( +flinkDep, +JobState.RUNNING, +configManager.getDeployConfig( +flinkDep.getMetadata(), flinkDep.getSpec())); +} else { + ReconciliationUtils.updateStatusForAlreadyUpgraded(flinkDep); +} + +status.getJobStatus() + .setState(org.apache.flink.api.common.JobStatus.RECONCILING.name()); + status.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.DEPLOYING); +} else if (deployedGeneration.equals(currentSpecGeneration)) { Review Comment: I think we have a problem here, we can only do the thing when the deployedGeneration equal to the `currentSpecGeneration` or the `upgradeTargetGeneration`. If they are not equal, then we do not know how to bring the job do the desired version. It's mainly the concern you mentioned in the description -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283: URL: https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r908605561 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/ReconciliationUtils.java: ## @@ -352,22 +382,33 @@ ErrorStatusUpdateControl toErrorStatusUpdateControl( return ErrorStatusUpdateControl.noStatusUpdate(); } -public static Long getUpgradeTargetGeneration(FlinkDeployment deployment) { +/** + * Get spec generation for the current in progress upgrade. + * + * @param resource Flink resource. + * @return The spec generation for the upgrade. + */ +public static Long getUpgradeTargetGeneration(AbstractFlinkResource resource) { var lastSpecWithMeta = -deployment -.getStatus() +resource.getStatus() .getReconciliationStatus() .deserializeLastReconciledSpecWithMeta(); if (lastSpecWithMeta == null || lastSpecWithMeta.f1 == null) { // For first deployments and when migrating from before this feature simply return // current generation -return deployment.getMetadata().getGeneration(); +return resource.getMetadata().getGeneration(); } return lastSpecWithMeta.f1.get("metadata").get("generation").asLong(-1L); Review Comment: Now the specWithMeta's meta is a `ObjectNode`, which will make it no schema and difficult to use, do you think we need to improve it here ? Like directly return a Bean object ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283: URL: https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r908605561 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/ReconciliationUtils.java: ## @@ -352,22 +382,33 @@ ErrorStatusUpdateControl toErrorStatusUpdateControl( return ErrorStatusUpdateControl.noStatusUpdate(); } -public static Long getUpgradeTargetGeneration(FlinkDeployment deployment) { +/** + * Get spec generation for the current in progress upgrade. + * + * @param resource Flink resource. + * @return The spec generation for the upgrade. + */ +public static Long getUpgradeTargetGeneration(AbstractFlinkResource resource) { var lastSpecWithMeta = -deployment -.getStatus() +resource.getStatus() .getReconciliationStatus() .deserializeLastReconciledSpecWithMeta(); if (lastSpecWithMeta == null || lastSpecWithMeta.f1 == null) { // For first deployments and when migrating from before this feature simply return // current generation -return deployment.getMetadata().getGeneration(); +return resource.getMetadata().getGeneration(); } return lastSpecWithMeta.f1.get("metadata").get("generation").asLong(-1L); Review Comment: Now the specWithMeta's meta is a `ObjectNode`, which will make it no schema and difficult to use, do you think we need to improve it here ? ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractDeploymentObserver.java: ## @@ -279,21 +279,35 @@ private void checkIfAlreadyUpgraded(FlinkDeployment flinkDep, Context context) { Long upgradeTargetGeneration = ReconciliationUtils.getUpgradeTargetGeneration(flinkDep); +Long currentSpecGeneration = flinkDep.getMetadata().getGeneration(); + if (deployedGeneration.equals(upgradeTargetGeneration)) { -logger.info( -"Last reconciled generation is already deployed, setting reconciliation status to " -+ ReconciliationState.DEPLOYED); +logger.info("Pending upgrade is already deployed, updating status."); var firstDeploy = status.getReconciliationStatus().getLastReconciledSpec() == null; -var conf = -firstDeploy -? configManager.getDeployConfig( -flinkDep.getMetadata(), flinkDep.getSpec()) -: configManager.getObserveConfig(flinkDep); +if (firstDeploy) { + ReconciliationUtils.updateForSpecReconciliationSuccess( +flinkDep, +JobState.RUNNING, +configManager.getDeployConfig( +flinkDep.getMetadata(), flinkDep.getSpec())); +} else { + ReconciliationUtils.updateStatusForAlreadyUpgraded(flinkDep); +} + +status.getJobStatus() + .setState(org.apache.flink.api.common.JobStatus.RECONCILING.name()); + status.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.DEPLOYING); +} else if (deployedGeneration.equals(currentSpecGeneration)) { Review Comment: I think we have a problem here, we can only do the thing when the deployedGeneration equal to the `currentSpecGeneration` or the `upgradeTargetGeneration`. If they are not equal, then we do not know how to bring the job do the desired version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28287) Should TaskManagerRunner need a ShutdownHook
Title: Message Title JieFang.He updated an issue Flink / FLINK-28287 Should TaskManagerRunner need a ShutdownHook Change By: JieFang.He TaskManagerRunner has a close method,but did not call when it stop.Some resources in TaskManagerRunner come with ShutdownHook, but some resources do not, such as rpcSystem, which causes the temporary file flink-rpc-akka_*.jar to not be deleted when stop.Should TaskManagerRunner need a ShutdownHook to call the close method to release all resources Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28287) Should TaskManagerRunner need a ShutdownHook
Title: Message Title JieFang.He created an issue Flink / FLINK-28287 Should TaskManagerRunner need a ShutdownHook Issue Type: Improvement Affects Versions: 1.14.0 Assignee: Unassigned Created: 29/Jun/22 00:06 Priority: Major Reporter: JieFang.He TaskManagerRunner has a close method,but did not call when it stop. Some resources in TaskManagerRunner come with ShutdownHook, but some resources do not, such as rpcSystem, which causes the temporary file flink-rpc-akka_*.jar to not be deleted when stop. Should TaskManagerRunner need a ShutdownHook to call the close method Add Comment
[GitHub] [flink] syhily commented on pull request #19473: [FLINK-27199][Connector/Pulsar] Bump pulsar to 2.10.0
syhily commented on PR #19473: URL: https://github.com/apache/flink/pull/19473#issuecomment-1169400631 > Thanks for the patch @syhily. I notice that this PR touches a lot of components including Pulsar client version, docs even default value of sink options. What about breaking these changes into different commits or PRs? I'm afraid squashing them into one commit is way too ambiguous. OK, I'll split this PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27155) Reduce multiple reads to the same Changelog file in the same taskmanager during restore
Title: Message Title Roman Khachatryan commented on FLINK-27155 Re: Reduce multiple reads to the same Changelog file in the same taskmanager during restore Thanks for clarifying! I moved the comments and responses to the document. It looks like we're close to a consensus now (I've discussed the design with Yuan Mei, though she'll probably take a closer look). So if you have a PR alredy, we could start reviewing it. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] rkhachatryan commented on a diff in pull request #19679: [FLINK-23143][state/changelog] Support state migration for ChangelogS…
rkhachatryan commented on code in PR #19679: URL: https://github.com/apache/flink/pull/19679#discussion_r909048188 ## flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java: ## @@ -477,6 +478,50 @@ KeyGroupedInternalPriorityQueue create( } } +@Override +public S upgradeKeyedState( +TypeSerializer namespaceSerializer, StateDescriptor stateDescriptor) +throws Exception { +StateFactory stateFactory = getStateFactory(stateDescriptor); +Tuple2> registerResult = +tryRegisterKvStateInformation(stateDescriptor, namespaceSerializer, noTransform()); + Preconditions.checkState(kvStateInformation.containsKey(stateDescriptor.getName())); +kvStateInformation.computeIfPresent( +stateDescriptor.getName(), +(stateName, kvStateInfo) -> +new RocksDbKvStateInfo( +kvStateInfo.columnFamilyHandle, +new RegisteredKeyValueStateBackendMetaInfo<>( +kvStateInfo.metaInfo.snapshot(; +return stateFactory.createState( +stateDescriptor, registerResult, RocksDBKeyedStateBackend.this); Review Comment: I think we are all on the same page now and it makes sense to document this design somehwere, along with other aspects of this feature - for future reference (as well as to make sure we're indeed on the same page). WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16419) Avoid to recommit transactions which are known committed successfully to Kafka upon recovery
Title: Message Title Yordan Pavlov commented on FLINK-16419 Re: Avoid to recommit transactions which are known committed successfully to Kafka upon recovery Martijn Visser As advised I have switched to KafkaSink and Flink 1.14.2. However I am still seeing the following error on recovering from a checkpoint: switched from INITIALIZING to FAILED with failure cause: org.apache.kafka.common.KafkaException: Unhandled error in EndTxnResponse: The producer attempted to use a producer id which is not currently assigned to its transactional id. at org.apache.kafka.clients.producer.internals.TransactionManager$EndTxnHandler.handleResponse(TransactionManager.java:1362) at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1074) at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:569) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:425) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:311) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244) at java.base/java.lang.Thread.run(Unknown Source) Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] flinkbot commented on pull request #20097: [FLINK-28284][Connectors/Jdbc] Add JdbcSink with new format
flinkbot commented on PR #20097: URL: https://github.com/apache/flink/pull/20097#issuecomment-1169146748 ## CI report: * 8bcbd05a6b7eeb2000685b729f860eae3c1b50aa UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18096) Generated avro formats should support user specified name and namespace
Title: Message Title Robin Neufeld commented on FLINK-18096 Re: Generated avro formats should support user specified name and namespace Without being able to customize the namespace and record name that Flink generates, it's very difficult to use this in a large organization. At my work we have a schema registry that is used by many teams, and everyone pre-registers their schemas. There are conventions about what the namespace and record name should be. We also enforce backwards and forwards compatibility to schema changes. When using this Flink connector/format, it tries to edit the name/namespace on the schema, which fails since it counts as backwards incompatible. It seems like a small detail, but it is a major obstacle for using this implementation. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] eskabetxe commented on pull request #20097: [FLINK-28284] Add JdbcSink with new format
eskabetxe commented on PR #20097: URL: https://github.com/apache/flink/pull/20097#issuecomment-1169141734 @leonardBang @MartijnVisser could you check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28284) Add JdbcSink based on new Sink (sink2)
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28284 Add JdbcSink based on new Sink (sink2) Change By: ASF GitHub Bot Labels: pull-request-available Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] eskabetxe opened a new pull request, #20097: [FLINK-28284] Add JdbcSink with new format
eskabetxe opened a new pull request, #20097: URL: https://github.com/apache/flink/pull/20097 ## What is the purpose of the change Add a JdbcSink with new format (sink2) ## Brief change log - *JdbcSink* the new sink - *JdbcSinkWriter* the writer used by the new sink - *JdbcQueryStatement* the query and preparestatement that will be used - *JdbcWriterStatement* the writer statement that will write to jdbc - *SimpleJdbcWriterStatement* a simple implementation of JdbcWriterStatement ## Verifying this change This change added tests and can be verified as follows: - JdbcITCase.testInsertWithSinkTo - JdbcITCase.testObjectReuseWithSinkTo ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: ( no ) - The runtime per-record code paths (performance sensitive): (no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes ) - If yes, how is the feature documented? ( JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28268) Elasticsearch6SinkE2ECase failed with no space left on device
Title: Message Title Martijn Visser closed an issue as Not A Problem Flink / FLINK-28268 Elasticsearch6SinkE2ECase failed with no space left on device Change By: Martijn Visser Resolution: Not A Problem Status: In Progress Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Closed] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing
Title: Message Title Martijn Visser closed an issue as Fixed Merged in: master: 5d94fe6c128f2fa9e3531ab8b169e8a2e21bc391 release-1.15: 7557a282314b29808ea5552aaf69134a7ebf9d08 release-1.14: 852713c52e1c4ce8d27905f5d3dedaed2015d4eb Flink / FLINK-28263 TPC-DS Bash e2e tests don't clean-up after completing Change By: Martijn Visser Fix Version/s: 1.16.0 Fix Version/s: 1.15.2 Fix Version/s: 1.14.6 Resolution: Fixed Status: In Progress Closed Add Comment This message was sent by Atlassian Jira (v8.20.
[GitHub] [flink] MartijnVisser merged pull request #20096: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser merged PR #20096: URL: https://github.com/apache/flink/pull/20096 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #20095: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser merged PR #20095: URL: https://github.com/apache/flink/pull/20095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser opened a new pull request, #20096: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser opened a new pull request, #20096: URL: https://github.com/apache/flink/pull/20096 Unchanged backport of https://github.com/apache/flink/pull/20083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org