[GitHub] [flink] KarmaGYZ commented on pull request #20042: [FLINK-19358][runtime] Make the job id distinct in application mode w…
KarmaGYZ commented on PR #20042: URL: https://github.com/apache/flink/pull/20042#issuecomment-1169561842 Thanks for the comments. PR updated. @xintongsong @wangyang0918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28282) Planner free in flink-table-store-codegen
Title: Message Title Jingsong Lee updated an issue Flink / FLINK-28282 Planner free in flink-table-store-codegen Change By: Jingsong Lee Parent: FLINK-28073 Issue Type: Improvement Sub-task Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28292) Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28292 Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory Change By: ASF GitHub Bot Labels: pull-request-available Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-table-store] JingsongLi opened a new pull request, #180: [FLINK-28292] Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory
JingsongLi opened a new pull request, #180: URL: https://github.com/apache/flink-table-store/pull/180 These APIs were changed significantly in 1.14 and 1.15 and are not compatible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28292) Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory
Title: Message Title Jingsong Lee created an issue Flink / FLINK-28292 Get rid of BulkReaderFormatFactory and BulkWriterFormatFactory Issue Type: Sub-task Assignee: Jingsong Lee Components: Table Store Created: 29/Jun/22 05:48 Fix Versions: table-store-0.2.0 Priority: Major Reporter: Jingsong Lee These APIs were changed significantly in 1.14 and 1.15 and are not compatible Add Comment
[jira] [Updated] (FLINK-28073) Table Store supports Flink 1.14
Title: Message Title Jingsong Lee updated an issue Flink / FLINK-28073 Table Store supports Flink 1.14 Change By: Jingsong Lee Priority: Minor Critical Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] 6591812 commented on pull request #20072: [FLINK-28136][runtime] Implement ExecutionTimeBasedSlowTaskDetector
6591812 commented on PR #20072: URL: https://github.com/apache/flink/pull/20072#issuecomment-1169530980 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on a diff in pull request #20075: [FLINK-27789][network] Disable overdraft buffer for LegacySource
1996fanrui commented on code in PR #20075: URL: https://github.com/apache/flink/pull/20075#discussion_r909185339 ## flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java: ## @@ -385,11 +385,16 @@ public Task( shuffleEnvironment.createShuffleIOOwnerContext( taskNameWithSubtaskAndId, executionId, metrics.getIOMetricGroup()); +boolean overdraftBufferEnabled = +!nameOfInvokableClass.equals( + "org.apache.flink.streaming.runtime.tasks.SourceStreamTask"); Review Comment: Updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile
JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile URL: https://github.com/apache/flink-table-store/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a diff in pull request #20056: [FLINK-28142][runtime] Enrich TaskManagerLocation with node information
wangyang0918 commented on code in PR #20056:
URL: https://github.com/apache/flink/pull/20056#discussion_r909164865
##
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/decorators/InitTaskManagerDecorator.java:
##
@@ -149,7 +153,16 @@ private Container decorateMainContainer(Container
container) {
.withResources(resourceRequirements);
// Merge fields
-
mainContainerBuilder.addAllToPorts(getContainerPorts()).addAllToEnv(getCustomizedEnvs());
+mainContainerBuilder
+.addAllToPorts(getContainerPorts())
+.addAllToEnv(getCustomizedEnvs())
+.addNewEnv()
+.withName(ENV_FLINK_POD_NODE_ID)
+.withValueFrom(
Review Comment:
I would like to suggest also setting the `ENV_FLINK_POD_NODE_ID` in
standalone on K8s.
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/kubernetes/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-28286) move “enablechangelog” constant out of flink-streaming-java module
Title: Message Title
Yanfei Lei updated an issue
Flink / FLINK-28286
move “enablechangelog” constant out of flink-streaming-java module
Change By:
Yanfei Lei
Some methods in the flink-runtime module want to use {{StreamConfig.ENABLE_CHANGE_LOG_STATE_BACKEND}} constant(in flink-streaming-java module), but flink-runtime should not depend on flink-streaming-java. We should move {{ENABLE_CHANGE_LOG_STATE_BACKEND }} to a right place. {{{ } { } }} See this [discussion|https://github.com/apache/flink/pull/19907#discussion_r902485780] for more details. {{}}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] Myasuka commented on pull request #19998: [FLINK-27822][docs-zh] Translate the doc of checkpoint/savepoint guar…
Myasuka commented on PR #19998: URL: https://github.com/apache/flink/pull/19998#issuecomment-1169492816 > Hi @Myasuka , It seems that test can't be triggered, have you ever come across this ? No, @zentol do you know what happened here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on pull request #19998: [FLINK-27822][docs-zh] Translate the doc of checkpoint/savepoint guar…
zoltar9264 commented on PR #19998: URL: https://github.com/apache/flink/pull/19998#issuecomment-1169491021 Hi @Myasuka , It seems that test can't be triggered, have you ever come across this ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a diff in pull request #20057: [FLINK-28213][runtime] StreamExecutionEnvironment configure method support override pipeline.jars option
zhuzhurk commented on code in PR #20057:
URL: https://github.com/apache/flink/pull/20057#discussion_r909160467
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java:
##
@@ -1023,6 +1025,19 @@ public void configure(ReadableConfig configuration,
ClassLoader classLoader) {
.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH,
flag));
+// merge PipelineOptions.JARS, user maybe set this option in high
level such as table
+// module, so here need to merge the jars from both configuration
object
+configuration
+.getOptional(PipelineOptions.JARS)
+.ifPresent(
+jars ->
+ConfigUtils.mergeCollectionsToConfig(
+this.configuration,
+PipelineOptions.JARS,
+new HashSet<>(jars),
Review Comment:
```suggestion
Collections.unmodifiableCollection(jars),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-28206) EOFException on Checkpoint Recovery
Title: Message Title Yun Tang commented on FLINK-28206 Re: EOFException on Checkpoint Recovery uharaqo After you checked the _metadata, I think the problem is related with the serialized RetrievableStateHandle stored in high-availability.storageDir. Since the store path is named as random string, it might not be easy to find which one points to your checkpoint-58310. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
Title: Message Title Yun Tang commented on FLINK-24932 Re: Frocksdb cannot run on Apple M1 Sebastian Mattheis, from my knowledge, RocksDB community has a benchmark-tool, however, they seem do not have public procedure to run the benchmark with specific parameters before any release (maybe Facebook has some internal procedure to avoid performance regression, but they did not make it public). Since RocksDB-6.20.3 already has performance regression compared with previous RocksDB-5.17, I think another 10% regression is a blocker here. I think your pick of 541463 looks good. Did you ever consider to pick RocksDB's community change of supporting Apple Silicon to RocksDB-6.20.3 just as I suggested? Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-table-store] JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile
JingsongLi closed pull request #178: [FLINK-28244] Introduce changelog file for DataFile URL: https://github.com/apache/flink-table-store/pull/178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
Title: Message Title
jiulong.zhu updated an issue
Flink / FLINK-28291
Add kerberos delegation token renewer feature instead of logged from keytab individually
Change By:
jiulong.zhu
h2. 1. Design LifeCyclic LifeCycle of delegation token in RM: # Container starts with DT given by client. # Enable delegation token renewer by: ## set {{security.kerberos.token.renew.enabled}} true, default false. And ## specify {{security.kerberos.login.keytab}} and {{security.kerberos.login.principal}} # When enabled delegation token renewer, the renewer thread will re-obtain tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider now). Then the renewer thread will broadcast new tokens to RM locally, all JMs and all TMs by RPCGateway. # RM process adds new tokens in context by UserGroupInformation. LifeCyclic LifeCycle of delegation token in JM / TM: # TaskManager starts with keytab stored in remote hdfs. # When registered successfully, JM / TM get the current tokens of RM boxed by {{JobMasterRegistrationSuccess}} / {{{}TaskExecutorRegistrationSuccess{}}}. # JM / TM process add new tokens in context by UserGroupInformation.It’s too heavy and unnecessary to retrieval leader of ResourceManager by HAService, so DelegationTokenManager is instanced by ResourceManager. So DelegationToken can hold the reference of ResourceManager, instead of RM RPCGateway or self gateway.h2. 2. Test # No local junit test. It’s too heavy to build junit environments including KDC and local hadoop. # Cluster teststep 1: Specify krb5.conf with short token lifetime(ticket_lifetime, renew_lifetime) when submitting flink application.```{{flink run -yD security.kerberos.token.renew.enabled=true -yD security.kerberos.krb5-conf.path= /home/work/krb5.conf -yD security.kerberos.login.use-ticket-cache=false ...}}```step 2: Watch token identifier changelog and synchronizer between rm and worker.>> In RM / JM log, 2022-06-28 15:13:03,509 INFO org.apache.flink.runtime.util.HadoopUtils [] - New token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) created in KerberosDelegationToken, and next schedule delay is 64799880 ms. 2022-06-28 15:13:03,529 INFO org.apache.flink.runtime.util.HadoopUtils [] - Updating delegation tokens for current user. 2022-06-28 15:13:04,729 INFO org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52101 for work on ha-hdfs:newfyyy) from RM.… 2022-06-29 09:13:03,732 INFO org.apache.flink.runtime.util.HadoopUtils [] - New token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy) created in KerberosDelegationToken, and next schedule delay is 64800045 ms.2022-06-29 09:13:03,805 INFO org.apache.flink.runtime.util.HadoopUtils [] - Updating delegation tokens for current user. 2022-06-29 09:13:03,806 INFO org.apache.flink.runtime.util.HadoopUtils [] - JobMaster receives new token (HDFS_DELEGATION_TOKEN token 52310 for work on ha-hdfs:newfyyy) from RM.>> In TM log, 2022-06-28 15:13:17,983 INFO
[jira] [Created] (FLINK-28291) Add kerberos delegation token renewer feature instead of logged from keytab individually
Title: Message Title jiulong.zhu created an issue Flink / FLINK-28291 Add kerberos delegation token renewer feature instead of logged from keytab individually Issue Type: New Feature Affects Versions: 1.13.5 Assignee: Unassigned Attachments: 0001-add-KerberosDelegationTokenManager-to-renew-token-pe.patch Components: Deployment / YARN Created: 29/Jun/22 02:52 Fix Versions: 1.13.5 Priority: Minor Reporter: jiulong.zhu 1. Design LifeCyclic of delegation token in RM: Container starts with DT given by client. Enable delegation token renewer by: set security.kerberos.token.renew.enabled true, default false. And specify security.kerberos.login.keytab and security.kerberos.login.principal When enabled delegation token renewer, the renewer thread will re-obtain tokens from DelegationTokenProvider(only HadoopFSDelegationTokenProvider now). Then the renewer thread
[jira] [Commented] (FLINK-26979) JsonFunctionsITCase.test failed on azure
Title: Message Title Huang Xingbo commented on FLINK-26979 Re: JsonFunctionsITCase.test failed on azure https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37329=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo updated an issue Flink / FLINK-28269 Kubernetes test failed with permission denied Change By: Huang Xingbo Affects Version/s: 1.15.0 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37328=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37329=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37327=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28198) CassandraConnectorITCase#testRaiseCassandraRequestsTimeouts fails with timeout
Title: Message Title Huang Xingbo commented on FLINK-28198 Re: CassandraConnectorITCase#testRaiseCassandraRequestsTimeouts fails with timeout https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37313=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37313=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] masteryhx commented on a diff in pull request #19679: [FLINK-23143][state/changelog] Support state migration for ChangelogS…
masteryhx commented on code in PR #19679:
URL: https://github.com/apache/flink/pull/19679#discussion_r909148359
##
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java:
##
@@ -477,6 +478,50 @@ KeyGroupedInternalPriorityQueue create(
}
}
+@Override
+public S upgradeKeyedState(
+TypeSerializer namespaceSerializer, StateDescriptor
stateDescriptor)
+throws Exception {
+StateFactory stateFactory = getStateFactory(stateDescriptor);
+Tuple2> registerResult =
+tryRegisterKvStateInformation(stateDescriptor,
namespaceSerializer, noTransform());
+
Preconditions.checkState(kvStateInformation.containsKey(stateDescriptor.getName()));
+kvStateInformation.computeIfPresent(
+stateDescriptor.getName(),
+(stateName, kvStateInfo) ->
+new RocksDbKvStateInfo(
+kvStateInfo.columnFamilyHandle,
+new RegisteredKeyValueStateBackendMetaInfo<>(
+kvStateInfo.metaInfo.snapshot(;
+return stateFactory.createState(
+stateDescriptor, registerResult,
RocksDBKeyedStateBackend.this);
Review Comment:
Sure, I will also try to document the design and share it in the ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37306=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28290) SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError
Title: Message Title Huang Xingbo commented on FLINK-28290 Re: SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError cc Shengkai Fang Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28290) SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError
Title: Message Title Huang Xingbo created an issue Flink / FLINK-28290 SessionManagerTest.testSessionNumberLimit failed with AssertionFailedError Issue Type: Bug Affects Versions: 1.16.0 Assignee: Unassigned Components: Table SQL / Gateway Created: 29/Jun/22 02:42 Labels: test-stability Priority: Major Reporter: Huang Xingbo 2022-06-28T11:40:17.0099766Z Jun 28 11:40:17 [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.192 s <<< FAILURE! - in org.apache.flink.table.gateway.service.session.SessionManagerTest 2022-06-28T11:40:17.0101180Z Jun 28 11:40:17 [ERROR] org.apache.flink.table.gateway.service.session.SessionManagerTest.testSessionNumberLimit Time elapsed: 0.115 s <<< FAILURE! 2022-06-28T11:40:17.0103132Z Jun 28 11:40:17 org.opentest4j.AssertionFailedError: Failed to create session, the count of active sessions exceeds the max count: 3 ==> Expected org.apache.flink.table.gateway.api.utils.SqlGatewayException to be thrown, but nothing was thrown. 2022-06-28T11:40:17.0104111Z Jun 28 11:40:17 at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:71) 2022-06-28T11:40:17.0105112Z Jun 28 11:40:17 at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:41) 2022-06-28T11:40:17.0105830Z Jun 28 11:40:17 at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3098) 2022-06-28T11:40:17.0106714Z Jun 28 11:40:17 at
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Huang Xingbo commented on FLINK-28269 Re: Kubernetes test failed with permission denied https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37302=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] fsk119 commented on a diff in pull request #16108: [FLINK-22821][core] Stabilize NetUtils#getAvailablePort in order to avoid wrongly allocating any used ports
fsk119 commented on code in PR #16108:
URL: https://github.com/apache/flink/pull/16108#discussion_r909145606
##
flink-core/src/main/java/org/apache/flink/util/FileLock.java:
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.util;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/** A file lock used for avoiding race condition among multiple
threads/processes. */
+@Internal
+public class FileLock {
+private static final String TEMP_DIR =
System.getProperty("java.io.tmpdir");
+private final File file;
+private FileOutputStream outputStream;
+private java.nio.channels.FileLock lock;
+
+/**
+ * Initialize a FileLock using a file located at fullPath.
+ *
+ * @param fullPath The path of the locking file
+ */
+public FileLock(String fullPath) {
+Preconditions.checkNotNull(fullPath, "fullPath should not be null");
+Path path = Paths.get(fullPath);
+String normalizedFileName =
normalizeFileName(path.getFileName().toString());
+if (normalizedFileName.isEmpty()) {
+throw new IllegalArgumentException("There are no legal characters
in the file name");
+}
+this.file =
+path.getParent() == null
+? new File(normalizedFileName)
Review Comment:
Currently when the path.getParent() is null, the path of the File is a
relative path. It means the FileLock will write a file into the current user
directory according to the java doc of the `java.io.File`. If I use the
`NetUtils#getAvaliablePort` in the test, the file will be created in the Flink
source dir. For example, I add a test in the flink-connector-hive, the path of
the `FileLock` is
`/Users/shengkai/Work/flink/flink-connector-hive/orgapacheflinkutilNetUtils60481`
. The idea will automatically add the file into the change list, which is very
annoying.
So I think why we don't create the file in the `TEMP_DIR` if the parent is
null? I think it has two benefits:
1. It's much clean for the test that uses the `NetUtils#getAvaliablePort`.
2. It's better for different modules to check the port conflicts. For
example, module A and module B run tests in parallel and both wants to use the
same port. Currently the two module will create the file under their directoy
rather than in the same directory.
Any opinion are welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a diff in pull request #16108: [FLINK-22821][core] Stabilize NetUtils#getAvailablePort in order to avoid wrongly allocating any used ports
fsk119 commented on code in PR #16108:
URL: https://github.com/apache/flink/pull/16108#discussion_r909145606
##
flink-core/src/main/java/org/apache/flink/util/FileLock.java:
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.util;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/** A file lock used for avoiding race condition among multiple
threads/processes. */
+@Internal
+public class FileLock {
+private static final String TEMP_DIR =
System.getProperty("java.io.tmpdir");
+private final File file;
+private FileOutputStream outputStream;
+private java.nio.channels.FileLock lock;
+
+/**
+ * Initialize a FileLock using a file located at fullPath.
+ *
+ * @param fullPath The path of the locking file
+ */
+public FileLock(String fullPath) {
+Preconditions.checkNotNull(fullPath, "fullPath should not be null");
+Path path = Paths.get(fullPath);
+String normalizedFileName =
normalizeFileName(path.getFileName().toString());
+if (normalizedFileName.isEmpty()) {
+throw new IllegalArgumentException("There are no legal characters
in the file name");
+}
+this.file =
+path.getParent() == null
+? new File(normalizedFileName)
Review Comment:
Currently when the path.getParent() is null, the path of the File is a
relative path. It means the FileLock will write a file into the current user
directory according to the java doc of the `java.io.File`. If I use the
`NetUtils#getAvaliablePort` in the test, the file will be created in the Flink
source dir. For example, I add a test in the flink-connector-hive, the path of
the `FileLock` is
`/Users/shengkai/Work/flink/flink-connector-hive/orgapacheflinkutilNetUtils60481`
. The idea will automatically add the file into the change list, which is very
annoying.
So I think think why we don't create the file in the `TEMP_DIR` if the
parent is null? I think it has two benefits:
1. It's much clean for the test that uses the `NetUtils#getAvaliablePort`.
2. It's better for different modules to check the port conflicts. For
example, module A and module B run tests in parallel and both wants to use the
same port. Currently the two module will create the file under their directoy
rather than in the same directory.
Any opinion are welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 closed pull request #74: [FLINK-26828][example] add new examples sub module and create KMeans example
lindong28 closed pull request #74: [FLINK-26828][example] add new examples sub module and create KMeans example URL: https://github.com/apache/flink-ml/pull/74 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #74: [FLINK-26828][example] add new examples sub module and create KMeans example
lindong28 commented on PR #74: URL: https://github.com/apache/flink-ml/pull/74#issuecomment-1169469111 Thanks for the PR. We have added examples and the corresponding doc in https://github.com/apache/flink-ml/pull/119. I will close this PR as discussed offline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26721) PulsarSourceITCase.testSavepoint failed on azure pipeline
Title: Message Title Huang Xingbo commented on FLINK-26721 Re: PulsarSourceITCase.testSavepoint failed on azure pipeline https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37289=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-ml] lindong28 merged pull request #121: [FLINK-27715] Add pyflink examples
lindong28 merged PR #121: URL: https://github.com/apache/flink-ml/pull/121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #121: [FLINK-27715] Add pyflink examples
lindong28 commented on PR #121: URL: https://github.com/apache/flink-ml/pull/121#issuecomment-1169468374 Thanks for the update. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20098: [FLINK-28288][table][python] Support decode and encode in Table API
flinkbot commented on PR #20098: URL: https://github.com/apache/flink/pull/20098#issuecomment-1169465743 ## CI report: * aaf82b9952009cc3fc131cd86d0f0201a5986855 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Change By: ASF GitHub Bot Labels: pull-request-available Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] deadwind4 opened a new pull request, #20098: [FLINK-28288][table][python] Support decode and encode in Table API
deadwind4 opened a new pull request, #20098: URL: https://github.com/apache/flink/pull/20098 ## What is the purpose of the change Support decode and encode built-in functions in Table API. ## Brief change log - *Support decode and encode built-in functions in Table API.* - *Support decode and encode built-in functions in Python Table API.* ## Verifying this change This change added tests and can be verified as follows: *(example:)* - *Added test that validates that decode and encode in Table API work* - *Added test that validates that decode and encode in Python Table API work* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive)no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28289) Introduce Spark2 Reader for table store
Title: Message Title Jingsong Lee created an issue Flink / FLINK-28289 Introduce Spark2 Reader for table store Issue Type: Sub-task Assignee: Unassigned Components: Table Store Created: 29/Jun/22 02:20 Fix Versions: table-store-0.2.0 Priority: Major Reporter: Jingsong Lee Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-27907) Implement disk read and write logic for hybrid shuffle
Title: Message Title Weijie Guo updated an issue Flink / FLINK-27907 Implement disk read and write logic for hybrid shuffle Change By: Weijie Guo Implement disk read and write logic for hybrid shuffle , . In order to access the disk as sequentially as possible , Introduce HsMemoryDataSpiller, HsResultPartitionReadScheduler and HsSubpartitionFileReader. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-27907) Implement disk read and write logic for hybrid shuffle
Title: Message Title Weijie Guo updated an issue Flink / FLINK-27907 Implement disk read and write logic for hybrid shuffle Change By: Weijie Guo Implement disk read and write logic for hybrid shuffle, In order to access the disk as sequentially as possible Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-27907) Implement disk read and write logic for hybrid shuffle
Title: Message Title Weijie Guo updated an issue Flink / FLINK-27907 Implement disk read and write logic for hybrid shuffle Change By: Weijie Guo Summary: Abstract and implement Implement disk read and write logic for hybrid shuffle Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-27155) Reduce multiple reads to the same Changelog file in the same taskmanager during restore
Title: Message Title Feifan Wang commented on FLINK-27155 Re: Reduce multiple reads to the same Changelog file in the same taskmanager during restore Thanks Roman Khachatryan and Yuan Mei, I did have an implementation already, but it was based on 1.15.0. I noticed there is some change on master, and I also need modify the implementation as your comments. I will submit a pr after finished those. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title LuNing Wang updated an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Change By: LuNing Wang External issue ID: 28701 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title LuNing Wang updated an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Change By: LuNing Wang External issue ID: 28701 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28288) Support decode and encode built-in function in the Table API
Title: Message Title LuNing Wang created an issue Flink / FLINK-28288 Support decode and encode built-in function in the Table API Issue Type: Improvement Affects Versions: 1.15.0 Assignee: Unassigned Components: API / Python, Table SQL / API Created: 29/Jun/22 02:12 Fix Versions: 1.16.0 Priority: Major Reporter: LuNing Wang Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] HuangXingBo commented on a diff in pull request #19878: [FLINK-27584][python] Support non-keyed co-broadcast processing
HuangXingBo commented on code in PR #19878:
URL: https://github.com/apache/flink/pull/19878#discussion_r906952038
##
flink-python/src/main/java/org/apache/flink/streaming/api/transformations/python/PythonBroadcastStateTransformation.java:
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations.python;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.configuration.Configuration;
+import
org.apache.flink.streaming.api.functions.python.DataStreamPythonFunctionInfo;
+import
org.apache.flink.streaming.api.transformations.AbstractBroadcastStateTransformation;
+import org.apache.flink.streaming.api.utils.ByteArrayWrapper;
+import org.apache.flink.streaming.api.utils.ByteArrayWrapperSerializer;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * A {@link Transformation} representing a Python Co-Broadcast-Process
operation, which will be
+ * translated into different operations by {@link
+ *
org.apache.flink.streaming.runtime.translators.python.PythonBroadcastStateTransformationTranslator}.
+ */
+public class PythonBroadcastStateTransformation
+extends AbstractBroadcastStateTransformation {
+
+private final Configuration configuration;
+private final DataStreamPythonFunctionInfo dataStreamPythonFunctionInfo;
+
+public PythonBroadcastStateTransformation(
+String name,
+Configuration configuration,
+DataStreamPythonFunctionInfo dataStreamPythonFunctionInfo,
+Transformation regularInput,
+Transformation broadcastInput,
+List> broadcastStateDescriptors,
+TypeInformation outTypeInfo,
+int parallelism) {
+super(
+name,
+regularInput,
+broadcastInput,
+broadcastStateDescriptors,
+outTypeInfo,
+parallelism);
+this.configuration = configuration;
+this.dataStreamPythonFunctionInfo = dataStreamPythonFunctionInfo;
+updateManagedMemoryStateBackendUseCase(false);
Review Comment:
Could the step put in the PythonConfigUtil just like other python
transformations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #121: [FLINK-27715] Add pyflink examples
yunfengzhou-hub commented on code in PR #121:
URL: https://github.com/apache/flink-ml/pull/121#discussion_r909134520
##
flink-ml-python/pyflink/examples/ml/__init__.py:
##
@@ -15,3 +15,80 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+from py4j.java_gateway import JavaClass, get_java_class, JavaObject
+from pyflink.java_gateway import get_gateway
+from pyflink.util import java_utils
+from pyflink.util.java_utils import to_jarray, load_java_class
+
+
+def add_jars_to_context_class_loader(jar_urls):
+"""
+Add jars to Python gateway server for local compilation and local
execution (i.e. minicluster).
+There are many component in Flink which won't be added to classpath by
default. e.g. Kafka
+connector, JDBC connector, CSV format etc. This utility function can be
used to hot load the
+jars.
+
+:param jar_urls: The list of jar urls.
+"""
+gateway = get_gateway()
+# validate and normalize
+jar_urls = [gateway.jvm.java.net.URL(url) for url in jar_urls]
+context_classloader =
gateway.jvm.Thread.currentThread().getContextClassLoader()
+existing_urls = []
+class_loader_name = context_classloader.getClass().getName()
+if class_loader_name == "java.net.URLClassLoader":
+existing_urls = set([url.toString() for url in
context_classloader.getURLs()])
+if all([url.toString() in existing_urls for url in jar_urls]):
+# if urls all existed, no need to create new class loader.
+return
+
+URLClassLoaderClass = load_java_class("java.net.URLClassLoader")
+if is_instance_of(context_classloader, URLClassLoaderClass):
+if class_loader_name ==
"org.apache.flink.runtime.execution.librarycache." \
+
"FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader":
+ensureInner =
context_classloader.getClass().getDeclaredMethod("ensureInner", None)
+ensureInner.setAccessible(True)
+context_classloader = ensureInner.invoke(context_classloader, None)
+
+addURL = URLClassLoaderClass.getDeclaredMethod(
+"addURL",
+to_jarray(
+gateway.jvm.Class,
+[load_java_class("java.net.URL")]))
+addURL.setAccessible(True)
+
+for url in jar_urls:
+addURL.invoke(context_classloader,
to_jarray(get_gateway().jvm.Object, [url]))
+
+else:
+context_classloader = create_url_class_loader(jar_urls,
context_classloader)
+
gateway.jvm.Thread.currentThread().setContextClassLoader(context_classloader)
+
+
+def is_instance_of(java_object, java_class):
+gateway = get_gateway()
+if isinstance(java_class, str):
+param = java_class
+elif isinstance(java_class, JavaClass):
+param = get_java_class(java_class)
+elif isinstance(java_class, JavaObject):
+if not is_instance_of(java_class, gateway.jvm.Class):
+param = java_class.getClass()
+else:
+param = java_class
+else:
+raise TypeError(
+"java_class must be a string, a JavaClass, or a JavaObject")
+
+return
gateway.jvm.org.apache.flink.api.python.shaded.py4j.reflection.TypeUtil.isInstanceOf(
+param, java_object)
+
+
+def create_url_class_loader(urls, parent_class_loader):
+gateway = get_gateway()
+url_class_loader = gateway.jvm.java.net.URLClassLoader(
+to_jarray(gateway.jvm.java.net.URL, urls), parent_class_loader)
+return url_class_loader
+
+
+java_utils.add_jars_to_context_class_loader = add_jars_to_context_class_loader
Review Comment:
Yes. It is a walk-around to a bug related to FLINK-15635 and FLINK-28002.
I'll add a TODO to avoid overwriting `pyflink.util.java_utils` after this bug
is fixed and released.
And according to offline discussion, I'll try to reuse the
`add_jars_to_context_class_loader` method across `pyflink/ml/__init__.py` and
`pyflink/examples/ml/__init__.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #20003: [FLINK-28080][runtime] Introduce MutableURLClassLoader as parent class of FlinkUserClassLoader and SafetyNetWrapperClassLoader
lsyldliu commented on PR #20003: URL: https://github.com/apache/flink/pull/20003#issuecomment-1169449211 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #20057: [FLINK-28213][runtime] StreamExecutionEnvironment configure method support override pipeline.jars option
lsyldliu commented on PR #20057: URL: https://github.com/apache/flink/pull/20057#issuecomment-1169448529 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28172) Scatter dstl files into separate directories by job id
Title: Message Title Feifan Wang commented on FLINK-28172 Re: Scatter dstl files into separate directories by job id Thanks very much Roman Khachatryan. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] zoltar9264 commented on pull request #20093: [FLINK-28172][changelog] Scatter dstl files into separate directories…
zoltar9264 commented on PR #20093: URL: https://github.com/apache/flink/pull/20093#issuecomment-1169447489 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-9970) Add ASCII/CHR function for Table API
Title: Message Title Dian Fu closed an issue as Duplicate Closing this ticket as it's already supported in FLINK-28092. Flink / FLINK-9970 Add ASCII/CHR function for Table API Change By: Dian Fu Resolution: Duplicate Status: Open Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-28092) Support ASCII and CHR built-in function in the Table API
Title: Message Title Dian Fu updated an issue Flink / FLINK-28092 Support ASCII and CHR built-in function in the Table API Change By: Dian Fu Affects Version/s: 1.15.0 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] xintongsong commented on pull request #20031: [FLINK-27906][runtime] Introduce HsDataIndex.
xintongsong commented on PR #20031: URL: https://github.com/apache/flink/pull/20031#issuecomment-1169445372 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28092) Support ASCII and CHR built-in function in the Table API
Title: Message Title Dian Fu closed an issue as Fixed Merged to master via e7e120acf9258ed8ef3c416fed6ed59cc0a06855 Flink / FLINK-28092 Support ASCII and CHR built-in function in the Table API Change By: Dian Fu Assignee: LuNing Wang Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] dianfu closed pull request #19988: [FLINK-28092][table] Support ASCII and CHR built-in function in the Table API
dianfu closed pull request #19988: [FLINK-28092][table] Support ASCII and CHR built-in function in the Table API URL: https://github.com/apache/flink/pull/19988 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on pull request #19988: [FLINK-28092][table] Support ASCII and CHR built-in function in the Table API
dianfu commented on PR #19988: URL: https://github.com/apache/flink/pull/19988#issuecomment-1169444531 LGTM. Merging... > If we migrate all built-in functions to the new stack, we could create a new Jira ticket to target this. Agree. Let's do that in a separate ticket if we found it necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on pull request #19998: [FLINK-27822][docs-zh] Translate the doc of checkpoint/savepoint guar…
zoltar9264 commented on PR #19998: URL: https://github.com/apache/flink/pull/19998#issuecomment-1169443111 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r909107158
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractDeploymentObserver.java:
##
@@ -279,21 +279,35 @@ private void checkIfAlreadyUpgraded(FlinkDeployment
flinkDep, Context context) {
Long upgradeTargetGeneration =
ReconciliationUtils.getUpgradeTargetGeneration(flinkDep);
+Long currentSpecGeneration =
flinkDep.getMetadata().getGeneration();
+
if (deployedGeneration.equals(upgradeTargetGeneration)) {
-logger.info(
-"Last reconciled generation is already
deployed, setting reconciliation status to "
-+ ReconciliationState.DEPLOYED);
+logger.info("Pending upgrade is already deployed,
updating status.");
var firstDeploy =
status.getReconciliationStatus().getLastReconciledSpec() == null;
-var conf =
-firstDeploy
-? configManager.getDeployConfig(
-flinkDep.getMetadata(),
flinkDep.getSpec())
-:
configManager.getObserveConfig(flinkDep);
+if (firstDeploy) {
+
ReconciliationUtils.updateForSpecReconciliationSuccess(
+flinkDep,
+JobState.RUNNING,
+configManager.getDeployConfig(
+flinkDep.getMetadata(),
flinkDep.getSpec()));
+} else {
+
ReconciliationUtils.updateStatusForAlreadyUpgraded(flinkDep);
+}
+
+status.getJobStatus()
+
.setState(org.apache.flink.api.common.JobStatus.RECONCILING.name());
+
status.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.DEPLOYING);
+} else if
(deployedGeneration.equals(currentSpecGeneration)) {
Review Comment:
What about also record the updateTargetSpec before execute the upgrade or
submit. So that, we can always have the right base to compare. And it will not
changed by user. In the observer, we compare it with the deployedGeneration.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] tweise commented on pull request #281: [FLINK-26891] Record important deployment events
tweise commented on PR #281: URL: https://github.com/apache/flink-kubernetes-operator/pull/281#issuecomment-1169419765 > The PR looks pretty good! > > My only question whether we should trigger events before or after executing the actions using the Flink service. > > For example now we trigger a suspended event before trying to cancel the job. So it might happen that the event is there but the job is not suspended. Of course we could have the reverse situation if we changed the order. Just wanted to discuss this explicitly. > > What do you think @tweise ? It should be OK to have the event logged first. If the subsequent action fails, we have a trace that it was attempted and any subsequent error would yield anther event so that there should not be any confusion. I pulled the reason codes into `EventRecorder`. (I did not add them to the listener as the listener does not actually reference them and it would also be a bit unusual to have them there from the operator code caller perspective.) @gyfora @morhidi PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nyingping commented on pull request #20028: [hotfix][docs] Remove redundant "." from sample code
nyingping commented on PR #20028: URL: https://github.com/apache/flink/pull/20028#issuecomment-1169414641 @Myasuka @MartijnVisser Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r909093534
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractDeploymentObserver.java:
##
@@ -279,21 +279,35 @@ private void checkIfAlreadyUpgraded(FlinkDeployment
flinkDep, Context context) {
Long upgradeTargetGeneration =
ReconciliationUtils.getUpgradeTargetGeneration(flinkDep);
+Long currentSpecGeneration =
flinkDep.getMetadata().getGeneration();
+
if (deployedGeneration.equals(upgradeTargetGeneration)) {
-logger.info(
-"Last reconciled generation is already
deployed, setting reconciliation status to "
-+ ReconciliationState.DEPLOYED);
+logger.info("Pending upgrade is already deployed,
updating status.");
var firstDeploy =
status.getReconciliationStatus().getLastReconciledSpec() == null;
-var conf =
-firstDeploy
-? configManager.getDeployConfig(
-flinkDep.getMetadata(),
flinkDep.getSpec())
-:
configManager.getObserveConfig(flinkDep);
+if (firstDeploy) {
+
ReconciliationUtils.updateForSpecReconciliationSuccess(
+flinkDep,
+JobState.RUNNING,
+configManager.getDeployConfig(
+flinkDep.getMetadata(),
flinkDep.getSpec()));
+} else {
+
ReconciliationUtils.updateStatusForAlreadyUpgraded(flinkDep);
+}
+
+status.getJobStatus()
+
.setState(org.apache.flink.api.common.JobStatus.RECONCILING.name());
+
status.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.DEPLOYING);
+} else if
(deployedGeneration.equals(currentSpecGeneration)) {
Review Comment:
I think we have a problem here, we can only do the thing when the
deployedGeneration equal to the `currentSpecGeneration` or the
`upgradeTargetGeneration`. If they are not equal, then we do not know how to
bring the job do the desired version. It's mainly the concern you mentioned in
the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r908605561
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/ReconciliationUtils.java:
##
@@ -352,22 +382,33 @@ ErrorStatusUpdateControl toErrorStatusUpdateControl(
return ErrorStatusUpdateControl.noStatusUpdate();
}
-public static Long getUpgradeTargetGeneration(FlinkDeployment deployment) {
+/**
+ * Get spec generation for the current in progress upgrade.
+ *
+ * @param resource Flink resource.
+ * @return The spec generation for the upgrade.
+ */
+public static Long getUpgradeTargetGeneration(AbstractFlinkResource
resource) {
var lastSpecWithMeta =
-deployment
-.getStatus()
+resource.getStatus()
.getReconciliationStatus()
.deserializeLastReconciledSpecWithMeta();
if (lastSpecWithMeta == null || lastSpecWithMeta.f1 == null) {
// For first deployments and when migrating from before this
feature simply return
// current generation
-return deployment.getMetadata().getGeneration();
+return resource.getMetadata().getGeneration();
}
return
lastSpecWithMeta.f1.get("metadata").get("generation").asLong(-1L);
Review Comment:
Now the specWithMeta's meta is a `ObjectNode`, which will make it no schema
and difficult to use, do you think we need to improve it here ? Like directly
return a Bean object ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #283: [FLINK-28228] Never skip generations when observing already upgraded deployment
Aitozi commented on code in PR #283:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/283#discussion_r908605561
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/ReconciliationUtils.java:
##
@@ -352,22 +382,33 @@ ErrorStatusUpdateControl toErrorStatusUpdateControl(
return ErrorStatusUpdateControl.noStatusUpdate();
}
-public static Long getUpgradeTargetGeneration(FlinkDeployment deployment) {
+/**
+ * Get spec generation for the current in progress upgrade.
+ *
+ * @param resource Flink resource.
+ * @return The spec generation for the upgrade.
+ */
+public static Long getUpgradeTargetGeneration(AbstractFlinkResource
resource) {
var lastSpecWithMeta =
-deployment
-.getStatus()
+resource.getStatus()
.getReconciliationStatus()
.deserializeLastReconciledSpecWithMeta();
if (lastSpecWithMeta == null || lastSpecWithMeta.f1 == null) {
// For first deployments and when migrating from before this
feature simply return
// current generation
-return deployment.getMetadata().getGeneration();
+return resource.getMetadata().getGeneration();
}
return
lastSpecWithMeta.f1.get("metadata").get("generation").asLong(-1L);
Review Comment:
Now the specWithMeta's meta is a `ObjectNode`, which will make it no schema
and difficult to use, do you think we need to improve it here ?
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/deployment/AbstractDeploymentObserver.java:
##
@@ -279,21 +279,35 @@ private void checkIfAlreadyUpgraded(FlinkDeployment
flinkDep, Context context) {
Long upgradeTargetGeneration =
ReconciliationUtils.getUpgradeTargetGeneration(flinkDep);
+Long currentSpecGeneration =
flinkDep.getMetadata().getGeneration();
+
if (deployedGeneration.equals(upgradeTargetGeneration)) {
-logger.info(
-"Last reconciled generation is already
deployed, setting reconciliation status to "
-+ ReconciliationState.DEPLOYED);
+logger.info("Pending upgrade is already deployed,
updating status.");
var firstDeploy =
status.getReconciliationStatus().getLastReconciledSpec() == null;
-var conf =
-firstDeploy
-? configManager.getDeployConfig(
-flinkDep.getMetadata(),
flinkDep.getSpec())
-:
configManager.getObserveConfig(flinkDep);
+if (firstDeploy) {
+
ReconciliationUtils.updateForSpecReconciliationSuccess(
+flinkDep,
+JobState.RUNNING,
+configManager.getDeployConfig(
+flinkDep.getMetadata(),
flinkDep.getSpec()));
+} else {
+
ReconciliationUtils.updateStatusForAlreadyUpgraded(flinkDep);
+}
+
+status.getJobStatus()
+
.setState(org.apache.flink.api.common.JobStatus.RECONCILING.name());
+
status.setJobManagerDeploymentStatus(JobManagerDeploymentStatus.DEPLOYING);
+} else if
(deployedGeneration.equals(currentSpecGeneration)) {
Review Comment:
I think we have a problem here, we can only do the thing when the
deployedGeneration equal to the `currentSpecGeneration` or the
`upgradeTargetGeneration`. If they are not equal, then we do not know how to
bring the job do the desired version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-28287) Should TaskManagerRunner need a ShutdownHook
Title: Message Title JieFang.He updated an issue Flink / FLINK-28287 Should TaskManagerRunner need a ShutdownHook Change By: JieFang.He TaskManagerRunner has a close method，but did not call when it stop.Some resources in TaskManagerRunner come with ShutdownHook, but some resources do not, such as rpcSystem, which causes the temporary file flink-rpc-akka_*.jar to not be deleted when stop.Should TaskManagerRunner need a ShutdownHook to call the close method to release all resources Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (FLINK-28287) Should TaskManagerRunner need a ShutdownHook
Title: Message Title JieFang.He created an issue Flink / FLINK-28287 Should TaskManagerRunner need a ShutdownHook Issue Type: Improvement Affects Versions: 1.14.0 Assignee: Unassigned Created: 29/Jun/22 00:06 Priority: Major Reporter: JieFang.He TaskManagerRunner has a close method，but did not call when it stop. Some resources in TaskManagerRunner come with ShutdownHook, but some resources do not, such as rpcSystem, which causes the temporary file flink-rpc-akka_*.jar to not be deleted when stop. Should TaskManagerRunner need a ShutdownHook to call the close method Add Comment
[GitHub] [flink] syhily commented on pull request #19473: [FLINK-27199][Connector/Pulsar] Bump pulsar to 2.10.0
syhily commented on PR #19473: URL: https://github.com/apache/flink/pull/19473#issuecomment-1169400631 > Thanks for the patch @syhily. I notice that this PR touches a lot of components including Pulsar client version, docs even default value of sink options. What about breaking these changes into different commits or PRs? I'm afraid squashing them into one commit is way too ambiguous. OK, I'll split this PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27155) Reduce multiple reads to the same Changelog file in the same taskmanager during restore
Title: Message Title Roman Khachatryan commented on FLINK-27155 Re: Reduce multiple reads to the same Changelog file in the same taskmanager during restore Thanks for clarifying! I moved the comments and responses to the document. It looks like we're close to a consensus now (I've discussed the design with Yuan Mei, though she'll probably take a closer look). So if you have a PR alredy, we could start reviewing it. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] rkhachatryan commented on a diff in pull request #19679: [FLINK-23143][state/changelog] Support state migration for ChangelogS…
rkhachatryan commented on code in PR #19679:
URL: https://github.com/apache/flink/pull/19679#discussion_r909048188
##
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java:
##
@@ -477,6 +478,50 @@ KeyGroupedInternalPriorityQueue create(
}
}
+@Override
+public S upgradeKeyedState(
+TypeSerializer namespaceSerializer, StateDescriptor
stateDescriptor)
+throws Exception {
+StateFactory stateFactory = getStateFactory(stateDescriptor);
+Tuple2> registerResult =
+tryRegisterKvStateInformation(stateDescriptor,
namespaceSerializer, noTransform());
+
Preconditions.checkState(kvStateInformation.containsKey(stateDescriptor.getName()));
+kvStateInformation.computeIfPresent(
+stateDescriptor.getName(),
+(stateName, kvStateInfo) ->
+new RocksDbKvStateInfo(
+kvStateInfo.columnFamilyHandle,
+new RegisteredKeyValueStateBackendMetaInfo<>(
+kvStateInfo.metaInfo.snapshot(;
+return stateFactory.createState(
+stateDescriptor, registerResult,
RocksDBKeyedStateBackend.this);
Review Comment:
I think we are all on the same page now and it makes sense to document this
design somehwere, along with other aspects of this feature - for future
reference (as well as to make sure we're indeed on the same page).
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-16419) Avoid to recommit transactions which are known committed successfully to Kafka upon recovery
Title: Message Title Yordan Pavlov commented on FLINK-16419 Re: Avoid to recommit transactions which are known committed successfully to Kafka upon recovery Martijn Visser As advised I have switched to KafkaSink and Flink 1.14.2. However I am still seeing the following error on recovering from a checkpoint: switched from INITIALIZING to FAILED with failure cause: org.apache.kafka.common.KafkaException: Unhandled error in EndTxnResponse: The producer attempted to use a producer id which is not currently assigned to its transactional id. at org.apache.kafka.clients.producer.internals.TransactionManager$EndTxnHandler.handleResponse(TransactionManager.java:1362) at org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1074) at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:569) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:425) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:311) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244) at java.base/java.lang.Thread.run(Unknown Source) Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] flinkbot commented on pull request #20097: [FLINK-28284][Connectors/Jdbc] Add JdbcSink with new format
flinkbot commented on PR #20097: URL: https://github.com/apache/flink/pull/20097#issuecomment-1169146748 ## CI report: * 8bcbd05a6b7eeb2000685b729f860eae3c1b50aa UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18096) Generated avro formats should support user specified name and namespace
Title: Message Title Robin Neufeld commented on FLINK-18096 Re: Generated avro formats should support user specified name and namespace Without being able to customize the namespace and record name that Flink generates, it's very difficult to use this in a large organization. At my work we have a schema registry that is used by many teams, and everyone pre-registers their schemas. There are conventions about what the namespace and record name should be. We also enforce backwards and forwards compatibility to schema changes. When using this Flink connector/format, it tries to edit the name/namespace on the schema, which fails since it counts as backwards incompatible. It seems like a small detail, but it is a major obstacle for using this implementation. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] eskabetxe commented on pull request #20097: [FLINK-28284] Add JdbcSink with new format
eskabetxe commented on PR #20097: URL: https://github.com/apache/flink/pull/20097#issuecomment-1169141734 @leonardBang @MartijnVisser could you check this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28284) Add JdbcSink based on new Sink (sink2)
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28284 Add JdbcSink based on new Sink (sink2) Change By: ASF GitHub Bot Labels: pull-request-available Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] eskabetxe opened a new pull request, #20097: [FLINK-28284] Add JdbcSink with new format
eskabetxe opened a new pull request, #20097: URL: https://github.com/apache/flink/pull/20097 ## What is the purpose of the change Add a JdbcSink with new format (sink2) ## Brief change log - *JdbcSink* the new sink - *JdbcSinkWriter* the writer used by the new sink - *JdbcQueryStatement* the query and preparestatement that will be used - *JdbcWriterStatement* the writer statement that will write to jdbc - *SimpleJdbcWriterStatement* a simple implementation of JdbcWriterStatement ## Verifying this change This change added tests and can be verified as follows: - JdbcITCase.testInsertWithSinkTo - JdbcITCase.testObjectReuseWithSinkTo ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: ( no ) - The runtime per-record code paths (performance sensitive): (no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes ) - If yes, how is the feature documented? ( JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28268) Elasticsearch6SinkE2ECase failed with no space left on device
Title: Message Title Martijn Visser closed an issue as Not A Problem Flink / FLINK-28268 Elasticsearch6SinkE2ECase failed with no space left on device Change By: Martijn Visser Resolution: Not A Problem Status: In Progress Closed Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Closed] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing
Title: Message Title Martijn Visser closed an issue as Fixed Merged in: master: 5d94fe6c128f2fa9e3531ab8b169e8a2e21bc391 release-1.15: 7557a282314b29808ea5552aaf69134a7ebf9d08 release-1.14: 852713c52e1c4ce8d27905f5d3dedaed2015d4eb Flink / FLINK-28263 TPC-DS Bash e2e tests don't clean-up after completing Change By: Martijn Visser Fix Version/s: 1.16.0 Fix Version/s: 1.15.2 Fix Version/s: 1.14.6 Resolution: Fixed Status: In Progress Closed Add Comment This message was sent by Atlassian Jira
[GitHub] [flink] MartijnVisser merged pull request #20096: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser merged PR #20096: URL: https://github.com/apache/flink/pull/20096 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #20095: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser merged PR #20095: URL: https://github.com/apache/flink/pull/20095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser opened a new pull request, #20096: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser opened a new pull request, #20096: URL: https://github.com/apache/flink/pull/20096 Unchanged backport of https://github.com/apache/flink/pull/20083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser opened a new pull request, #20095: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser opened a new pull request, #20095: URL: https://github.com/apache/flink/pull/20095 Unchanged backport of https://github.com/apache/flink/pull/20083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #20083: [FLINK-28263][TPCDS][Tests] Clean-up generated data folder by TPCDS test
MartijnVisser merged PR #20083: URL: https://github.com/apache/flink/pull/20083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-28206) EOFException on Checkpoint Recovery
Title: Message Title
uharaqo edited a comment on FLINK-28206
Re: EOFException on Checkpoint Recovery
Thanks [~yunta] . I was able to can read the metadata file (checkpoint: 58130) which caused the EOF error: {noformat}result = {CheckpointMetadata@955} "Checkpoint Metadata" checkpointId = 58130 operatorStates = {ArrayList@957} size = 11 0 = {OperatorState@960} "OperatorState(operatorID: 5323682bad0857bbc5dc34e8a0198f36, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 3607732189)" 1 = {OperatorState@961} "OperatorState(operatorID: b0d7d72b41ddffa0e746c98bfade0740, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 11901811)" 2 = {OperatorState@962} "OperatorState(operatorID: 799506d4ab267dabfd7e5e151c32b771, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 0, total size (bytes): 0)" 3 = {OperatorState@963} "OperatorState(operatorID: bc764cd8ddf7a0cff126f51c16239658, parallelism: 64, maxParallelism: 128, coordinatorState: 528 bytes, sub task states: 64, total size (bytes): 16400)" 4 = {OperatorState@964} "OperatorState(operatorID: 6ac701675b3507fc60576d5eb1d2a02a, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 0, total size (bytes): 0)" 5 = {OperatorState@965} "OperatorState(operatorID: 4f12c5e521389e5e6c62d38d49148502, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 0)" 6 = {OperatorState@966} "OperatorState(operatorID: 2963852293169ba90d9d1e7d6308db5c, parallelism: 64, maxParallelism: 128, coordinatorState: 144 bytes, sub task states: 64, total size (bytes): 15296)" 7 = {OperatorState@967} "OperatorState(operatorID: 30ddfdac31a80b264a11886ca4e04ea8, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 15868786)" 8 = {OperatorState@968} "OperatorState(operatorID: ee7af715df0b4d5489c7e945244bab03, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 646402101)" 9 = {OperatorState@969} "OperatorState(operatorID: feca28aff5a3958840bee985ee7de4d3, parallelism: 64, maxParallelism: 128, coordinatorState: 128 bytes, sub task states: 64, total size (bytes): 15264)" 10 = {OperatorState@970} "OperatorState(operatorID: 99f06a2f6abbb4a5ff119455ad01a44b, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 0)" masterStates = {Collections$EmptyList@958} size = 0{noformat}We use Kubernetes and flink-s3-fs-presto. The network was stable at the time and we were able to restart the job by ignoring the savepoint and checkpoint. Do you think the EOF error was caused by the Presto plugin?
Add Comment
[GitHub] [flink] MartijnVisser commented on pull request #20083: [FLINK-28263][TPCDS][Tests] Clean-up generated folders by TPCDS test
MartijnVisser commented on PR #20083: URL: https://github.com/apache/flink/pull/20083#issuecomment-1169074724 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28269) Kubernetes test failed with permission denied
Title: Message Title Martijn Visser commented on FLINK-28269 Re: Kubernetes test failed with permission denied Yang Wang Robert Metzger I've been doing some investigation on this but I do have one question: Do you know if it was it a deliberate decision to use the none driver over the other existing ones? Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] zentol commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog
zentol commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r908766389
##
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##
@@ -685,13 +687,27 @@ public CompletableFuture submitTask(
jobManagerConnection.getClassLoaderHandle();
PartitionProducerStateChecker partitionStateChecker =
jobManagerConnection.getPartitionStateChecker();
+TernaryBoolean changelogEnabled = TernaryBoolean.FALSE;
+try {
+changelogEnabled =
+InstantiationUtil.readObjectFromConfig(
+taskInformation.getTaskConfiguration(),
+"enablechangelog",
+getClass().getClassLoader());
Review Comment:
I would put it into `flink-core` personally; not sure if there is a huge
benefit in spreading config options around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing
Title: Message Title ASF GitHub Bot updated an issue Flink / FLINK-28263 TPC-DS Bash e2e tests don't clean-up after completing Change By: ASF GitHub Bot Labels: pull-request-available test-stability Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink] MartijnVisser commented on pull request #20083: [FLINK-28263][TPCDS][Tests] Clean-up generated folders by TPCDS test
MartijnVisser commented on PR #20083: URL: https://github.com/apache/flink/pull/20083#issuecomment-1169016409 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28216) Hadoop S3FileSystemFactory does not honor fs.s3.impl
Title: Message Title Martijn Visser commented on FLINK-28216 Re: Hadoop S3FileSystemFactory does not honor fs.s3.impl The S3 Hadoop plugin is deliberately a small footprint plugin. I wouldn't be in favour of expanding it for EMRFS. I think it would be a better idea to create a separate plugin/implementation, specifically for EMRFS only. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-27852) Contribute k8s operator to OperatorHub
Title: Message Title Márton Balassi commented on FLINK-27852 Re: Contribute k8s operator to OperatorHub Thanks, James Busche and Ted Chang assigned the ticket accordingly. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-28206) EOFException on Checkpoint Recovery
Title: Message Title
uharaqo commented on FLINK-28206
Re: EOFException on Checkpoint Recovery
Thanks Yun Tang . I was able to read the metadata file (checkpoint: 58130) which caused the EOF error:
result = {CheckpointMetadata@955} "Checkpoint Metadata"
checkpointId = 58130
operatorStates = {ArrayList@957} size = 11
0 = {OperatorState@960} "OperatorState(operatorID: 5323682bad0857bbc5dc34e8a0198f36, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 3607732189)"
1 = {OperatorState@961} "OperatorState(operatorID: b0d7d72b41ddffa0e746c98bfade0740, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 11901811)"
2 = {OperatorState@962} "OperatorState(operatorID: 799506d4ab267dabfd7e5e151c32b771, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 0, total size (bytes): 0)"
3 = {OperatorState@963} "OperatorState(operatorID: bc764cd8ddf7a0cff126f51c16239658, parallelism: 64, maxParallelism: 128, coordinatorState: 528 bytes, sub task states: 64, total size (bytes): 16400)"
4 = {OperatorState@964} "OperatorState(operatorID: 6ac701675b3507fc60576d5eb1d2a02a, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 0, total size (bytes): 0)"
5 = {OperatorState@965} "OperatorState(operatorID: 4f12c5e521389e5e6c62d38d49148502, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 0)"
6 = {OperatorState@966} "OperatorState(operatorID: 2963852293169ba90d9d1e7d6308db5c, parallelism: 64, maxParallelism: 128, coordinatorState: 144 bytes, sub task states: 64, total size (bytes): 15296)"
7 = {OperatorState@967} "OperatorState(operatorID: 30ddfdac31a80b264a11886ca4e04ea8, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 15868786)"
8 = {OperatorState@968} "OperatorState(operatorID: ee7af715df0b4d5489c7e945244bab03, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 646402101)"
9 = {OperatorState@969} "OperatorState(operatorID: feca28aff5a3958840bee985ee7de4d3, parallelism: 64, maxParallelism: 128, coordinatorState: 128 bytes, sub task states: 64, total size (bytes): 15264)"
10 = {OperatorState@970} "OperatorState(operatorID: 99f06a2f6abbb4a5ff119455ad01a44b, parallelism: 64, maxParallelism: 128, coordinatorState: (none), sub task states: 64, total size (bytes): 0)"
masterStates = {Collections$EmptyList@958} size = 0
We use Kubernetes and flink-s3-fs-presto. The network was stable at the time and we were able to restart the job by ignoring the savepoint and checkpoint. Do you think the EOF error was caused by the Presto plugin?
[jira] [Assigned] (FLINK-27852) Contribute k8s operator to OperatorHub
Title: Message Title Márton Balassi assigned an issue to James Busche Flink / FLINK-27852 Contribute k8s operator to OperatorHub Change By: Márton Balassi Assignee: James Busche Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
Title: Message Title Sebastian Mattheis commented on FLINK-24932 Re: Frocksdb cannot run on Apple M1 Yun Tang Also, please have a look into the commit on my current branch if something is suspicious to you: https://github.com/smattheis/frocksdb/commit/541463f0fea489a05d6c2ea724b0b5acc566c4ec Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
Title: Message Title
Sebastian Mattheis commented on FLINK-24932
Re: Frocksdb cannot run on Apple M1
Yun Tang I have run flink-benchmarks locally before and I think bisect'ing a regression of <10% on a Desktop-based machine is like chasing a Heisenbug and would require me to do nothing on that machine for hours if not days given the compile time of frocksdb. That is practically impossible. I could use a cloud machine but I had really bad experiences doing sensitive performance measurements on cloud machines ... ("noisy neighbors") So not sure about that. So my questions:
Do you know any performance benchmarks from RocksDB development/community that could give some evidence that there is no regression from 6.20.3 to 6.29.5 or, opposite case, that there is some regression that confirms the observation in the flink-benchmarks?
Is the observed performance regression a blocker to have it in 1.16 release?
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Assigned] (FLINK-27852) Contribute k8s operator to OperatorHub
Title: Message Title Márton Balassi assigned an issue to Unassigned Flink / FLINK-27852 Contribute k8s operator to OperatorHub Change By: Márton Balassi Assignee: Márton Balassi Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[GitHub] [flink-kubernetes-operator] mbalassi merged pull request #284: [tooling] Use 7 letter hash in release scripts
mbalassi merged PR #284: URL: https://github.com/apache/flink-kubernetes-operator/pull/284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
