[jira] [Created] (FLINK-28603) Flink runtime.rpc.RpcServiceUtils code style
wuqingzhi created FLINK-28603: - Summary: Flink runtime.rpc.RpcServiceUtils code style Key: FLINK-28603 URL: https://issues.apache.org/jira/browse/FLINK-28603 Project: Flink Issue Type: Improvement Components: Runtime / RPC Reporter: wuqingzhi Hello, I found a code style problem in , which is located in RpcServiceUtils, where nextNameOffset should be capitalized and separated by an underscore. eg: private static final AtomicLong nextNameOffset = new AtomicLong(0L); -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28121) Translate "Extension Points" and "Full Stack Example" in "User-defined Sources & Sinks" page
[ https://issues.apache.org/jira/browse/FLINK-28121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568334#comment-17568334 ] hunter commented on FLINK-28121: [~yckkcy] Hi,Chengkai,I have completed the translation of the above part, you can see this pr (https://github.com/apache/flink/pull/20283), looking forward to your joining this pr > Translate "Extension Points" and "Full Stack Example" in "User-defined > Sources & Sinks" page > - > > Key: FLINK-28121 > URL: https://issues.apache.org/jira/browse/FLINK-28121 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Reporter: Chengkai Yang >Assignee: Chengkai Yang >Priority: Minor > > The links are > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/#extension-points > and > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/#full-stack-example -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] hunter-cloud09 commented on pull request #20283: [FLINK-28122][docs-zh] Translate "User-defined Sources & Sinks" page into Chinese
hunter-cloud09 commented on PR #20283: URL: https://github.com/apache/flink/pull/20283#issuecomment-1188620526 @PatrickRen Hi，Patrick，I just saw that FLINK-28121 has already translated the following content -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #132: [FLINK-28571]Add Chi-squared test as Transformer to ml.feature
yunfengzhou-hub commented on code in PR #132:
URL: https://github.com/apache/flink-ml/pull/132#discussion_r924035970
##
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/ChiSqTestTransformerTest.java:
##
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.ml.feature.chisqtest.ChiSqTestTransformer;
+import org.apache.flink.ml.util.TestUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import
org.apache.flink.streaming.api.environment.ExecutionCheckpointingOptions;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+/** Tests {@link ChiSqTestTransformerTest}. */
+public class ChiSqTestTransformerTest extends AbstractTestBase {
+@Rule public final TemporaryFolder tempFolder = new TemporaryFolder();
+private StreamTableEnvironment tEnv;
+private Table inputTable;
+
+private final List samples =
+Arrays.asList(
+Row.of(0., 5, 1.),
+Row.of(2., 6, 2.),
+Row.of(1., 7, 2.),
+Row.of(1., 5, 4.),
+Row.of(0., 5, 1.),
+Row.of(2., 6, 2.),
+Row.of(1., 7, 2.),
+Row.of(1., 5, 4.),
+Row.of(2., 5, 1.),
+Row.of(0., 5, 2.),
+Row.of(0., 5, 2.),
+Row.of(1., 9, 4.),
+Row.of(1., 9, 3.));
Review Comment:
We would expect that the functionality of Flink ML algorithms could be
aligned with existing practices. Have you referenced the function of any other
machine learning libraries?
If not, could you please reference the ChisqTest in
[Alink](https://github.com/alibaba/Alink)? For example, Alink supports labels
to be of integer type, while the current implementation in this PR only
supports doubles.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/chisqtest/ChiSqTestTransformer.java:
##
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.chisqtest;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.functions.RichMapPartitionFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import
[jira] [Created] (FLINK-28602) Changelog cannot close stream normally while enabling compression
Hangxiang Yu created FLINK-28602:
Summary: Changelog cannot close stream normally while enabling
compression
Key: FLINK-28602
URL: https://issues.apache.org/jira/browse/FLINK-28602
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.15.1, 1.16.0
Reporter: Hangxiang Yu
Fix For: 1.16.0, 1.15.2
While enabling compression, Changelog part will wrap output stream using
StreamCompressionDecorator#decorateWithCompression.
As the comment said, "IMPORTANT: For streams returned by this method, \{@link
OutputStream#close()} is not propagated to the inner stream. The inner stream
must be closed separately.".
But StateChangeFsUploader will not close inner stream if wrapped stream has
been closed.
So the upload will not complete when enabling compression even if it returns
success.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28307) FLIP-241: Completed Jobs Information Enhancement
[ https://issues.apache.org/jira/browse/FLINK-28307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-28307. Release Note: We have enhanced the experiences of viewing completed jobs’ information in this release. - JobManager / HistoryServer WebUI now provides detailed execution time metrics, including durations tasks spend in each execution state and the accumulated busy / idle / back-pressured time during running. - JobManager / HistoryServer WebUI now provides aggregation of major SubTask metrics, grouped by Task or TaskManager. - JobManager / HistoryServer WebUI now provides more environmental information, including environment variables, JVM options and classpath. - HistoryServer now supports browsing logs from external log archiving services. For more details: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/advanced/historyserver#log-integration Resolution: Done > FLIP-241: Completed Jobs Information Enhancement > > > Key: FLINK-28307 > URL: https://issues.apache.org/jira/browse/FLINK-28307 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST, Runtime / Web Frontend >Reporter: Yangze Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > Streaming and Batch users have different interests in probing a job. While > streaming users mainly care about the instant status of a running job (tps, > delay, backpressure, etc.), batch users care more about the overall job > status during the entire execution (queueing / execution time, total data > amount, etc.). > As Flink grows into a unified streaming & batch processor and is adopted by > more and more batch users, the experiences in inspecting completed jobs has > become more important than ever. > We compared Flink with other popular batch processors, and spotted several > potential improvements. Most of these changes involves WebUI & REST API > changes, which should be discussed and voted on as FLIPs. However, creating > separated FLIPs for each of the improvement might be overkill, because > changes needed by each improvement are quite small. Thus, we include all > these potential improvements in this one FLIP. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28555) Update history server documentation
[ https://issues.apache.org/jira/browse/FLINK-28555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-28555. Fix Version/s: 1.16.0 Resolution: Done master (1.16): 0394a01ebb4f1c6b3c3534acc58dd2467c8196e1 > Update history server documentation > --- > > Key: FLINK-28555 > URL: https://issues.apache.org/jira/browse/FLINK-28555 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Fix For: 1.16.0 > > > Update the history server documentation, w.r.t the log-viewing integration > and the supported rest apis. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] xintongsong closed pull request #20297: [FLINK-28307][doc] Update history server docs w.r.t. FLIP-241.
xintongsong closed pull request #20297: [FLINK-28307][doc] Update history server docs w.r.t. FLIP-241. URL: https://github.com/apache/flink/pull/20297 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fredia commented on a diff in pull request #20217: [FLINK-27693][state] Support local recovery for non-materialized part
fredia commented on code in PR #20217:
URL: https://github.com/apache/flink/pull/20217#discussion_r924060396
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/LocalStateRegistry.java:
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.concurrent.Executor;
+import java.util.concurrent.RejectedExecutionException;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** This registry manages handles which is written for local recovery. */
+public class LocalStateRegistry {
Review Comment:
Nice suggestion, another tick for monitoring the space taken by the
changelog could be created later, but we may not have enough time to finish it
in v1.16(Changelog V2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-28499) resource leak when job failed with unknown status In Application Mode
[ https://issues.apache.org/jira/browse/FLINK-28499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568322#comment-17568322 ] Yang Wang commented on FLINK-28499: --- Maybe we could improve this by deleting the init error pods when recovering after JobManager failover. > resource leak when job failed with unknown status In Application Mode > - > > Key: FLINK-28499 > URL: https://issues.apache.org/jira/browse/FLINK-28499 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.13.1 >Reporter: lihe ma >Priority: Major > Attachments: cluster-pod-error.png > > > I found a job restarted for thousands of times, and jobmanager tried to > create a new taskmanager pod every time. The jobmanager restarted because > submitted with duplicate job id[1] (we preset the jobId rather than > generate), but I hadn't save the logs unfortunately. > this job requires one taskmanager pod in normal circumstances, but thousands > of pods were leaked finally. you can find the screenshot in the attachment. > > In application mode, cluster resources will be released when job finished in > succeeded, failed or canceled status[2][3] . When some exception happen, the > job may be terminated in unknown status[4] . > In this case, the job exited with unknown status , without releasing > taskmanager pods. So is it reasonable to not release taskmanager when job > exited in unknown status ? > > > one line in original logs: > 2022-07-01 09:45:40,712 [main] INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Terminating cluster > entrypoint process KubernetesApplicationClusterEntrypoint with exit code 1445. > > [1] > [https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L452] > [2] > [https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L90-L91] > [3] > [https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/application/ApplicationDispatcherBootstrap.java#L175] > [4] > [https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ApplicationStatus.java#L39] > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] fredia commented on a diff in pull request #20217: [FLINK-27693][state] Support local recovery for non-materialized part
fredia commented on code in PR #20217:
URL: https://github.com/apache/flink/pull/20217#discussion_r924056241
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements
ChangelogStateBackendHandle {
+private static final long serialVersionUID = 1L;
+private static final Logger LOG =
+LoggerFactory.getLogger(ChangelogStateBackendLocalHandle.class);
+private final List localMaterialized;
+private final List localNonMaterialized;
+private final ChangelogStateBackendHandleImpl remoteHandle;
+
+public ChangelogStateBackendLocalHandle(
+List localMaterialized,
+List localNonMaterialized,
+ChangelogStateBackendHandleImpl remoteHandle) {
+this.localMaterialized = localMaterialized;
+this.localNonMaterialized = localNonMaterialized;
+this.remoteHandle = remoteHandle;
+}
+
+@Override
+public List getMaterializedStateHandles() {
+return localMaterialized;
+}
+
+@Override
+public List getNonMaterializedStateHandles() {
+return localNonMaterialized;
+}
+
+@Override
+public long getMaterializationID() {
+return remoteHandle.getMaterializationID();
+}
+
+@Override
+public ChangelogStateBackendHandle rebound(long checkpointId) {
+return remoteHandle.rebound(checkpointId);
+}
+
+public List getRemoteMaterializedStateHandles() {
+return remoteHandle.getMaterializedStateHandles();
+}
+
+public List getRemoteNonMaterializedStateHandles() {
+return remoteHandle.getNonMaterializedStateHandles();
+}
+
+@Override
+public long getCheckpointId() {
+return remoteHandle.getCheckpointId();
+}
+
+@Override
+public void registerSharedStates(SharedStateRegistry stateRegistry, long
checkpointID) {
+remoteHandle.registerSharedStates(stateRegistry, checkpointID);
+}
+
+@Override
+public long getCheckpointedSize() {
+return remoteHandle.getCheckpointedSize();
+}
+
+@Override
+public KeyGroupRange getKeyGroupRange() {
+return remoteHandle.getKeyGroupRange();
+}
+
+@Nullable
+@Override
+public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+throw new UnsupportedOperationException(
+"This is a local state handle for the TM side only.");
+}
+
+@Override
+public StateHandleID getStateHandleId() {
+return remoteHandle.getStateHandleId();
+}
+
+@Override
+public void discardState() throws Exception {}
Review Comment:
> 1. (Changelog)TaskLocalStateStore calls this no-op method
Yes, `TaskLocalStateStore` call no-op method, but `TaskLocalStateStore`
would delete the whole chk folder(`changelog_chk_` and `chk_`) when confirm
checkpoint.
> 4. As opposed to "normal" local state, local changelog state is not
discarded on recovery (because of 1); but it will be discarded on the 1st
checkpoint confirmation
Like "normal" local state, local changelog state would **be discarded** on
recovery, it would be delete when `deleteDirectory()`, because of
`recursive=true`.
```java
protected void deleteDirectory(File directory) throws IOException {
Path path = new Path(directory.toURI());
FileSystem fileSystem = path.getFileSystem();
if (fileSystem.exists(path)) {
fileSystem.delete(path, true);
}
}
```
The total discard logic of local dstl files is:
1. Register files to `TaskChangelogRegistry#startTracking()` on
[GitHub] [flink] xintongsong commented on pull request #20297: [FLINK-28307][doc] Update history server docs w.r.t. FLIP-241.
xintongsong commented on PR #20297: URL: https://github.com/apache/flink/pull/20297#issuecomment-1188580620 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28579) Supports predicate testing for new columns
[ https://issues.apache.org/jira/browse/FLINK-28579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28579. Resolution: Fixed master: 4d36d8dcb5d20c9517e3532d8aa42aa828ac4c23 > Supports predicate testing for new columns > -- > > Key: FLINK-28579 > URL: https://issues.apache.org/jira/browse/FLINK-28579 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > The currently added column, if there is a filter on it, will cause an error > in the RowDataToObjectArrayConverter because the number of columns is not > correct > We can make BinaryTableStats supports evolution from shorter rowData. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #223: [FLINK-28579] Supports predicate testing for new columns
JingsongLi merged PR #223: URL: https://github.com/apache/flink-table-store/pull/223 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28601) Support FeatureHasher in FlinkML
[ https://issues.apache.org/jira/browse/FLINK-28601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28601: --- Labels: pull-request-available (was: ) > Support FeatureHasher in FlinkML > > > Key: FLINK-28601 > URL: https://issues.apache.org/jira/browse/FLINK-28601 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: weibo zhao >Priority: Major > Labels: pull-request-available > > Support FeatureHasher in FlinkML. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] weibozhao opened a new pull request, #133: [FLINK-28601] Support FeatureHasher in FlinkML
weibozhao opened a new pull request, #133: URL: https://github.com/apache/flink-ml/pull/133 Support FeatureHasher in FlinkML -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #20297: [FLINK-28307][doc] Update history server docs w.r.t. FLIP-241.
xintongsong commented on PR #20297: URL: https://github.com/apache/flink/pull/20297#issuecomment-1188571408 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28601) Support FeatureHasher in FlinkML
weibo zhao created FLINK-28601: -- Summary: Support FeatureHasher in FlinkML Key: FLINK-28601 URL: https://issues.apache.org/jira/browse/FLINK-28601 Project: Flink Issue Type: New Feature Components: Library / Machine Learning Reporter: weibo zhao Support FeatureHasher in FlinkML. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20304: [FLINK-28600][connectors] Support FilterPushDown in flink-connector-jdbc
flinkbot commented on PR #20304: URL: https://github.com/apache/flink/pull/20304#issuecomment-1188570834 ## CI report: * 5a43fbab2b9c1fe1780df4d77cf1b5de9dad1915 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28600) Support FilterPushDown in flink-connector-jdbc
[ https://issues.apache.org/jira/browse/FLINK-28600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28600: --- Labels: pull-request-available (was: ) > Support FilterPushDown in flink-connector-jdbc > -- > > Key: FLINK-28600 > URL: https://issues.apache.org/jira/browse/FLINK-28600 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: Hailin Wang >Priority: Major > Labels: pull-request-available > > Support FilterPushDown in flink-connector-jdbc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] hailin0 opened a new pull request, #20304: [FLINK-28600][connectors] Support FilterPushDown in flink-connector-jdbc
hailin0 opened a new pull request, #20304: URL: https://github.com/apache/flink/pull/20304 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-28600) Support FilterPushDown in flink-connector-jdbc
Hailin Wang created FLINK-28600: --- Summary: Support FilterPushDown in flink-connector-jdbc Key: FLINK-28600 URL: https://issues.apache.org/jira/browse/FLINK-28600 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: Hailin Wang Support FilterPushDown in flink-connector-jdbc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20303: [FLINK-28599][table-planner] Adding FlinkJoinToMultiJoinRule to support translating left/right outer join to multi join
flinkbot commented on PR #20303: URL: https://github.com/apache/flink/pull/20303#issuecomment-1188565617 ## CI report: * 15dfbaf1a0ab15dd8ff38b6b4660b46bbc2d48e7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28599) Adding FlinkJoinToMultiJoinRule to support translating left/right outer join to multi join
[
https://issues.apache.org/jira/browse/FLINK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-28599:
---
Labels: pull-request-available (was: )
> Adding FlinkJoinToMultiJoinRule to support translating left/right outer join
> to multi join
> ---
>
> Key: FLINK-28599
> URL: https://issues.apache.org/jira/browse/FLINK-28599
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Yunhong Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> Now, Flink use Calcite's rule
> {code:java}
> JOIN_TO_MULTI_JOIN{code}
> to convert multiple joins into a join set, which can be used by join
> reorder. However, calcite's rule can not adapte to all outer joins. For left
> or right outer join, if they meet certain conditions, it can also be
> converted to multi join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20302: [hotfix][table-runtime] Avoid NPE for SliceAssigner and improve error message
flinkbot commented on PR #20302: URL: https://github.com/apache/flink/pull/20302#issuecomment-1188562448 ## CI report: * 32453a6ad95e01d3a1785d1207d2ebb64756fa20 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] swuferhong opened a new pull request, #20303: [FLINK-28599][table-planner] Adding FlinkJoinToMultiJoinRule to support translating left/right outer join to multi join
swuferhong opened a new pull request, #20303: URL: https://github.com/apache/flink/pull/20303 ## What is the purpose of the change For some join cases, like `SELECT * FROM T1 LEFT OUTER JOIN T2 ON a = c LEFT OUTER JOIN (SELECT * FROM T3) ON a = e`, tables `T1`, `T2` and `T3` can be translated to a multi join set because of changing their order can get the same join results. However, now, flink use calcite's default `JOIN_TO_MULTI_JOIN` rule, this rule will not put any outer join in the same multi join set. So, this pr is aims to add `FlinkJoinToMultiJoinRule` to support translating left/outer join to one multi join set while they meet correct conditions. ## Brief change log - Adding new rule `FlinkJoinToMultiJoinRule`. - Adding more test cases in `FlinkJoinToMultiJoinRuleTest` to verify left/right outer join. ## Verifying this change - Adding more test cases in `FlinkJoinToMultiJoinRuleTest`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? no docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-28533) SchemaChange supports updateColumnNullability and updateColumnComment
[ https://issues.apache.org/jira/browse/FLINK-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-28533. Resolution: Fixed master: d9b3eeef306efa10a8c4f1a5d09be11a33849ead > SchemaChange supports updateColumnNullability and updateColumnComment > - > > Key: FLINK-28533 > URL: https://issues.apache.org/jira/browse/FLINK-28533 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.2.0 >Reporter: Jane Chan >Assignee: Jane Chan >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.2.0 > > > * Already supported > ** update / remove table options > ** add column / change column type > * More to support > ** update nullability of column > ** update column comment -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #215: [FLINK-28533] SchemaChange supports updateColumnNullability and updateColumnComment
JingsongLi merged PR #215: URL: https://github.com/apache/flink-table-store/pull/215 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lincoln-lil opened a new pull request, #20302: [hotfix][table-runtime] Avoid NPE for SliceAssigner and improve error message
lincoln-lil opened a new pull request, #20302: URL: https://github.com/apache/flink/pull/20302 ## What is the purpose of the change NPE may occurs at LocalSlicingWindowAggOperator.processElement when an input row with null timestamp column, it would be better to avoid a NPE and improve the error message (just like WatermarkAssignerOperator does) ## Brief change log Add null check for AbstractSliceAssigner ## Verifying this change existing tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with @Public(Evolving): (no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568300#comment-17568300
]
Yuan Mei commented on FLINK-24932:
--
Hey, [~yunta] , I think it would be good to resolve this issue within Flink
1.16, however, not a must need (it is marked as a minor priority).
We are now more occupied by release 1.16, as the freezing date is close. If
this issue is something urgent for you, I agree with [~smattheis] that you can
take this ticket over.
> Frocksdb cannot run on Apple M1
> ---
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Sebastian Mattheis
>Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
>
[GitHub] [flink] 1996fanrui commented on a diff in pull request #20233: [FLINK-28474][checkpoint] Simplify the ChannelStateWriterImpl due to concurrent unaligned checkpoint isn't supported
1996fanrui commented on code in PR #20233:
URL: https://github.com/apache/flink/pull/20233#discussion_r923351868
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriterImpl.java:
##
@@ -60,89 +63,78 @@
public class ChannelStateWriterImpl implements ChannelStateWriter {
private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriterImpl.class);
-private static final int DEFAULT_MAX_CHECKPOINTS =
-1000; // includes max-concurrent-checkpoints + checkpoints to be
aborted (scheduled via
-// mailbox)
Review Comment:
Hi @pnowojski , thanks for your review.
The `checkpointId < nextExpectedCheckpointId` may happen. For example,
`AlternatingWaitingForFirstBarrierUnaligned#barrierReceived` don't check
whether checkpointId is aborted, if received a barrier, it will call the
ChannelStateWriter.start().
- If `checkpointId < ongoingCheckpointId`, it should throw exception.
- If `checkpointId >= ongoingCheckpointId && checkpointId <
nextExpectedCheckpointId` that is `checkpointId >= ongoingCheckpointId &&
abortedCheckpointIds.contains(checkpointId)`, it should ignore the checkpoint.
- If `checkpointId >= ongoingCheckpointId &&
!abortedCheckpointIds.contains(checkpointId)`, the ongoingCheckpointId should
be updated, and start this checkpoint.
`checkpointId >= ongoingCheckpointId && checkpointId <
nextExpectedCheckpointId` and `checkpointId < ongoingCheckpointId` are
processed differently. So, we should use `ongoingCheckpointId`, and just update
it within `ChannelStateWriterImpl#start`
We should ignore checkpoint when the checkpointId is aborted, so we need
the `NavigableSet abortedCheckpointIds;`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] hunter-cloud09 commented on pull request #20283: [FLINK-28122][docs-zh] Translate "User-defined Sources & Sinks" page into Chinese
hunter-cloud09 commented on PR #20283: URL: https://github.com/apache/flink/pull/20283#issuecomment-1188552304 @PatrickRen yep, but the latter part seems a bit heavy and may take some times -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on a diff in pull request #20233: [FLINK-28474][checkpoint] Simplify the ChannelStateWriterImpl due to concurrent unaligned checkpoint isn't supported
1996fanrui commented on code in PR #20233:
URL: https://github.com/apache/flink/pull/20233#discussion_r923351868
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriterImpl.java:
##
@@ -60,89 +63,78 @@
public class ChannelStateWriterImpl implements ChannelStateWriter {
private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriterImpl.class);
-private static final int DEFAULT_MAX_CHECKPOINTS =
-1000; // includes max-concurrent-checkpoints + checkpoints to be
aborted (scheduled via
-// mailbox)
Review Comment:
Hi @pnowojski , thanks for your review.
The `checkpointId < nextExpectedCheckpointId` may happen. For example,
`AlternatingWaitingForFirstBarrierUnaligned#barrierReceived` don't check
whether checkpointId is aborted, if received a barrier, it will call the
ChannelStateWriter.start().
- If `checkpointId < ongoingCheckpointId`, it should throw exception.
- If `checkpointId < nextExpectedCheckpointId` that is `checkpointId >=
ongoingCheckpointId && abortedCheckpointIds.contains(checkpointId)`, it should
ignore the checkpoint.
- If `checkpointId >= ongoingCheckpointId &&
!abortedCheckpointIds.contains(checkpointId)`, the ongoingCheckpointId should
be updated, and start this checkpoint.
So, we should use `ongoingCheckpointId`, and just update it within
`ChannelStateWriterImpl#start`
We should ignore checkpoint when the checkpointId is aborted, so we need
the `NavigableSet abortedCheckpointIds;`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #20283: [FLINK-28122][docs-zh] Translate "User-defined Sources & Sinks" page into Chinese
PatrickRen commented on PR #20283: URL: https://github.com/apache/flink/pull/20283#issuecomment-1188548478 @hunter-cloud09 No worries Would you like to translate the remain part of the doc? It's better to finish the entire page in one PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] hunter-cloud09 commented on pull request #20283: [FLINK-28122][docs-zh] Translate "User-defined Sources & Sinks" page into Chinese
hunter-cloud09 commented on PR #20283: URL: https://github.com/apache/flink/pull/20283#issuecomment-1188546768 @PatrickRen I'm so sorry for these problems. For these problems, they have been solved in the new commit. Please help me review again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28267) KafkaSourceLegacyITCase.testBrokerFailure hang on azure
[ https://issues.apache.org/jira/browse/FLINK-28267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568294#comment-17568294 ] Huang Xingbo commented on FLINK-28267: -- https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=38349=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203 > KafkaSourceLegacyITCase.testBrokerFailure hang on azure > --- > > Key: FLINK-28267 > URL: https://issues.apache.org/jira/browse/FLINK-28267 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.16.0 >Reporter: Zhu Zhu >Assignee: Qingsheng Ren >Priority: Critical > > "main" #1 prio=5 os_prio=0 tid=0x7f5ae000b800 nid=0x22c2 waiting on > condition [0x7f5ae8398000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xa5565010> (a > java.util.concurrent.CompletableFuture$Signaller) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) > at > java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) > at > java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) > at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:67) > at > org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBrokerFailureTest(KafkaConsumerTestBase.java:1506) > at > org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testBrokerFailure(KafkaSourceLegacyITCase.java:94) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37247=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=41526 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28267) KafkaSourceLegacyITCase.testBrokerFailure hang on azure
[ https://issues.apache.org/jira/browse/FLINK-28267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568292#comment-17568292 ] Huang Xingbo commented on FLINK-28267: -- https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=38347=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203 > KafkaSourceLegacyITCase.testBrokerFailure hang on azure > --- > > Key: FLINK-28267 > URL: https://issues.apache.org/jira/browse/FLINK-28267 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.16.0 >Reporter: Zhu Zhu >Assignee: Qingsheng Ren >Priority: Critical > > "main" #1 prio=5 os_prio=0 tid=0x7f5ae000b800 nid=0x22c2 waiting on > condition [0x7f5ae8398000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xa5565010> (a > java.util.concurrent.CompletableFuture$Signaller) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) > at > java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) > at > java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) > at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:67) > at > org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBrokerFailureTest(KafkaConsumerTestBase.java:1506) > at > org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testBrokerFailure(KafkaSourceLegacyITCase.java:94) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37247=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=41526 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28267) KafkaSourceLegacyITCase.testBrokerFailure hang on azure
[ https://issues.apache.org/jira/browse/FLINK-28267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568293#comment-17568293 ] Huang Xingbo commented on FLINK-28267: -- Hi [~renqs] Any updates on the progress？ > KafkaSourceLegacyITCase.testBrokerFailure hang on azure > --- > > Key: FLINK-28267 > URL: https://issues.apache.org/jira/browse/FLINK-28267 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.16.0 >Reporter: Zhu Zhu >Assignee: Qingsheng Ren >Priority: Critical > > "main" #1 prio=5 os_prio=0 tid=0x7f5ae000b800 nid=0x22c2 waiting on > condition [0x7f5ae8398000] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xa5565010> (a > java.util.concurrent.CompletableFuture$Signaller) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) > at > java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) > at > java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) > at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:67) > at > org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runBrokerFailureTest(KafkaConsumerTestBase.java:1506) > at > org.apache.flink.connector.kafka.source.KafkaSourceLegacyITCase.testBrokerFailure(KafkaSourceLegacyITCase.java:94) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37247=logs=aa18c3f6-13b8-5f58-86bb-c1cffb239496=502fb6c0-30a2-5e49-c5c2-a00fa3acb203=41526 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] LadyForest commented on a diff in pull request #223: [FLINK-28579] Supports predicate testing for new columns
LadyForest commented on code in PR #223:
URL: https://github.com/apache/flink-table-store/pull/223#discussion_r924013592
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/stats/BinaryTableStats.java:
##
@@ -59,8 +59,12 @@ public BinaryTableStats(
}
public FieldStats[] fields(FieldStatsArraySerializer converter) {
+return fields(converter, null);
Review Comment:
L#54, annotate parameter`cacheArray` as `@Nullable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-28529) ChangelogPeriodicMaterializationSwitchStateBackendITCase.testSwitchFromDisablingToEnablingInClaimMode failed with CheckpointException: Checkpoint expired before comple
[
https://issues.apache.org/jira/browse/FLINK-28529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568291#comment-17568291
]
Huang Xingbo commented on FLINK-28529:
--
Hi [~Yanfei Lei], I have merged the PR
https://github.com/apache/flink/pull/19864 . You can go on your work.
> ChangelogPeriodicMaterializationSwitchStateBackendITCase.testSwitchFromDisablingToEnablingInClaimMode
> failed with CheckpointException: Checkpoint expired before completing
> ---
>
> Key: FLINK-28529
> URL: https://issues.apache.org/jira/browse/FLINK-28529
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Yanfei Lei
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-07-12T04:30:49.9912088Z Jul 12 04:30:49 [ERROR]
> ChangelogPeriodicMaterializationSwitchStateBackendITCase.testSwitchFromDisablingToEnablingInClaimMode
> Time elapsed: 617.048 s <<< ERROR!
> 2022-07-12T04:30:49.9913108Z Jul 12 04:30:49
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired
> before completing.
> 2022-07-12T04:30:49.9913880Z Jul 12 04:30:49 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2022-07-12T04:30:49.9914606Z Jul 12 04:30:49 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-07-12T04:30:49.9915572Z Jul 12 04:30:49 at
> org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationSwitchStateBackendITCase.testSwitchFromDisablingToEnablingInClaimMode(ChangelogPeriodicMaterializationSwitchStateBackendITCase.java:125)
> 2022-07-12T04:30:49.9916483Z Jul 12 04:30:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-07-12T04:30:49.9917377Z Jul 12 04:30:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-07-12T04:30:49.9918121Z Jul 12 04:30:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-07-12T04:30:49.9918788Z Jul 12 04:30:49 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-07-12T04:30:49.9919456Z Jul 12 04:30:49 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-07-12T04:30:49.9920193Z Jul 12 04:30:49 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-07-12T04:30:49.9920923Z Jul 12 04:30:49 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-07-12T04:30:49.9921630Z Jul 12 04:30:49 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-07-12T04:30:49.9922326Z Jul 12 04:30:49 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-07-12T04:30:49.9923023Z Jul 12 04:30:49 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-07-12T04:30:49.9923708Z Jul 12 04:30:49 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-07-12T04:30:49.9924449Z Jul 12 04:30:49 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-07-12T04:30:49.9925124Z Jul 12 04:30:49 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-07-12T04:30:49.9925912Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-07-12T04:30:49.9926742Z Jul 12 04:30:49 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-07-12T04:30:49.9928142Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-07-12T04:30:49.9928715Z Jul 12 04:30:49 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-07-12T04:30:49.9929311Z Jul 12 04:30:49 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-07-12T04:30:49.9929863Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-07-12T04:30:49.9930376Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-07-12T04:30:49.9930911Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-07-12T04:30:49.9931441Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-07-12T04:30:49.9931975Z Jul 12 04:30:49 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-07-12T04:30:49.9932493Z Jul 12 04:30:49 at
>
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #130: [FLINK-28502] Add Transformer for RegexTokenizer
yunfengzhou-hub commented on code in PR #130:
URL: https://github.com/apache/flink-ml/pull/130#discussion_r924009394
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/regextokenizer/RegexTokenizerParams.java:
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.regextokenizer;
+
+import org.apache.flink.ml.common.param.HasInputCol;
+import org.apache.flink.ml.common.param.HasOutputCol;
+import org.apache.flink.ml.param.BooleanParam;
+import org.apache.flink.ml.param.IntParam;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+
+/**
+ * Params for {@link RegexTokenizer}.
+ *
+ * @param The class type of this instance.
+ */
+public interface RegexTokenizerParams extends HasInputCol,
HasOutputCol {
+IntParam MIN_TOKEN_LENGTH =
+new IntParam("minTokenLength", "Minimum token length", 1,
ParamValidators.gtEq(0));
+BooleanParam GAPS = new BooleanParam("gaps", "Set regex to match gaps or
tokens", true);
+
+StringParam PATTERN = new StringParam("pattern", "Regex pattern used for
tokenizing", "\\s+");
+
+BooleanParam TO_LOWERCASE =
+new BooleanParam(
+"toLowercase",
+"Whether to convert all characters to lowercase before
tokenizing",
+true);
+
+default T setMinTokenLength(int value) {
+return set(MIN_TOKEN_LENGTH, value);
+}
+
+default Integer getMinTokenLength() {
Review Comment:
we can make this method return `int` instead of `Integer`, as `HasMaxIter`
has done, and remove the `intValue()` in tests.
##
flink-ml-examples/src/main/java/org/apache/flink/ml/examples/feature/RegexTokenizerExample.java:
##
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.examples.feature;
+
+import org.apache.flink.ml.feature.regextokenizer.RegexTokenizer;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.CloseableIterator;
+
+import java.util.Arrays;
+
+/** Simple program that creates a RegexTokenizer instance and uses it for
feature engineering. */
+public class RegexTokenizerExample {
+public static void main(String[] args) {
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tEnv = StreamTableEnvironment.create(env);
+
+// Generates input data.
+DataStream inputStream =
+env.fromElements(Row.of("Test for tokenization."),
Row.of("Te,st. punct"));
+Table inputTable = tEnv.fromDataStream(inputStream).as("input");
+
+// Creates a RegexTokenizer object and initializes its parameters.
+RegexTokenizer regexTokenizer =
+new
RegexTokenizer().setInputCol("input").setOutputCol("output");
Review Comment:
It might be better to add a `setPattern()` here to show the distinguished
function of RegexTokenizer from Tokenizer.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/regextokenizer/RegexTokenizerParams.java:
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the
[jira] [Commented] (FLINK-28177) Elasticsearch6DynamicSinkITCase.testWritingDocumentsNoPrimaryKey failed with 503 Service Unavailable
[
https://issues.apache.org/jira/browse/FLINK-28177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568290#comment-17568290
]
Huang Xingbo commented on FLINK-28177:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=38339=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9
> Elasticsearch6DynamicSinkITCase.testWritingDocumentsNoPrimaryKey failed with
> 503 Service Unavailable
>
>
> Key: FLINK-28177
> URL: https://issues.apache.org/jira/browse/FLINK-28177
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-06-21T07:39:23.9065585Z Jun 21 07:39:23 [ERROR] Tests run: 4, Failures:
> 0, Errors: 2, Skipped: 0, Time elapsed: 43.125 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase
> 2022-06-21T07:39:23.9068457Z Jun 21 07:39:23 [ERROR]
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase.testWritingDocumentsNoPrimaryKey
> Time elapsed: 8.697 s <<< ERROR!
> 2022-06-21T07:39:23.9069955Z Jun 21 07:39:23
> java.util.concurrent.ExecutionException:
> org.apache.flink.table.api.TableException: Failed to wait job finish
> 2022-06-21T07:39:23.9071135Z Jun 21 07:39:23 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2022-06-21T07:39:23.9072225Z Jun 21 07:39:23 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-06-21T07:39:23.9073408Z Jun 21 07:39:23 at
> org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:118)
> 2022-06-21T07:39:23.9075081Z Jun 21 07:39:23 at
> org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:81)
> 2022-06-21T07:39:23.9076560Z Jun 21 07:39:23 at
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase.testWritingDocumentsNoPrimaryKey(Elasticsearch6DynamicSinkITCase.java:286)
> 2022-06-21T07:39:23.9078535Z Jun 21 07:39:23 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-06-21T07:39:23.9079534Z Jun 21 07:39:23 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-06-21T07:39:23.9080702Z Jun 21 07:39:23 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-06-21T07:39:23.9081838Z Jun 21 07:39:23 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-06-21T07:39:23.9082942Z Jun 21 07:39:23 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-06-21T07:39:23.9084127Z Jun 21 07:39:23 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-06-21T07:39:23.9085246Z Jun 21 07:39:23 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-06-21T07:39:23.9086380Z Jun 21 07:39:23 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-06-21T07:39:23.9087812Z Jun 21 07:39:23 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-06-21T07:39:23.9088843Z Jun 21 07:39:23 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-06-21T07:39:23.9089823Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-06-21T07:39:23.9103797Z Jun 21 07:39:23 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-06-21T07:39:23.9105022Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-06-21T07:39:23.9106065Z Jun 21 07:39:23 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-06-21T07:39:23.9107500Z Jun 21 07:39:23 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-06-21T07:39:23.9108591Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-06-21T07:39:23.9109575Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-06-21T07:39:23.9110606Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-06-21T07:39:23.9111634Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-06-21T07:39:23.9112653Z Jun 21 07:39:23 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-06-21T07:39:23.9113922Z Jun 21 07:39:23 at
>
[GitHub] [flink] TanYuxin-tyx commented on pull request #20294: [FLINK-28514][network] Remove data flush in SortMergeResultPartition
TanYuxin-tyx commented on PR #20294: URL: https://github.com/apache/flink/pull/20294#issuecomment-1188536886 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28599) Adding FlinkJoinToMultiJoinRule to support translating left/right outer join to multi join
[
https://issues.apache.org/jira/browse/FLINK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yunhong Zheng updated FLINK-28599:
--
Summary: Adding FlinkJoinToMultiJoinRule to support translating left/right
outer join to multi join (was: Adding FlinkJoinToMultiJoinRule to support
left/right outer join be translated to multi join)
> Adding FlinkJoinToMultiJoinRule to support translating left/right outer join
> to multi join
> ---
>
> Key: FLINK-28599
> URL: https://issues.apache.org/jira/browse/FLINK-28599
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Yunhong Zheng
>Priority: Major
> Fix For: 1.16.0
>
>
> Now, Flink use Calcite's rule
> {code:java}
> JOIN_TO_MULTI_JOIN{code}
> to convert multiple joins into a join set, which can be used by join
> reorder. However, calcite's rule can not adapte to all outer joins. For left
> or right outer join, if they meet certain conditions, it can also be
> converted to multi join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] fredia commented on a diff in pull request #20300: [WIP][FLINK-28597][state] Discard non-initial checkpoints without a delay
fredia commented on code in PR #20300:
URL: https://github.com/apache/flink/pull/20300#discussion_r924011917
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/SharedStateRegistryImpl.java:
##
@@ -192,6 +208,8 @@ public void registerAll(
@Override
public void registerAllAfterRestored(CompletedCheckpoint checkpoint,
RestoreMode mode) {
Review Comment:
This method is called by
`CompletedCheckpoint#registerSharedStatesAfterRestored()`, and
`registerSharedStatesAfterRestored()` is called by
`CheckpointCoordinator#restoreSavepoint()` and `DEFAULT_FACTORY`.
I'm not sure if the call in `DEFAULT_FACTORY` has other effects. If a job
failover and restarts, does `highestRestoredCheckpointID` update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] PatrickRen commented on a diff in pull request #20283: [FLINK-28122][docs-zh] Translate "User-defined Sources & Sinks" page into Chinese
PatrickRen commented on code in PR #20283:
URL: https://github.com/apache/flink/pull/20283#discussion_r924002285
##
docs/content.zh/docs/dev/table/sourcesSinks.md:
##
@@ -24,113 +24,84 @@ specific language governing permissions and limitations
under the License.
-->
-# User-defined Sources & Sinks
+# 用户自定义 Sources & Sinks
-_Dynamic tables_ are the core concept of Flink's Table & SQL API for
processing both bounded and unbounded
-data in a unified fashion.
+*动态表*是 Flink Table & SQL API的核心概念，用于统一有界和无界数据的处理。
-Because dynamic tables are only a logical concept, Flink does not own the data
itself. Instead, the content
-of a dynamic table is stored in external systems (such as databases, key-value
stores, message queues) or files.
+动态表只是一个逻辑概念，因此 Flink 并不拥有数据。相应的，动态表的内容存储在外部系统（ 如数据库、键值存储、消息队列 ）或文件中。
-_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from
and to an external system. In
-the documentation, sources and sinks are often summarized under the term
_connector_.
+*动态 sources* 和*动态 sinks* 可用于从外部系统读取数据和向外部系统写入数据。在文档中，sources 和 sinks 常在术语*连接器*
下进行总结。
-Flink provides pre-defined connectors for Kafka, Hive, and different file
systems. See the [connector section]({{< ref "docs/connectors/table/overview"
>}})
-for more information about built-in table sources and sinks.
+Flink 为 Kafka、Hive 和不同的文件系统提供了预定义的连接器。有关内置 table sources 和 sinks
的更多信息，请参阅[连接器部分]({{< ref "docs/connectors/table/overview" >}})
-This page focuses on how to develop a custom, user-defined connector.
+本页重点介绍如何开发自定义的用户定义连接器。
-Overview
+概述
-In many cases, implementers don't need to create a new connector from scratch
but would like to slightly
-modify existing connectors or hook into the existing stack. In other cases,
implementers would like to
-create specialized connectors.
+在许多情况下，开发人员不需要从头开始创建新的连接器，而是希望稍微修改现有的连接器或 hook 到现有的
stack。在其他情况下，开发人员希望创建专门的连接器。
-This section helps for both kinds of use cases. It explains the general
architecture of table connectors
-from pure declaration in the API to runtime code that will be executed on the
cluster.
+本节对这两种用例都有帮助。它解释了表连接器的一般体系结构，从 API 中的纯粹声明到在集群上执行的运行时代码
-The filled arrows show how objects are transformed to other objects from one
stage to the next stage during
-the translation process.
+实心箭头展示了在转换过程中对象如何从一个阶段到下一个阶段转换为其他对象。
{{< img width="90%" src="/fig/table_connectors.svg" alt="Translation of table
connectors" >}}
-### Metadata
+### 元数据
-Both Table API and SQL are declarative APIs. This includes the declaration of
tables. Thus, executing
-a `CREATE TABLE` statement results in updated metadata in the target catalog.
+Table API 和 SQL 都是声明式 API。这包括表的声明。因此，执行 `CREATE TABLE` 语句会导致目标 catalog 中的元数据更新。
-For most catalog implementations, physical data in the external system is not
modified for such an
-operation. Connector-specific dependencies don't have to be present in the
classpath yet. The options declared
-in the `WITH` clause are neither validated nor otherwise interpreted.
+对于大多数目录实现，外部系统中的物理数据不会针对此类操作进行修改。特定于连接器的依赖项不必存在于类路径中。在 `WITH`
子句中声明的选项既不被验证也不被解释。
Review Comment:
"目录" -> "catalog" to keep consistency in the context
##
docs/content.zh/docs/dev/table/sourcesSinks.md:
##
@@ -24,113 +24,84 @@ specific language governing permissions and limitations
under the License.
-->
-# User-defined Sources & Sinks
+# 用户自定义 Sources & Sinks
-_Dynamic tables_ are the core concept of Flink's Table & SQL API for
processing both bounded and unbounded
-data in a unified fashion.
+*动态表*是 Flink Table & SQL API的核心概念，用于统一有界和无界数据的处理。
-Because dynamic tables are only a logical concept, Flink does not own the data
itself. Instead, the content
-of a dynamic table is stored in external systems (such as databases, key-value
stores, message queues) or files.
+动态表只是一个逻辑概念，因此 Flink 并不拥有数据。相应的，动态表的内容存储在外部系统（ 如数据库、键值存储、消息队列 ）或文件中。
-_Dynamic sources_ and _dynamic sinks_ can be used to read and write data from
and to an external system. In
-the documentation, sources and sinks are often summarized under the term
_connector_.
+*动态 sources* 和*动态 sinks* 可用于从外部系统读取数据和向外部系统写入数据。在文档中，sources 和 sinks 常在术语*连接器*
下进行总结。
-Flink provides pre-defined connectors for Kafka, Hive, and different file
systems. See the [connector section]({{< ref "docs/connectors/table/overview"
>}})
-for more information about built-in table sources and sinks.
+Flink 为 Kafka、Hive 和不同的文件系统提供了预定义的连接器。有关内置 table sources 和 sinks
的更多信息，请参阅[连接器部分]({{< ref "docs/connectors/table/overview" >}})
-This page focuses on how to develop a custom, user-defined connector.
+本页重点介绍如何开发自定义的用户定义连接器。
-Overview
+概述
-In many cases, implementers don't need to create a new connector from scratch
but would like to slightly
-modify existing connectors or hook into the existing stack. In other cases,
implementers would like to
-create specialized connectors.
+在许多情况下，开发人员不需要从头开始创建新的连接器，而是希望稍微修改现有的连接器或 hook 到现有的
[jira] [Updated] (FLINK-28599) Adding FlinkJoinToMultiJoinRule to support left/right outer join be translated to multi join
[
https://issues.apache.org/jira/browse/FLINK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yunhong Zheng updated FLINK-28599:
--
Summary: Adding FlinkJoinToMultiJoinRule to support left/right outer join
be translated to multi join (was: Adding FlinkJoinToMultiJoinRule to support
left/right outer join can be translated to multi join)
> Adding FlinkJoinToMultiJoinRule to support left/right outer join be
> translated to multi join
>
>
> Key: FLINK-28599
> URL: https://issues.apache.org/jira/browse/FLINK-28599
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Yunhong Zheng
>Priority: Major
> Fix For: 1.16.0
>
>
> Now, Flink use Calcite's rule
> {code:java}
> JOIN_TO_MULTI_JOIN{code}
> to convert multiple joins into a join set, which can be used by join
> reorder. However, calcite's rule can not adapte to all outer joins. For left
> or right outer join, if they meet certain conditions, it can also be
> converted to multi join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28599) Adding FlinkJoinToMultiJoinRule to support left/right outer join can be translated to multi join
[
https://issues.apache.org/jira/browse/FLINK-28599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568287#comment-17568287
]
Yunhong Zheng commented on FLINK-28599:
---
Hi, [~godfreyhe] , could you assign this to me ! Thanks !
> Adding FlinkJoinToMultiJoinRule to support left/right outer join can be
> translated to multi join
>
>
> Key: FLINK-28599
> URL: https://issues.apache.org/jira/browse/FLINK-28599
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Yunhong Zheng
>Priority: Major
> Fix For: 1.16.0
>
>
> Now, Flink use Calcite's rule
> {code:java}
> JOIN_TO_MULTI_JOIN{code}
> to convert multiple joins into a join set, which can be used by join
> reorder. However, calcite's rule can not adapte to all outer joins. For left
> or right outer join, if they meet certain conditions, it can also be
> converted to multi join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-28599) Adding FlinkJoinToMultiJoinRule to support left/right outer join can be translated to multi join
Yunhong Zheng created FLINK-28599:
-
Summary: Adding FlinkJoinToMultiJoinRule to support left/right
outer join can be translated to multi join
Key: FLINK-28599
URL: https://issues.apache.org/jira/browse/FLINK-28599
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Affects Versions: 1.16.0
Reporter: Yunhong Zheng
Fix For: 1.16.0
Now, Flink use Calcite's rule
{code:java}
JOIN_TO_MULTI_JOIN{code}
to convert multiple joins into a join set, which can be used by join reorder.
However, calcite's rule can not adapte to all outer joins. For left or right
outer join, if they meet certain conditions, it can also be converted to multi
join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-28598) ClusterEntryPoint can't get the real exit reason when shutting down
zlzhang0122 created FLINK-28598: --- Summary: ClusterEntryPoint can't get the real exit reason when shutting down Key: FLINK-28598 URL: https://issues.apache.org/jira/browse/FLINK-28598 Project: Flink Issue Type: Bug Components: Deployment / YARN, Runtime / Task Affects Versions: 1.15.1, 1.14.2 Reporter: zlzhang0122 When the cluster is starting and some error occurs, the ClusterEntryPoint will shutDown the cluster asynchronous, but if it can't get a Throwable, the shutDown reason will be null, but actually if it's a user code problem and this may happen. I think we can get the real exit reason caused by user code and pass it to the diagnostics parameter, this may help users a lot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20301: [FLINK-28544][python][e2e] Clean up python environment after e2e tests
flinkbot commented on PR #20301: URL: https://github.com/apache/flink/pull/20301#issuecomment-1188526008 ## CI report: * c07d636eb2f802c86d6873071372838c60ef3910 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] WencongLiu commented on a diff in pull request #20236: [FLINK-27769][sql-gateway]Introduce the REST endpoint framework
WencongLiu commented on code in PR #20236:
URL: https://github.com/apache/flink/pull/20236#discussion_r924006512
##
flink-runtime/src/main/java/org/apache/flink/runtime/rest/versioning/APIVersion.java:
##
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.rest.versioning;
+
+/** Interface for all versions of the REST API. */
+public interface APIVersion {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] liyubin117 commented on pull request #134: [FLINK-27207] Support built-in parquet format
liyubin117 commented on PR #134: URL: https://github.com/apache/flink-table-store/pull/134#issuecomment-1188528042 @JingsongLi Thanks for your kind reminds, I will refactor parquet formats to adapt the new framework in the late days. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28544) Elasticsearch6SinkE2ECase failed with no space left on device
[
https://issues.apache.org/jira/browse/FLINK-28544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-28544:
---
Labels: pull-request-available test-stability (was: test-stability)
> Elasticsearch6SinkE2ECase failed with no space left on device
> -
>
> Key: FLINK-28544
> URL: https://issues.apache.org/jira/browse/FLINK-28544
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch, Tests
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {code:java}
> 2022-07-13T02:49:13.5455800Z Jul 13 02:49:13 [ERROR] Tests run: 1, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 49.38 s <<< FAILURE! - in
> org.apache.flink.streaming.tests.Elasticsearch6SinkE2ECase
> 2022-07-13T02:49:13.5465965Z Jul 13 02:49:13 [ERROR]
> org.apache.flink.streaming.tests.Elasticsearch6SinkE2ECase Time elapsed:
> 49.38 s <<< ERROR!
> 2022-07-13T02:49:13.5466765Z Jul 13 02:49:13 java.lang.RuntimeException:
> Failed to build JobManager image
> 2022-07-13T02:49:13.5467621Z Jul 13 02:49:13 at
> org.apache.flink.connector.testframe.container.FlinkTestcontainersConfigurator.configureJobManagerContainer(FlinkTestcontainersConfigurator.java:67)
> 2022-07-13T02:49:13.5468645Z Jul 13 02:49:13 at
> org.apache.flink.connector.testframe.container.FlinkTestcontainersConfigurator.configure(FlinkTestcontainersConfigurator.java:147)
> 2022-07-13T02:49:13.5469564Z Jul 13 02:49:13 at
> org.apache.flink.connector.testframe.container.FlinkContainers$Builder.build(FlinkContainers.java:197)
> 2022-07-13T02:49:13.5470467Z Jul 13 02:49:13 at
> org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment.(FlinkContainerTestEnvironment.java:88)
> 2022-07-13T02:49:13.5471424Z Jul 13 02:49:13 at
> org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment.(FlinkContainerTestEnvironment.java:51)
> 2022-07-13T02:49:13.5472504Z Jul 13 02:49:13 at
> org.apache.flink.streaming.tests.ElasticsearchSinkE2ECaseBase.(ElasticsearchSinkE2ECaseBase.java:58)
> 2022-07-13T02:49:13.5473388Z Jul 13 02:49:13 at
> org.apache.flink.streaming.tests.Elasticsearch6SinkE2ECase.(Elasticsearch6SinkE2ECase.java:36)
> 2022-07-13T02:49:13.5474161Z Jul 13 02:49:13 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 2022-07-13T02:49:13.5474905Z Jul 13 02:49:13 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 2022-07-13T02:49:13.5475756Z Jul 13 02:49:13 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 2022-07-13T02:49:13.5476734Z Jul 13 02:49:13 at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 2022-07-13T02:49:13.5477495Z Jul 13 02:49:13 at
> org.junit.platform.commons.util.ReflectionUtils.newInstance(ReflectionUtils.java:550)
> 2022-07-13T02:49:13.5478313Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.ConstructorInvocation.proceed(ConstructorInvocation.java:56)
> 2022-07-13T02:49:13.5479220Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-07-13T02:49:13.5480165Z Jul 13 02:49:13 at
> org.junit.jupiter.api.extension.InvocationInterceptor.interceptTestClassConstructor(InvocationInterceptor.java:73)
> 2022-07-13T02:49:13.5481038Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-07-13T02:49:13.5481944Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-07-13T02:49:13.5482875Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-07-13T02:49:13.5483764Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-07-13T02:49:13.5484642Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-07-13T02:49:13.5486123Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-07-13T02:49:13.5488185Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:77)
> 2022-07-13T02:49:13.543Z Jul 13 02:49:13 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeTestClassConstructor(ClassBasedTestDescriptor.java:355)
>
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #215: [FLINK-28533] SchemaChange supports updateColumnNullability and updateColumnComment
JingsongLi commented on code in PR #215:
URL: https://github.com/apache/flink-table-store/pull/215#discussion_r924001994
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/schema/DataType.java:
##
@@ -45,6 +45,22 @@ public LogicalType logicalType() {
return logicalType;
}
+public static DataType copy(DataType type, boolean isNullable) {
Review Comment:
maybe a `public abstract DataType copy(boolean isNullable);`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangXingBo opened a new pull request, #20301: [FLINK-28544][python][e2e] Clean up python environment after e2e tests
HuangXingBo opened a new pull request, #20301: URL: https://github.com/apache/flink/pull/20301 ## What is the purpose of the change *This pull request will clean up python environment after e2e tests* ## Brief change log - *Clean up python environment after e2e tests* ## Verifying this change This change added tests and can be verified as follows: - *e2e tests* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] swuferhong commented on pull request #20084: [FLINK-27988][table-planner] Let HiveTableSource extend from SupportStatisticsReport
swuferhong commented on PR #20084: URL: https://github.com/apache/flink/pull/20084#issuecomment-1188522296 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-27162) RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed on azure
[
https://issues.apache.org/jira/browse/FLINK-27162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo closed FLINK-27162.
Resolution: Fixed
Merged into master via bf983ab1f9bdc7ef1daed15397abf0b032ad2919
> RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed on
> azure
> ---
>
> Key: FLINK-27162
> URL: https://issues.apache.org/jira/browse/FLINK-27162
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Yun Gao
>Assignee: Anton Kalashnikov
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 2022-04-08T16:21:37.7382295Z Apr 08 16:21:37 [ERROR] Tests run: 2, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 19.21 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase
> 2022-04-08T16:21:37.7383825Z Apr 08 16:21:37 [ERROR]
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState
> Time elapsed: 9.642 s <<< ERROR!
> 2022-04-08T16:21:37.7385362Z Apr 08 16:21:37
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint was
> canceled because a barrier from newer checkpoint was received.
> 2022-04-08T16:21:37.7386479Z Apr 08 16:21:37 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2022-04-08T16:21:37.7387206Z Apr 08 16:21:37 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-04-08T16:21:37.7388026Z Apr 08 16:21:37 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.runJobAndGetCheckpoint(RescaleCheckpointManuallyITCase.java:196)
> 2022-04-08T16:21:37.7389054Z Apr 08 16:21:37 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingKeyedState(RescaleCheckpointManuallyITCase.java:137)
> 2022-04-08T16:21:37.7390072Z Apr 08 16:21:37 at
> org.apache.flink.test.checkpointing.RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState(RescaleCheckpointManuallyITCase.java:115)
> 2022-04-08T16:21:37.7391320Z Apr 08 16:21:37 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-04-08T16:21:37.7392401Z Apr 08 16:21:37 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-04-08T16:21:37.7393916Z Apr 08 16:21:37 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-04-08T16:21:37.7394662Z Apr 08 16:21:37 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-04-08T16:21:37.7395293Z Apr 08 16:21:37 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-04-08T16:21:37.7396038Z Apr 08 16:21:37 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-04-08T16:21:37.7396749Z Apr 08 16:21:37 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-04-08T16:21:37.7397458Z Apr 08 16:21:37 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-04-08T16:21:37.7398164Z Apr 08 16:21:37 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-04-08T16:21:37.7398844Z Apr 08 16:21:37 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-04-08T16:21:37.7399505Z Apr 08 16:21:37 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-04-08T16:21:37.7400182Z Apr 08 16:21:37 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-04-08T16:21:37.7400804Z Apr 08 16:21:37 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-04-08T16:21:37.7401492Z Apr 08 16:21:37 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-04-08T16:21:37.7402605Z Apr 08 16:21:37 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-04-08T16:21:37.7403783Z Apr 08 16:21:37 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-04-08T16:21:37.7404514Z Apr 08 16:21:37 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-04-08T16:21:37.7405180Z Apr 08 16:21:37 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-04-08T16:21:37.7405784Z Apr 08 16:21:37 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-04-08T16:21:37.7406537Z Apr 08 16:21:37 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-04-08T16:21:37.7407256Z Apr 08
[GitHub] [flink] HuangXingBo closed pull request #19864: [FLINK-27162][runtime] Trigger non-periodic checkpoint in 'timer' thread
HuangXingBo closed pull request #19864: [FLINK-27162][runtime] Trigger non-periodic checkpoint in 'timer' thread URL: https://github.com/apache/flink/pull/19864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #20297: [FLINK-28307][doc] Update history server docs w.r.t. FLIP-241.
xintongsong commented on PR #20297: URL: https://github.com/apache/flink/pull/20297#issuecomment-1188517346 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28562) Rocksdb state backend is too slow when using AggregateFunction
[
https://issues.apache.org/jira/browse/FLINK-28562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568279#comment-17568279
]
Yue Shang commented on FLINK-28562:
---
ok,
i will reach out to the community for help.
thank you.
> Rocksdb state backend is too slow when using AggregateFunction
> --
>
> Key: FLINK-28562
> URL: https://issues.apache.org/jira/browse/FLINK-28562
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.13.2, 1.14.3
> Environment: {code:java}
> final ParameterTool params = ParameterTool.fromArgs(args);
> // set up the execution environment
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> // get input data
> DataStream source = env.addSource(new SourceFunction() {
> @Override
> public void run(SourceContext ctx) throws Exception {
> Random rd = new Random();
> while (true){
> UserTag userTag = new UserTag();
> userTag.setUserId(rd.nextLong());
> userTag.setMetricName(UUID.randomUUID().toString());
> userTag.setTagDimension(UUID.randomUUID().toString());
> userTag.setTagValue(rd.nextDouble());
> userTag.setTagTime( new Long(new Date().getTime()).intValue());
> userTag.setMonitorTime(new Long(new Date().getTime()).intValue());
> userTag.setAggregationPeriod("5s");
> userTag.setAggregationType("sum");
> userTag.setTimePeriod("hour");
> userTag.setDataType("number");
> userTag.setBaseTime(1657803600);
> userTag.setTopic(UUID.randomUUID().toString());
> for(int i = 0;i<100;i++){
> userTag.setUserTagName(UUID.randomUUID() + "-"+i);
> ctx.collect(userTag);
> }
> Thread.sleep(1);
> }
> }
> @Override
> public void cancel() {
> }
> });
> source.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
> .aggregate(new AggregateFunction,
> List>(){
> @Override
> public Map createAccumulator() {
> return new HashMap<>();
> }
> @Override
> public Map add(UserTag userTag, Map UserTag> stringUserTagMap) {
> stringUserTagMap.put(userTag.getUserTagName(),userTag);
> return stringUserTagMap;
> }
> @Override
> public List getResult(Map
> stringUserTagMap) {
> return new ArrayList<>(stringUserTagMap.values());
> }
> @Override
> public Map merge(Map acc1,
> Map acc2) {
> acc1.putAll(acc2);
> return acc1;
> }
> })
> .setParallelism(1)
> .name("NewUserTagAggregation_5s")
> .print().setParallelism(2);
> // execute program
> env.execute(); {code}
> !image-2022-07-15-14-19-21-200.png!
> !image-2022-07-15-14-19-41-678.png!
>Reporter: Yue Shang
>Priority: Major
> Attachments: image-2022-07-15-14-19-21-200.png,
> image-2022-07-15-14-19-41-678.png, image-2022-07-18-19-56-45-650.png,
> image-2022-07-18-19-59-18-872.png, image-2022-07-18-20-34-54-407.png,
> image-2022-07-18-20-36-23-021.png
>
>
> Rocksdb state backend is too slow when using AggregateFunction.
> just only supports 300 traffic per second use Map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] luoyuxia commented on pull request #20283: [FLINK-28122][docs-zh] Translate "User-defined Sources & Sinks" page into Chinese
luoyuxia commented on PR #20283: URL: https://github.com/apache/flink/pull/20283#issuecomment-1188507492 @PatrickRen Could you please help review when you are free? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on a diff in pull request #20103: [FLINK-28178][runtime-web] Show the delegated StateBackend and whethe…
zoltar9264 commented on code in PR #20103: URL: https://github.com/apache/flink/pull/20103#discussion_r923989858 ## flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/job-checkpoints.component.html: ## @@ -551,15 +551,15 @@ -Changelog state-backend -Enabled -Disabled +State Changelog Review Comment: Hi @Myasuka , how do you think about “Changelog of Flink State” ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on a diff in pull request #20103: [FLINK-28178][runtime-web] Show the delegated StateBackend and whethe…
zoltar9264 commented on code in PR #20103:
URL: https://github.com/apache/flink/pull/20103#discussion_r923988243
##
flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/checkpoints/CheckpointConfigInfo.java:
##
@@ -65,6 +65,8 @@ public class CheckpointConfigInfo implements ResponseBody {
public static final String FIELD_NAME_CHECKPOINTS_AFTER_TASKS_FINISH =
"checkpoints_after_tasks_finish";
+public static final String FIELD_NAME_STATE_CHANGELOG = "state_changelog";
Review Comment:
Hi @Myasuka , I change it to 'state_changelog_enabled'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #216: [FLINK-28482] num-sorted-run.stop-trigger introduced a unstable merging
JingsongLi commented on code in PR #216:
URL: https://github.com/apache/flink-table-store/pull/216#discussion_r923982350
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/CoreOptions.java:
##
@@ -429,6 +438,10 @@ public int maxFileNum() {
return options.get(COMPACTION_MAX_FILE_NUM);
}
+public Integer maxSortedRunNum() {
+return options.get(COMPACTION_MAX_SORTED_RUN_NUM);
+}
+
public boolean enableChangelogFile() {
Review Comment:
Maybe we should set the default value of `numLevels` to
`numSortedRunStopTrigger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-28590) flink on yarn checkpoint exception
[ https://issues.apache.org/jira/browse/FLINK-28590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangMu updated FLINK-28590: --- Component/s: Deployment / YARN Runtime / Checkpointing > flink on yarn checkpoint exception > -- > > Key: FLINK-28590 > URL: https://issues.apache.org/jira/browse/FLINK-28590 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN, Runtime / Checkpointing >Affects Versions: 1.15.0, 1.15.1 >Reporter: wangMu >Priority: Major > Attachments: image-2022-07-18-17-55-54-408.png > > > When I submit using flink on yarn on cdh6.2.1, the jobmanager log prints the > following exception： > !image-2022-07-18-17-55-54-408.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] xinbinhuang commented on pull request #20289: [FLINK-23633][FLIP-150][connector/common] HybridSource: Support dynamic stop position in FileSource
xinbinhuang commented on PR #20289: URL: https://github.com/apache/flink/pull/20289#issuecomment-1188474251 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on a diff in pull request #20275: [FLINK-26029][Runtime/Checkpointing] Generalize the checkpoint protocol of OperatorCoordinator
lindong28 commented on code in PR #20275:
URL: https://github.com/apache/flink/pull/20275#discussion_r923952394
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorEventDispatcherImpl.java:
##
@@ -82,6 +83,10 @@ public void registerEventHandler(OperatorID operator,
OperatorEventHandler handl
}
}
+Set getRegisteredOperators() {
Review Comment:
Could we explicitly mark methods as `private/protected/public` for
readability and consistency?
##
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##
@@ -245,7 +262,13 @@ public void notifyCheckpointAborted(long checkpointId) {
// checkpoint coordinator time thread.
// we can remove the delegation once the checkpoint coordinator runs
fully in the
// scheduler's main thread executor
-mainThreadExecutor.execute(() ->
coordinator.notifyCheckpointAborted(checkpointId));
+mainThreadExecutor.execute(
+() -> {
+subtaskGatewayMap
+.values()
+.forEach(x ->
x.openGatewayAndUnmarkCheckpoint(checkpointId));
+coordinator.notifyCheckpointAborted(checkpointId);
Review Comment:
This method does not call `eventValve.openValveAndUnmarkCheckpoint()` prior
to this PR. Could you explain why it is now necessary to call
`openGatewayAndUnmarkCheckpoint()`?
##
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##
@@ -277,29 +300,33 @@ private void checkpointCoordinatorInternal(
final long checkpointId, final CompletableFuture result) {
mainThreadExecutor.assertRunningInMainThread();
-final CompletableFuture coordinatorCheckpoint = new
CompletableFuture<>();
-
-FutureUtils.assertNoException(
-coordinatorCheckpoint.handleAsync(
-(success, failure) -> {
-if (failure != null) {
-result.completeExceptionally(failure);
-} else if (eventValve.tryShutValve(checkpointId)) {
-
completeCheckpointOnceEventsAreDone(checkpointId, result, success);
-} else {
-// if we cannot shut the valve, this means the
checkpoint
-// has been aborted before, so the future is
already
-// completed exceptionally. but we try to
complete it here
-// again, just in case, as a safety net.
-result.completeExceptionally(
-new FlinkException("Cannot shut event
valve"));
-}
-return null;
-},
-mainThreadExecutor));
-
try {
-eventValve.markForCheckpoint(checkpointId);
+subtaskGatewayMap.values().forEach(x ->
x.markForCheckpoint(checkpointId));
+
+final CompletableFuture coordinatorCheckpoint = new
CompletableFuture<>();
+
+coordinatorCheckpoint.whenComplete(
+(success, failure) -> {
+if (failure != null) {
+result.completeExceptionally(failure);
+} else {
+closeGateways(checkpointId, result);
+}
+});
+
+FutureUtils.assertNoException(
+coordinatorCheckpoint.handleAsync(
+(success, failure) -> {
+if (failure != null) {
+result.completeExceptionally(failure);
Review Comment:
This line seems to duplicate the logic at line 311 above.
##
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java:
##
@@ -308,6 +335,26 @@ private void checkpointCoordinatorInternal(
}
}
+private void closeGateways(final long checkpointId, final
CompletableFuture result) {
+Set closedGateways = new HashSet<>();
+for (CloseableSubtaskGateway gateway : subtaskGatewayMap.values()) {
+if (!gateway.tryCloseGateway(checkpointId)) {
+
closedGateways.forEach(CloseableSubtaskGateway::openGatewayAndUnmarkCheckpoint);
Review Comment:
Is there any case where `closedGateways` is not empty at this point? If no,
could it be better to throw IllegalStateException if `closedGateways()` is not
empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
[GitHub] [flink] flinkbot commented on pull request #20300: [WIP][FLINK-28597][state] Discard non-initial checkpoints without a delay
flinkbot commented on PR #20300: URL: https://github.com/apache/flink/pull/20300#issuecomment-1188454872 ## CI report: * acc4f2e262947cdda78eea45aaa1f1bca976e16b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28597) Empty checkpoint folders not deleted on job cancellation if their shared state is still in use
[ https://issues.apache.org/jira/browse/FLINK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28597: --- Labels: pull-request-available (was: ) > Empty checkpoint folders not deleted on job cancellation if their shared > state is still in use > -- > > Key: FLINK-28597 > URL: https://issues.apache.org/jira/browse/FLINK-28597 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.16.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > After FLINK-25872, SharedStateRegistry registers all state handles, including > private ones. > Once the state isn't use AND the checkpoint is subsumed, it will actually be > discarded. > This is done to prevent premature deletion when recovering in CLAIM mode: > 1. RocksDB native savepoint folder (shared state is stored in chk-xx folder > so it might fail the deletion) > 2. Initial non-changelog checkpoint when switching to changelog-based > checkpoints (private state of the initial checkpoint might be included into > later checkpoints and its deletion would invalidate them) > Additionally, checkpoint folders are not deleted for a longer time which > might be confusing. > In case of a crash, more folders will remain. > cc: [~Yanfei Lei], [~ym] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] rkhachatryan opened a new pull request, #20300: [WIP][FLINK-28597][state] Discard non-initial checkpoints without a delay
rkhachatryan opened a new pull request, #20300: URL: https://github.com/apache/flink/pull/20300 ## What is the purpose of the change ``` Newly created checkpoints can be discarded right after subsumption. But the initial checkpoint needs to be kept until all of its private AND shared state is not in use. Keeping any checkpoint for longer leaves its folder undeleted on job cancellation (and also on crash or JM failover). This change limits createdByCheckpointID tracking to only initial checkpoints. ``` ## Verifying this change **TBD** ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28597) Empty checkpoint folders not deleted on job cancellation if their shared state is still in use
[ https://issues.apache.org/jira/browse/FLINK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan updated FLINK-28597: -- Description: After FLINK-25872, SharedStateRegistry registers all state handles, including private ones. Once the state isn't use AND the checkpoint is subsumed, it will actually be discarded. This is done to prevent premature deletion when recovering in CLAIM mode: 1. RocksDB native savepoint folder (shared state is stored in chk-xx folder so it might fail the deletion) 2. Initial non-changelog checkpoint when switching to changelog-based checkpoints (private state of the initial checkpoint might be included into later checkpoints and its deletion would invalidate them) Additionally, checkpoint folders are not deleted for a longer time which might be confusing. In case of a crash, more folders will remain. cc: [~Yanfei Lei], [~ym] was: After FLINK-25872, SharedStateRegistry registers all state handles, including private ones. Once the state isn't use AND the checkpoint is subsumed, it will actually be discarded. This is done to prevent premature deletion when recovering in CLAIM mode: 1. RocksDB native savepoint folder (shared state is stored in chk-xx folder so it might fail the deletion) 2. Initial non-changelog checkpoint when switching to changelog-based checkpoints (private state of the initial checkpoint might be included into later checkpoints and its deletion would invalidate them) Additionally, checkpoint folders are not deleted for a longer time which might be confusing. In case of a crash, more folders will remain. cc: [~Yanfei Lei], [~ym] > Empty checkpoint folders not deleted on job cancellation if their shared > state is still in use > -- > > Key: FLINK-28597 > URL: https://issues.apache.org/jira/browse/FLINK-28597 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.16.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Fix For: 1.16.0 > > > After FLINK-25872, SharedStateRegistry registers all state handles, including > private ones. > Once the state isn't use AND the checkpoint is subsumed, it will actually be > discarded. > This is done to prevent premature deletion when recovering in CLAIM mode: > 1. RocksDB native savepoint folder (shared state is stored in chk-xx folder > so it might fail the deletion) > 2. Initial non-changelog checkpoint when switching to changelog-based > checkpoints (private state of the initial checkpoint might be included into > later checkpoints and its deletion would invalidate them) > Additionally, checkpoint folders are not deleted for a longer time which > might be confusing. > In case of a crash, more folders will remain. > cc: [~Yanfei Lei], [~ym] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28597) Empty checkpoint folders not deleted on job cancellation if their shared state is still in use
Roman Khachatryan created FLINK-28597: - Summary: Empty checkpoint folders not deleted on job cancellation if their shared state is still in use Key: FLINK-28597 URL: https://issues.apache.org/jira/browse/FLINK-28597 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing Affects Versions: 1.16.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan Fix For: 1.16.0 After FLINK-25872, SharedStateRegistry registers all state handles, including private ones. Once the state isn't use AND the checkpoint is subsumed, it will actually be discarded. This is done to prevent premature deletion when recovering in CLAIM mode: 1. RocksDB native savepoint folder (shared state is stored in chk-xx folder so it might fail the deletion) 2. Initial non-changelog checkpoint when switching to changelog-based checkpoints (private state of the initial checkpoint might be included into later checkpoints and its deletion would invalidate them) Additionally, checkpoint folders are not deleted for a longer time which might be confusing. In case of a crash, more folders will remain. cc: [~Yanfei Lei], [~ym] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27701) HashMapStateBackendWindowITCase. testAggregateWindowStateReader failed with Not all required tasks are currently running
[
https://issues.apache.org/jira/browse/FLINK-27701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-27701:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> HashMapStateBackendWindowITCase. testAggregateWindowStateReader failed with
> Not all required tasks are currently running
> -
>
> Key: FLINK-27701
> URL: https://issues.apache.org/jira/browse/FLINK-27701
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: stale-major, test-stability
>
> {code:java}
> 2022-05-19T11:04:27.4331524Z May 19 11:04:27 [ERROR] Tests run: 9, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 29.034 s <<< FAILURE! - in
> org.apache.flink.state.api.HashMapStateBackendWindowITCase
> 2022-05-19T11:04:27.4333055Z May 19 11:04:27 [ERROR]
> org.apache.flink.state.api.HashMapStateBackendWindowITCase.testAggregateWindowStateReader
> Time elapsed: 0.105 s <<< ERROR!
> 2022-05-19T11:04:27.4333765Z May 19 11:04:27 java.lang.RuntimeException:
> Failed to take savepoint
> 2022-05-19T11:04:27.4334405Z May 19 11:04:27 at
> org.apache.flink.state.api.utils.SavepointTestBase.takeSavepoint(SavepointTestBase.java:68)
> 2022-05-19T11:04:27.4335375Z May 19 11:04:27 at
> org.apache.flink.state.api.SavepointWindowReaderITCase.testAggregateWindowStateReader(SavepointWindowReaderITCase.java:149)
> 2022-05-19T11:04:27.4338106Z May 19 11:04:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-05-19T11:04:27.4339140Z May 19 11:04:27 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-05-19T11:04:27.4339854Z May 19 11:04:27 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-05-19T11:04:27.4340560Z May 19 11:04:27 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-05-19T11:04:27.4341746Z May 19 11:04:27 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-05-19T11:04:27.4342797Z May 19 11:04:27 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-05-19T11:04:27.4343717Z May 19 11:04:27 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-05-19T11:04:27.4344909Z May 19 11:04:27 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-05-19T11:04:27.4345993Z May 19 11:04:27 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-05-19T11:04:27.4346981Z May 19 11:04:27 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-05-19T11:04:27.4347590Z May 19 11:04:27 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-05-19T11:04:27.4348200Z May 19 11:04:27 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-05-19T11:04:27.4348856Z May 19 11:04:27 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-05-19T11:04:27.4349484Z May 19 11:04:27 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-05-19T11:04:27.4350118Z May 19 11:04:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-05-19T11:04:27.4350899Z May 19 11:04:27 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-05-19T11:04:27.4352057Z May 19 11:04:27 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-05-19T11:04:27.4353154Z May 19 11:04:27 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-05-19T11:04:27.4354153Z May 19 11:04:27 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-05-19T11:04:27.4354936Z May 19 11:04:27 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-05-19T11:04:27.4355560Z May 19 11:04:27 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-05-19T11:04:27.4356167Z May 19 11:04:27 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-05-19T11:04:27.4356775Z May 19
[jira] [Updated] (FLINK-24434) PyFlink YARN per-job on Docker test fails on Azure
[
https://issues.apache.org/jira/browse/FLINK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-24434:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> PyFlink YARN per-job on Docker test fails on Azure
> --
>
> Key: FLINK-24434
> URL: https://issues.apache.org/jira/browse/FLINK-24434
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Deployment / YARN
>Affects Versions: 1.14.4, 1.15.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24669=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=070ff179-953e-5bda-71fa-d6599415701c=23186
> {code}
> Sep 30 18:20:22 Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=mapred, access=WRITE,
> inode="/":hdfs:hadoop:drwxr-xr-x
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:318)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:219)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1663)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1647)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1606)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3039)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1079)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:652)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> Sep 30 18:20:22 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:850)
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:793)
> Sep 30 18:20:22 at java.security.AccessController.doPrivileged(Native
> Method)
> Sep 30 18:20:22 at javax.security.auth.Subject.doAs(Subject.java:422)
> Sep 30 18:20:22 at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2489)
> Sep 30 18:20:22
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
> Sep 30 18:20:22 at org.apache.hadoop.ipc.Client.call(Client.java:1435)
> Sep 30 18:20:22 at org.apache.hadoop.ipc.Client.call(Client.java:1345)
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> Sep 30 18:20:22 at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> Sep 30 18:20:22 at com.sun.proxy.$Proxy12.mkdirs(Unknown Source)
> Sep 30 18:20:22 at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:583)
> Sep 30 18:20:22 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Sep 30 18:20:22 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Sep 30 18:20:22 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Sep 30 18:20:22 at java.lang.reflect.Method.invoke(Method.java:498)
> Sep 30 18:20:22 at
>
[jira] [Updated] (FLINK-25119) The name of freeSlots parameter in SlotSharingSlotAllocator#determineParallelism method is named incorrectly.
[ https://issues.apache.org/jira/browse/FLINK-25119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-25119: --- Labels: auto-deprioritized-minor pull-request-available (was: pull-request-available stale-minor) Priority: Not a Priority (was: Minor) This issue was labeled "stale-minor" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Minor, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > The name of freeSlots parameter in > SlotSharingSlotAllocator#determineParallelism method is named incorrectly. > - > > Key: FLINK-25119 > URL: https://issues.apache.org/jira/browse/FLINK-25119 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.13.0, 1.14.0, 1.13.1, 1.13.2, 1.13.3 >Reporter: Cassie >Priority: Not a Priority > Labels: auto-deprioritized-minor, pull-request-available > Attachments: image-2021-12-01-11-18-34-572.png > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > determineParallism方法中的第二个参数freeSlots，即为在slotpool中空闲的slot，但实际使用时，有的调用方传入的是slotpool中all > slot(allocated slot and free slot)，有的调用方传入的是slotpool中free > slot，因此参数名直接定义成freeSlots，不够准确，影响对determineParallism逻辑的理解。 > !image-2021-12-01-11-18-34-572.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-25962) Flink generated Avro schemas can't be parsed using Python
[
https://issues.apache.org/jira/browse/FLINK-25962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-25962:
---
Labels: auto-deprioritized-major pull-request-available stale-major (was:
auto-deprioritized-major pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Flink generated Avro schemas can't be parsed using Python
> -
>
> Key: FLINK-25962
> URL: https://issues.apache.org/jira/browse/FLINK-25962
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.14.3
>Reporter: Ryan Skraba
>Priority: Major
> Labels: auto-deprioritized-major, pull-request-available,
> stale-major
>
> Flink currently generates Avro schemas as records with the top-level name
> {{"record"}}
> Unfortunately, there is some inconsistency between Avro implementations in
> different languages that may prevent this record from being read, notably
> Python, which generates the error:
> *avro.schema.SchemaParseException: record is a reserved type name*
> (See the comment on FLINK-18096 for the full stack trace).
> The Java SDK accepts this name, and there's an [ongoing
> discussion|https://lists.apache.org/thread/0wmgyx6z69gy07lvj9ndko75752b8cn2]
> about what the expected behaviour should be. This should be clarified and
> fixed in Avro, of course.
> Regardless of the resolution, the best practice (which is used almost
> everywhere else in the Flink codebase) is to explicitly specify a top-level
> namespace for an Avro record. We should use a default like:
> {{{}org.apache.flink.avro.generated{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-27695) KafkaTransactionLogITCase failed on azure due to Could not find a valid Docker environment
[
https://issues.apache.org/jira/browse/FLINK-27695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-27695:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> KafkaTransactionLogITCase failed on azure due to Could not find a valid
> Docker environment
> --
>
> Key: FLINK-27695
> URL: https://issues.apache.org/jira/browse/FLINK-27695
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors / Kafka
>Affects Versions: 1.14.4
>Reporter: Huang Xingbo
>Priority: Major
> Labels: stale-major, test-stability
>
> {code:java}
> 022-05-19T02:04:23.9190098Z May 19 02:04:23 [ERROR] Tests run: 1, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 7.404 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase
> 2022-05-19T02:04:23.9191182Z May 19 02:04:23 [ERROR]
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase Time
> elapsed: 7.404 s <<< ERROR!
> 2022-05-19T02:04:23.9192250Z May 19 02:04:23 java.lang.IllegalStateException:
> Could not find a valid Docker environment. Please see logs and check
> configuration
> 2022-05-19T02:04:23.9193144Z May 19 02:04:23 at
> org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$4(DockerClientProviderStrategy.java:156)
> 2022-05-19T02:04:23.9194653Z May 19 02:04:23 at
> java.util.Optional.orElseThrow(Optional.java:290)
> 2022-05-19T02:04:23.9196179Z May 19 02:04:23 at
> org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:148)
> 2022-05-19T02:04:23.9197995Z May 19 02:04:23 at
> org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:146)
> 2022-05-19T02:04:23.9199486Z May 19 02:04:23 at
> org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:188)
> 2022-05-19T02:04:23.9200666Z May 19 02:04:23 at
> org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:101)
> 2022-05-19T02:04:23.9202109Z May 19 02:04:23 at
> com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:107)
> 2022-05-19T02:04:23.9203065Z May 19 02:04:23 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:316)
> 2022-05-19T02:04:23.9204641Z May 19 02:04:23 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1066)
> 2022-05-19T02:04:23.9205765Z May 19 02:04:23 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> 2022-05-19T02:04:23.9206568Z May 19 02:04:23 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2022-05-19T02:04:23.9207497Z May 19 02:04:23 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-05-19T02:04:23.9208246Z May 19 02:04:23 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-05-19T02:04:23.9208887Z May 19 02:04:23 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2022-05-19T02:04:23.9209691Z May 19 02:04:23 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2022-05-19T02:04:23.9210490Z May 19 02:04:23 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2022-05-19T02:04:23.9211246Z May 19 02:04:23 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2022-05-19T02:04:23.9211989Z May 19 02:04:23 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2022-05-19T02:04:23.9212682Z May 19 02:04:23 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2022-05-19T02:04:23.9213391Z May 19 02:04:23 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2022-05-19T02:04:23.9214305Z May 19 02:04:23 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 2022-05-19T02:04:23.9215044Z May 19 02:04:23 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> 2022-05-19T02:04:23.9215809Z May 19 02:04:23 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> 2022-05-19T02:04:23.9216576Z May 19 02:04:23 at
>
[jira] [Commented] (FLINK-28596) Support writing arrays to postgres array columns in Flink SQL JDBC connector
[ https://issues.apache.org/jira/browse/FLINK-28596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568218#comment-17568218 ] Bobby Richard commented on FLINK-28596: --- Looking at the code it looks like the PostgresRowConverter supports reading arrays from postgres, but not writing them. Looks simple enough to add by calling setArray on the preparedstatement from the PostgresRowConverter. Any thoughts [~martijnvisser] ? > Support writing arrays to postgres array columns in Flink SQL JDBC connector > > > Key: FLINK-28596 > URL: https://issues.apache.org/jira/browse/FLINK-28596 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.15.0 >Reporter: Bobby Richard >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28596) Support writing arrays to postgres array columns in Flink SQL JDBC connector
Bobby Richard created FLINK-28596: - Summary: Support writing arrays to postgres array columns in Flink SQL JDBC connector Key: FLINK-28596 URL: https://issues.apache.org/jira/browse/FLINK-28596 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Affects Versions: 1.15.0 Reporter: Bobby Richard -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28595) KafkaSource should not read metadata of unmatched regex topics
Afzal Mazhar created FLINK-28595: Summary: KafkaSource should not read metadata of unmatched regex topics Key: FLINK-28595 URL: https://issues.apache.org/jira/browse/FLINK-28595 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Reporter: Afzal Mazhar When we use a regex to subscribe to topics, the current connector gets a list of all topics, then runs describe against all of them, and finally filters by the regex pattern. This is not performant, as well as could possibly trigger audit alarms against sensitive topics that do not match the regex. Proposed fix: move the regex filtering from the TopicPatternSubscriber's set() down into KafkaSubscriberUtils getAllTopicMetadata(). Get the list of topics, filter by pattern (if any), then get metadata. Create appropriate tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] gyfora merged pull request #321: [FLINK-27714][hotfix] Update NOTICE file
gyfora merged PR #321: URL: https://github.com/apache/flink-kubernetes-operator/pull/321 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi opened a new pull request, #321: [FLINK-27914][hotfix] Update NOTICE file
morhidi opened a new pull request, #321: URL: https://github.com/apache/flink-kubernetes-operator/pull/321 ## What is the purpose of the change Update NOTICE file with dependency changes from [FLINK-27914] ## Brief change log ``` -- io.javaoperatorsdk:operator-framework-core:2.1.4 -- io.javaoperatorsdk:operator-framework:2.1.4 +- io.javaoperatorsdk:operator-framework-core:3.0.3 +- io.javaoperatorsdk:operator-framework-framework-core:0.2.0 +- io.javaoperatorsdk:operator-framework:3.0.3 ``` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lindong28 commented on a diff in pull request #20275: [FLINK-26029][Runtime/Checkpointing] Generalize the checkpoint protocol of OperatorCoordinator
lindong28 commented on code in PR #20275:
URL: https://github.com/apache/flink/pull/20275#discussion_r923461372
##
flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/CloseableSubtaskGateway.java:
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.operators.coordination;
+
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
+import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.util.concurrent.FutureUtils;
+
+import javax.annotation.Nullable;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * An {@link
org.apache.flink.runtime.operators.coordination.OperatorCoordinator.SubtaskGateway}
+ * that can be temporarily closed, blocking events from going through,
buffering them, and releasing
+ * them later. It is used for "alignment" of operator event streams with
checkpoint barrier
+ * injection, similar to how the input channels are aligned during a common
checkpoint.
+ *
+ * This class is NOT thread safe, but assumed to be used in a single
threaded context. To guard
+ * that, one can register a "main thread executor" (as used by the mailbox
components like RPC
+ * components) via {@link
#setMainThreadExecutorForValidation(ComponentMainThreadExecutor)}.
+ */
+class CloseableSubtaskGateway implements OperatorCoordinator.SubtaskGateway {
+
+/** The wrapped gateway that actually performs the event-sending
operation. */
+private final OperatorCoordinator.SubtaskGateway innerGateway;
+
+private static final long NO_CHECKPOINT = Long.MIN_VALUE;
+
+private final List blockedEvents = new ArrayList<>();
+
+private long currentCheckpointId;
+
+private long lastCheckpointId;
+
+private boolean isClosed;
+
+@Nullable private ComponentMainThreadExecutor mainThreadExecutor;
+
+CloseableSubtaskGateway(OperatorCoordinator.SubtaskGateway innerGateway) {
+this.innerGateway = innerGateway;
+this.currentCheckpointId = NO_CHECKPOINT;
+this.lastCheckpointId = Long.MIN_VALUE;
+}
+
+void setMainThreadExecutorForValidation(ComponentMainThreadExecutor
mainThreadExecutor) {
+this.mainThreadExecutor = mainThreadExecutor;
+}
+
+/**
+ * Marks the gateway for the next checkpoint. This remembers the
checkpoint ID and will only
+ * allow closing the gateway for this specific checkpoint.
+ *
+ * This is the gateway's mechanism to detect situations where multiple
coordinator
+ * checkpoints would be attempted overlapping, which is currently not
supported (the gateway
+ * doesn't keep a list of events blocked per checkpoint). It also helps to
identify situations
+ * where the checkpoint was aborted even before the gateway was closed (by
finding out that the
+ * {@code currentCheckpointId} was already reset to {@code NO_CHECKPOINT}.
+ */
+void markForCheckpoint(long checkpointId) {
+checkRunsInMainThread();
+
+if (currentCheckpointId != NO_CHECKPOINT && currentCheckpointId !=
checkpointId) {
+throw new IllegalStateException(
+String.format(
+"Cannot mark for checkpoint %d, already marked for
checkpoint %d",
+checkpointId, currentCheckpointId));
+}
+if (checkpointId > lastCheckpointId) {
+currentCheckpointId = checkpointId;
+lastCheckpointId = checkpointId;
+} else {
+throw new IllegalStateException(
+String.format(
+"Regressing checkpoint IDs. Previous checkpointId
= %d, new checkpointId = %d",
+lastCheckpointId, checkpointId));
+}
+}
+
+/**
+ * Closes the gateway. All events sent through this gateway are blocked
until the gateway is
+ * re-opened. If the gateway is already closed, this does nothing.
+ *
+ * @return True if the gateway is closed, false if the checkpointId is
incorrect.
+ */
+boolean
[GitHub] [flink] maosuhan commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
maosuhan commented on PR #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-1187643038 protobuf can also work in sql client now  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] maosuhan commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
maosuhan commented on PR #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-1187639170 @libenchao Thanks for your detailed code review. It is really helpful. There are still some unresolved conversations left, could you take a look them again? Does the code look good for you now? I will start to draft a documentation once all the code is ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] maosuhan commented on a diff in pull request #14376: [FLINK-18202][PB format] New Format of protobuf
maosuhan commented on code in PR #14376:
URL: https://github.com/apache/flink/pull/14376#discussion_r923499714
##
flink-formats/flink-protobuf/pom.xml:
##
@@ -0,0 +1,146 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-formats
+ 1.16-SNAPSHOT
+
+
+ flink-protobuf
+ Flink : Formats : Protobuf
+
+ jar
+
+
+
+ 3.0.11
+
+
+
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+ provided
+
+
+
+ org.apache.flink
+ flink-table-common
+ ${project.version}
+ provided
+
+
+
+ com.google.protobuf
+ protobuf-java
+ ${protoc.version}
+
+
+
+ org.codehaus.janino
+ janino
+
+ ${janino.version}
+ provided
+
+
+
+
+ org.apache.flink
+
flink-table-planner_${scala.binary.version}
+ ${project.version}
+ test
+
+
+
+ org.apache.flink
+
flink-table-planner_${scala.binary.version}
+ ${project.version}
+ test
+ test-jar
+
+
+
+
+ org.apache.flink
+ flink-test-utils
+ ${project.version}
+ test
+
+
+
+
+
+
+ com.github.os72
+ protoc-jar-maven-plugin
+ 3.11.4
+
+
+ generate-sources
+
+ run
+
+
+
${protoc.version}
+
+
src/test/proto
+
+
+
+
java
+
none
+
target/test-proto-sources
+
+
+
+
+
+
+
+ org.codehaus.mojo
+
build-helper-maven-plugin
+ 3.0.0
+
+
+ generate-sources
+
+ add-source
+
+
+
+
target/test-proto-sources
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24932) Frocksdb cannot run on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568049#comment-17568049
]
Sebastian Mattheis edited comment on FLINK-24932 at 7/18/22 3:01 PM:
-
[~yunta] Building FRocksDB on M1 silicon is not the problem. However, building
the binaries for PPC64 poses a problem as Travis CI pipeline failed repeatedly
due timeouts where various attempts to avoid or work around the timeout did not
work: [https://app.travis-ci.com/github/smattheis/frocksdb/builds] (I have no
other PPC64 system to use and QEMU emulation did not work on neither on Linux
x86_64 and nor on Mac OS x86_64 or arm64).
Other than that, is anything wrong with the branch that I created
[https://github.com/smattheis/frocksdb/tree/FRocksDB-6.20.3-M1-fix] ? It would
have been helpful to get some review comments instead of creating a new branch.
Also I have been working on this issue on side out of courtesy, as this issue
has no priority for my employer and I have actually no time to allocate for it.
As you added some tests that I didn't add and you have some more urgency, I
would suggest it makes more sense that you continue with your branch on this
issue where doing cross-platform builds and following release process remains.
[~yunta] Could you please assign the issue to yourself?
was (Author: JIRAUSER284806):
[~yunta] Building FRocksDB on M1 silicon is not the problem. However, building
the binaries for PPC64 poses a problem as Travis CI pipeline failed repeatedly
due timeouts where various attempts to avoid or work around the timeout did not
work: [https://app.travis-ci.com/github/smattheis/frocksdb/builds] (I have no
other PPC64 system to use and QEMU emulation did not work on neither on Linux
x86_64 and nor on Mac OS x86_64 or arm64).
Other than that, is anything wrong with the branch that I created
[https://github.com/smattheis/frocksdb/tree/FRocksDB-6.20.3-M1-fix] ? It would
have been helpful to get some review comments instead of creating a new branch.
Also I have been working on this issue on side out of courtesy, as this issue
has no priority for my employer and I have actually no time to allocate for it.
As you added some tests that I didn't add and have some more urgency, I would
suggest it makes more sense that you continue with your branch on this issue
where doing cross-platform builds and following release process remains.
[~yunta] Could you please assign the issue to yourself?
> Frocksdb cannot run on Apple M1
> ---
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Sebastian Mattheis
>Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by:
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568049#comment-17568049
]
Sebastian Mattheis commented on FLINK-24932:
[~yunta] Building FRocksDB on M1 silicon is not the problem. However, building
the binaries for PPC64 poses a problem as Travis CI pipeline failed repeatedly
due timeouts where various attempts to avoid or work around the timeout did not
work: [https://app.travis-ci.com/github/smattheis/frocksdb/builds] (I have no
other PPC64 system to use and QEMU emulation did not work on neither on Linux
x86_64 and nor on Mac OS x86_64 or arm64).
Other than that, is anything wrong with the branch that I created
[https://github.com/smattheis/frocksdb/tree/FRocksDB-6.20.3-M1-fix] ? It would
have been helpful to get some review comments instead of creating a new branch.
Also I have been working on this issue on side out of courtesy, as this issue
has no priority for my employer and I have actually no time to allocate for it.
As you added some tests that I didn't add and have some more urgency, I would
suggest it makes more sense that you continue with your branch on this issue
where doing cross-platform builds and following release process remains.
[~yunta] Could you please assign the issue to yourself?
> Frocksdb cannot run on Apple M1
> ---
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Sebastian Mattheis
>Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
[GitHub] [flink] zstraw commented on pull request #17956: [FLINK-18779][table sql/planner]Support the SupportsFilterPushDown for LookupTableSource
zstraw commented on PR #17956: URL: https://github.com/apache/flink/pull/17956#issuecomment-1187595881 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] 1996fanrui commented on pull request #20233: [FLINK-28474][checkpoint] Simplify the ChannelStateWriterImpl due to concurrent unaligned checkpoint isn't supported
1996fanrui commented on PR #20233: URL: https://github.com/apache/flink/pull/20233#issuecomment-1187589584 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] WencongLiu commented on pull request #20236: [FLINK-27769][sql-gateway]Introduce the REST endpoint framework
WencongLiu commented on PR #20236: URL: https://github.com/apache/flink/pull/20236#issuecomment-1187588579 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-12655) SlotCountExceedingParallelismTest failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-12655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568037#comment-17568037
]
Wencong Liu commented on FLINK-12655:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=38355=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8]
> SlotCountExceedingParallelismTest failed on Travis
> --
>
> Key: FLINK-12655
> URL: https://issues.apache.org/jira/browse/FLINK-12655
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: test-stability
>
> https://travis-ci.org/apache/flink/jobs/538166458
> {code}
> akka.actor.RobustActorSystemTest
> 12:42:34.415 [INFO]
> 12:42:34.415 [INFO] Results:
> 12:42:34.415 [INFO]
> 12:42:34.415 [ERROR] Errors:
> 12:42:34.420 [ERROR]
> SlotCountExceedingParallelismTest.testNoSlotSharingAndBlockingResultBoth:91->submitJobGraphAndWait:97
> » JobExecution
> 12:42:34.420 [ERROR]
> SlotCountExceedingParallelismTest.testNoSlotSharingAndBlockingResultReceiver:84->submitJobGraphAndWait:97
> » JobExecution
> 12:42:34.420 [ERROR]
> SlotCountExceedingParallelismTest.testNoSlotSharingAndBlockingResultSender:77->submitJobGraphAndWait:97
> »
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] 1996fanrui commented on pull request #20233: [FLINK-28474][checkpoint] Simplify the ChannelStateWriterImpl due to concurrent unaligned checkpoint isn't supported
1996fanrui commented on PR #20233: URL: https://github.com/apache/flink/pull/20233#issuecomment-1187561410 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28518) Exception: AssertionError: Cannot add expression of different type to set in sub-query with ROW type
[
https://issues.apache.org/jira/browse/FLINK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568023#comment-17568023
]
Martijn Visser commented on FLINK-28518:
[~KoylubaevNT] A Calcite upgrade is not an easy task; it's currently tracked
under FLINK-28518
> Exception: AssertionError: Cannot add expression of different type to set in
> sub-query with ROW type
>
>
> Key: FLINK-28518
> URL: https://issues.apache.org/jira/browse/FLINK-28518
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.1
>Reporter: Koylubaev Nikita
>Priority: Major
> Attachments: SubQueryRowTypeTest.scala, test.sql
>
>
> All scripts is attached to file: test.sql
> Create 2 tables:
>
> {code:java}
> SET
> 'sql-client.execution.result-mode' = 'tableau';
> SET
> 'execution.runtime-mode' = 'batch';
> SET
> 'sql-client.execution.mode' = 'streaming';
> SET
> 'parallelism.default' = '8';
> SET
> 'table.dml-sync' = 'true';
> CREATE
> TEMPORARY TABLE fl (
> `id` INT,
> `name` STRING)
> WITH (
> 'connector' = 'faker',
> 'number-of-rows' = '10',
> 'rows-per-second' = '1',
> 'fields.id.expression' = '#{number.numberBetween ''0'',''10''}',
> 'fields.name.expression' = '#{superhero.name}');
> CREATE
> TEMPORARY TABLE application (
> `id` INT,
> `fl_id` INT,
> `num` INT,
> `db` DOUBLE)
> WITH (
> 'connector' = 'faker',
> 'number-of-rows' = '100',
> 'rows-per-second' = '100',
> 'fields.id.expression' = '#{number.numberBetween ''0'',''100''}',
> 'fields.fl_id.expression' = '#{number.numberBetween ''0'',''10''}',
> 'fields.num.expression' = '#{number.numberBetween
> ''-2147483648'',''2147483647''}',
> 'fields.db.expression' = '#{number.randomDouble ''3'',''-1000'',''1000''}');
> {code}
> The next SQL throw exception:
> {code:java}
> select fl.name,
>(select (COLLECT(application.num), COLLECT(application.db))
> from application
> where fl.id = application.fl_id)
> from fl;{code}
> Error stack trace is (I marked what is different in type: it's just NOT NULL):
>
>
> {code:java}
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(INTEGER id, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" name, RecordType(INTEGER MULTISET EXPR$0, DOUBLE MULTISET EXPR$1)
> $f0) NOT NULL
> expression type is RecordType(INTEGER id, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" name, RecordType(INTEGER MULTISET EXPR$0, DOUBLE MULTISET EXPR$1)
> NOT NULL $f0) NOT NULL
> set is
> rel#129:LogicalCorrelate.NONE.any.[](left=HepRelVertex#119,right=HepRelVertex#128,correlation=$cor0,joinType=left,requiredColumns={0})
> expression is LogicalProject(id=[$0], name=[$1], $f0=[ROW($2, $3)])
> LogicalCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{0}])
> LogicalTableScan(table=[[default_catalog, default_database, fl]])
> LogicalAggregate(group=[{}], agg#0=[COLLECT($0)], agg#1=[COLLECT($1)])
> LogicalProject(num=[$2], db=[$3])
> LogicalFilter(condition=[=($cor0.id, $1)])
> LogicalTableScan(table=[[default_catalog, default_database,
> application]])
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28518) Exception: AssertionError: Cannot add expression of different type to set in sub-query with ROW type
[
https://issues.apache.org/jira/browse/FLINK-28518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568014#comment-17568014
]
Koylubaev Nikita commented on FLINK-28518:
--
[~martijnvisser] I found that error was in calcite. And they resolved it in
1.30.0 version.
Is it possible to upgrade version calcite in flink project from 1.26.0 to
1.30.0?
> Exception: AssertionError: Cannot add expression of different type to set in
> sub-query with ROW type
>
>
> Key: FLINK-28518
> URL: https://issues.apache.org/jira/browse/FLINK-28518
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.1
>Reporter: Koylubaev Nikita
>Priority: Major
> Attachments: SubQueryRowTypeTest.scala, test.sql
>
>
> All scripts is attached to file: test.sql
> Create 2 tables:
>
> {code:java}
> SET
> 'sql-client.execution.result-mode' = 'tableau';
> SET
> 'execution.runtime-mode' = 'batch';
> SET
> 'sql-client.execution.mode' = 'streaming';
> SET
> 'parallelism.default' = '8';
> SET
> 'table.dml-sync' = 'true';
> CREATE
> TEMPORARY TABLE fl (
> `id` INT,
> `name` STRING)
> WITH (
> 'connector' = 'faker',
> 'number-of-rows' = '10',
> 'rows-per-second' = '1',
> 'fields.id.expression' = '#{number.numberBetween ''0'',''10''}',
> 'fields.name.expression' = '#{superhero.name}');
> CREATE
> TEMPORARY TABLE application (
> `id` INT,
> `fl_id` INT,
> `num` INT,
> `db` DOUBLE)
> WITH (
> 'connector' = 'faker',
> 'number-of-rows' = '100',
> 'rows-per-second' = '100',
> 'fields.id.expression' = '#{number.numberBetween ''0'',''100''}',
> 'fields.fl_id.expression' = '#{number.numberBetween ''0'',''10''}',
> 'fields.num.expression' = '#{number.numberBetween
> ''-2147483648'',''2147483647''}',
> 'fields.db.expression' = '#{number.randomDouble ''3'',''-1000'',''1000''}');
> {code}
> The next SQL throw exception:
> {code:java}
> select fl.name,
>(select (COLLECT(application.num), COLLECT(application.db))
> from application
> where fl.id = application.fl_id)
> from fl;{code}
> Error stack trace is (I marked what is different in type: it's just NOT NULL):
>
>
> {code:java}
> [ERROR] Could not execute SQL statement. Reason:
> java.lang.AssertionError: Cannot add expression of different type to set:
> set type is RecordType(INTEGER id, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" name, RecordType(INTEGER MULTISET EXPR$0, DOUBLE MULTISET EXPR$1)
> $f0) NOT NULL
> expression type is RecordType(INTEGER id, VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" name, RecordType(INTEGER MULTISET EXPR$0, DOUBLE MULTISET EXPR$1)
> NOT NULL $f0) NOT NULL
> set is
> rel#129:LogicalCorrelate.NONE.any.[](left=HepRelVertex#119,right=HepRelVertex#128,correlation=$cor0,joinType=left,requiredColumns={0})
> expression is LogicalProject(id=[$0], name=[$1], $f0=[ROW($2, $3)])
> LogicalCorrelate(correlation=[$cor0], joinType=[left],
> requiredColumns=[{0}])
> LogicalTableScan(table=[[default_catalog, default_database, fl]])
> LogicalAggregate(group=[{}], agg#0=[COLLECT($0)], agg#1=[COLLECT($1)])
> LogicalProject(num=[$2], db=[$3])
> LogicalFilter(condition=[=($cor0.id, $1)])
> LogicalTableScan(table=[[default_catalog, default_database,
> application]])
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568010#comment-17568010
]
Yun Tang commented on FLINK-24932:
--
BTW, I noticed that the CI on circle seems disabled on ververica/frocksdb,
[~smattheis] or [~ym] could have a look at that problem, and we can leverage
existing tests to ensure our codebase is correct.
> Frocksdb cannot run on Apple M1
> ---
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Sebastian Mattheis
>Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
>
[jira] [Commented] (FLINK-24932) Frocksdb cannot run on Apple M1
[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568009#comment-17568009
]
Yun Tang commented on FLINK-24932:
--
[~smattheis] In case of that you are not so familiar with FRocksDB, I created a
PR https://github.com/ververica/frocksdb/pull/50 to enable building FRocksJava
on apple silicon. I think you can refer to that PR to continue the work, and I
could get the binaries on my local M1-MacBook.
> Frocksdb cannot run on Apple M1
> ---
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Sebastian Mattheis
>Priority: Minor
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
>
[GitHub] [flink] 1996fanrui commented on a diff in pull request #20233: [FLINK-28474][checkpoint] Simplify the ChannelStateWriterImpl due to concurrent unaligned checkpoint isn't supported
1996fanrui commented on code in PR #20233:
URL: https://github.com/apache/flink/pull/20233#discussion_r923351868
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriterImpl.java:
##
@@ -60,89 +63,78 @@
public class ChannelStateWriterImpl implements ChannelStateWriter {
private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriterImpl.class);
-private static final int DEFAULT_MAX_CHECKPOINTS =
-1000; // includes max-concurrent-checkpoints + checkpoints to be
aborted (scheduled via
-// mailbox)
Review Comment:
Hi @pnowojski , thanks for your review.
The `checkpointId < nextExpectedCheckpointId` may happen. For example,
`AlternatingWaitingForFirstBarrierUnaligned#barrierReceived` don't check
whether checkpointId is aborted, if received a barrier, it will call the
ChannelStateWriter.start().
So, we should use `ongoingCheckpointId`, and just update it within
`ChannelStateWriterImpl#start`
We should ignore checkpoint when the checkpointId is aborted, so we need
the `NavigableSet abortedCheckpointIds;`.
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/channel/ChannelStateWriterImpl.java:
##
@@ -60,89 +63,78 @@
public class ChannelStateWriterImpl implements ChannelStateWriter {
private static final Logger LOG =
LoggerFactory.getLogger(ChannelStateWriterImpl.class);
-private static final int DEFAULT_MAX_CHECKPOINTS =
-1000; // includes max-concurrent-checkpoints + checkpoints to be
aborted (scheduled via
-// mailbox)
private final String taskName;
private final ChannelStateWriteRequestExecutor executor;
-private final ConcurrentMap results;
-private final int maxCheckpoints;
-/**
- * Creates a {@link ChannelStateWriterImpl} with {@link
#DEFAULT_MAX_CHECKPOINTS} as {@link
- * #maxCheckpoints}.
- */
-public ChannelStateWriterImpl(
-String taskName, int subtaskIndex, CheckpointStorageWorkerView
streamFactoryResolver) {
-this(taskName, subtaskIndex, streamFactoryResolver,
DEFAULT_MAX_CHECKPOINTS);
-}
+private final Object lock = new Object();
+
+@GuardedBy("lock")
+private long ongoingCheckpointId;
+
+@GuardedBy("lock")
+private final NavigableSet abortedCheckpointIds;
+
+// The result of ongoingCheckpointId, the checkpoint that CheckpointId is
less than
+// ongoingCheckpointId should be aborted due to concurrent unaligned
checkpoint is currently not
+// supported.
+@GuardedBy("lock")
+private ChannelStateWriteResult result;
/**
* Creates a {@link ChannelStateWriterImpl} with {@link
ChannelStateSerializerImpl default}
* {@link ChannelStateSerializer}, and a {@link
ChannelStateWriteRequestExecutorImpl}.
*
* @param taskName
* @param streamFactoryResolver a factory to obtain output stream factory
for a given checkpoint
- * @param maxCheckpoints maximum number of checkpoints to be written
currently or finished but
- * not taken yet.
*/
-ChannelStateWriterImpl(
-String taskName,
-int subtaskIndex,
-CheckpointStorageWorkerView streamFactoryResolver,
-int maxCheckpoints) {
+public ChannelStateWriterImpl(
+String taskName, int subtaskIndex, CheckpointStorageWorkerView
streamFactoryResolver) {
this(
taskName,
-new ConcurrentHashMap<>(maxCheckpoints),
new ChannelStateWriteRequestExecutorImpl(
taskName,
new ChannelStateWriteRequestDispatcherImpl(
taskName,
subtaskIndex,
streamFactoryResolver,
-new ChannelStateSerializerImpl())),
-maxCheckpoints);
+new ChannelStateSerializerImpl(;
}
-ChannelStateWriterImpl(
-String taskName,
-ConcurrentMap results,
-ChannelStateWriteRequestExecutor executor,
-int maxCheckpoints) {
+ChannelStateWriterImpl(String taskName, ChannelStateWriteRequestExecutor
executor) {
this.taskName = taskName;
-this.results = results;
-this.maxCheckpoints = maxCheckpoints;
this.executor = executor;
+this.abortedCheckpointIds = new TreeSet<>();
+this.ongoingCheckpointId = 0;
}
@Override
public void start(long checkpointId, CheckpointOptions checkpointOptions) {
LOG.debug("{} starting checkpoint {} ({})", taskName, checkpointId,
checkpointOptions);
-ChannelStateWriteResult result = new ChannelStateWriteResult();
-ChannelStateWriteResult put =
-results.computeIfAbsent(
+synchronized (lock) {
+if
[GitHub] [flink] lincoln-lil commented on a diff in pull request #20243: [FLINK-28491][table-planner] Introduce APPROX_COUNT_DISTINCT aggregate function for batch sql
lincoln-lil commented on code in PR #20243: URL: https://github.com/apache/flink/pull/20243#discussion_r923368739 ## tools/maven/suppressions.xml: ## @@ -45,6 +45,9 @@ under the License. + Review Comment: nit: the comment seems incorrect, and move it to line 39 maybe better -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
