Re: [PR] [FLINK-34593][release] Add release note for version 1.19 [flink]
lincoln-lil merged PR #24505: URL: https://github.com/apache/flink/pull/24505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add GPG key for 1.19.0 release [flink-docker]
lincoln-lil merged PR #187: URL: https://github.com/apache/flink-docker/pull/187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527940518
##
flink-process-function-parent/flink-process-function/src/main/java/org/apache/flink/process/impl/ExecutionEnvironmentImpl.java:
##
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.impl;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.RuntimeExecutionMode;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.InvalidTypesException;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.v2.FromDataSource;
+import org.apache.flink.api.connector.v2.Source;
+import org.apache.flink.api.connector.v2.WrappedSource;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.typeutils.MissingTypeInfo;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.configuration.ExecutionOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import
org.apache.flink.connector.datagen.functions.FromElementsGeneratorFunction;
+import org.apache.flink.connector.datagen.source.DataGeneratorSource;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.core.execution.PipelineExecutor;
+import org.apache.flink.core.execution.PipelineExecutorFactory;
+import org.apache.flink.core.execution.PipelineExecutorServiceLoader;
+import org.apache.flink.process.api.ExecutionEnvironment;
+import org.apache.flink.process.api.stream.NonKeyedPartitionStream;
+import org.apache.flink.process.impl.stream.NonKeyedPartitionStreamImpl;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamGraphGenerator;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import
org.apache.flink.streaming.runtime.translators.ProcessFunctionSinkTransformationTranslator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+
+import static
org.apache.flink.streaming.api.graph.StreamGraphGenerator.DEFAULT_TIME_CHARACTERISTIC;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** The implementation of {@link ExecutionEnvironment}. */
+public class ExecutionEnvironmentImpl implements ExecutionEnvironment {
+private final List> transformations = new ArrayList<>();
+
+private final ExecutionConfig executionConfig;
+
+/** Settings that control the checkpointing behavior. */
+private final CheckpointConfig checkpointCfg;
+
+private final Configuration configuration;
+
+private final ClassLoader userClassloader;
+
+private final PipelineExecutorServiceLoader executorServiceLoader;
+
+static {
+try {
+// All transformation translator must be put to a map in
StreamGraphGenerator, but
+// streaming-java is not depend on process-function module, using
reflect to handle
+// this.
+
ProcessFunctionSinkTransformationTranslator.registerSinkTransformationTranslator();
+} catch (Exception e) {
+throw new RuntimeException(
+"Can not register process function transformation
translator.");
+}
+}
+
+/**
+ * The environment of the context (local by default, cluster if invoked
through command line).
+ */
+private static ExecutionEnvironmentFactory contextEnvironmentFactory =
null;
+
+public static ExecutionEnvironment newInstance() {
+if (contextEnvironmentFactory != null) {
+
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527936405
##
flink-process-function-parent/flink-process-function/src/test/java/org/apache/flink/process/impl/ExecutionEnvironmentImplTest.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.impl;
+
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.v2.SourceUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import org.apache.flink.process.api.ExecutionEnvironment;
+import org.apache.flink.process.api.stream.NonKeyedPartitionStream;
+import org.apache.flink.process.impl.stream.StreamTestUtils;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamNode;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collection;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link ExecutionEnvironmentImpl}. */
+class ExecutionEnvironmentImplTest {
Review Comment:
I have added a new test in `ExecutionEnvironmentImplTest`.
As for the testing of `execute`: It's a bit special, or hard to test with a
unit test. I will introduce integration tests through other PR, which will be
covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527936405
##
flink-process-function-parent/flink-process-function/src/test/java/org/apache/flink/process/impl/ExecutionEnvironmentImplTest.java:
##
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.impl;
+
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.v2.SourceUtils;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import org.apache.flink.process.api.ExecutionEnvironment;
+import org.apache.flink.process.api.stream.NonKeyedPartitionStream;
+import org.apache.flink.process.impl.stream.StreamTestUtils;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamNode;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collection;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** Tests for {@link ExecutionEnvironmentImpl}. */
+class ExecutionEnvironmentImplTest {
Review Comment:
I have added a new test in `ExecutionEnvironmentImplTest`.
As for the testing of `execute`: It's a bit special, or hard to test with a
unit test. I will introduce integration tests through other PR, which will be
covered in due course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34713) Updates the docs stable version
[
https://issues.apache.org/jira/browse/FLINK-34713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34713:
Summary: Updates the docs stable version (was: CLONE - Updates the docs
stable version)
> Updates the docs stable version
> ---
>
> Key: FLINK-34713
> URL: https://issues.apache.org/jira/browse/FLINK-34713
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> Update docs to "stable" in {{docs/config.toml}} in the branch of the
> _just-released_ version:
> * Change V{{{}ersion{}}} from {{{}x.y-SNAPSHOT }}to \{{{}x.y.z{}}}, i.e.
> {{1.6-SNAPSHOT}} to {{1.6.0}}
> * Change V{{{}ersionTitle{}}} from {{x.y-SNAPSHOT}} to {{{}x.y{}}}, i.e.
> {{1.6-SNAPSHOT}} to {{1.6}}
> * Change Branch from {{master}} to {{{}release-x.y{}}}, i.e. {{master}} to
> {{release-1.6}}
> * Change {{baseURL}} from
> {{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
> to
> {{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
> * Change {{javadocs_baseurl}} from
> {{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
> to
> {{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
> * Change {{IsStable}} to {{true}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34714) Start End of Life discussion thread for now outdated Flink minor version
[ https://issues.apache.org/jira/browse/FLINK-34714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-34714: Summary: Start End of Life discussion thread for now outdated Flink minor version (was: CLONE - Start End of Life discussion thread for now outdated Flink minor version) > Start End of Life discussion thread for now outdated Flink minor version > > > Key: FLINK-34714 > URL: https://issues.apache.org/jira/browse/FLINK-34714 > Project: Flink > Issue Type: Sub-task >Reporter: Matthias Pohl >Priority: Major > > The idea is to discuss whether we should do a final release for the now not > supported minor version in the community. Such a minor release shouldn't be > covered by the current minor version release managers. Their only > responsibility is to trigger the discussion. > The intention of a final patch release for the now unsupported Flink minor > version is to flush out all the fixes that didn't end up in the previous > release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34714) Start End of Life discussion thread for now outdated Flink minor version
[ https://issues.apache.org/jira/browse/FLINK-34714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-34714: Reporter: Lincoln Lee (was: Matthias Pohl) > Start End of Life discussion thread for now outdated Flink minor version > > > Key: FLINK-34714 > URL: https://issues.apache.org/jira/browse/FLINK-34714 > Project: Flink > Issue Type: Sub-task >Reporter: Lincoln Lee >Priority: Major > > The idea is to discuss whether we should do a final release for the now not > supported minor version in the community. Such a minor release shouldn't be > covered by the current minor version release managers. Their only > responsibility is to trigger the discussion. > The intention of a final patch release for the now unsupported Flink minor > version is to flush out all the fixes that didn't end up in the previous > release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34713) Updates the docs stable version
[
https://issues.apache.org/jira/browse/FLINK-34713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34713:
Reporter: Lincoln Lee (was: Sergey Nuyanzin)
> Updates the docs stable version
> ---
>
> Key: FLINK-34713
> URL: https://issues.apache.org/jira/browse/FLINK-34713
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Priority: Major
>
> Update docs to "stable" in {{docs/config.toml}} in the branch of the
> _just-released_ version:
> * Change V{{{}ersion{}}} from {{{}x.y-SNAPSHOT }}to \{{{}x.y.z{}}}, i.e.
> {{1.6-SNAPSHOT}} to {{1.6.0}}
> * Change V{{{}ersionTitle{}}} from {{x.y-SNAPSHOT}} to {{{}x.y{}}}, i.e.
> {{1.6-SNAPSHOT}} to {{1.6}}
> * Change Branch from {{master}} to {{{}release-x.y{}}}, i.e. {{master}} to
> {{release-1.6}}
> * Change {{baseURL}} from
> {{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
> to
> {{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
> * Change {{javadocs_baseurl}} from
> {{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
> to
> {{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
> * Change {{IsStable}} to {{true}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34711) Other announcements
[
https://issues.apache.org/jira/browse/FLINK-34711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34711:
Reporter: Lincoln Lee (was: Sergey Nuyanzin)
> Other announcements
> ---
>
> Key: FLINK-34711
> URL: https://issues.apache.org/jira/browse/FLINK-34711
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Priority: Major
>
> h3. Recordkeeping
> Use [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink]
> to seed the information about the release into future project reports.
> (Note: Only PMC members have access report releases. If you do not have
> access, ask on the mailing list for assistance.)
> h3. Flink blog
> Major or otherwise important releases should have a blog post. Write one if
> needed for this particular release. Minor releases that don’t introduce new
> major functionality don’t necessarily need to be blogged (see [flink-web PR
> #581 for Flink 1.15.3|https://github.com/apache/flink-web/pull/581] as an
> example for a minor release blog post).
> Please make sure that the release notes of the documentation (see section
> "Review and update documentation") are linked from the blog post of a major
> release.
> We usually include the names of all contributors in the announcement blog
> post. Use the following command to get the list of contributors:
> {code}
> # first line is required to make sort first with uppercase and then lower
> export LC_ALL=C
> export FLINK_PREVIOUS_RELEASE_BRANCH=
> export FLINK_CURRENT_RELEASE_BRANCH=
> # e.g.
> # export FLINK_PREVIOUS_RELEASE_BRANCH=release-1.17
> # export FLINK_CURRENT_RELEASE_BRANCH=release-1.18
> git log $(git merge-base master $FLINK_PREVIOUS_RELEASE_BRANCH)..$(git
> show-ref --hash ${FLINK_CURRENT_RELEASE_BRANCH}) --pretty=format:"%an%n%cn" |
> sort -u | paste -sd, | sed "s/\,/\, /g"
> {code}
> h3. Social media
> Tweet, post on Facebook, LinkedIn, and other platforms. Ask other
> contributors to do the same.
> h3. Flink Release Wiki page
> Add a summary of things that went well or that went not so well during the
> release process. This can include feedback from contributors but also more
> generic things like the release have taken longer than initially anticipated
> (and why) to give a bit of context to the release process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34711) Other announcements
[
https://issues.apache.org/jira/browse/FLINK-34711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34711:
Summary: Other announcements (was: CLONE - Other announcements)
> Other announcements
> ---
>
> Key: FLINK-34711
> URL: https://issues.apache.org/jira/browse/FLINK-34711
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> h3. Recordkeeping
> Use [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink]
> to seed the information about the release into future project reports.
> (Note: Only PMC members have access report releases. If you do not have
> access, ask on the mailing list for assistance.)
> h3. Flink blog
> Major or otherwise important releases should have a blog post. Write one if
> needed for this particular release. Minor releases that don’t introduce new
> major functionality don’t necessarily need to be blogged (see [flink-web PR
> #581 for Flink 1.15.3|https://github.com/apache/flink-web/pull/581] as an
> example for a minor release blog post).
> Please make sure that the release notes of the documentation (see section
> "Review and update documentation") are linked from the blog post of a major
> release.
> We usually include the names of all contributors in the announcement blog
> post. Use the following command to get the list of contributors:
> {code}
> # first line is required to make sort first with uppercase and then lower
> export LC_ALL=C
> export FLINK_PREVIOUS_RELEASE_BRANCH=
> export FLINK_CURRENT_RELEASE_BRANCH=
> # e.g.
> # export FLINK_PREVIOUS_RELEASE_BRANCH=release-1.17
> # export FLINK_CURRENT_RELEASE_BRANCH=release-1.18
> git log $(git merge-base master $FLINK_PREVIOUS_RELEASE_BRANCH)..$(git
> show-ref --hash ${FLINK_CURRENT_RELEASE_BRANCH}) --pretty=format:"%an%n%cn" |
> sort -u | paste -sd, | sed "s/\,/\, /g"
> {code}
> h3. Social media
> Tweet, post on Facebook, LinkedIn, and other platforms. Ask other
> contributors to do the same.
> h3. Flink Release Wiki page
> Add a summary of things that went well or that went not so well during the
> release process. This can include feedback from contributors but also more
> generic things like the release have taken longer than initially anticipated
> (and why) to give a bit of context to the release process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34687) Home Page of Flink CDC Documentation
[ https://issues.apache.org/jira/browse/FLINK-34687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu resolved FLINK-34687. Resolution: Resolved Resolved in flink-cdc(master) via : aa9e91ce4599d8c5a51df82cea4e31aed7d2f634 > Home Page of Flink CDC Documentation > > > Key: FLINK-34687 > URL: https://issues.apache.org/jira/browse/FLINK-34687 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Qingsheng Ren >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34687][cdc][docs] Build the home page of Flink CDC documentation [flink-cdc]
leonardBang merged PR #3156: URL: https://github.com/apache/flink-cdc/pull/3156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34648] add waitChangeResultRequest and WaitChangeResultResponse to avoid RPC timeout. [flink-cdc]
BaoPiao commented on PR #3128: URL: https://github.com/apache/flink-cdc/pull/3128#issuecomment-2003029727 > @BaoPiao Yeah, this is because SchemaRegistryRequestHandler doesn't clear pendingSchemaChanges when SchemaOperator restarted, I've added one commit [7aba6a4](https://github.com/apache/flink-cdc/commit/7aba6a4489486af9fece954f561dca0c9c443006) to fixed it. Should we clear the flushedSinkWriters as well? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34710) Apache mailing lists announcements
[
https://issues.apache.org/jira/browse/FLINK-34710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34710:
Summary: Apache mailing lists announcements (was: CLONE - Apache mailing
lists announcements)
> Apache mailing lists announcements
> --
>
> Key: FLINK-34710
> URL: https://issues.apache.org/jira/browse/FLINK-34710
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> Announce on the {{dev@}} mailing list that the release has been finished.
> Announce on the release on the {{user@}} mailing list, listing major
> improvements and contributions.
> Announce the release on the [annou...@apache.org|mailto:annou...@apache.org]
> mailing list.
> {panel}
> {panel}
> |{{From: Release Manager}}
> {{To: d...@flink.apache.org, u...@flink.apache.org, user...@flink.apache.org,
> annou...@apache.org}}
> {{Subject: [ANNOUNCE] Apache Flink 1.2.3 released}}
>
> {{The Apache Flink community is very happy to announce the release of Apache
> Flink 1.2.3, which is the third bugfix release for the Apache Flink 1.2
> series.}}
>
> {{Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.}}
>
> {{The release is available for download at:}}
> {{[https://flink.apache.org/downloads.html]}}
>
> {{Please check out the release blog post for an overview of the improvements
> for this bugfix release:}}
> {{}}
>
> {{The full release notes are available in Jira:}}
> {{}}
>
> {{We would like to thank all contributors of the Apache Flink community who
> made this release possible!}}
>
> {{Feel free to reach out to the release managers (or respond to this thread)
> with feedback on the release process. Our goal is to constantly improve the
> release process. Feedback on what could be improved or things that didn't go
> so well are appreciated.}}
>
> {{Regards,}}
> {{Release Manager}}|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34710) Apache mailing lists announcements
[
https://issues.apache.org/jira/browse/FLINK-34710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34710:
Reporter: Lincoln Lee (was: Sergey Nuyanzin)
> Apache mailing lists announcements
> --
>
> Key: FLINK-34710
> URL: https://issues.apache.org/jira/browse/FLINK-34710
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Priority: Major
>
> Announce on the {{dev@}} mailing list that the release has been finished.
> Announce on the release on the {{user@}} mailing list, listing major
> improvements and contributions.
> Announce the release on the [annou...@apache.org|mailto:annou...@apache.org]
> mailing list.
> {panel}
> {panel}
> |{{From: Release Manager}}
> {{To: d...@flink.apache.org, u...@flink.apache.org, user...@flink.apache.org,
> annou...@apache.org}}
> {{Subject: [ANNOUNCE] Apache Flink 1.2.3 released}}
>
> {{The Apache Flink community is very happy to announce the release of Apache
> Flink 1.2.3, which is the third bugfix release for the Apache Flink 1.2
> series.}}
>
> {{Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.}}
>
> {{The release is available for download at:}}
> {{[https://flink.apache.org/downloads.html]}}
>
> {{Please check out the release blog post for an overview of the improvements
> for this bugfix release:}}
> {{}}
>
> {{The full release notes are available in Jira:}}
> {{}}
>
> {{We would like to thank all contributors of the Apache Flink community who
> made this release possible!}}
>
> {{Feel free to reach out to the release managers (or respond to this thread)
> with feedback on the release process. Our goal is to constantly improve the
> release process. Feedback on what could be improved or things that didn't go
> so well are appreciated.}}
>
> {{Regards,}}
> {{Release Manager}}|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34708) Merge website pull request
[ https://issues.apache.org/jira/browse/FLINK-34708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-34708: Summary: Merge website pull request (was: CLONE - Merge website pull request) > Merge website pull request > -- > > Key: FLINK-34708 > URL: https://issues.apache.org/jira/browse/FLINK-34708 > Project: Flink > Issue Type: Sub-task >Reporter: Lincoln Lee >Assignee: lincoln lee >Priority: Major > > Merge the website pull request to [list the > release|http://flink.apache.org/downloads.html]. Make sure to regenerate the > website as well, as it isn't build automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34709) Remove outdated versions
[
https://issues.apache.org/jira/browse/FLINK-34709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34709:
Summary: Remove outdated versions (was: CLONE - Remove outdated versions)
> Remove outdated versions
>
>
> Key: FLINK-34709
> URL: https://issues.apache.org/jira/browse/FLINK-34709
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> h4. dist.apache.org
> For a new major release remove all release files older than 2 versions, e.g.,
> when releasing 1.7, remove all releases <= 1.5.
> For a new bugfix version remove all release files for previous bugfix
> releases in the same series, e.g., when releasing 1.7.1, remove the 1.7.0
> release.
> # If you have not already, check out the Flink section of the {{release}}
> repository on {{[dist.apache.org|http://dist.apache.org/]}} via Subversion.
> In a fresh directory:
> {code}
> svn checkout https://dist.apache.org/repos/dist/release/flink
> --depth=immediates
> cd flink
> {code}
> # Remove files for outdated releases and commit the changes.
> {code}
> svn remove flink-
> svn commit
> {code}
> # Verify that files are
> [removed|https://dist.apache.org/repos/dist/release/flink]
> (!) Remember to remove the corresponding download links from the website.
> h4. CI
> Disable the cron job for the now-unsupported version from
> (tools/azure-pipelines/[build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
> in the respective branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34712) CLONE - Update reference data for Migration Tests
[
https://issues.apache.org/jira/browse/FLINK-34712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee reassigned FLINK-34712:
---
Assignee: lincoln lee (was: Sergey Nuyanzin)
> CLONE - Update reference data for Migration Tests
> -
>
> Key: FLINK-34712
> URL: https://issues.apache.org/jira/browse/FLINK-34712
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.18.1
>
>
> Update migration tests in master to cover migration from new version. Since
> 1.18, this step could be done automatically with the following steps. For
> more information please refer to [this
> page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
> # {*}On the published release tag (e.g., release-1.16.0){*}, run
> {panel}
> {panel}
> |{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data
> -Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|
> The version (1.16 in the command above) should be replaced with the target
> one.
> # Modify the content of the file
> [apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
> to the latest version (it would be "v1_16" if sticking to the example where
> 1.16.0 was released).
> # Commit the modification in step a and b with "{_}[release] Generate
> reference data for state migration tests based on release-1.xx.0{_}" to the
> corresponding release branch (e.g. {{release-1.16}} in our example), replace
> "xx" with the actual version (in this example "16"). You should use the Jira
> issue ID in case of [release] as the commit message's prefix if you have a
> dedicated Jira issue for this task.
> # Cherry-pick the commit to the master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34712) CLONE - Update reference data for Migration Tests
[
https://issues.apache.org/jira/browse/FLINK-34712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34712:
Reporter: Lincoln Lee (was: Sergey Nuyanzin)
> CLONE - Update reference data for Migration Tests
> -
>
> Key: FLINK-34712
> URL: https://issues.apache.org/jira/browse/FLINK-34712
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.18.1
>
>
> Update migration tests in master to cover migration from new version. Since
> 1.18, this step could be done automatically with the following steps. For
> more information please refer to [this
> page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
> # {*}On the published release tag (e.g., release-1.16.0){*}, run
> {panel}
> {panel}
> |{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data
> -Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|
> The version (1.16 in the command above) should be replaced with the target
> one.
> # Modify the content of the file
> [apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
> to the latest version (it would be "v1_16" if sticking to the example where
> 1.16.0 was released).
> # Commit the modification in step a and b with "{_}[release] Generate
> reference data for state migration tests based on release-1.xx.0{_}" to the
> corresponding release branch (e.g. {{release-1.16}} in our example), replace
> "xx" with the actual version (in this example "16"). You should use the Jira
> issue ID in case of [release] as the commit message's prefix if you have a
> dedicated Jira issue for this task.
> # Cherry-pick the commit to the master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34712) CLONE - Update reference data for Migration Tests
[
https://issues.apache.org/jira/browse/FLINK-34712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34712:
Fix Version/s: 1.20.0
1.19.1
(was: 1.19.0)
(was: 1.18.1)
> CLONE - Update reference data for Migration Tests
> -
>
> Key: FLINK-34712
> URL: https://issues.apache.org/jira/browse/FLINK-34712
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0, 1.19.1
>
>
> Update migration tests in master to cover migration from new version. Since
> 1.18, this step could be done automatically with the following steps. For
> more information please refer to [this
> page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
> # {*}On the published release tag (e.g., release-1.16.0){*}, run
> {panel}
> {panel}
> |{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data
> -Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|
> The version (1.16 in the command above) should be replaced with the target
> one.
> # Modify the content of the file
> [apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
> to the latest version (it would be "v1_16" if sticking to the example where
> 1.16.0 was released).
> # Commit the modification in step a and b with "{_}[release] Generate
> reference data for state migration tests based on release-1.xx.0{_}" to the
> corresponding release branch (e.g. {{release-1.16}} in our example), replace
> "xx" with the actual version (in this example "16"). You should use the Jira
> issue ID in case of [release] as the commit message's prefix if you have a
> dedicated Jira issue for this task.
> # Cherry-pick the commit to the master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34712) Update reference data for Migration Tests
[
https://issues.apache.org/jira/browse/FLINK-34712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34712:
Summary: Update reference data for Migration Tests (was: CLONE - Update
reference data for Migration Tests)
> Update reference data for Migration Tests
> -
>
> Key: FLINK-34712
> URL: https://issues.apache.org/jira/browse/FLINK-34712
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0, 1.19.1
>
>
> Update migration tests in master to cover migration from new version. Since
> 1.18, this step could be done automatically with the following steps. For
> more information please refer to [this
> page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
> # {*}On the published release tag (e.g., release-1.16.0){*}, run
> {panel}
> {panel}
> |{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data
> -Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|
> The version (1.16 in the command above) should be replaced with the target
> one.
> # Modify the content of the file
> [apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
> to the latest version (it would be "v1_16" if sticking to the example where
> 1.16.0 was released).
> # Commit the modification in step a and b with "{_}[release] Generate
> reference data for state migration tests based on release-1.xx.0{_}" to the
> corresponding release branch (e.g. {{release-1.16}} in our example), replace
> "xx" with the actual version (in this example "16"). You should use the Jira
> issue ID in case of [release] as the commit message's prefix if you have a
> dedicated Jira issue for this task.
> # Cherry-pick the commit to the master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34709) Remove outdated versions
[
https://issues.apache.org/jira/browse/FLINK-34709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34709:
Reporter: Lincoln Lee (was: Sergey Nuyanzin)
> Remove outdated versions
>
>
> Key: FLINK-34709
> URL: https://issues.apache.org/jira/browse/FLINK-34709
> Project: Flink
> Issue Type: Sub-task
>Reporter: Lincoln Lee
>Priority: Major
>
> h4. dist.apache.org
> For a new major release remove all release files older than 2 versions, e.g.,
> when releasing 1.7, remove all releases <= 1.5.
> For a new bugfix version remove all release files for previous bugfix
> releases in the same series, e.g., when releasing 1.7.1, remove the 1.7.0
> release.
> # If you have not already, check out the Flink section of the {{release}}
> repository on {{[dist.apache.org|http://dist.apache.org/]}} via Subversion.
> In a fresh directory:
> {code}
> svn checkout https://dist.apache.org/repos/dist/release/flink
> --depth=immediates
> cd flink
> {code}
> # Remove files for outdated releases and commit the changes.
> {code}
> svn remove flink-
> svn commit
> {code}
> # Verify that files are
> [removed|https://dist.apache.org/repos/dist/release/flink]
> (!) Remember to remove the corresponding download links from the website.
> h4. CI
> Disable the cron job for the now-unsupported version from
> (tools/azure-pipelines/[build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
> in the respective branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34707) CLONE - Update japicmp configuration
[
https://issues.apache.org/jira/browse/FLINK-34707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34707:
Fix Version/s: 1.20.0
1.19.1
(was: 1.19.0)
(was: 1.18.1)
> CLONE - Update japicmp configuration
>
>
> Key: FLINK-34707
> URL: https://issues.apache.org/jira/browse/FLINK-34707
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0, 1.19.1
>
>
> Update the japicmp reference version and wipe exclusions / enable API
> compatibility checks for {{@PublicEvolving}} APIs on the corresponding
> SNAPSHOT branch with the {{update_japicmp_configuration.sh}} script (see
> below).
> For a new major release (x.y.0), run the same command also on the master
> branch for updating the japicmp reference version and removing out-dated
> exclusions in the japicmp configuration.
> Make sure that all Maven artifacts are already pushed to Maven Central.
> Otherwise, there's a risk that CI fails due to missing reference artifacts.
> {code:bash}
> tools $ NEW_VERSION=$RELEASE_VERSION releasing/update_japicmp_configuration.sh
> tools $ cd ..$ git add *$ git commit -m "Update japicmp configuration for
> $RELEASE_VERSION" {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34708) CLONE - Merge website pull request
[ https://issues.apache.org/jira/browse/FLINK-34708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee reassigned FLINK-34708: --- Assignee: lincoln lee > CLONE - Merge website pull request > -- > > Key: FLINK-34708 > URL: https://issues.apache.org/jira/browse/FLINK-34708 > Project: Flink > Issue Type: Sub-task >Reporter: Sergey Nuyanzin >Assignee: lincoln lee >Priority: Major > > Merge the website pull request to [list the > release|http://flink.apache.org/downloads.html]. Make sure to regenerate the > website as well, as it isn't build automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34707) Update japicmp configuration
[
https://issues.apache.org/jira/browse/FLINK-34707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34707:
Summary: Update japicmp configuration (was: CLONE - Update japicmp
configuration)
> Update japicmp configuration
>
>
> Key: FLINK-34707
> URL: https://issues.apache.org/jira/browse/FLINK-34707
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0, 1.19.1
>
>
> Update the japicmp reference version and wipe exclusions / enable API
> compatibility checks for {{@PublicEvolving}} APIs on the corresponding
> SNAPSHOT branch with the {{update_japicmp_configuration.sh}} script (see
> below).
> For a new major release (x.y.0), run the same command also on the master
> branch for updating the japicmp reference version and removing out-dated
> exclusions in the japicmp configuration.
> Make sure that all Maven artifacts are already pushed to Maven Central.
> Otherwise, there's a risk that CI fails due to missing reference artifacts.
> {code:bash}
> tools $ NEW_VERSION=$RELEASE_VERSION releasing/update_japicmp_configuration.sh
> tools $ cd ..$ git add *$ git commit -m "Update japicmp configuration for
> $RELEASE_VERSION" {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34708) CLONE - Merge website pull request
[ https://issues.apache.org/jira/browse/FLINK-34708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lincoln lee updated FLINK-34708: Reporter: Lincoln Lee (was: Sergey Nuyanzin) > CLONE - Merge website pull request > -- > > Key: FLINK-34708 > URL: https://issues.apache.org/jira/browse/FLINK-34708 > Project: Flink > Issue Type: Sub-task >Reporter: Lincoln Lee >Assignee: lincoln lee >Priority: Major > > Merge the website pull request to [list the > release|http://flink.apache.org/downloads.html]. Make sure to regenerate the > website as well, as it isn't build automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34648] add waitChangeResultRequest and WaitChangeResultResponse to avoid RPC timeout. [flink-cdc]
yanghuaiGit commented on PR #3128: URL: https://github.com/apache/flink-cdc/pull/3128#issuecomment-2003023705 如果schemaRegister的pending里有多个待执行schema事件，isSchemaChangeApplying代表的是正在执行schema变更事件，但是无法得知是哪个schema事件执行，isSchemaChangeApplying无法和客户端的事件绑定，导致schemaoperator可能需要等待所有的schema事件执行完才会转为非阻塞 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-34707) CLONE - Update japicmp configuration
[
https://issues.apache.org/jira/browse/FLINK-34707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee reassigned FLINK-34707:
---
Assignee: lincoln lee (was: Sergey Nuyanzin)
> CLONE - Update japicmp configuration
>
>
> Key: FLINK-34707
> URL: https://issues.apache.org/jira/browse/FLINK-34707
> Project: Flink
> Issue Type: Sub-task
>Reporter: Sergey Nuyanzin
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0, 1.18.1
>
>
> Update the japicmp reference version and wipe exclusions / enable API
> compatibility checks for {{@PublicEvolving}} APIs on the corresponding
> SNAPSHOT branch with the {{update_japicmp_configuration.sh}} script (see
> below).
> For a new major release (x.y.0), run the same command also on the master
> branch for updating the japicmp reference version and removing out-dated
> exclusions in the japicmp configuration.
> Make sure that all Maven artifacts are already pushed to Maven Central.
> Otherwise, there's a risk that CI fails due to missing reference artifacts.
> {code:bash}
> tools $ NEW_VERSION=$RELEASE_VERSION releasing/update_japicmp_configuration.sh
> tools $ cd ..$ git add *$ git commit -m "Update japicmp configuration for
> $RELEASE_VERSION" {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34714) CLONE - Start End of Life discussion thread for now outdated Flink minor version
lincoln lee created FLINK-34714: --- Summary: CLONE - Start End of Life discussion thread for now outdated Flink minor version Key: FLINK-34714 URL: https://issues.apache.org/jira/browse/FLINK-34714 Project: Flink Issue Type: Sub-task Reporter: Matthias Pohl The idea is to discuss whether we should do a final release for the now not supported minor version in the community. Such a minor release shouldn't be covered by the current minor version release managers. Their only responsibility is to trigger the discussion. The intention of a final patch release for the now unsupported Flink minor version is to flush out all the fixes that didn't end up in the previous release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34711) CLONE - Other announcements
lincoln lee created FLINK-34711:
---
Summary: CLONE - Other announcements
Key: FLINK-34711
URL: https://issues.apache.org/jira/browse/FLINK-34711
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
h3. Recordkeeping
Use [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink] to
seed the information about the release into future project reports.
(Note: Only PMC members have access report releases. If you do not have access,
ask on the mailing list for assistance.)
h3. Flink blog
Major or otherwise important releases should have a blog post. Write one if
needed for this particular release. Minor releases that don’t introduce new
major functionality don’t necessarily need to be blogged (see [flink-web PR
#581 for Flink 1.15.3|https://github.com/apache/flink-web/pull/581] as an
example for a minor release blog post).
Please make sure that the release notes of the documentation (see section
"Review and update documentation") are linked from the blog post of a major
release.
We usually include the names of all contributors in the announcement blog post.
Use the following command to get the list of contributors:
{code}
# first line is required to make sort first with uppercase and then lower
export LC_ALL=C
export FLINK_PREVIOUS_RELEASE_BRANCH=
export FLINK_CURRENT_RELEASE_BRANCH=
# e.g.
# export FLINK_PREVIOUS_RELEASE_BRANCH=release-1.17
# export FLINK_CURRENT_RELEASE_BRANCH=release-1.18
git log $(git merge-base master $FLINK_PREVIOUS_RELEASE_BRANCH)..$(git show-ref
--hash ${FLINK_CURRENT_RELEASE_BRANCH}) --pretty=format:"%an%n%cn" | sort -u |
paste -sd, | sed "s/\,/\, /g"
{code}
h3. Social media
Tweet, post on Facebook, LinkedIn, and other platforms. Ask other contributors
to do the same.
h3. Flink Release Wiki page
Add a summary of things that went well or that went not so well during the
release process. This can include feedback from contributors but also more
generic things like the release have taken longer than initially anticipated
(and why) to give a bit of context to the release process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34706) Promote release 1.19
[
https://issues.apache.org/jira/browse/FLINK-34706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee reassigned FLINK-34706:
---
Assignee: lincoln lee (was: Jing Ge)
> Promote release 1.19
>
>
> Key: FLINK-34706
> URL: https://issues.apache.org/jira/browse/FLINK-34706
> Project: Flink
> Issue Type: New Feature
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
>
> Once the release has been finalized (FLINK-32920), the last step of the
> process is to promote the release within the project and beyond. Please wait
> for 24h after finalizing the release in accordance with the [ASF release
> policy|http://www.apache.org/legal/release-policy.html#release-announcements].
> *Final checklist to declare this issue resolved:*
> # Website pull request to [list the
> release|http://flink.apache.org/downloads.html] merged
> # Release announced on the user@ mailing list.
> # Blog post published, if applicable.
> # Release recorded in
> [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
> # Release announced on social media.
> # Completion declared on the dev@ mailing list.
> # Update Homebrew:
> [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] (seems to be done
> automatically - at least for minor releases for both minor and major
> releases)
> # Updated the japicmp configuration
> ** corresponding SNAPSHOT branch japicmp reference version set to the just
> released version, and API compatibiltity checks for {{@PublicEvolving}} was
> enabled
> ** (minor version release only) master branch japicmp reference version set
> to the just released version
> ** (minor version release only) master branch japicmp exclusions have been
> cleared
> # Update the list of previous version in {{docs/config.toml}} on the master
> branch.
> # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch
> of the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released)
> # Update stable and master alias in
> [https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
> # Open discussion thread for End of Life for Unsupported version (i.e. 1.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34706) Promote release 1.19
[
https://issues.apache.org/jira/browse/FLINK-34706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-34706:
Reporter: Lincoln Lee (was: Sergey Nuyanzin)
> Promote release 1.19
>
>
> Key: FLINK-34706
> URL: https://issues.apache.org/jira/browse/FLINK-34706
> Project: Flink
> Issue Type: New Feature
>Affects Versions: 1.18.0
>Reporter: Lincoln Lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
>
> Once the release has been finalized (FLINK-32920), the last step of the
> process is to promote the release within the project and beyond. Please wait
> for 24h after finalizing the release in accordance with the [ASF release
> policy|http://www.apache.org/legal/release-policy.html#release-announcements].
> *Final checklist to declare this issue resolved:*
> # Website pull request to [list the
> release|http://flink.apache.org/downloads.html] merged
> # Release announced on the user@ mailing list.
> # Blog post published, if applicable.
> # Release recorded in
> [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
> # Release announced on social media.
> # Completion declared on the dev@ mailing list.
> # Update Homebrew:
> [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] (seems to be done
> automatically - at least for minor releases for both minor and major
> releases)
> # Updated the japicmp configuration
> ** corresponding SNAPSHOT branch japicmp reference version set to the just
> released version, and API compatibiltity checks for {{@PublicEvolving}} was
> enabled
> ** (minor version release only) master branch japicmp reference version set
> to the just released version
> ** (minor version release only) master branch japicmp exclusions have been
> cleared
> # Update the list of previous version in {{docs/config.toml}} on the master
> branch.
> # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch
> of the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released)
> # Update stable and master alias in
> [https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
> # Open discussion thread for End of Life for Unsupported version (i.e. 1.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34713) CLONE - Updates the docs stable version
lincoln lee created FLINK-34713:
---
Summary: CLONE - Updates the docs stable version
Key: FLINK-34713
URL: https://issues.apache.org/jira/browse/FLINK-34713
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Update docs to "stable" in {{docs/config.toml}} in the branch of the
_just-released_ version:
* Change V{{{}ersion{}}} from {{{}x.y-SNAPSHOT }}to \{{{}x.y.z{}}}, i.e.
{{1.6-SNAPSHOT}} to {{1.6.0}}
* Change V{{{}ersionTitle{}}} from {{x.y-SNAPSHOT}} to {{{}x.y{}}}, i.e.
{{1.6-SNAPSHOT}} to {{1.6}}
* Change Branch from {{master}} to {{{}release-x.y{}}}, i.e. {{master}} to
{{release-1.6}}
* Change {{baseURL}} from
{{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
to
{{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
* Change {{javadocs_baseurl}} from
{{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
to
{{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
* Change {{IsStable}} to {{true}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34712) CLONE - Update reference data for Migration Tests
lincoln lee created FLINK-34712:
---
Summary: CLONE - Update reference data for Migration Tests
Key: FLINK-34712
URL: https://issues.apache.org/jira/browse/FLINK-34712
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin
Fix For: 1.19.0, 1.18.1
Update migration tests in master to cover migration from new version. Since
1.18, this step could be done automatically with the following steps. For more
information please refer to [this
page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
# {*}On the published release tag (e.g., release-1.16.0){*}, run
{panel}
{panel}
|{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data
-Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|
The version (1.16 in the command above) should be replaced with the target one.
# Modify the content of the file
[apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
to the latest version (it would be "v1_16" if sticking to the example where
1.16.0 was released).
# Commit the modification in step a and b with "{_}[release] Generate
reference data for state migration tests based on release-1.xx.0{_}" to the
corresponding release branch (e.g. {{release-1.16}} in our example), replace
"xx" with the actual version (in this example "16"). You should use the Jira
issue ID in case of [release] as the commit message's prefix if you have a
dedicated Jira issue for this task.
# Cherry-pick the commit to the master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34710) CLONE - Apache mailing lists announcements
lincoln lee created FLINK-34710:
---
Summary: CLONE - Apache mailing lists announcements
Key: FLINK-34710
URL: https://issues.apache.org/jira/browse/FLINK-34710
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Announce on the {{dev@}} mailing list that the release has been finished.
Announce on the release on the {{user@}} mailing list, listing major
improvements and contributions.
Announce the release on the [annou...@apache.org|mailto:annou...@apache.org]
mailing list.
{panel}
{panel}
|{{From: Release Manager}}
{{To: d...@flink.apache.org, u...@flink.apache.org, user...@flink.apache.org,
annou...@apache.org}}
{{Subject: [ANNOUNCE] Apache Flink 1.2.3 released}}
{{The Apache Flink community is very happy to announce the release of Apache
Flink 1.2.3, which is the third bugfix release for the Apache Flink 1.2
series.}}
{{Apache Flink® is an open-source stream processing framework for distributed,
high-performing, always-available, and accurate data streaming applications.}}
{{The release is available for download at:}}
{{[https://flink.apache.org/downloads.html]}}
{{Please check out the release blog post for an overview of the improvements
for this bugfix release:}}
{{}}
{{The full release notes are available in Jira:}}
{{}}
{{We would like to thank all contributors of the Apache Flink community who
made this release possible!}}
{{Feel free to reach out to the release managers (or respond to this thread)
with feedback on the release process. Our goal is to constantly improve the
release process. Feedback on what could be improved or things that didn't go so
well are appreciated.}}
{{Regards,}}
{{Release Manager}}|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34709) CLONE - Remove outdated versions
lincoln lee created FLINK-34709:
---
Summary: CLONE - Remove outdated versions
Key: FLINK-34709
URL: https://issues.apache.org/jira/browse/FLINK-34709
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
h4. dist.apache.org
For a new major release remove all release files older than 2 versions, e.g.,
when releasing 1.7, remove all releases <= 1.5.
For a new bugfix version remove all release files for previous bugfix releases
in the same series, e.g., when releasing 1.7.1, remove the 1.7.0 release.
# If you have not already, check out the Flink section of the {{release}}
repository on {{[dist.apache.org|http://dist.apache.org/]}} via Subversion. In
a fresh directory:
{code}
svn checkout https://dist.apache.org/repos/dist/release/flink --depth=immediates
cd flink
{code}
# Remove files for outdated releases and commit the changes.
{code}
svn remove flink-
svn commit
{code}
# Verify that files are
[removed|https://dist.apache.org/repos/dist/release/flink]
(!) Remember to remove the corresponding download links from the website.
h4. CI
Disable the cron job for the now-unsupported version from
(tools/azure-pipelines/[build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
in the respective branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34708) CLONE - Merge website pull request
lincoln lee created FLINK-34708: --- Summary: CLONE - Merge website pull request Key: FLINK-34708 URL: https://issues.apache.org/jira/browse/FLINK-34708 Project: Flink Issue Type: Sub-task Reporter: Sergey Nuyanzin Merge the website pull request to [list the release|http://flink.apache.org/downloads.html]. Make sure to regenerate the website as well, as it isn't build automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34707) CLONE - Update japicmp configuration
lincoln lee created FLINK-34707:
---
Summary: CLONE - Update japicmp configuration
Key: FLINK-34707
URL: https://issues.apache.org/jira/browse/FLINK-34707
Project: Flink
Issue Type: Sub-task
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin
Fix For: 1.19.0, 1.18.1
Update the japicmp reference version and wipe exclusions / enable API
compatibility checks for {{@PublicEvolving}} APIs on the corresponding SNAPSHOT
branch with the {{update_japicmp_configuration.sh}} script (see below).
For a new major release (x.y.0), run the same command also on the master branch
for updating the japicmp reference version and removing out-dated exclusions in
the japicmp configuration.
Make sure that all Maven artifacts are already pushed to Maven Central.
Otherwise, there's a risk that CI fails due to missing reference artifacts.
{code:bash}
tools $ NEW_VERSION=$RELEASE_VERSION releasing/update_japicmp_configuration.sh
tools $ cd ..$ git add *$ git commit -m "Update japicmp configuration for
$RELEASE_VERSION" {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34706) Promote release 1.19
lincoln lee created FLINK-34706:
---
Summary: Promote release 1.19
Key: FLINK-34706
URL: https://issues.apache.org/jira/browse/FLINK-34706
Project: Flink
Issue Type: New Feature
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin
Assignee: Jing Ge
Once the release has been finalized (FLINK-32920), the last step of the process
is to promote the release within the project and beyond. Please wait for 24h
after finalizing the release in accordance with the [ASF release
policy|http://www.apache.org/legal/release-policy.html#release-announcements].
*Final checklist to declare this issue resolved:*
# Website pull request to [list the
release|http://flink.apache.org/downloads.html] merged
# Release announced on the user@ mailing list.
# Blog post published, if applicable.
# Release recorded in
[reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
# Release announced on social media.
# Completion declared on the dev@ mailing list.
# Update Homebrew: [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request]
(seems to be done automatically - at least for minor releases for both minor
and major releases)
# Updated the japicmp configuration
** corresponding SNAPSHOT branch japicmp reference version set to the just
released version, and API compatibiltity checks for {{@PublicEvolving}} was
enabled
** (minor version release only) master branch japicmp reference version set to
the just released version
** (minor version release only) master branch japicmp exclusions have been
cleared
# Update the list of previous version in {{docs/config.toml}} on the master
branch.
# Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch of
the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released)
# Update stable and master alias in
[https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
# Open discussion thread for End of Life for Unsupported version (i.e. 1.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [build] fix jackson conflicts among cdc connectors [flink-cdc]
Shawn-Hx commented on code in PR #2987: URL: https://github.com/apache/flink-cdc/pull/2987#discussion_r1527909731 ## pom.xml: ## @@ -95,6 +96,36 @@ under the License. + +com.fasterxml.jackson.core Review Comment: Thanks for your contribution. I think using [jackson-bom](https://github.com/FasterXML/jackson-bom) here is more clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34668][checkpoint] Report operator state handle of file merging directory to JM [flink]
flinkbot commented on PR #24513: URL: https://github.com/apache/flink/pull/24513#issuecomment-2003011938 ## CI report: * 8cc6814c6ea3b5dee6280f350ec293cb06ece658 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34684 & 34683][docs] Add licenses & contributing docs in Flink CDC [flink-cdc]
LYanquan commented on code in PR #3159: URL: https://github.com/apache/flink-cdc/pull/3159#discussion_r1527773536 ## docs/content/docs/developer-guide/contribute-to-flink-cdc.md: ## @@ -23,3 +23,62 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Contributing + +Flink CDC is developed by an open and friendly community and welcomes anyone who wants to help out in any way. +There are several ways to interact with the community and contribute to Flink CDC including asking questions, filing +bug reports, proposing new features, joining discussions on the mailing lists, contributing code or documentation, +improving website, testing release candidates and writing corresponding blog etc. + +## What do you want to do + +Contributing to Flink CDC goes beyond writing code for the project. Here are different opportunities to help the +project as follows. + +| Area| Further information | +|:|:--| +| Report Bug | To report a problem, open an issue in [Flink jira](https://issues.apache.org/jira/projects/FLINK/issues) and select `Flink CDC` in `Component/s`. Please give detailed information about the problem you encountered and, if possible, add a description that helps to reproduce the problem. | +| Contribute Code | Read the Code Contribution Guide | +| Code Reviews| Read the Code Review Guide | Review Comment: ~~We should update the actual page addresses of these two links.~~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34668) Report State handle of file merging directory to JM
[ https://issues.apache.org/jira/browse/FLINK-34668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34668: --- Labels: pull-request-available (was: ) > Report State handle of file merging directory to JM > --- > > Key: FLINK-34668 > URL: https://issues.apache.org/jira/browse/FLINK-34668 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Zakelly Lan >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34668][checkpoint] Report operator state handle of file merging directory to JM [flink]
fredia opened a new pull request, #24513: URL: https://github.com/apache/flink/pull/24513 ## What is the purpose of the change Operator states are stored in `taskownd/` dir when file merging is enabled, this PR reports the operator state handle of file merging directory to JM `SharedStateRegistry`. ## Brief change log - Introduce `FileMergingOperatorStreamStateHandle` , `EmptyFileMergingOperatorStreamStateHandle` and `DirectoryStreamStateHandle`. - Still report file merging directory to JM even if operator state is empty. ## Verifying this change This change added tests and can be verified as follows: - `OperatorStateBackendTest#testFileMergingSnapshotEmpty` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( checkpointing ) - The S3 file system connector: ( no ) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422: URL: https://github.com/apache/flink/pull/24422#discussion_r1527893607 ## flink-process-function-parent/flink-process-function-api/src/main/java/org/apache/flink/process/api/stream/KeyedPartitionStream.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.process.api.stream; + +import org.apache.flink.annotation.Experimental; +import org.apache.flink.api.connector.v2.Sink; +import org.apache.flink.api.java.functions.KeySelector; +import org.apache.flink.process.api.function.OneInputStreamProcessFunction; +import org.apache.flink.process.api.function.TwoInputBroadcastStreamProcessFunction; +import org.apache.flink.process.api.function.TwoInputNonBroadcastStreamProcessFunction; +import org.apache.flink.process.api.function.TwoOutputStreamProcessFunction; +import org.apache.flink.process.api.stream.NonKeyedPartitionStream.TwoNonKeyedPartitionStreams; + +/** This interface represents a stream that each parallel task processes the same data. */ Review Comment: Ai, good catch!. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [build] fix inconsistent Kafka shading among cdc connectors [flink-cdc]
link3280 commented on code in PR #2988: URL: https://github.com/apache/flink-cdc/pull/2988#discussion_r1527893588 ## flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-tidb-cdc/pom.xml: ## @@ -60,6 +60,12 @@ under the License. + +org.apache.kafka + + com.ververica.cdc.connectors.shaded.org.apache.kafka Review Comment: I found no shaded pattern has migrated from `com.ververica.cdc.connectors.shaded` yet. Maybe we should do it all together in an other PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527888923
##
flink-process-function-parent/flink-process-function-api/src/main/java/org/apache/flink/process/api/stream/BroadcastStream.java:
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.api.stream;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.java.functions.KeySelector;
+import
org.apache.flink.process.api.function.TwoInputBroadcastStreamProcessFunction;
+
+/** This interface represents a stream that each parallel task processes the
same data. */
+@Experimental
+public interface BroadcastStream {
+/**
+ * Apply a two input operation to this and other {@link
KeyedPartitionStream}.
+ *
+ * @param other {@link KeyedPartitionStream} to perform operation with two
input.
+ * @param processFunction to perform operation.
+ * @return new stream with this operation.
+ */
+ NonKeyedPartitionStream connectAndProcess(
+KeyedPartitionStream other,
+TwoInputBroadcastStreamProcessFunction
processFunction);
+
+/**
+ * Apply a two input operation to this and other {@link
NonKeyedPartitionStream}.
+ *
+ * @param other {@link NonKeyedPartitionStream} to perform operation with
two input.
+ * @param processFunction to perform operation.
+ * @return new stream with this operation.
+ */
+ NonKeyedPartitionStream connectAndProcess(
+NonKeyedPartitionStream other,
+TwoInputBroadcastStreamProcessFunction
processFunction);
+
+/**
+ * Apply a two input operation to this and other {@link
KeyedPartitionStream}.
+ *
+ * This method is used to avoid shuffle after applying the process
function. It is required
+ * that for the record from non-broadcast input, the new {@link
KeySelector} must extract the
+ * same key as the original {@link KeySelector}s on the {@link
KeyedPartitionStream}. For the
+ * record from broadcast input, the output key from keyed partition itself
instead of new key
+ * selector, so it is safe already.
Review Comment:
Yes, the clearer the better. But I'm not sure what is not clear, could you
give me some proposed changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [build] fix inconsistent Kafka shading among cdc connectors [flink-cdc]
link3280 commented on code in PR #2988: URL: https://github.com/apache/flink-cdc/pull/2988#discussion_r1527886955 ## flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-tidb-cdc/pom.xml: ## @@ -60,6 +60,12 @@ under the License. + +org.apache.kafka + + com.ververica.cdc.connectors.shaded.org.apache.kafka Review Comment: You're right. I didn't notice the package name had changed. Hold on a sec. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527883895
##
flink-core-api/pom.xml:
##
@@ -0,0 +1,60 @@
+
+
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-parent
+ 1.20-SNAPSHOT
+
+
+ flink-core-api
+ Flink : Core API
+
+ jar
+
+
+
+ org.apache.flink
+ flink-annotations
+ ${project.version}
+
+
+
+
+
+
Review Comment:
Make sense. I first introduced it and thinking that this PR-moved class from
flink-core would have a corresponding test class, but it doesn't. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527879616
##
flink-process-function-parent/flink-process-function/src/main/java/org/apache/flink/process/impl/ExecutionEnvironmentImpl.java:
##
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.impl;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.RuntimeExecutionMode;
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.functions.InvalidTypesException;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.v2.FromDataSource;
+import org.apache.flink.api.connector.v2.Source;
+import org.apache.flink.api.connector.v2.WrappedSource;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.typeutils.MissingTypeInfo;
+import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.DeploymentOptions;
+import org.apache.flink.configuration.ExecutionOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import
org.apache.flink.connector.datagen.functions.FromElementsGeneratorFunction;
+import org.apache.flink.connector.datagen.source.DataGeneratorSource;
+import org.apache.flink.core.execution.DefaultExecutorServiceLoader;
+import org.apache.flink.core.execution.JobClient;
+import org.apache.flink.core.execution.PipelineExecutor;
+import org.apache.flink.core.execution.PipelineExecutorFactory;
+import org.apache.flink.core.execution.PipelineExecutorServiceLoader;
+import org.apache.flink.process.api.ExecutionEnvironment;
+import org.apache.flink.process.api.stream.NonKeyedPartitionStream;
+import org.apache.flink.process.impl.stream.NonKeyedPartitionStreamImpl;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.graph.StreamGraph;
+import org.apache.flink.streaming.api.graph.StreamGraphGenerator;
+import org.apache.flink.streaming.api.transformations.SourceTransformation;
+import
org.apache.flink.streaming.runtime.translators.ProcessFunctionSinkTransformationTranslator;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+
+import static
org.apache.flink.streaming.api.graph.StreamGraphGenerator.DEFAULT_TIME_CHARACTERISTIC;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** The implementation of {@link ExecutionEnvironment}. */
+public class ExecutionEnvironmentImpl implements ExecutionEnvironment {
+private final List> transformations = new ArrayList<>();
+
+private final ExecutionConfig executionConfig;
+
+/** Settings that control the checkpointing behavior. */
+private final CheckpointConfig checkpointCfg;
+
+private final Configuration configuration;
+
+private final ClassLoader userClassloader;
+
+private final PipelineExecutorServiceLoader executorServiceLoader;
+
+static {
+try {
+// All transformation translator must be put to a map in
StreamGraphGenerator, but
+// streaming-java is not depend on process-function module, using
reflect to handle
+// this.
+
ProcessFunctionSinkTransformationTranslator.registerSinkTransformationTranslator();
+} catch (Exception e) {
+throw new RuntimeException(
+"Can not register process function transformation
translator.");
+}
+}
+
+/**
+ * The environment of the context (local by default, cluster if invoked
through command line).
+ */
+private static ExecutionEnvironmentFactory contextEnvironmentFactory =
null;
+
+public static ExecutionEnvironment newInstance() {
+if (contextEnvironmentFactory != null) {
+
Re: [PR] [build] fix inconsistent Kafka shading among cdc connectors [flink-cdc]
leonardBang commented on code in PR #2988: URL: https://github.com/apache/flink-cdc/pull/2988#discussion_r1527879357 ## flink-cdc-connect/flink-cdc-source-connectors/flink-sql-connector-tidb-cdc/pom.xml: ## @@ -60,6 +60,12 @@ under the License. + +org.apache.kafka + + com.ververica.cdc.connectors.shaded.org.apache.kafka Review Comment: Maybe we should change ` com.ververica.` to `org.apache.flink` after we donated to Apache? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34679][cdc] add doc under core-concept. [flink-cdc]
LYanquan commented on PR #3153: URL: https://github.com/apache/flink-cdc/pull/3153#issuecomment-2002977044 address it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527870916
##
flink-process-function-parent/flink-process-function/src/main/java/org/apache/flink/streaming/api/transformations/ProcessFunctionSinkTransformation.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.process.impl.stream.DataStream;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import org.apache.commons.compress.utils.Lists;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@link Transformation} for sink-v2 in process function.
+ *
+ * @param The input type of the {@link SinkWriter}
+ * @param The output type of the {@link Sink}
+ */
+@Internal
+public class ProcessFunctionSinkTransformation
Review Comment:
`SinkTransformation` and `DataStream(v1 version)` are tightly coupled.
Unfortunately, V1 and V2 can't be converted to each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527867946
##
flink-process-function-parent/flink-process-function/src/main/java/org/apache/flink/process/impl/operators/ProcessOperator.java:
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.impl.operators;
+
+import org.apache.flink.process.api.function.OneInputStreamProcessFunction;
+import org.apache.flink.process.impl.common.OutputCollector;
+import org.apache.flink.process.impl.common.TimestampCollector;
+import org.apache.flink.process.impl.context.DefaultNonPartitionedContext;
+import org.apache.flink.process.impl.context.DefaultRuntimeContext;
+import org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+/** Operator for {@link OneInputStreamProcessFunction}. */
+public class ProcessOperator
+extends AbstractUdfStreamOperator>
+implements OneInputStreamOperator, BoundedOneInput {
+
+protected transient DefaultRuntimeContext context;
+
+protected transient DefaultNonPartitionedContext
nonPartitionedContext;
+
+protected transient TimestampCollector outputCollector;
+
+public ProcessOperator(OneInputStreamProcessFunction
userFunction) {
+super(userFunction);
+
+chainingStrategy = ChainingStrategy.ALWAYS;
Review Comment:
https://github.com/apache/flink/blob/841f23c73e4399df91112dd11ddca74f45ea5b37/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SimpleOperatorFactory.java#L92-L98
IIRC, `AbstractStreamOperator` is always a `SetupableStreamOperator` with
the default chainingStrategy `HEAD`. 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34689][MySQL][Feature] check binlog_row_value_optoins [flink-cdc]
leonardBang commented on PR #3148: URL: https://github.com/apache/flink-cdc/pull/3148#issuecomment-2002970708 Thanks @SML0127 for the contribution, @ruanhang1993 Would you like to help review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
reswqa commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1527863818
##
flink-process-function-parent/flink-process-function-api/src/main/java/org/apache/flink/process/api/function/ProcessFunction.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.api.function;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.functions.Function;
+
+/** Base class for all user defined process functions. */
+@Experimental
+public interface ProcessFunction extends Function {
+/**
+ * Initialization method for the function. It is called before the actual
working methods (like
+ * processRecord) and thus suitable for one time setup work.
+ *
+ * By default, this method does nothing.
+ *
+ * @throws Exception Implementations may forward exceptions, which are
caught by the runtime.
+ * When the runtime catches an exception, it aborts the task and lets
the fail-over logic
+ * decide whether to retry the task execution.
+ */
+default void open() throws Exception {}
Review Comment:
I just didn't realize what we need to do in `OpenContext` at the moment?
Since the V2 API is experimental, we can introduce such contexts to it before
it becomes public, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34702) Rank should not convert to StreamExecDuplicate when the input is not insert only
[
https://issues.apache.org/jira/browse/FLINK-34702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827843#comment-17827843
]
lincoln lee commented on FLINK-34702:
-
[~jackylau] Thank you for reporting this!
This should be an improvement but not a bug. Currently, some operators do not
support processing changelog input, and `StreamPhysicalDeduplicate` is one of
them.
For the solution, can we consider taking a step forward, allowing
`StreamPhysicalDeduplicate` to support changelog input (let the physical
operator support this input, just like we added support changelog input for
Window TVF Aggregation in 1.19), instead of patching with a switch during the
optimization phase to bypass it, WDYT?
> Rank should not convert to StreamExecDuplicate when the input is not insert
> only
>
>
> Key: FLINK-34702
> URL: https://issues.apache.org/jira/browse/FLINK-34702
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.20.0
>Reporter: Jacky Lau
>Priority: Major
> Fix For: 1.20.0
>
>
> {code:java}
> @Test
> def testSimpleFirstRowOnBuiltinProctime1(): Unit = {
> val sqlQuery =
> """
> |SELECT *
> |FROM (
> | SELECT *,
> |ROW_NUMBER() OVER (PARTITION BY a ORDER BY PROCTIME() ASC) as
> rowNum
> | FROM (select a, count(b) as b from MyTable group by a)
> |)
> |WHERE rowNum = 1
> """.stripMargin
> util.verifyExecPlan(sqlQuery)
> } {code}
> Exception:
> org.apache.flink.table.api.TableException: StreamPhysicalDeduplicate doesn't
> support consuming update changes which is produced by node
> GroupAggregate(groupBy=[a], select=[a, COUNT(b) AS b])
> because the StreamPhysicalDeduplicate can not consuming update changes now
> while StreamExecRank can.
> so we should not convert the FlinkLogicalRank to StreamPhysicalDeduplicate in
> this case. and we can defer whether input contains update change in the
> "optimize the physical plan" phase.
> so we can add an option to solve it. and when the StreamPhysicalDeduplicate
> can support consuming update changes , we can deprecate it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34689) check binlog_row_value_optoins
[
https://issues.apache.org/jira/browse/FLINK-34689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827840#comment-17827840
]
Lee SeungMin commented on FLINK-34689:
--
Assign this pr to me please
> check binlog_row_value_optoins
> --
>
> Key: FLINK-34689
> URL: https://issues.apache.org/jira/browse/FLINK-34689
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
>Reporter: Lee SeungMin
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-03-15-12-56-49-344.png
>
>
> When {{binlog_row_value_optoins}} is set to {{{}PARTIAL_JSON{}}},
> the update operator remains as {{{}Update_rows_partial{}}}.
> Flink CDC does not parse this event because {{Update_row_partial}} binlog
> event is mapped to {{PARTIAL_UPDATE_ROWS_EVENT}} and Flink CDC do not handle
> that event type
>
> Example of Update_row_partial (when {{binlog_row_value_optoins}} =
> {{PARTIAL_JSON)}}
> !image-2024-03-15-12-56-49-344.png|width=1015,height=30!
> So, we have to check {{binlog_row_value_optoins}} before starting.
>
>
> Cretae PR: [[MySQL][Feature] check binlog_row_value_optoins by SML0127 · Pull
> Request #3148 · apache/flink-cdc
> (github.com)|https://github.com/apache/flink-cdc/pull/3148]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34680) "Connectors - Overview" Page for Flink CDC Documentation
[ https://issues.apache.org/jira/browse/FLINK-34680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34680: --- Labels: pull-request-available (was: ) > "Connectors - Overview" Page for Flink CDC Documentation > - > > Key: FLINK-34680 > URL: https://issues.apache.org/jira/browse/FLINK-34680 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34705) org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase FAILURE
[
https://issues.apache.org/jira/browse/FLINK-34705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
baihailiang updated FLINK-34705:
Description:
{code:java}
// code placeholder
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 111.197
s <<< FAILURE! - in
org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase
[ERROR] org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase
Time elapsed: 111.197 s <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Container startup failed
at
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:345)
at
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
at
org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
at
org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229)
at
org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$4(ExtensionValuesStore.java:86)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.computeValue(ExtensionValuesStore.java:223)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:211)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.evaluate(ExtensionValuesStore.java:191)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.access$100(ExtensionValuesStore.java:171)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:89)
at
org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
at
org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$10(ClassBasedTestDescriptor.java:381)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:381)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:205)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:80)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolH
[jira] [Updated] (FLINK-34548) FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction
[ https://issues.apache.org/jira/browse/FLINK-34548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34548: --- Labels: pull-request-available (was: ) > FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and > ProcessFunction > - > > Key: FLINK-34548 > URL: https://issues.apache.org/jira/browse/FLINK-34548 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Major > Labels: pull-request-available > > This is the ticket for FLIP-409: DataStream V2 Building Blocks: DataStream, > Partitioning and ProcessFunction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34705) org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase FAILURE
baihailiang created FLINK-34705:
---
Summary:
org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase FAILURE
Key: FLINK-34705
URL: https://issues.apache.org/jira/browse/FLINK-34705
Project: Flink
Issue Type: Bug
Components: Connectors / ElasticSearch
Affects Versions: 1.16.2
Environment: Environment is x86 machine
Reporter: baihailiang
{code:java}
// code placeholder
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 111.197
s <<< FAILURE! - in
org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase
[ERROR] org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase
Time elapsed: 111.197 s <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Container startup failed
at
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:345)
at
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:326)
at
org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
at
org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229)
at
org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$4(ExtensionValuesStore.java:86)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.computeValue(ExtensionValuesStore.java:223)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:211)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.evaluate(ExtensionValuesStore.java:191)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.access$100(ExtensionValuesStore.java:171)
at
org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:89)
at
org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
at
org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at
org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$10(ClassBasedTestDescriptor.java:381)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:381)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:205)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:80)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.jav
Re: [PR] [FLINK-34548][api] Introduce DataStream, Partitioning and ProcessFunction [flink]
WencongLiu commented on code in PR #24422:
URL: https://github.com/apache/flink/pull/24422#discussion_r1523101423
##
flink-process-function-parent/flink-process-function-api/src/main/java/org/apache/flink/process/api/stream/BroadcastStream.java:
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.api.stream;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.java.functions.KeySelector;
+import
org.apache.flink.process.api.function.TwoInputBroadcastStreamProcessFunction;
+
+/** This interface represents a stream that each parallel task processes the
same data. */
+@Experimental
+public interface BroadcastStream {
+/**
+ * Apply a two input operation to this and other {@link
KeyedPartitionStream}.
+ *
+ * @param other {@link KeyedPartitionStream} to perform operation with two
input.
+ * @param processFunction to perform operation.
+ * @return new stream with this operation.
+ */
+ NonKeyedPartitionStream connectAndProcess(
+KeyedPartitionStream other,
+TwoInputBroadcastStreamProcessFunction
processFunction);
+
+/**
+ * Apply a two input operation to this and other {@link
NonKeyedPartitionStream}.
+ *
+ * @param other {@link NonKeyedPartitionStream} to perform operation with
two input.
+ * @param processFunction to perform operation.
+ * @return new stream with this operation.
+ */
+ NonKeyedPartitionStream connectAndProcess(
+NonKeyedPartitionStream other,
+TwoInputBroadcastStreamProcessFunction
processFunction);
+
+/**
+ * Apply a two input operation to this and other {@link
KeyedPartitionStream}.
+ *
+ * This method is used to avoid shuffle after applying the process
function. It is required
+ * that for the record from non-broadcast input, the new {@link
KeySelector} must extract the
+ * same key as the original {@link KeySelector}s on the {@link
KeyedPartitionStream}. For the
+ * record from broadcast input, the output key from keyed partition itself
instead of new key
+ * selector, so it is safe already.
Review Comment:
For `KeyedPartitionStream` and `BroadcastStream`, we may need to explain
more details on why we need a new key selector, why it avoid shuffle process,
and the concept of `safe`.
##
flink-process-function-parent/flink-process-function/src/main/java/org/apache/flink/process/impl/utils/StreamUtils.java:
##
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.process.impl.utils;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.v2.Sink;
+import org.apache.flink.api.connector.v2.WrappedSink;
+import org.apache.flink.api.java.Utils;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.process.api.function.OneInputStreamProcessFunction;
+import
org.apache.flink.process.api.function.TwoInputBroadcastStreamProcessFunction;
+import
org.apache.flink.process.api.function.TwoInputNonBroadcastStreamProcessFunction;
+import org.apache.flink.process.api.function.TwoOutputStreamProcessFunction;
+import org.apache.flink.process.impl.stream.DataStream;
+import org.apache.flink.pr
Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]
flinkbot commented on PR #24512: URL: https://github.com/apache/flink/pull/24512#issuecomment-2002894151 ## CI report: * 79ca29c5dd966fe4af885c9e5b18a737c5e5b6e9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]
LadyForest opened a new pull request, #24512: URL: https://github.com/apache/flink/pull/24512 ## What is the purpose of the change This PR is cherry-picked from https://github.com/apache/flink/pull/24390 ## Brief change log - Fix unstable TableSourceITCase#testTableHintWithLogicalTableScanReuse - Moves the staging dir configuration into builder for easier testing ## Verifying this change FileSystemOutputFormatTest#testGetUniqueStagingDirectory ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34684 & 34683][docs] Add licenses & contributing docs in Flink CDC [flink-cdc]
LYanquan commented on code in PR #3159: URL: https://github.com/apache/flink-cdc/pull/3159#discussion_r1527773536 ## docs/content/docs/developer-guide/contribute-to-flink-cdc.md: ## @@ -23,3 +23,62 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Contributing + +Flink CDC is developed by an open and friendly community and welcomes anyone who wants to help out in any way. +There are several ways to interact with the community and contribute to Flink CDC including asking questions, filing +bug reports, proposing new features, joining discussions on the mailing lists, contributing code or documentation, +improving website, testing release candidates and writing corresponding blog etc. + +## What do you want to do + +Contributing to Flink CDC goes beyond writing code for the project. Here are different opportunities to help the +project as follows. + +| Area| Further information | +|:|:--| +| Report Bug | To report a problem, open an issue in [Flink jira](https://issues.apache.org/jira/projects/FLINK/issues) and select `Flink CDC` in `Component/s`. Please give detailed information about the problem you encountered and, if possible, add a description that helps to reproduce the problem. | +| Contribute Code | Read the Code Contribution Guide | +| Code Reviews| Read the Code Review Guide | Review Comment: We should update the actual page addresses of these two links. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34684 & 34683][docs] Add licenses & contributing docs in Flink CDC [flink-cdc]
LYanquan commented on code in PR #3159: URL: https://github.com/apache/flink-cdc/pull/3159#discussion_r1527773536 ## docs/content/docs/developer-guide/contribute-to-flink-cdc.md: ## @@ -23,3 +23,62 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Contributing + +Flink CDC is developed by an open and friendly community and welcomes anyone who wants to help out in any way. +There are several ways to interact with the community and contribute to Flink CDC including asking questions, filing +bug reports, proposing new features, joining discussions on the mailing lists, contributing code or documentation, +improving website, testing release candidates and writing corresponding blog etc. + +## What do you want to do + +Contributing to Flink CDC goes beyond writing code for the project. Here are different opportunities to help the +project as follows. + +| Area| Further information | +|:|:--| +| Report Bug | To report a problem, open an issue in [Flink jira](https://issues.apache.org/jira/projects/FLINK/issues) and select `Flink CDC` in `Component/s`. Please give detailed information about the problem you encountered and, if possible, add a description that helps to reproduce the problem. | +| Contribute Code | Read the Code Contribution Guide | +| Code Reviews| Read the Code Review Guide | Review Comment: We should update the actual addresses of these two links. ## docs/content/docs/developer-guide/contribute-to-flink-cdc.md: ## @@ -23,3 +23,62 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Contributing + +Flink CDC is developed by an open and friendly community and welcomes anyone who wants to help out in any way. +There are several ways to interact with the community and contribute to Flink CDC including asking questions, filing +bug reports, proposing new features, joining discussions on the mailing lists, contributing code or documentation, +improving website, testing release candidates and writing corresponding blog etc. + +## What do you want to do + +Contributing to Flink CDC goes beyond writing code for the project. Here are different opportunities to help the +project as follows. + +| Area| Further information | +|:|:--| +| Report Bug | To report a problem, open an issue in [Flink jira](https://issues.apache.org/jira/projects/FLINK/issues) and select `Flink CDC` in `Component/s`. Please give detailed information about the problem you encountered and, if possible, add a description that helps to reproduce the problem. | +| Contribute Code | Read the Code Contribution Guide | +| Code Reviews| Read the Code Review Guide | +| Support Users | Reply to questions on the flink user mailing list, check the latest is
Re: [PR] [FLINK-34679][cdc] add doc under core-concept. [flink-cdc]
leonardBang commented on code in PR #3153: URL: https://github.com/apache/flink-cdc/pull/3153#discussion_r1527763427 ## docs/content/docs/core-concept/data-pipeline.md: ## @@ -23,3 +23,80 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Definition +Since events in Flink CDC flow from the upstream to the downstream in a pipeline manner, the data synchronization task is also referred as a Data Pipeline. Review Comment: we can use **Data Pipeline** to highlight ## docs/content/docs/core-concept/route.md: ## @@ -23,3 +23,52 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Definition +Route specifies the rule of matching a list of source-table and mapping to sink-table. The most typical scenario is the merge of sub-databases and sub-tables, routing multiple upstream source tables to the same sink table. + +# Parameters +To describe a route, the follows are required: + +| parameter | meaning | +|---|--| +| source-table | Source table id, supports regular expressions | +| sink-table | Sink table id, supports regular expressions | +| description | Routing rule description(optional, default value provided)| + +A route module can contain a list of source-table/sink-table rules. + +# Example +## one to one +if synchronize the table `web_order` in the database `mydb` to a Doris table `ods_web_order`, we can use this yaml file to define this route： + +```yaml +route: +source-table: mydb.web_order +sink-table: mydb.ods_web_order +description: sync table to one destination table with given prefix ods_ +``` + +## many to one Review Comment: Route multiple Data Source tables to one Data Sink table ## docs/content/docs/core-concept/data-pipeline.md: ## @@ -23,3 +23,80 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Definition +Since events in Flink CDC flow from the upstream to the downstream in a pipeline manner, the data synchronization task is also referred as a Data Pipeline. + +# Parameters +A pipeline corresponds to a chain of operators in Flink. +To describe a Data Pipeline, the following parts are required: +- [source](data-source.md) +- [sink](data-sink.md) +- [pipeline](data-pipeline.md#global-parameters) (global parameters) + +the following parts are optional: +- [route](route.md) +- [transform](transform.md) + +# Example +## Only required +We could use this concise yaml file to define a pipeline: + +```yaml + source: + type: mysql + hostname: localhost + port: 3306 + username: root + password: 123456 + tables: app_db.\.* + + sink: + type: doris + fenodes: 127.0.0.1:8030 + username: root + password: "" + + pipeline: + name: Sync MySQL Database to Doris + parallelism: 2 +``` + +## With optional +We could use this complicated yaml file to define a pipeline: + +```yaml + source: + type: mysql + hostname: localhost + port: 3306 + username: root + password: 123456 + tables: app_db.\.* + + sink: + type: doris + fenodes: 127.0.0.1:8030 + username: root + password: "" + route: + - source-table: app_db.orders + sink-table: ods_db.ods_orders + - source-table: app_db.shipments + sink-table: ods_db.ods_shipments + - source-table: app_db.products + sink-table: ods_db.ods_products + + pipeline: + name: Sync MySQL Database to Doris + parallelism: 2 +``` + +# Global Parameters +The following parameters are global parameters of the pipeline: Review Comment: The following config options of Data Pipeline level are supported: ## docs/content/docs/core-concept/data-pipeline.md: ## @@ -23,3 +23,80 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Definition +Since events in Flink CDC flow from the upstream to the downstream in a pipeline manner, the data synchronization task is also referred as a Data Pipeline. Review Comment: the data synchronization task -> the whole ETL task is referred as a ** Data Pipeline ** ## docs/content/docs/core-concept/route.md: ## @@ -23,3 +23,52 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Definition +Route specifies the rule of matching a list of source-table and mapping to sink-table. The most typical scenario is the merge of sub-databases and sub-tables, routing multiple upstream source tables to the same sink t
[jira] [Created] (FLINK-34704) Process checkpoint barrier in AsyncWaitOperator when the element queue is full
Zakelly Lan created FLINK-34704: --- Summary: Process checkpoint barrier in AsyncWaitOperator when the element queue is full Key: FLINK-34704 URL: https://issues.apache.org/jira/browse/FLINK-34704 Project: Flink Issue Type: Improvement Components: Runtime / Task Reporter: Zakelly Lan As discussed in https://lists.apache.org/thread/4f7ywn29kdv4302j2rq3fkxc6pc8myr2 . Maybe it is better to provide such a new `yield` that can process mail with low priority in the mailbox executor. More discussion needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]
flinkbot commented on PR #24511: URL: https://github.com/apache/flink/pull/24511#issuecomment-2002875245 ## CI report: * 09a83bf383c13f3262a61279871cc0e9882f6e47 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34678][cdc] Add introduction page for Flink CDC docs [flink-cdc]
leonardBang commented on code in PR #3158: URL: https://github.com/apache/flink-cdc/pull/3158#discussion_r1527756592 ## docs/content/docs/get-started/introduction.md: ## @@ -23,3 +23,98 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Welcome to Flink CDC 🎉 + +Flink CDC is a stream data integration framework that aims to provide users with Review Comment: Flink CDC is a streaming data integration tool ## docs/content/docs/get-started/introduction.md: ## @@ -23,3 +23,98 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Welcome to Flink CDC 🎉 + +Flink CDC is a stream data integration framework that aims to provide users with +a more robust API. It allows users to configure their data synchronization logic +through customized Flink operators and job submission tools. The framework +prioritizes optimizing the task submission process and offers enhanced +functionalities such as whole database synchronization, sharding, and schema +change synchronization. Review Comment: please keep same with homepage key feature section. ## docs/content/docs/get-started/introduction.md: ## @@ -23,3 +23,98 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Welcome to Flink CDC 🎉 + +Flink CDC is a stream data integration framework that aims to provide users with +a more robust API. It allows users to configure their data synchronization logic +through customized Flink operators and job submission tools. The framework +prioritizes optimizing the task submission process and offers enhanced Review Comment: The framework prioritizes > Flink CDC prioritizes ## docs/content/docs/get-started/introduction.md: ## @@ -23,3 +23,98 @@ KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> + +# Welcome to Flink CDC 🎉 + +Flink CDC is a stream data integration framework that aims to provide users with +a more robust API. It allows users to configure their data synchronization logic +through customized Flink operators and job submission tools. The framework Review Comment: It allows users to describe their ETL pipeline logic via YAML elegantly and help users automatically generating customized Flink operators and submitting job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-29114][connector][filesystem] Fix issue of file overwriting caused by multiple writes to the same sink table and shared staging directory [flink]
LadyForest opened a new pull request, #24511: URL: https://github.com/apache/flink/pull/24511 ## What is the purpose of the change This PR is cherry-picked from https://github.com/apache/flink/pull/24390 ## Brief change log - Fix unstable TableSourceITCase#testTableHintWithLogicalTableScanReuse - Moves the staging dir configuration into builder for easier testing ## Verifying this change FileSystemOutputFormatTest#testGetUniqueStagingDirectory ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializer: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34687][cdc][docs] Build the home page of Flink CDC documentation [flink-cdc]
PatrickRen commented on code in PR #3156:
URL: https://github.com/apache/flink-cdc/pull/3156#discussion_r1527754771
##
docs/content/_index.md:
##
@@ -22,17 +22,179 @@ specific language governing permissions and limitations
under the License.
-->
-
+
+
+
+
+Flink CDC
+
+
+The streaming data integration tool
+
+
+
+
+
+
+
+
+
+
+
+
+ {{< img src="/fig/cdc-flow.png" alt="Flink CDC Flow">}}
+
+
+
+
+
+
-
-
-Flink CDC: Change Data Capture Solution Of Apache Flink
-
-Set of source connectors for Apache Flink® directly
ingesting changes coming from different databases using Change Data
Capture(CDC).
+
+
+
+
+ What is
Flink CDC?
+
+
+
+Flink CDC is a distributed data integration tool for real time
data and batch data.
+Flink CDC brings the simplicity and elegance of data integration
via YAML to describe
+the data movement and transformation.
+
+
+ {{< img src="/fig/index-yaml-example.png" alt="Flink CDC Example">}}
+
+
+
+
+
-Flink CDC integrates Debezium as the engine to capture data changes. So it can
fully leverage the ability of Debezium. See more about what is
[Debezium](https://github.com/debezium/debezium).
-{{< img src="/fig/cdc-flow.png" alt="Stateful Functions" width="50%" >}}
+
+
+
+
+
+ Key
Features
+
+
+
+
+
+
+
+
+
+
+ Change Data Capture
+
+
+FLink CDC supports distributed scanning of historical data of
database and then automatically switches to change data capturing. The switch
uses the incremental snapshot algorithm which ensure the switch action does not
lock the database.
+
+
+
+
+ Schema Evolution
+
+
+Flink CDC will automatically create downstream table using the
inferred table structure based on upstream table. Flink CDC will automatically
apply upstream DDL to downstream systems during change data capturing.
+
+
+
+ Streaming Pipeline
+
+
+Flink CDC jobs run in streaming mode by default, providing
sub-second end-to-end latency in real-time binlog synchronization scenarios,
this feature effectively ensures data freshness for downstream businesses.
+
+
+
+
+
+
+
+
+
+
+
+
+ Data Transformation
+
+
+Flink CDC will soon support data transform operations of ETL,
including column projection, computed column, filter expression and classical
scalar functions.
+
+
+
+ Full Database Sync
+
+ Flink CDC supports synchronizing all
tables of source database instance to downstream in one job, user can config
the captured database list and table list.
Review Comment:
```suggestion
Flink CDC supports synchronizing all
tables of source database instance to downstream in one job by configuring the
captured database list and table list.
```
##
docs/content/_index.md:
##
@@ -22,17 +22,179 @@ specific language governing permissions and limitations
under the License.
-->
-
+
+
+
+
+Flink CDC
+
+
+The streaming data integration tool
+
+
+
+
+
+
+
+
+
+
+
+
+ {{< img src="/fig/cdc-flow.png" alt="Flink CDC Flow">}}
+
+
+
+
+
+
-
-
-Flink CDC: Change Data Capture Solution Of Apache Flink
-
-Set of source connectors for Apache Flink® directly
ingesting changes coming from different databases using Change Data
Capture(CDC).
+
+
+
+
+ What is
Flink CDC?
+
+
+
+Flink CDC is a distributed data integration tool for real time
data and batch data.
+Flink CDC brings the simplicity and elegance of data integration
via YAML to describe
+the data movement and transformation.
+
+
+ {{< img src="/fig/index-yaml-example.png" alt="Flink CDC Example">}}
+
+
+
+
+
-Flink CDC integrates Debezium as the engine to capture data changes. So it can
fully leverage the ability of Debezium. See more about what is
[Debezium](https://github.com/debezium/debezium).
-{{< img src="/fig/cdc-flow.png" alt="Stateful Functions" width="50%" >}}
+
+
+
+
+
+ Key
Features
+
+
+
+
+
+
+
+
+
+
+ Change Data Capture
+
+
+FLink CDC supports distributed scanning of historical data of
database and then automatically switches to change data capturing. The s
[PR] [FLINK-34684 & 34683][docs] Add licenses & contributing docs in Flink CDC [flink-cdc]
ruanhang1993 opened a new pull request, #3159: URL: https://github.com/apache/flink-cdc/pull/3159 This PR adds licenses & contributing docs to Flink CDC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34684) "Developer Guide - Licenses" Page for Flink CDC Documentation
[ https://issues.apache.org/jira/browse/FLINK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34684: --- Labels: pull-request-available (was: ) > "Developer Guide - Licenses" Page for Flink CDC Documentation > - > > Key: FLINK-34684 > URL: https://issues.apache.org/jira/browse/FLINK-34684 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Qingsheng Ren >Assignee: Hang Ruan >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [cdc][cli][2934]Submit CDC Job To Flink K8S Native Application Mode [flink-cdc]
czy006 commented on code in PR #3093:
URL: https://github.com/apache/flink-cdc/pull/3093#discussion_r1527747030
##
flink-cdc-cli/src/main/java/org/apache/flink/cdc/cli/CliFrontendOptions.java:
##
@@ -46,6 +46,16 @@ public class CliFrontendOptions {
.desc("JARs to be submitted together with the pipeline")
.build();
+public static final Option TARGET =
+Option.builder("t")
+.longOpt("target")
+.hasArg()
+.desc(
+"The deployment target for the execution. This can
take one of the following values "
Review Comment:
https://issues.apache.org/jira/browse/FLINK-34677 Maybe can in there, WDHY?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34648] add waitChangeResultRequest and WaitChangeResultResponse to avoid RPC timeout. [flink-cdc]
fanqiejiang8 commented on PR #3128: URL: https://github.com/apache/flink-cdc/pull/3128#issuecomment-2002855730 @lvyanquan @BaoPiao When task parallelism is multiple, task retry all sent SchemaRegistry task more the RefreshPendingListsRequest request, that each request will not cause data inconsistency pendingSchemaChanges -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [build] fix jackson conflicts among cdc connectors [flink-cdc]
link3280 commented on PR #2987: URL: https://github.com/apache/flink-cdc/pull/2987#issuecomment-2002817974 ping @leonardBang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [build] fix inconsistent Kafka shading among cdc connectors [flink-cdc]
link3280 commented on PR #2988: URL: https://github.com/apache/flink-cdc/pull/2988#issuecomment-2002816885 ping @leonardBang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34679][cdc] add doc under core-concept. [flink-cdc]
LYanquan commented on PR #3153: URL: https://github.com/apache/flink-cdc/pull/3153#issuecomment-2002777037 @leonardBang @PatrickRen PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34678) "Introduction" Page for Flink CDC Documentation
[ https://issues.apache.org/jira/browse/FLINK-34678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34678: --- Labels: pull-request-available (was: ) > "Introduction" Page for Flink CDC Documentation > --- > > Key: FLINK-34678 > URL: https://issues.apache.org/jira/browse/FLINK-34678 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34678][cdc] Add introduction page for Flink CDC docs [flink-cdc]
PatrickRen opened a new pull request, #3158: URL: https://github.com/apache/flink-cdc/pull/3158 This pull request adds introduction page for Flink CDC docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32074][checkpoint] Merge file across checkpoints [flink]
masteryhx commented on PR #24497: URL: https://github.com/apache/flink/pull/24497#issuecomment-2002763228 Thanks for the update. Merged 841f23c7 into master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-32074) Support file merging across checkpoints
[ https://issues.apache.org/jira/browse/FLINK-32074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hangxiang Yu resolved FLINK-32074. -- Resolution: Fixed Merged 841f23c7 into master > Support file merging across checkpoints > --- > > Key: FLINK-32074 > URL: https://issues.apache.org/jira/browse/FLINK-32074 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.18.0 >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32074][checkpoint] Merge file across checkpoints [flink]
masteryhx closed pull request #24497: [FLINK-32074][checkpoint] Merge file across checkpoints URL: https://github.com/apache/flink/pull/24497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-34334) Add sub-task level RocksDB file count metric
[
https://issues.apache.org/jira/browse/FLINK-34334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hangxiang Yu resolved FLINK-34334.
--
Resolution: Fixed
Merged
[{{0921968}}|https://github.com/apache/flink/commit/0921968bb7d38bfac0b7899ec974a9744a721b22]
into master.
> Add sub-task level RocksDB file count metric
>
>
> Key: FLINK-34334
> URL: https://issues.apache.org/jira/browse/FLINK-34334
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.18.0
>Reporter: Jufang He
>Assignee: Jufang He
>Priority: Major
> Labels: pull-request-available
> Attachments: img_v3_027i_7ed0b8ba-3f12-48eb-aab3-cc368ac47cdg.jpg
>
>
> In our production environment, we encountered the problem of task deploy
> failure. The root cause was that too many sst files of a single sub-task led
> to too much task deployment information(OperatorSubtaskState), and then
> caused akka request timeout in the task deploy phase. Therefore, I wanted to
> add sub-task level RocksDB file count metrics. It is convenient to avoid
> performance problems caused by too many sst files in time.
> RocksDB has provided the JNI
> (https://javadoc.io/doc/org.rocksdb/rocksdbjni/6.20.3/org/rocksdb/RocksDB.html#getColumnFamilyMetaData
> ()) We can easily retrieve the file count and report it via metrics reporter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34334][state] Add sub-task level RocksDB file count metrics [flink]
masteryhx closed pull request #24322: [FLINK-34334][state] Add sub-task level RocksDB file count metrics URL: https://github.com/apache/flink/pull/24322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add announcement blog post for Flink 1.19 [flink-web]
lincoln-lil commented on PR #721: URL: https://github.com/apache/flink-web/pull/721#issuecomment-2002757799 Finalize the commits before merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34687][cdc][docs] Build the home page of Flink CDC documentation [flink-cdc]
xleoken commented on PR #3156: URL: https://github.com/apache/flink-cdc/pull/3156#issuecomment-2002754671 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34671][cdc] update README.md file to update links and description. [flink-cdc]
LYanquan commented on PR #3152: URL: https://github.com/apache/flink-cdc/pull/3152#issuecomment-2002750194 @leonardBang @PatrickRen PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20578) Cannot create empty array using ARRAY[]
[ https://issues.apache.org/jira/browse/FLINK-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827826#comment-17827826 ] Nathan Taylor Armstrong Lewis commented on FLINK-20578: --- Does anyone know of a workaround to create an empty array literal until this issue is addressed? > Cannot create empty array using ARRAY[] > --- > > Key: FLINK-20578 > URL: https://issues.apache.org/jira/browse/FLINK-20578 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.11.2 >Reporter: Fabian Hueske >Assignee: Eric Xiao >Priority: Major > Labels: pull-request-available, stale-assigned, starter > Fix For: 1.20.0 > > Attachments: Screen Shot 2022-10-25 at 10.50.42 PM.png, Screen Shot > 2022-10-25 at 10.50.47 PM.png, Screen Shot 2022-10-25 at 11.01.06 PM.png, > Screen Shot 2022-10-26 at 2.28.49 PM.png, image-2022-10-26-14-42-08-468.png, > image-2022-10-26-14-42-57-579.png > > > Calling the ARRAY function without an element (`ARRAY[]`) results in an error > message. > Is that the expected behavior? > How can users create empty arrays? -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34660][checkpoint] Parse cluster configuration for AutoRescalingITCase#testCheckpointRescalingInKeyedState [flink]
masteryhx commented on PR #24504: URL: https://github.com/apache/flink/pull/24504#issuecomment-2002742851 Thanks for the review. Merged e79df354 into master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34660][checkpoint] Parse cluster configuration for AutoRescalingITCase#testCheckpointRescalingInKeyedState [flink]
masteryhx closed pull request #24504: [FLINK-34660][checkpoint] Parse cluster configuration for AutoRescalingITCase#testCheckpointRescalingInKeyedState URL: https://github.com/apache/flink/pull/24504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-34571) SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed due an assertion
[
https://issues.apache.org/jira/browse/FLINK-34571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo closed FLINK-34571.
--
Fix Version/s: 1.19.1
Resolution: Fixed
> SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed
> due an assertion
> --
>
> Key: FLINK-34571
> URL: https://issues.apache.org/jira/browse/FLINK-34571
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Fix For: 1.20.0, 1.19.1
>
>
> https://github.com/apache/flink/actions/runs/8134965216/job/8875618#step:10:8586
> {code}
> Error: 02:39:36 02:39:36.688 [ERROR] Tests run: 9, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 13.68 s <<< FAILURE! -- in
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest
> Error: 02:39:36 02:39:36.689 [ERROR]
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError
> -- Time elapsed: 0.174 s <<< FAILURE!
> Mar 04 02:39:36 org.opentest4j.AssertionFailedError:
> Mar 04 02:39:36
> Mar 04 02:39:36 Expecting value to be true but was false
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Mar 04 02:39:36 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Mar 04 02:39:36 at
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError(SortMergeResultPartitionReadSchedulerTest.java:225)
> Mar 04 02:39:36 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 04 02:39:36 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34571) SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed due an assertion
[
https://issues.apache.org/jira/browse/FLINK-34571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825198#comment-17825198
]
Weijie Guo edited comment on FLINK-34571 at 3/18/24 1:36 AM:
-
master(1.20) via fcf61b9108942a892aa93765178a81f2c0e787c5.
release-1.19: 4f7f6a97f799a4a30d1aea57fc0af5d07dc0eeed.
was (Author: weijie guo):
master(1.20) via fcf61b9108942a892aa93765178a81f2c0e787c5.
Will backport this to release-1.19 after it released.
> SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError failed
> due an assertion
> --
>
> Key: FLINK-34571
> URL: https://issues.apache.org/jira/browse/FLINK-34571
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0, 1.20.0
>Reporter: Matthias Pohl
>Assignee: Weijie Guo
>Priority: Critical
> Labels: github-actions, pull-request-available, test-stability
> Fix For: 1.20.0
>
>
> https://github.com/apache/flink/actions/runs/8134965216/job/8875618#step:10:8586
> {code}
> Error: 02:39:36 02:39:36.688 [ERROR] Tests run: 9, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 13.68 s <<< FAILURE! -- in
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest
> Error: 02:39:36 02:39:36.689 [ERROR]
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError
> -- Time elapsed: 0.174 s <<< FAILURE!
> Mar 04 02:39:36 org.opentest4j.AssertionFailedError:
> Mar 04 02:39:36
> Mar 04 02:39:36 Expecting value to be true but was false
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Mar 04 02:39:36 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Mar 04 02:39:36 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Mar 04 02:39:36 at
> org.apache.flink.runtime.io.network.partition.SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError(SortMergeResultPartitionReadSchedulerTest.java:225)
> Mar 04 02:39:36 at java.lang.reflect.Method.invoke(Method.java:498)
> Mar 04 02:39:36 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Mar 04 02:39:36 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [BP-1.19][FLINK-34571][test] Fix flaky test SortMergeResultPartitionReadSchedulerTest.testOnReadBufferRequestError [flink]
reswqa merged PR #24479: URL: https://github.com/apache/flink/pull/24479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Flink 34569][e2e] fail fast if AWS cli container fails to start [flink]
robobario commented on code in PR #24491:
URL: https://github.com/apache/flink/pull/24491#discussion_r1527607029
##
flink-end-to-end-tests/test-scripts/common_s3_operations.sh:
##
@@ -58,7 +64,11 @@ function aws_cli_stop() {
if [[ $AWSCLI_CONTAINER_ID ]]; then
aws_cli_stop
fi
-aws_cli_start
+aws_cli_start || aws_cli_start
Review Comment:
I've added in a failsafe to kill/remove if `CONTAINER_ID` is non-empty after
the docker run fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-34633) Support unnesting array constants
[
https://issues.apache.org/jira/browse/FLINK-34633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui reassigned FLINK-34633:
---
Assignee: Jeyhun Karimov
> Support unnesting array constants
> -
>
> Key: FLINK-34633
> URL: https://issues.apache.org/jira/browse/FLINK-34633
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Planner
>Affects Versions: 1.18.1
>Reporter: Xingcan Cui
>Assignee: Jeyhun Karimov
>Priority: Minor
> Labels: pull-request-available
>
> It seems that the current planner doesn't support using UNNEST on array
> constants.(x)
> {code:java}
> SELECT * FROM UNNEST(ARRAY[1,2,3]);{code}
>
> The following query can't be compiled.(x)
> {code:java}
> SELECT * FROM (VALUES('a')) CROSS JOIN UNNEST(ARRAY[1, 2, 3]){code}
>
> The rewritten version works. (/)
> {code:java}
> SELECT * FROM (SELECT *, ARRAY[1,2,3] AS A FROM (VALUES('a'))) CROSS JOIN
> UNNEST(A){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34633][table] Support unnesting array constants [flink]
flinkbot commented on PR #24510: URL: https://github.com/apache/flink/pull/24510#issuecomment-2002539941 ## CI report: * e239874a6d95df04d4c5919b09fc800b82cacc44 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
