[PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
GOODBOY008 opened a new pull request, #3331: URL: https://github.com/apache/flink-cdc/pull/3331 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35387) PG CDC source support heart beat
Hongshun Wang created FLINK-35387:
-
Summary: PG CDC source support heart beat
Key: FLINK-35387
URL: https://issues.apache.org/jira/browse/FLINK-35387
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: Hongshun Wang
Fix For: cdc-3.2.0
Though, document of PG CDC [1] has heartbeat.interval.ms, but it's not valid.
The reason is bellow.
In debezium dos says: For the connector to detect and process events from a
heartbeat table, you must add the table to the PostgreSQL publication specified
by the
[publication.name|https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-publication-name]
property. If this publication predates your Debezium deployment, the connector
uses the publications as defined. If the publication is not already configured
to automatically replicate changes {{FOR ALL TABLES}} in the database, you must
explicitly add the heartbeat table to the publication[2].
Thus, if you want use heart beat in cdc:
1. add a heartbeat table to publication: ALTER PUBLICATION __
ADD TABLE {_}{_};
2. set heartbeatInterval
3. add
debezium.[{{heartbeat.action.query}}|https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-heartbeat-action-query]
[3]
However, when I use it it CDC, some exception occurs:
{code:java}
Caused by: java.lang.NullPointerException
at
io.debezium.heartbeat.HeartbeatFactory.createHeartbeat(HeartbeatFactory.java:55)
at io.debezium.pipeline.EventDispatcher.(EventDispatcher.java:127)
at io.debezium.pipeline.EventDispatcher.(EventDispatcher.java:94){code}
!https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5292b7c63c883d1620bbf7d3875a3a5b158e70b814913bc360a414d3de9277d871abf3af1cbd75249eddaaa1b37c2b2f5421a918fb1a2f0f3853c0ce41721e620699d98626fa2281948c58faa63edf8ebfc653b69905bac42?tmpCode=9193555a-7bf3-4335-9427-b59c1dfe1931!
It seems CDC don't add a HeartbeatConnectionProvider when configure
PostgresEventDispatcher:
{code:java}
//org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresSourceFetchTaskContext#configure
this.postgresDispatcher =
new PostgresEventDispatcher<>(
dbzConfig,
topicSelector,
schema,
queue,
dbzConfig.getTableFilters().dataCollectionFilter(),
DataChangeEvent::new,
metadataProvider,
schemaNameAdjuster); {code}
in debezium, when PostgresConnectorTask start, it will do it
{code:java}
//io.debezium.connector.postgresql.PostgresConnectorTask#start
final PostgresEventDispatcher dispatcher = new
PostgresEventDispatcher<>(
connectorConfig,
topicNamingStrategy,
schema,
queue,
connectorConfig.getTableFilters().dataCollectionFilter(),
DataChangeEvent::new,
PostgresChangeRecordEmitter::updateSchema,
metadataProvider,
connectorConfig.createHeartbeat(
topicNamingStrategy,
schemaNameAdjuster,
() -> new
PostgresConnection(connectorConfig.getJdbcConfig(),
PostgresConnection.CONNECTION_GENERAL),
exception -> {
String sqlErrorId = exception.getSQLState();
switch (sqlErrorId) {
case "57P01":
// Postgres error admin_shutdown, see
https://www.postgresql.org/docs/12/errcodes-appendix.html
throw new DebeziumException("Could not
execute heartbeat action query (Error: " + sqlErrorId + ")", exception);
case "57P03":
// Postgres error cannot_connect_now,
see https://www.postgresql.org/docs/12/errcodes-appendix.html
throw new RetriableException("Could not
execute heartbeat action query (Error: " + sqlErrorId + ")", exception);
default:
break;
}
}),
schemaNameAdjuster,
signalProcessor); {code}
Thus, this jira will add this.
[1]
https://nightlies.apache.org/flink/flink-cdc-docs-master/docs/connectors/legacy-flink-cdc-sources/postgres-cdc/
[2]
https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-heartbeat-interval-ms
[3]
[jira] [Assigned] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable
[ https://issues.apache.org/jira/browse/FLINK-35386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-35386: - Assignee: Zhongqiang Gong (was: Qingsheng Ren) > Build Flink CDC 3.1 Documentation and mark it as stable > --- > > Key: FLINK-35386 > URL: https://issues.apache.org/jira/browse/FLINK-35386 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Qingsheng Ren >Assignee: Zhongqiang Gong >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > This is part of Flink CDC 3.1.0 release. We need to update the Github Action > configuration to build release-3.1 doc and mark it as stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable
[ https://issues.apache.org/jira/browse/FLINK-35386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren resolved FLINK-35386. --- Resolution: Fixed > Build Flink CDC 3.1 Documentation and mark it as stable > --- > > Key: FLINK-35386 > URL: https://issues.apache.org/jira/browse/FLINK-35386 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Qingsheng Ren >Assignee: Zhongqiang Gong >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > This is part of Flink CDC 3.1.0 release. We need to update the Github Action > configuration to build release-3.1 doc and mark it as stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable
[ https://issues.apache.org/jira/browse/FLINK-35386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847155#comment-17847155 ] Qingsheng Ren commented on FLINK-35386: --- flink-cdc master: 999027fbb1bc02109e8a79ac25dd2a43c0483f84 release-3.1: 4c0cafe9f4171f304ca9ed87d46e92c274ea5d88 > Build Flink CDC 3.1 Documentation and mark it as stable > --- > > Key: FLINK-35386 > URL: https://issues.apache.org/jira/browse/FLINK-35386 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > This is part of Flink CDC 3.1.0 release. We need to update the Github Action > configuration to build release-3.1 doc and mark it as stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
PatrickRen merged PR #3317: URL: https://github.com/apache/flink-cdc/pull/3317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
PatrickRen commented on PR #3317: URL: https://github.com/apache/flink-cdc/pull/3317#issuecomment-2116703527 > Is this pr will be merged after release vote passed? Yes. The documentation can be adjusted anytime -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35049) Implement List/Map Async State API for ForStStateBackend
[ https://issues.apache.org/jira/browse/FLINK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei reassigned FLINK-35049: -- Assignee: Yanfei Lei > Implement List/Map Async State API for ForStStateBackend > > > Key: FLINK-35049 > URL: https://issues.apache.org/jira/browse/FLINK-35049 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Hangxiang Yu >Assignee: Yanfei Lei >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
PatrickRen merged PR #3330: URL: https://github.com/apache/flink-cdc/pull/3330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
PatrickRen commented on PR #3329: URL: https://github.com/apache/flink-cdc/pull/3329#issuecomment-2116680813 Already covered by #3330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
PatrickRen closed pull request #3329: [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable URL: https://github.com/apache/flink-cdc/pull/3329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35030][runtime] Introduce Epoch Manager under async execution [flink]
fredia merged PR #24748: URL: https://github.com/apache/flink/pull/24748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
GOODBOY008 opened a new pull request, #3330: URL: https://github.com/apache/flink-cdc/pull/3330 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35351][checkpoint] Fix fail during restore from unaligned chec… [flink]
ldadima commented on code in PR #24784:
URL: https://github.com/apache/flink/pull/24784#discussion_r1603118510
##
flink-tests/src/test/java/org/apache/flink/test/checkpointing/UnalignedCheckpointCustomRescaleITCase.java:
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.JobStatus;
+import org.apache.flink.api.common.functions.Partitioner;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.minicluster.MiniClusterConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.functions.sink.SinkFunction;
+import org.apache.flink.util.FileUtils;
+
+import org.junit.After;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Collection;
+
+import static
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.createLocalEnvironment;
+import static org.junit.Assert.fail;
+
+/** Integration test for performing rescale of unaligned checkpoint with
custom partitioner. */
+public class UnalignedCheckpointCustomRescaleITCase {
Review Comment:
I though about it, but in `UnalignedCheckpointRescaleITCase` used `Long` as
stream data. But for the regular repeatability of the test it was necessary to
use the `String`. I will try to add test to `UnalignedCheckpointRescaleITCase`,
but I'm not sure. Also in `UnalignedCheckpointRescaleITCase` rescale full graph
(change parallelism for all vertexes), but need to change only one vertex, not
all. I have no ideas, how to add new Topology for this test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [hotfix] Fix compilation error due to constructor removal of MockSourceReader [flink-benchmarks]
Zakelly merged PR #89: URL: https://github.com/apache/flink-benchmarks/pull/89 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable
[ https://issues.apache.org/jira/browse/FLINK-35386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35386: --- Labels: pull-request-available (was: ) > Build Flink CDC 3.1 Documentation and mark it as stable > --- > > Key: FLINK-35386 > URL: https://issues.apache.org/jira/browse/FLINK-35386 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > This is part of Flink CDC 3.1.0 release. We need to update the Github Action > configuration to build release-3.1 doc and mark it as stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35386][cdc][docs] Build release-3.1 documentation and mark it as stable [flink-cdc]
PatrickRen opened a new pull request, #3329: URL: https://github.com/apache/flink-cdc/pull/3329 This pull request configures GitHub Action to build release-3.1 documentation and mark it as stable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable
Qingsheng Ren created FLINK-35386: - Summary: Build Flink CDC 3.1 Documentation and mark it as stable Key: FLINK-35386 URL: https://issues.apache.org/jira/browse/FLINK-35386 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Qingsheng Ren Assignee: Qingsheng Ren Fix For: cdc-3.1.0 This is part of Flink CDC 3.1.0 release. We need to update the Github Action configuration to build release-3.1 doc and mark it as stable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Add release announcement for Flink CDC 3.1.0 [flink-web]
PatrickRen commented on code in PR #739: URL: https://github.com/apache/flink-web/pull/739#discussion_r1604319112 ## docs/content/posts/2024-MM-DD-release-cdc-3.1.0.md: ## @@ -0,0 +1,67 @@ +--- +title: "Apache Flink CDC 3.1.0 Release Announcement" +date: "2024-MM-DDT08:00:00.000Z" +authors: +- renqs: + name: "Qingsheng Ren" + twitter: "renqstuite" +aliases: +- /news/2024/MM/DD/release-cdc-3.1.0.html +--- + +The Apache Flink community is excited to announce the release of Flink CDC 3.1.0! This is the first release after the community accepted the donation of Flink CDC as a sub-project of Apache Flink, with exciting new features such as transform and table merging. The eco-system of Flink CDC keeps expanding, including new Kafka and Paimon pipeline sinks and enhancement to existing connectors. + +We'd like to invite you to check out [Flink CDC documentation](https://nightlies.apache.org/flink/flink-cdc-docs-stable) and have a try on [the quickstart tutorial](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/docs/get-started/introduction) to explore the world of Flink CDC. Also we encourage you to [download the release](https://flink.apache.org/downloads.html) and share your feedback with the community through the Flink [mailing lists](https://flink.apache.org/community.html#mailing-lists) or [JIRA](https://issues.apache.org/jira/browse/flink)! We hope you like the new release and we’d be eager to learn about your experience with it. Review Comment: Sure! I'll create another PR for download links -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add release announcement for Flink CDC 3.1.0 [flink-web]
PatrickRen commented on code in PR #739: URL: https://github.com/apache/flink-web/pull/739#discussion_r1604313283 ## docs/content/posts/2024-MM-DD-release-cdc-3.1.0.md: ## @@ -0,0 +1,67 @@ +--- +title: "Apache Flink CDC 3.1.0 Release Announcement" +date: "2024-MM-DDT08:00:00.000Z" +authors: +- renqs: + name: "Qingsheng Ren" + twitter: "renqstuite" +aliases: +- /news/2024/MM/DD/release-cdc-3.1.0.html +--- + +The Apache Flink community is excited to announce the release of Flink CDC 3.1.0! This is the first release after the community accepted the donation of Flink CDC as a sub-project of Apache Flink, with exciting new features such as transform and table merging. The eco-system of Flink CDC keeps expanding, including new Kafka and Paimon pipeline sinks and enhancement to existing connectors. + +We'd like to invite you to check out [Flink CDC documentation](https://nightlies.apache.org/flink/flink-cdc-docs-stable) and have a try on [the quickstart tutorial](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/docs/get-started/introduction) to explore the world of Flink CDC. Also we encourage you to [download the release](https://flink.apache.org/downloads.html) and share your feedback with the community through the Flink [mailing lists](https://flink.apache.org/community.html#mailing-lists) or [JIRA](https://issues.apache.org/jira/browse/flink)! We hope you like the new release and we’d be eager to learn about your experience with it. + +## Highlights + +### Transformation Support in Pipeline + +Flink CDC 3.1.0 introduces the ability of making transformations in the CDC pipeline. By incorporating a `transform` section within the YAML pipeline definitions, users can now easily apply a variety of transformations to data change event from source, including projections, calculations, and addition of constant columns, enhancing the effectiveness of data intergration pipelines. Leveraging an SQL-like syntax for defining these transformations, the new feature ensures that users can quickly adapt to and utilize it. + +### Table Merging Support + +Flink CDC 3.1.0 now suuports merging multiple tables into one by configuring `route` in the YAML pipeline definition. It is a prevalent occurrence where business data is partitioned across tables even databases due to the substantial volume. By configuring `route`s that mapping multiple tables into one, data change events will be merged into the same destination table. Moreover, schema changes on source tables will also be applied to the destination. + +### Connectors + + Distributions of MySQL / Oracle / OceanBase / Db2 connectors + +Unfortunately due to the license incompatibility, we cannot ship JDBC drivers of the following connectors together with our binary release: + +- Db2 +- MySQL +- Oracle +- OceanBase + +Please manually download the corresponding JDBC driver into `$FLINK_CDC_HOME/lib` and `$FLINK_HOME/lib`, and specify their paths when submiting YAML pipelines with `--jar`, or make sure they are under the classpath if you are using Flink SQL. Review Comment: Yep I will rewrite this part. There are two ways to involve MySQL JDBC driver: 1. Put it into `FLINK_HOME/lib` directly 2. Download to anywhere locally and specify the path with `--jar`. It is not necessarily in `FLINK_CDC_HOME/lib` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-35309) Enable Notice file ci check and fix Notice
[ https://issues.apache.org/jira/browse/FLINK-35309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren resolved FLINK-35309. --- Resolution: Fixed > Enable Notice file ci check and fix Notice > --- > > Key: FLINK-35309 > URL: https://issues.apache.org/jira/browse/FLINK-35309 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Blocker > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > Changes: > * Add ci to check Notice file > * Fix Notice file issue -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34990) [feature][cdc-connector][oracle] Oracle cdc support newly add table
[ https://issues.apache.org/jira/browse/FLINK-34990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34990: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > [feature][cdc-connector][oracle] Oracle cdc support newly add table > --- > > Key: FLINK-34990 > URL: https://issues.apache.org/jira/browse/FLINK-34990 > Project: Flink > Issue Type: New Feature > Components: Flink CDC >Reporter: Xin Gong >Assignee: Xin Gong >Priority: Minor > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > [feature][cdc-connector][oracle] Oracle cdc support newly add table -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-34912) Replace all occurrences of com.ververica in the project with org.apache.flink
[ https://issues.apache.org/jira/browse/FLINK-34912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren closed FLINK-34912. - Resolution: Duplicate > Replace all occurrences of com.ververica in the project with org.apache.flink > - > > Key: FLINK-34912 > URL: https://issues.apache.org/jira/browse/FLINK-34912 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Xiao Huang >Priority: Minor > Labels: pull-request-available > Fix For: cdc-3.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34908) [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision for timestamp
[ https://issues.apache.org/jira/browse/FLINK-34908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34908: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > [Feature][Pipeline] Mysql pipeline to doris and starrocks will lost precision > for timestamp > --- > > Key: FLINK-34908 > URL: https://issues.apache.org/jira/browse/FLINK-34908 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xin Gong >Assignee: Xin Gong >Priority: Minor > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > flink cdc pipeline will decide timestamp zone by config of pipeline. I found > mysql2doris and mysql2starracks will specific datetime format > -MM-dd HH:mm:ss, it will cause lost datatime precision. I think we > should't set fixed datetime format, just return LocalDateTime object. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34904) [Feature] submit Flink CDC pipeline job to yarn application cluster.
[ https://issues.apache.org/jira/browse/FLINK-34904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34904: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > [Feature] submit Flink CDC pipeline job to yarn application cluster. > > > Key: FLINK-34904 > URL: https://issues.apache.org/jira/browse/FLINK-34904 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: ZhengYu Chen >Priority: Minor > Fix For: cdc-3.2.0 > > > support flink cdc cli submit pipeline job to yarn application cluster.discuss > in FLINK-34853 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34752) "legacy-flink-cdc-sources" Page of TIDB for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34752: - Assignee: 牛一凡 > "legacy-flink-cdc-sources" Page of TIDB for Flink CDC Chinese Documentation. > > > Key: FLINK-34752 > URL: https://issues.apache.org/jira/browse/FLINK-34752 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: 牛一凡 >Priority: Minor > Fix For: cdc-3.1.0 > > > Translate legacy-flink-cdc-sources pages of > https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/tidb-cdc.md > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34752) "legacy-flink-cdc-sources" Page of TIDB for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34752: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "legacy-flink-cdc-sources" Page of TIDB for Flink CDC Chinese Documentation. > > > Key: FLINK-34752 > URL: https://issues.apache.org/jira/browse/FLINK-34752 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: 牛一凡 >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate legacy-flink-cdc-sources pages of > https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/tidb-cdc.md > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34750) "legacy-flink-cdc-sources" Page of Postgresql for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34750: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "legacy-flink-cdc-sources" Page of Postgresql for Flink CDC Chinese > Documentation. > -- > > Key: FLINK-34750 > URL: https://issues.apache.org/jira/browse/FLINK-34750 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: Hongshun Wang >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate legacy-flink-cdc-sources pages of > https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/postgres-cdc.md > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34750) "legacy-flink-cdc-sources" Page of Postgresql for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34750: - Assignee: Hongshun Wang > "legacy-flink-cdc-sources" Page of Postgresql for Flink CDC Chinese > Documentation. > -- > > Key: FLINK-34750 > URL: https://issues.apache.org/jira/browse/FLINK-34750 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: Hongshun Wang >Priority: Minor > Fix For: cdc-3.1.0 > > > Translate legacy-flink-cdc-sources pages of > https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/postgres-cdc.md > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34748) "legacy-flink-cdc-sources" Page of Oracle for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34748: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "legacy-flink-cdc-sources" Page of Oracle for Flink CDC Chinese Documentation. > -- > > Key: FLINK-34748 > URL: https://issues.apache.org/jira/browse/FLINK-34748 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: 牛一凡 >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate legacy-flink-cdc-sources pages of > https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34749) "legacy-flink-cdc-sources" Page of SQLServer for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34749: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "legacy-flink-cdc-sources" Page of SQLServer for Flink CDC Chinese > Documentation. > - > > Key: FLINK-34749 > URL: https://issues.apache.org/jira/browse/FLINK-34749 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: 牛一凡 >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate legacy-flink-cdc-sources pages of > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/sqlserver-cdc.md > > |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into > Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34748) "legacy-flink-cdc-sources" Page of Oracle for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34748: - Assignee: 牛一凡 > "legacy-flink-cdc-sources" Page of Oracle for Flink CDC Chinese Documentation. > -- > > Key: FLINK-34748 > URL: https://issues.apache.org/jira/browse/FLINK-34748 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: 牛一凡 >Priority: Minor > Fix For: cdc-3.1.0 > > > Translate legacy-flink-cdc-sources pages of > https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34747) "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34747: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation. > --- > > Key: FLINK-34747 > URL: https://issues.apache.org/jira/browse/FLINK-34747 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate legacy-flink-cdc-sources pages of > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md > > |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into > Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34749) "legacy-flink-cdc-sources" Page of SQLServer for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34749: - Assignee: 牛一凡 > "legacy-flink-cdc-sources" Page of SQLServer for Flink CDC Chinese > Documentation. > - > > Key: FLINK-34749 > URL: https://issues.apache.org/jira/browse/FLINK-34749 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: 牛一凡 >Priority: Minor > Fix For: cdc-3.1.0 > > > Translate legacy-flink-cdc-sources pages of > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/sqlserver-cdc.md > > |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into > Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34747) "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation.
[ https://issues.apache.org/jira/browse/FLINK-34747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34747: - Assignee: Xiao Huang > "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation. > --- > > Key: FLINK-34747 > URL: https://issues.apache.org/jira/browse/FLINK-34747 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: Xiao Huang >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate legacy-flink-cdc-sources pages of > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md > > |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into > Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35047][state] Shutdown StateExecutors when ForStKeyedStateBackend is disposed [flink]
ljz2051 commented on code in PR #24768:
URL: https://github.com/apache/flink/pull/24768#discussion_r1604301920
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStKeyedStateBackend.java:
##
@@ -147,15 +160,30 @@ public S createState(@Nonnull
StateDescriptor stateDes
@Override
@Nonnull
public StateExecutor createStateExecutor() {
-// TODO: Make io parallelism configurable
-return new ForStStateExecutor(4, db,
optionsContainer.getWriteOptions());
+synchronized (lock) {
Review Comment:
You means that `StateExecutor#shutdown `should be invoked in the `close`
method, not the `dispose` method ?
Maybe we need introduce aonther `closed` flag in ForStKeyedStateBackend to
prevent "createStateExecutor after close".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34735) "Developer Guide - Understanding Flink CDC API" Page for Flink CDC Chinese Documentation
[ https://issues.apache.org/jira/browse/FLINK-34735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34735: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "Developer Guide - Understanding Flink CDC API" Page for Flink CDC Chinese > Documentation > > > Key: FLINK-34735 > URL: https://issues.apache.org/jira/browse/FLINK-34735 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/developer-guide/understand-flink-cdc-api.md] > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34736) "Deployment - Standalone" Page for Flink CDC Chinese Documentation
[ https://issues.apache.org/jira/browse/FLINK-34736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34736: - Assignee: Vincent Woo > "Deployment - Standalone" Page for Flink CDC Chinese Documentation > -- > > Key: FLINK-34736 > URL: https://issues.apache.org/jira/browse/FLINK-34736 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: Vincent Woo >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/standalone.md] > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34736) "Deployment - Standalone" Page for Flink CDC Chinese Documentation
[ https://issues.apache.org/jira/browse/FLINK-34736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34736: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "Deployment - Standalone" Page for Flink CDC Chinese Documentation > -- > > Key: FLINK-34736 > URL: https://issues.apache.org/jira/browse/FLINK-34736 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Minor > Fix For: cdc-3.2.0 > > > Translate > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/standalone.md] > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34730) Translate pages of Flink CDC docs to Chinese
[ https://issues.apache.org/jira/browse/FLINK-34730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34730: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > Translate pages of Flink CDC docs to Chinese > > > Key: FLINK-34730 > URL: https://issues.apache.org/jira/browse/FLINK-34730 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation, Flink CDC >Reporter: LvYanquan >Priority: Minor > Fix For: cdc-3.2.0 > > > As https://issues.apache.org/jira/browse/FLINK-34677 introduced new structure > of Document for Flink CDC, it's also necessary to update the Chinese Document > for better user experience. > We need to translate the pages under content into Chinese, and update the > titles. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35354) Support host mapping in Flink tikv cdc
[ https://issues.apache.org/jira/browse/FLINK-35354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-35354: -- Fix Version/s: (was: cdc-3.1.0) > Support host mapping in Flink tikv cdc > -- > > Key: FLINK-35354 > URL: https://issues.apache.org/jira/browse/FLINK-35354 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0, cdc-3.2.0 >Reporter: ouyangwulin >Priority: Major > Fix For: cdc-3.2.0 > > > In tidb production environment deployment, there are usually two kinds of > network: internal network and public network. When we use pd mode in tikv, we > need to do network mapping, such as `spark.tispark.host_mapping` in > [https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md]. So I > think we need support `host_mapping` in our Flink tikv cdc connector. > > Add param: > tikv.host_mapping:192.168.0.2:8.8.8.8;192.168.0.3:9.9.9.9 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35259) FlinkCDC Pipeline transform can't deal timestamp field
[ https://issues.apache.org/jira/browse/FLINK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren resolved FLINK-35259. --- Resolution: Fixed > FlinkCDC Pipeline transform can't deal timestamp field > -- > > Key: FLINK-35259 > URL: https://issues.apache.org/jira/browse/FLINK-35259 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Wenkai Qi >Assignee: Wenkai Qi >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > When the original table contains fields of type Timestamp, it cannot be > converted properly. > When the added calculation columns contain fields of type Timestamp, it > cannot be converted properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35259) FlinkCDC Pipeline transform can't deal timestamp field
[ https://issues.apache.org/jira/browse/FLINK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841824#comment-17841824 ] Qingsheng Ren edited comment on FLINK-35259 at 5/17/24 3:24 AM: flink-cdc master: 0108d0e5d18c55873c3d67e9caad58b9d2148d6a release-3.1: e022f4d2592dedc6d570948213fd07f47c1e505c was (Author: renqs): flink-cdc master: 0108d0e5d18c55873c3d67e9caad58b9d2148d6a > FlinkCDC Pipeline transform can't deal timestamp field > -- > > Key: FLINK-35259 > URL: https://issues.apache.org/jira/browse/FLINK-35259 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: Wenkai Qi >Assignee: Wenkai Qi >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > When the original table contains fields of type Timestamp, it cannot be > converted properly. > When the added calculation columns contain fields of type Timestamp, it > cannot be converted properly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34944) Use Incremental Source Framework in Flink CDC OceanBase Source Connector
[ https://issues.apache.org/jira/browse/FLINK-34944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34944: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > Use Incremental Source Framework in Flink CDC OceanBase Source Connector > > > Key: FLINK-34944 > URL: https://issues.apache.org/jira/browse/FLINK-34944 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: He Wang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34944) Use Incremental Source Framework in Flink CDC OceanBase Source Connector
[ https://issues.apache.org/jira/browse/FLINK-34944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34944: - Assignee: He Wang > Use Incremental Source Framework in Flink CDC OceanBase Source Connector > > > Key: FLINK-34944 > URL: https://issues.apache.org/jira/browse/FLINK-34944 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: He Wang >Assignee: He Wang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34853) [Feature] submit Flink CDC pipeline job to k8s cluster.
[ https://issues.apache.org/jira/browse/FLINK-34853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34853: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > [Feature] submit Flink CDC pipeline job to k8s cluster. > --- > > Key: FLINK-34853 > URL: https://issues.apache.org/jira/browse/FLINK-34853 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Flink CDC Issue Import >Priority: Major > Labels: github-import, pull-request-available > Fix For: cdc-3.2.0 > > > ### Search before asking > - [X] I searched in the > [issues|https://github.com/ververica/flink-cdc-connectors/issues] and found > nothing similar. > ### Motivation > Currently, there is no a clear description to tell how to run a pipeline on > k8s. > So If there is no need to modify the code, please add some docs to guide > users how to submit job to a Kubernetes cluster. > If it's necessary to modify the code, you can submit a pr and add doc to fix > it. > ### Solution > _No response_ > ### Alternatives > _No response_ > ### Anything else? > _No response_ > ### Are you willing to submit a PR? > - [ ] I'm willing to submit a PR! > Imported from GitHub > Url: https://github.com/apache/flink-cdc/issues/2934 > Created by: [lvyanquan|https://github.com/lvyanquan] > Labels: enhancement, > Created at: Wed Dec 27 11:01:17 CST 2023 > State: open -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34737) "Deployment - Kubernetes" Page for Flink CDC Chinese Documentation
[ https://issues.apache.org/jira/browse/FLINK-34737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34737: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "Deployment - Kubernetes" Page for Flink CDC Chinese Documentation > -- > > Key: FLINK-34737 > URL: https://issues.apache.org/jira/browse/FLINK-34737 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Major > Fix For: cdc-3.2.0 > > > Translate > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/kubernetes.md] > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34738) "Deployment - YARN" Page for Flink CDC Chinese Documentation
[ https://issues.apache.org/jira/browse/FLINK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34738: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > "Deployment - YARN" Page for Flink CDC Chinese Documentation > > > Key: FLINK-34738 > URL: https://issues.apache.org/jira/browse/FLINK-34738 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation, Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: Vincent Woo >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.2.0 > > > Translate > [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/yarn.md] > into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34638) Support default value of table column.
[ https://issues.apache.org/jira/browse/FLINK-34638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34638: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > Support default value of table column. > -- > > Key: FLINK-34638 > URL: https://issues.apache.org/jira/browse/FLINK-34638 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Major > Fix For: cdc-3.2.0 > > > Support default value of table column of literal constant. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34614) Add e2e test for Flink CDC pipeline job.
[ https://issues.apache.org/jira/browse/FLINK-34614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren resolved FLINK-34614. --- Assignee: LvYanquan Resolution: Fixed > Add e2e test for Flink CDC pipeline job. > > > Key: FLINK-34614 > URL: https://issues.apache.org/jira/browse/FLINK-34614 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Assignee: LvYanquan >Priority: Major > Fix For: cdc-3.1.0 > > > Add e2e test for Flink CDC pipeline job. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34614) Add e2e test for Flink CDC pipeline job.
[ https://issues.apache.org/jira/browse/FLINK-34614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847138#comment-17847138 ] LvYanquan commented on FLINK-34614: --- done in pull-request https://github.com/apache/flink-cdc/pull/2997. > Add e2e test for Flink CDC pipeline job. > > > Key: FLINK-34614 > URL: https://issues.apache.org/jira/browse/FLINK-34614 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: cdc-3.1.0 >Reporter: LvYanquan >Priority: Major > Fix For: cdc-3.1.0 > > > Add e2e test for Flink CDC pipeline job. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34595) Fix ClassNotFoundException: com.ververica.cdc.common.utils.StringUtils
[ https://issues.apache.org/jira/browse/FLINK-34595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren updated FLINK-34595: -- Fix Version/s: cdc-3.2.0 (was: cdc-3.1.0) > Fix ClassNotFoundException: com.ververica.cdc.common.utils.StringUtils > -- > > Key: FLINK-34595 > URL: https://issues.apache.org/jira/browse/FLINK-34595 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: LvYanquan >Priority: Major > Fix For: cdc-3.2.0 > > > In [this > pr|[https://github.com/apache/flink-cdc/pull/2986/files#diff-cec13810c47e9465e4f2a72507f655b86f41579768b9924fe024aabc60b31d17R21|https://github.com/apache/flink-cdc/pull/2986/files#diff-cec13810c47e9465e4f2a72507f655b86f41579768b9924fe024aabc60b31d17R21[]], > we introduced > org.apache.flink.cdc.common.utils.StringUtils class of flink-cdc-common > module in flink-connector-mysql-cdc module. > However, the sub module flink-sql-connector-mysql-cdc doesn't include > flink-cdc-common module when packaging, so we can't find this class in sql > jar.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35047][state] Shutdown StateExecutors when ForStKeyedStateBackend is disposed [flink]
ljz2051 commented on code in PR #24768:
URL: https://github.com/apache/flink/pull/24768#discussion_r1604284072
##
flink-state-backends/flink-statebackend-forst/src/main/java/org/apache/flink/state/forst/ForStKeyedStateBackend.java:
##
@@ -147,15 +160,30 @@ public S createState(@Nonnull
StateDescriptor stateDes
@Override
@Nonnull
public StateExecutor createStateExecutor() {
-// TODO: Make io parallelism configurable
-return new ForStStateExecutor(4, db,
optionsContainer.getWriteOptions());
+synchronized (lock) {
+if (disposed) {
+throw new FlinkRuntimeException(
+"Attempt to create StateExecutor after
ForStKeyedStateBackend is disposed.");
+}
+// TODO: Make io parallelism configurable
+StateExecutor stateExecutor =
+new ForStStateExecutor(4, db,
optionsContainer.getWriteOptions());
+managedStateExecutors.add(stateExecutor);
+return stateExecutor;
+}
}
/** Should only be called by one thread, and only after all accesses to
the DB happened. */
@Override
public void dispose() {
-if (this.disposed) {
-return;
+synchronized (lock) {
+if (disposed) {
+return;
+}
+disposed = true;
+for (StateExecutor executor : managedStateExecutors) {
Review Comment:
Based on the current implementation, it is indeed doable. However, I think
that enclosing both `disposed` and `managedStateExecutors` within the
protective scope of the `lock` constitutes a more safe practice. Additionally,
the chance of a lock conflict between `createStateExecutor()` and `dispose()`
is very minimal, so the impact in this place is virtually imperceptible.
So I prefer to put this `for-loop` of `managedStateExecutors` inside the
`lock` scope.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [hotfix] Fix compilation error due to constructor removal of MockSourceReader [flink-benchmarks]
Zakelly opened a new pull request, #89: URL: https://github.com/apache/flink-benchmarks/pull/89 Due to this commit in main repo https://github.com/apache/flink/commit/41391065e1bf1476040039077049fbb98bec457e , there is a compilation error. This PR change to use another constructor of `MockSourceReader` to fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35343) NullPointerException in SourceReaderBase
[
https://issues.apache.org/jira/browse/FLINK-35343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zyh updated FLINK-35343:
Description:
h2. operation
I used flink batch to read data from Doris and write to Doris.
The flink job include two source task, one table join task and one sink task,
which like:
source: Table A
source: Table B
hashjoin: c= a join b
sink: c
h2.
h2. table properties
h3. source
properties.put("connector", "doris");
properties.put("fenodes", inputDataSource.getHttpUrl());
properties.put("table.identifier", inputDataSource.getDatabase() + "." +
sourceTable.getName());
properties.put("username", inputDataSource.getUsername());
properties.put("password", inputDataSource.getPassword());
h3. sink
properties.put("connector", "doris");
properties.put("fenodes", dataExplore.getOutputDataSource().getHttpUrl());
properties.put("table.identifier",
dataExplore.getOutputDataSource().getDatabase() + "." + tableName);
properties.put("username", dataExplore.getOutputDataSource().getUsername());
properties.put("password", dataExplore.getOutputDataSource().getPassword());
properties.put("sink.properties.format", "csv");
//列分隔符
properties.put("sink.properties.column_separator", "#cs_");
//行分隔符
properties.put("sink.properties.line_delimiter", "#ld_");
properties.put("sink.label-prefix", "doris_label" + UUID.randomUUID());
properties.put("sink.parallelism", "2");
h2. exception stack
{code:java}
Caused by: java.lang.NullPointerException at
org.apache.flink.connector.base.source.reader.SourceReaderBase.finishCurrentFetch(SourceReaderBase.java:194)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.moveToNextSplit(SourceReaderBase.java:208)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:173)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:131)
at
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419)
at
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931) at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) at
java.lang.Thread.run(Thread.java:748) {code}
h2. other
The problem only occur in flink local mode and deploy on k8s.
was:
h2. operation
I used flink batch to read data from Doris and write to Doris.
The flink job include two source task, one table join task and one sink task.
h2. exception stack
{code:java}
Caused by: java.lang.NullPointerException at
org.apache.flink.connector.base.source.reader.SourceReaderBase.finishCurrentFetch(SourceReaderBase.java:194)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.moveToNextSplit(SourceReaderBase.java:208)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:173)
at
org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:131)
at
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419)
at
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:550)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931) at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) at
java.lang.Thread.run(Thread.java:748) {code}
h2. other
The problem only occur in flink local mode and deploy on k8s.
>
[jira] [Commented] (FLINK-35356) Async reducing state
[ https://issues.apache.org/jira/browse/FLINK-35356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847126#comment-17847126 ] Zakelly Lan commented on FLINK-35356: - Merged into master via cc21eec066b3426a7eb0af5303c4590f08d65b2d > Async reducing state > > > Key: FLINK-35356 > URL: https://issues.apache.org/jira/browse/FLINK-35356 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-35356) Async reducing state
[ https://issues.apache.org/jira/browse/FLINK-35356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zakelly Lan resolved FLINK-35356. - Resolution: Fixed > Async reducing state > > > Key: FLINK-35356 > URL: https://issues.apache.org/jira/browse/FLINK-35356 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35356][State] State descriptor and implementation for async reducing state [flink]
Zakelly merged PR #24798: URL: https://github.com/apache/flink/pull/24798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
hanyuzheng7 commented on PR #24365: URL: https://github.com/apache/flink/pull/24365#issuecomment-2116497109 @dawidwys So We can use splitByWholeSeparatorPreserveAllTokens when the delimiter is not empty. When the delimiter is empty, do we need to handle this situation specially? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-35385) upgrader flink dependency version to 1.19
Hongshun Wang created FLINK-35385: - Summary: upgrader flink dependency version to 1.19 Key: FLINK-35385 URL: https://issues.apache.org/jira/browse/FLINK-35385 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: Hongshun Wang Fix For: cdc-3.2.0 Flink 1.19 was released on 2024-03-18 and the connectors have not yet caught up. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35384) Expose TaskIOMetricGroup to custom Partitioner via init Context
[
https://issues.apache.org/jira/browse/FLINK-35384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Zhen Wu updated FLINK-35384:
---
Summary: Expose TaskIOMetricGroup to custom Partitioner via init Context
(was: Expose TaskIOMetricGroup to custom Partitioner)
> Expose TaskIOMetricGroup to custom Partitioner via init Context
> ---
>
> Key: FLINK-35384
> URL: https://issues.apache.org/jira/browse/FLINK-35384
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Affects Versions: 1.9.4
>Reporter: Steven Zhen Wu
>Priority: Major
>
> I am trying to implement a custom range partitioner in the Flink Iceberg
> sink. Want to publish some counter metrics for certain scenarios. This is
> like the network metrics exposed in `TaskIOMetricGroup`.
> We can implement the range partitioner using the public interface from
> `DataStream`.
> {code}
> public DataStream partitionCustom(
> Partitioner partitioner, KeySelector keySelector)
> {code}
> We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
> `CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
> pubic `Partitioner` interface, where we can implement the custom range
> partitioner.
> `Partitioner` interface is a functional interface today. we can add a new
> default `setup` method without breaking the backward compatibility.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> *default void setup(TaskIOMetricGroup metrics) {}*
> int partition(K key, int numPartitions);
> }
> {code}
> I know public interface requires a FLIP process. will do that if the
> community agree with this feature request.
> Personally, `numPartitions` should be passed in the `setup` method too. But
> it is a breaking change that is NOT worth the benefit right now.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
> int partition(K key);
> }
> {code}
> That would be similar to `StreamPartitioner#setup()` method that we would
> need to modify for passing the metrics group.
> {code}
> @Internal
> public abstract class StreamPartitioner
> implements ChannelSelector>>,
> Serializable {
> @Override
> public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
> this.numberOfChannels = numberOfChannels;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35384) Expose TaskIOMetricGroup to custom Partitioner
[
https://issues.apache.org/jira/browse/FLINK-35384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847116#comment-17847116
]
Steven Zhen Wu edited comment on FLINK-35384 at 5/17/24 12:06 AM:
--
one potential risk of this type of API is that it is not extensible. if we want
to pass in another arg to partitioner, we need to break the compatibility or
add a new method.
{code}
default void setup(TaskIOMetricGroup metrics) {}
{code}
Maybe we can move to the context model that is widely used in Flink
{code}
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
int partition(K key, int numPartitions);
void init(Context context) {}
interface Context {
int numberOfChannels();
TaskIOMetricGroup metrics();
}
}
{code}
was (Author: stevenz3wu):
one potential risk of this type of API is that it is not extensible. if we want
to pass in another arg to partitioner, we need to break the compatibility or
add a new method.
{code}
default void setup(TaskIOMetricGroup metrics) {}
{code}
Maybe we can move to the context model that is widely used in Flink
{code}
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
int partition(K key, int numPartitions);
void init(Context context) {}
interface Context {
int numberOfChannels();
TaskIOMetricGroup metrics();
}
}
{code}
> Expose TaskIOMetricGroup to custom Partitioner
> --
>
> Key: FLINK-35384
> URL: https://issues.apache.org/jira/browse/FLINK-35384
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Affects Versions: 1.9.4
>Reporter: Steven Zhen Wu
>Priority: Major
>
> I am trying to implement a custom range partitioner in the Flink Iceberg
> sink. Want to publish some counter metrics for certain scenarios. This is
> like the network metrics exposed in `TaskIOMetricGroup`.
> We can implement the range partitioner using the public interface from
> `DataStream`.
> {code}
> public DataStream partitionCustom(
> Partitioner partitioner, KeySelector keySelector)
> {code}
> We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
> `CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
> pubic `Partitioner` interface, where we can implement the custom range
> partitioner.
> `Partitioner` interface is a functional interface today. we can add a new
> default `setup` method without breaking the backward compatibility.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> *default void setup(TaskIOMetricGroup metrics) {}*
> int partition(K key, int numPartitions);
> }
> {code}
> I know public interface requires a FLIP process. will do that if the
> community agree with this feature request.
> Personally, `numPartitions` should be passed in the `setup` method too. But
> it is a breaking change that is NOT worth the benefit right now.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
> int partition(K key);
> }
> {code}
> That would be similar to `StreamPartitioner#setup()` method that we would
> need to modify for passing the metrics group.
> {code}
> @Internal
> public abstract class StreamPartitioner
> implements ChannelSelector>>,
> Serializable {
> @Override
> public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
> this.numberOfChannels = numberOfChannels;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35384) Expose TaskIOMetricGroup to custom Partitioner
[
https://issues.apache.org/jira/browse/FLINK-35384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847116#comment-17847116
]
Steven Zhen Wu commented on FLINK-35384:
one potential risk of this type of API is that it is not extensible. if we want
to pass in another arg to partitioner, we need to break the compatibility or
add a new method.
{code}
default void setup(TaskIOMetricGroup metrics) {}
{code}
Maybe we can move to the context model that is widely used in Flink
{code}
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
int partition(K key, int numPartitions);
void init(Context context) {}
interface Context {
int numberOfChannels();
TaskIOMetricGroup metrics();
}
}
{code}
> Expose TaskIOMetricGroup to custom Partitioner
> --
>
> Key: FLINK-35384
> URL: https://issues.apache.org/jira/browse/FLINK-35384
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Affects Versions: 1.9.4
>Reporter: Steven Zhen Wu
>Priority: Major
>
> I am trying to implement a custom range partitioner in the Flink Iceberg
> sink. Want to publish some counter metrics for certain scenarios. This is
> like the network metrics exposed in `TaskIOMetricGroup`.
> We can implement the range partitioner using the public interface from
> `DataStream`.
> {code}
> public DataStream partitionCustom(
> Partitioner partitioner, KeySelector keySelector)
> {code}
> We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
> `CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
> pubic `Partitioner` interface, where we can implement the custom range
> partitioner.
> `Partitioner` interface is a functional interface today. we can add a new
> default `setup` method without breaking the backward compatibility.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> *default void setup(TaskIOMetricGroup metrics) {}*
> int partition(K key, int numPartitions);
> }
> {code}
> I know public interface requires a FLIP process. will do that if the
> community agree with this feature request.
> Personally, `numPartitions` should be passed in the `setup` method too. But
> it is a breaking change that is NOT worth the benefit right now.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
> int partition(K key);
> }
> {code}
> That would be similar to `StreamPartitioner#setup()` method that we would
> need to modify for passing the metrics group.
> {code}
> @Internal
> public abstract class StreamPartitioner
> implements ChannelSelector>>,
> Serializable {
> @Override
> public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
> this.numberOfChannels = numberOfChannels;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35378] [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunction [flink]
flinkbot commented on PR #24805: URL: https://github.com/apache/flink/pull/24805#issuecomment-2116209257 ## CI report: * 50b7df7bcec79ae1d990d8184eb77f32942f8e7a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35378) [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc
[ https://issues.apache.org/jira/browse/FLINK-35378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35378: --- Labels: pull-request-available (was: ) > [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc > --- > > Key: FLINK-35378 > URL: https://issues.apache.org/jira/browse/FLINK-35378 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Reporter: Martijn Visser >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > > https://cwiki.apache.org/confluence/pages/resumedraft.action?draftId=303794871=af4ace88-98b7-4a53-aece-cd67d2f91a15; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35378] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction [flink]
MartijnVisser opened a new pull request, #24805: URL: https://github.com/apache/flink/pull/24805 ## What is the purpose of the change Promote Unified Sink API V2 to Public and Deprecate SinkFunction ## Brief change log da30c2faaa487ca03304b90553326ab189fba699 Deprecate SinkFunction, SocketClientSink and TwoPhaseCommitSinkFunction aa94e3822323bbb49bbac81a06a7535aa55c625b Promote SinkV2 interfaces b9c692d3529d093aeb0166623e0584597cd2d075 Use new V2 DiscardingSink 2a11eeb437b6ddbdcf5c9e9fba9b0205ef88c220 Use new PrintSink 50b7df7bcec79ae1d990d8184eb77f32942f8e7a Promote PrintSink from `@PublicEvolving` to `@Public` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35384) Expose TaskIOMetricGroup to custom Partitioner
[
https://issues.apache.org/jira/browse/FLINK-35384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Zhen Wu updated FLINK-35384:
---
Summary: Expose TaskIOMetricGroup to custom Partitioner (was: Expose
metrics group to custom Partitioner)
> Expose TaskIOMetricGroup to custom Partitioner
> --
>
> Key: FLINK-35384
> URL: https://issues.apache.org/jira/browse/FLINK-35384
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Affects Versions: 1.9.4
>Reporter: Steven Zhen Wu
>Priority: Major
>
> I am trying to implement a custom range partitioner in the Flink Iceberg
> sink. Want to publish some counter metrics for certain scenarios. This is
> like the network metrics exposed in `TaskIOMetricGroup`.
> We can implement the range partitioner using the public interface from
> `DataStream`.
> {code}
> public DataStream partitionCustom(
> Partitioner partitioner, KeySelector keySelector)
> {code}
> We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
> `CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
> pubic `Partitioner` interface, where we can implement the custom range
> partitioner.
> `Partitioner` interface is a functional interface today. we can add a new
> default `setup` method without breaking the backward compatibility.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> *default void setup(TaskIOMetricGroup metrics) {}*
> int partition(K key, int numPartitions);
> }
> {code}
> I know public interface requires a FLIP process. will do that if the
> community agree with this feature request.
> Personally, `numPartitions` should be passed in the `setup` method too. But
> it is a breaking change that is NOT worth the benefit right now.
> {code}
> @Public
> @FunctionalInterface
> public interface Partitioner extends java.io.Serializable, Function {
> public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
> int partition(K key);
> }
> {code}
> That would be similar to `StreamPartitioner#setup()` method that we would
> need to modify for passing the metrics group.
> {code}
> @Internal
> public abstract class StreamPartitioner
> implements ChannelSelector>>,
> Serializable {
> @Override
> public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
> this.numberOfChannels = numberOfChannels;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35384) Expose metrics group to custom Partitioner
[
https://issues.apache.org/jira/browse/FLINK-35384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Zhen Wu updated FLINK-35384:
---
Description:
I am trying to implement a custom range partitioner in the Flink Iceberg sink.
Want to publish some counter metrics for certain scenarios. This is like the
network metrics exposed in `TaskIOMetricGroup`.
We can implement the range partitioner using the public interface from
`DataStream`.
{code}
public DataStream partitionCustom(
Partitioner partitioner, KeySelector keySelector)
{code}
We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
`CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
pubic `Partitioner` interface, where we can implement the custom range
partitioner.
`Partitioner` interface is a functional interface today. we can add a new
default `setup` method without breaking the backward compatibility.
{code}
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
*default void setup(TaskIOMetricGroup metrics) {}*
int partition(K key, int numPartitions);
}
{code}
I know public interface requires a FLIP process. will do that if the community
agree with this feature request.
Personally, `numPartitions` should be passed in the `setup` method too. But it
is a breaking change that is NOT worth the benefit right now.
{code}
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
int partition(K key);
}
{code}
That would be similar to `StreamPartitioner#setup()` method that we would need
to modify for passing the metrics group.
{code}
@Internal
public abstract class StreamPartitioner
implements ChannelSelector>>,
Serializable {
@Override
public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
this.numberOfChannels = numberOfChannels;
}
{code}
was:
I am trying to implement a custom range partitioner in the Flink Iceberg sink.
Want to publish some counter metrics for certain scenarios. This is like the
network metrics exposed in `TaskIOMetricGroup`.
We can implement the range partitioner using the public interface from
`DataStream`.
```
public DataStream partitionCustom(
Partitioner partitioner, KeySelector keySelector)
```
We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
`CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
pubic `Partitioner` interface, where we can implement the custom range
partitioner.
`Partitioner` interface is a functional interface today. we can add a new
default `setup` method without breaking the backward compatibility.
```
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
*default void setup(TaskIOMetricGroup metrics) {}*
int partition(K key, int numPartitions);
}
```
I know public interface requires a FLIP process. will do that if the community
agree with this feature request.
Personally, `numPartitions` should be passed in the `setup` method too. But it
is a breaking change that is NOT worth the benefit right now.
```
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
int partition(K key);
}
```
That would be similar to `StreamPartitioner#setup()` method that we would need
to modify for passing the metrics group.
```
@Internal
public abstract class StreamPartitioner
implements ChannelSelector>>,
Serializable {
@Override
public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
this.numberOfChannels = numberOfChannels;
}
```
> Expose metrics group to custom Partitioner
> --
>
> Key: FLINK-35384
> URL: https://issues.apache.org/jira/browse/FLINK-35384
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
>Affects Versions: 1.9.4
>Reporter: Steven Zhen Wu
>Priority: Major
>
> I am trying to implement a custom range partitioner in the Flink Iceberg
> sink. Want to publish some counter metrics for certain scenarios. This is
> like the network metrics exposed in `TaskIOMetricGroup`.
> We can implement the range partitioner using the public interface from
> `DataStream`.
> {code}
> public DataStream partitionCustom(
> Partitioner partitioner, KeySelector keySelector)
> {code}
> We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
> `CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
> pubic `Partitioner` interface, where we can implement the custom range
> partitioner.
>
[jira] [Created] (FLINK-35384) Expose metrics group to custom Partitioner
Steven Zhen Wu created FLINK-35384:
--
Summary: Expose metrics group to custom Partitioner
Key: FLINK-35384
URL: https://issues.apache.org/jira/browse/FLINK-35384
Project: Flink
Issue Type: Improvement
Components: Runtime / Metrics
Affects Versions: 1.9.4
Reporter: Steven Zhen Wu
I am trying to implement a custom range partitioner in the Flink Iceberg sink.
Want to publish some counter metrics for certain scenarios. This is like the
network metrics exposed in `TaskIOMetricGroup`.
We can implement the range partitioner using the public interface from
`DataStream`.
```
public DataStream partitionCustom(
Partitioner partitioner, KeySelector keySelector)
```
We can pass the `TaskIOMetricGroup` to the `StreamPartitioner` that
`CustomPartitionerWrapper` extends from. `CustomPartitionerWrapper` wraps the
pubic `Partitioner` interface, where we can implement the custom range
partitioner.
`Partitioner` interface is a functional interface today. we can add a new
default `setup` method without breaking the backward compatibility.
```
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
*default void setup(TaskIOMetricGroup metrics) {}*
int partition(K key, int numPartitions);
}
```
I know public interface requires a FLIP process. will do that if the community
agree with this feature request.
Personally, `numPartitions` should be passed in the `setup` method too. But it
is a breaking change that is NOT worth the benefit right now.
```
@Public
@FunctionalInterface
public interface Partitioner extends java.io.Serializable, Function {
public void setup(int numPartitions, TaskIOMetricGroup metrics) {}
int partition(K key);
}
```
That would be similar to `StreamPartitioner#setup()` method that we would need
to modify for passing the metrics group.
```
@Internal
public abstract class StreamPartitioner
implements ChannelSelector>>,
Serializable {
@Override
public void setup(int numberOfChannels, TaskIOMetricGroup metrics) {
this.numberOfChannels = numberOfChannels;
}
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-35383][docs] Update compatibility matrix for 1.19 release [flink]
flinkbot commented on PR #24804: URL: https://github.com/apache/flink/pull/24804#issuecomment-2115772472 ## CI report: * 47f36a38135422bac19966e3cf1b5208810d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35383][docs] Update compatibility matrix for 1.19 release [flink]
flinkbot commented on PR #24803: URL: https://github.com/apache/flink/pull/24803#issuecomment-2115770824 ## CI report: * b8865071b048a26da4ef39ff953ca627d919e1bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35383) Update compatibility matrix to include 1.19 release
[ https://issues.apache.org/jira/browse/FLINK-35383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh reassigned FLINK-35383: --- Assignee: Aleksandr Pilipenko > Update compatibility matrix to include 1.19 release > --- > > Key: FLINK-35383 > URL: https://issues.apache.org/jira/browse/FLINK-35383 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Aleksandr Pilipenko >Assignee: Aleksandr Pilipenko >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0, 1.19.1 > > > Update compatibility matrix in documentation to include Flink 1.19 release: > https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/upgrading/#compatibility-table > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/ops/upgrading/#%E5%85%BC%E5%AE%B9%E6%80%A7%E9%80%9F%E6%9F%A5%E8%A1%A8 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35383][docs] Update compatibility matrix for 1.19 release [flink]
z3d1k opened a new pull request, #24804: URL: https://github.com/apache/flink/pull/24804 ## What is the purpose of the change This pull requests updates the snapshot compatibility matrix for release 1.19. ## Brief change log Update compatibility matrix to include Flink 1.19 release ## Verifying this change This is a doc-only change. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-35383][docs] Update compatibility matrix for 1.19 release [flink]
z3d1k opened a new pull request, #24803: URL: https://github.com/apache/flink/pull/24803 ## What is the purpose of the change This pull requests updates the snapshot compatibility matrix for release 1.19. ## Brief change log Update compatibility matrix to include Flink 1.19 release ## Verifying this change This is a doc-only change. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35383) Update compatibility matrix to include 1.19 release
[ https://issues.apache.org/jira/browse/FLINK-35383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35383: --- Labels: pull-request-available (was: ) > Update compatibility matrix to include 1.19 release > --- > > Key: FLINK-35383 > URL: https://issues.apache.org/jira/browse/FLINK-35383 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Aleksandr Pilipenko >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0, 1.19.1 > > > Update compatibility matrix in documentation to include Flink 1.19 release: > https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/upgrading/#compatibility-table > https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/ops/upgrading/#%E5%85%BC%E5%AE%B9%E6%80%A7%E9%80%9F%E6%9F%A5%E8%A1%A8 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35383) Update compatibility matrix to include 1.19 release
Aleksandr Pilipenko created FLINK-35383: --- Summary: Update compatibility matrix to include 1.19 release Key: FLINK-35383 URL: https://issues.apache.org/jira/browse/FLINK-35383 Project: Flink Issue Type: Improvement Components: Documentation Affects Versions: 1.19.0 Reporter: Aleksandr Pilipenko Fix For: 1.20.0, 1.19.1 Update compatibility matrix in documentation to include Flink 1.19 release: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/ops/upgrading/#compatibility-table https://nightlies.apache.org/flink/flink-docs-release-1.19/zh/docs/ops/upgrading/#%E5%85%BC%E5%AE%B9%E6%80%A7%E9%80%9F%E6%9F%A5%E8%A1%A8 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35342) MaterializedTableStatementITCase test can check for wrong status
[
https://issues.apache.org/jira/browse/FLINK-35342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847032#comment-17847032
]
Feng Jin commented on FLINK-35342:
--
[~rskraba] Thank you for the reminder. I will run repeated tests on GitHub
Actions to confirm that this issue has been fixed correctly.
> MaterializedTableStatementITCase test can check for wrong status
>
>
> Key: FLINK-35342
> URL: https://issues.apache.org/jira/browse/FLINK-35342
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> * 1.20 AdaptiveScheduler / Test (module: table)
> https://github.com/apache/flink/actions/runs/9056197319/job/24879135605#step:10:12490
>
> It looks like
> {{MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume}}
> can be flaky, where the expected status is not yet RUNNING:
> {code}
> Error: 03:24:03 03:24:03.902 [ERROR] Tests run: 6, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 26.78 s <<< FAILURE! -- in
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase
> Error: 03:24:03 03:24:03.902 [ERROR]
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume(Path,
> RestClusterClient) -- Time elapsed: 3.850 s <<< FAILURE!
> May 13 03:24:03 org.opentest4j.AssertionFailedError:
> May 13 03:24:03
> May 13 03:24:03 expected: "RUNNING"
> May 13 03:24:03 but was: "CREATED"
> May 13 03:24:03 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> May 13 03:24:03 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> May 13 03:24:03 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> May 13 03:24:03 at
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume(MaterializedTableStatementITCase.java:650)
> May 13 03:24:03 at java.lang.reflect.Method.invoke(Method.java:498)
> May 13 03:24:03 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> May 13 03:24:03
> May 13 03:24:04 03:24:04.270 [INFO]
> May 13 03:24:04 03:24:04.270 [INFO] Results:
> May 13 03:24:04 03:24:04.270 [INFO]
> Error: 03:24:04 03:24:04.270 [ERROR] Failures:
> Error: 03:24:04 03:24:04.271 [ERROR]
> MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume:650
>
> May 13 03:24:04 expected: "RUNNING"
> May 13 03:24:04 but was: "CREATED"
> May 13 03:24:04 03:24:04.271 [INFO]
> Error: 03:24:04 03:24:04.271 [ERROR] Tests run: 82, Failures: 1, Errors: 0,
> Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
MartijnVisser commented on PR #24365: URL: https://github.com/apache/flink/pull/24365#issuecomment-2115689935 > To confirm, you suggest to support an empty delimiter and in that case split all characters Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25168) Azure failed due to unable to transfer maven artifacts
[
https://issues.apache.org/jira/browse/FLINK-25168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847027#comment-17847027
]
Ryan Skraba commented on FLINK-25168:
-
* 1.19 cron_snapshot_deployment_maven
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59585=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=7b3c1df5-9194-5183-5ebd-5567f52d5f8f=6027]
Another failure... it might be that repository.apache.org is unstable at the
moment? In the log, we can also see many retries (eventually succeeding) that
look like this:
{code}
03:23:16.738 [WARNING] Failed to upload checksum to
org/apache/flink/flink-dstl-dfs/1.19-SNAPSHOT/flink-dstl-dfs-1.19-20240516.031110-210-javadoc.jar.sha1
org.apache.maven.wagon.TransferFailedException: transfer failed for
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-dstl-dfs/1.19-SNAPSHOT/flink-dstl-dfs-1.19-20240516.031110-210-javadoc.jar.sha1,
status: 408 Request Timeout
at
org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put
(AbstractHttpClientWagon.java:835)
at
org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put
(AbstractHttpClientWagon.java:750)
at
org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put
(AbstractHttpClientWagon.java:722)
at
org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.put
(AbstractHttpClientWagon.java:716)
at
org.apache.maven.wagon.providers.http.wagon.shared.AbstractHttpClientWagon.putFromStream
(AbstractHttpClientWagon.java:710)
at org.eclipse.aether.transport.wagon.WagonTransporter$PutTaskRunner.run
(WagonTransporter.java:605)
{code}
> Azure failed due to unable to transfer maven artifacts
> --
>
> Key: FLINK-25168
> URL: https://issues.apache.org/jira/browse/FLINK-25168
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.13.3, 1.15.0
>Reporter: Yun Gao
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on
> project flink-tests: Failed to deploy artifacts: Could not transfer artifact
> org.apache.flink:flink-tests:jar:1.13-20211205.020632-728 from/to
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots): Failed to
> transfer file:
> https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-tests/1.13-SNAPSHOT/flink-tests-1.13-20211205.020632-728.jar.
> Return code is: 502, ReasonPhrase: Proxy Error. -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :flink-tests
> ##[error]Bash exited with code '1'.
> Finishing: Deploy maven snapshot
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27560=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7=97156
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26644) python StreamExecutionEnvironmentTests.test_generate_stream_graph_with_dependencies failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847026#comment-17847026
]
Ryan Skraba commented on FLINK-26644:
-
* 1.19 Java 8 / Test (module: python)
https://github.com/apache/flink/actions/runs/910540/job/25031151177#step:10:24395
> python
> StreamExecutionEnvironmentTests.test_generate_stream_graph_with_dependencies
> failed on azure
> ---
>
> Key: FLINK-26644
> URL: https://issues.apache.org/jira/browse/FLINK-26644
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.14.4, 1.15.0, 1.16.0, 1.19.0
>Reporter: Yun Gao
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> 2022-03-14T18:50:24.6842853Z Mar 14 18:50:24
> === FAILURES
> ===
> 2022-03-14T18:50:24.6844089Z Mar 14 18:50:24 _
> StreamExecutionEnvironmentTests.test_generate_stream_graph_with_dependencies _
> 2022-03-14T18:50:24.6844846Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6846063Z Mar 14 18:50:24 self =
> testMethod=test_generate_stream_graph_with_dependencies>
> 2022-03-14T18:50:24.6847104Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6847766Z Mar 14 18:50:24 def
> test_generate_stream_graph_with_dependencies(self):
> 2022-03-14T18:50:24.6848677Z Mar 14 18:50:24 python_file_dir =
> os.path.join(self.tempdir, "python_file_dir_" + str(uuid.uuid4()))
> 2022-03-14T18:50:24.6849833Z Mar 14 18:50:24 os.mkdir(python_file_dir)
> 2022-03-14T18:50:24.6850729Z Mar 14 18:50:24 python_file_path =
> os.path.join(python_file_dir, "test_stream_dependency_manage_lib.py")
> 2022-03-14T18:50:24.6852679Z Mar 14 18:50:24 with
> open(python_file_path, 'w') as f:
> 2022-03-14T18:50:24.6853646Z Mar 14 18:50:24 f.write("def
> add_two(a):\nreturn a + 2")
> 2022-03-14T18:50:24.6854394Z Mar 14 18:50:24 env = self.env
> 2022-03-14T18:50:24.6855019Z Mar 14 18:50:24
> env.add_python_file(python_file_path)
> 2022-03-14T18:50:24.6855519Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6856254Z Mar 14 18:50:24 def plus_two_map(value):
> 2022-03-14T18:50:24.6857045Z Mar 14 18:50:24 from
> test_stream_dependency_manage_lib import add_two
> 2022-03-14T18:50:24.6857865Z Mar 14 18:50:24 return value[0],
> add_two(value[1])
> 2022-03-14T18:50:24.6858466Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6858924Z Mar 14 18:50:24 def add_from_file(i):
> 2022-03-14T18:50:24.6859806Z Mar 14 18:50:24 with
> open("data/data.txt", 'r') as f:
> 2022-03-14T18:50:24.6860266Z Mar 14 18:50:24 return i[0],
> i[1] + int(f.read())
> 2022-03-14T18:50:24.6860879Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6862022Z Mar 14 18:50:24 from_collection_source =
> env.from_collection([('a', 0), ('b', 0), ('c', 1), ('d', 1),
> 2022-03-14T18:50:24.6863259Z Mar 14 18:50:24
> ('e', 2)],
> 2022-03-14T18:50:24.6864057Z Mar 14 18:50:24
> type_info=Types.ROW([Types.STRING(),
> 2022-03-14T18:50:24.6864651Z Mar 14 18:50:24
> Types.INT()]))
> 2022-03-14T18:50:24.6865150Z Mar 14 18:50:24
> from_collection_source.name("From Collection")
> 2022-03-14T18:50:24.6866212Z Mar 14 18:50:24 keyed_stream =
> from_collection_source.key_by(lambda x: x[1], key_type=Types.INT())
> 2022-03-14T18:50:24.6867083Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6867793Z Mar 14 18:50:24 plus_two_map_stream =
> keyed_stream.map(plus_two_map).name("Plus Two Map").set_parallelism(3)
> 2022-03-14T18:50:24.6868620Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6869412Z Mar 14 18:50:24 add_from_file_map =
> plus_two_map_stream.map(add_from_file).name("Add From File Map")
> 2022-03-14T18:50:24.6870239Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6870883Z Mar 14 18:50:24 test_stream_sink =
> add_from_file_map.add_sink(self.test_sink).name("Test Sink")
> 2022-03-14T18:50:24.6871803Z Mar 14 18:50:24
> test_stream_sink.set_parallelism(4)
> 2022-03-14T18:50:24.6872291Z Mar 14 18:50:24
> 2022-03-14T18:50:24.6872756Z Mar 14 18:50:24 archive_dir_path =
> os.path.join(self.tempdir, "archive_" + str(uuid.uuid4()))
> 2022-03-14T18:50:24.6873557Z Mar 14 18:50:24
> os.mkdir(archive_dir_path)
> 2022-03-14T18:50:24.6874817Z Mar 14 18:50:24 with
> open(os.path.join(archive_dir_path, "data.txt"), 'w') as f:
> 2022-03-14T18:50:24.6875414Z Mar 14 18:50:24 f.write("3")
> 2022-03-14T18:50:24.6875906Z Mar 14
[jira] [Commented] (FLINK-35342) MaterializedTableStatementITCase test can check for wrong status
[
https://issues.apache.org/jira/browse/FLINK-35342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847025#comment-17847025
]
Ryan Skraba commented on FLINK-35342:
-
I'm not entirely sure how this can be reproduced locally, but it can also be
observed in GitHub actions if that helps!
* 1.20 Default (Java 8) / Test (module: tests)
https://github.com/apache/flink/actions/runs/9108427134/job/25040554694#step:10:12099
* 1.20 Default (Java 8) / Test (module: tests)
https://github.com/apache/flink/actions/runs/9105407291/job/25031170942#step:10:11841
> MaterializedTableStatementITCase test can check for wrong status
>
>
> Key: FLINK-35342
> URL: https://issues.apache.org/jira/browse/FLINK-35342
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> * 1.20 AdaptiveScheduler / Test (module: table)
> https://github.com/apache/flink/actions/runs/9056197319/job/24879135605#step:10:12490
>
> It looks like
> {{MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume}}
> can be flaky, where the expected status is not yet RUNNING:
> {code}
> Error: 03:24:03 03:24:03.902 [ERROR] Tests run: 6, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 26.78 s <<< FAILURE! -- in
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase
> Error: 03:24:03 03:24:03.902 [ERROR]
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume(Path,
> RestClusterClient) -- Time elapsed: 3.850 s <<< FAILURE!
> May 13 03:24:03 org.opentest4j.AssertionFailedError:
> May 13 03:24:03
> May 13 03:24:03 expected: "RUNNING"
> May 13 03:24:03 but was: "CREATED"
> May 13 03:24:03 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> May 13 03:24:03 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> May 13 03:24:03 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> May 13 03:24:03 at
> org.apache.flink.table.gateway.service.MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume(MaterializedTableStatementITCase.java:650)
> May 13 03:24:03 at java.lang.reflect.Method.invoke(Method.java:498)
> May 13 03:24:03 at
> java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> May 13 03:24:03 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> May 13 03:24:03
> May 13 03:24:04 03:24:04.270 [INFO]
> May 13 03:24:04 03:24:04.270 [INFO] Results:
> May 13 03:24:04 03:24:04.270 [INFO]
> Error: 03:24:04 03:24:04.270 [ERROR] Failures:
> Error: 03:24:04 03:24:04.271 [ERROR]
> MaterializedTableStatementITCase.testAlterMaterializedTableSuspendAndResume:650
>
> May 13 03:24:04 expected: "RUNNING"
> May 13 03:24:04 but was: "CREATED"
> May 13 03:24:04 03:24:04.271 [INFO]
> Error: 03:24:04 03:24:04.271 [ERROR] Tests run: 82, Failures: 1, Errors: 0,
> Skipped: 0
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35380) ResumeCheckpointManuallyITCase hanging on tests
[
https://issues.apache.org/jira/browse/FLINK-35380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847022#comment-17847022
]
Ryan Skraba commented on FLINK-35380:
-
* 1.20 Default (Java 8) / Test (module: tests)
https://github.com/apache/flink/actions/runs/9108427134/job/25040554694#step:10:12099
> ResumeCheckpointManuallyITCase hanging on tests
>
>
> Key: FLINK-35380
> URL: https://issues.apache.org/jira/browse/FLINK-35380
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: test-stability
>
> * 1.20 Default (Java 8) / Test (module: tests)
> https://github.com/apache/flink/actions/runs/9105407291/job/25031170942#step:10:11841
>
> (This is a slightly different error, waiting in a different place than
> FLINK-28319)
> {code}
> May 16 03:23:58
> ==
> May 16 03:23:58 Process produced no output for 900 seconds.
> May 16 03:23:58
> ==
> ... snip until stack trace ...
> ay 16 03:23:58at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> May 16 03:23:58 at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> May 16 03:23:58 at
> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> May 16 03:23:58 at
> org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.runJobAndGetExternalizedCheckpoint(ResumeCheckpointManuallyITCase.java:410)
> May 16 03:23:58 at
> org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.testExternalizedCheckpoints(ResumeCheckpointManuallyITCase.java:378)
> May 16 03:23:58 at
> org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.testExternalizedCheckpoints(ResumeCheckpointManuallyITCase.java:318)
> May 16 03:23:58 at
> org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.testExternalizedFullRocksDBCheckpointsWithLocalRecoveryStandalone(ResumeCheckpointManuallyITCase.java:133)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35382) ChangelogCompatibilityITCase.testRestore fails with an NPE
[
https://issues.apache.org/jira/browse/FLINK-35382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847020#comment-17847020
]
Ryan Skraba commented on FLINK-35382:
-
[~lijinzhong] Do you think this is related to the changes made in FLINK-32080?
> ChangelogCompatibilityITCase.testRestore fails with an NPE
> --
>
> Key: FLINK-35382
> URL: https://issues.apache.org/jira/browse/FLINK-35382
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Ryan Skraba
>Priority: Critical
> Labels: test-stability
>
> * 1.20 Java 8 / Test (module: tests)
> https://github.com/apache/flink/actions/runs/9110398985/job/25045798401#step:10:8192
> It looks like there can be a [NullPointerException at this
> line|https://github.com/apache/flink/blob/9a5a99b1a30054268bbde36d565cbb1b81018890/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java#L666]
> causing a test failure:
> {code}
> Error: 10:36:23 10:36:23.312 [ERROR] Tests run: 9, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 19.31 s <<< FAILURE! -- in
> org.apache.flink.test.state.ChangelogCompatibilityITCase
> Error: 10:36:23 10:36:23.313 [ERROR]
> org.apache.flink.test.state.ChangelogCompatibilityITCase.testRestore[startWithChangelog=false,
> restoreWithChangelog=true, restoreFrom=CHECKPOINT, allowStore=true,
> allowRestore=true] -- Time elapsed: 1.492 s <<< ERROR!
> May 16 10:36:23 java.lang.RuntimeException:
> org.opentest4j.AssertionFailedError: Graph is in globally terminal state
> (FAILED)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.tryRun(ChangelogCompatibilityITCase.java:204)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.restoreAndValidate(ChangelogCompatibilityITCase.java:190)
> May 16 10:36:23 at java.util.Optional.ifPresent(Optional.java:159)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.testRestore(ChangelogCompatibilityITCase.java:118)
> May 16 10:36:23 at java.lang.reflect.Method.invoke(Method.java:498)
> May 16 10:36:23 Caused by: org.opentest4j.AssertionFailedError: Graph is in
> globally terminal state (FAILED)
> May 16 10:36:23 at
> org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:42)
> May 16 10:36:23 at
> org.junit.jupiter.api.Assertions.fail(Assertions.java:150)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.lambda$waitForAllTaskRunning$3(CommonTestUtils.java:214)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:151)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:209)
> May 16 10:36:23 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:182)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.submit(ChangelogCompatibilityITCase.java:284)
> May 16 10:36:23 at
> org.apache.flink.test.state.ChangelogCompatibilityITCase.tryRun(ChangelogCompatibilityITCase.java:197)
> May 16 10:36:23 ... 4 more
> May 16 10:36:23 Caused by: org.apache.flink.runtime.JobException:
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> NoRestartBackoffTimeStrategy
> May 16 10:36:23 at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:219)
> May 16 10:36:23 at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailureAndReport(ExecutionFailureHandler.java:166)
> May 16 10:36:23 at
> org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:121)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:279)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:270)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:263)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:788)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:765)
> May 16 10:36:23 at
> org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
> May 16 10:36:23
[jira] [Created] (FLINK-35382) ChangelogCompatibilityITCase.testRestore fails with an NPE
Ryan Skraba created FLINK-35382:
---
Summary: ChangelogCompatibilityITCase.testRestore fails with an NPE
Key: FLINK-35382
URL: https://issues.apache.org/jira/browse/FLINK-35382
Project: Flink
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Ryan Skraba
* 1.20 Java 8 / Test (module: tests)
https://github.com/apache/flink/actions/runs/9110398985/job/25045798401#step:10:8192
It looks like there can be a [NullPointerException at this
line|https://github.com/apache/flink/blob/9a5a99b1a30054268bbde36d565cbb1b81018890/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java#L666]
causing a test failure:
{code}
Error: 10:36:23 10:36:23.312 [ERROR] Tests run: 9, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 19.31 s <<< FAILURE! -- in
org.apache.flink.test.state.ChangelogCompatibilityITCase
Error: 10:36:23 10:36:23.313 [ERROR]
org.apache.flink.test.state.ChangelogCompatibilityITCase.testRestore[startWithChangelog=false,
restoreWithChangelog=true, restoreFrom=CHECKPOINT, allowStore=true,
allowRestore=true] -- Time elapsed: 1.492 s <<< ERROR!
May 16 10:36:23 java.lang.RuntimeException:
org.opentest4j.AssertionFailedError: Graph is in globally terminal state
(FAILED)
May 16 10:36:23 at
org.apache.flink.test.state.ChangelogCompatibilityITCase.tryRun(ChangelogCompatibilityITCase.java:204)
May 16 10:36:23 at
org.apache.flink.test.state.ChangelogCompatibilityITCase.restoreAndValidate(ChangelogCompatibilityITCase.java:190)
May 16 10:36:23 at java.util.Optional.ifPresent(Optional.java:159)
May 16 10:36:23 at
org.apache.flink.test.state.ChangelogCompatibilityITCase.testRestore(ChangelogCompatibilityITCase.java:118)
May 16 10:36:23 at java.lang.reflect.Method.invoke(Method.java:498)
May 16 10:36:23 Caused by: org.opentest4j.AssertionFailedError: Graph is in
globally terminal state (FAILED)
May 16 10:36:23 at
org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:42)
May 16 10:36:23 at
org.junit.jupiter.api.Assertions.fail(Assertions.java:150)
May 16 10:36:23 at
org.apache.flink.runtime.testutils.CommonTestUtils.lambda$waitForAllTaskRunning$3(CommonTestUtils.java:214)
May 16 10:36:23 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:151)
May 16 10:36:23 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:145)
May 16 10:36:23 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:209)
May 16 10:36:23 at
org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning(CommonTestUtils.java:182)
May 16 10:36:23 at
org.apache.flink.test.state.ChangelogCompatibilityITCase.submit(ChangelogCompatibilityITCase.java:284)
May 16 10:36:23 at
org.apache.flink.test.state.ChangelogCompatibilityITCase.tryRun(ChangelogCompatibilityITCase.java:197)
May 16 10:36:23 ... 4 more
May 16 10:36:23 Caused by: org.apache.flink.runtime.JobException:
org.apache.flink.runtime.JobException: Recovery is suppressed by
NoRestartBackoffTimeStrategy
May 16 10:36:23 at
org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:219)
May 16 10:36:23 at
org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.handleFailureAndReport(ExecutionFailureHandler.java:166)
May 16 10:36:23 at
org.apache.flink.runtime.executiongraph.failover.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:121)
May 16 10:36:23 at
org.apache.flink.runtime.scheduler.DefaultScheduler.recordTaskFailure(DefaultScheduler.java:279)
May 16 10:36:23 at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:270)
May 16 10:36:23 at
org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:263)
May 16 10:36:23 at
org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:788)
May 16 10:36:23 at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:765)
May 16 10:36:23 at
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
May 16 10:36:23 at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:496)
May 16 10:36:23 at java.lang.reflect.Method.invoke(Method.java:498)
May 16 10:36:23 at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:318)
May 16 10:36:23 at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
May 16
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
hanyuzheng7 commented on code in PR #24365:
URL: https://github.com/apache/flink/pull/24365#discussion_r1603638142
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##
@@ -1721,4 +1722,83 @@ private Stream arrayExceptTestCases() {
"Invalid input arguments. Expected signatures
are:\n"
+ "ARRAY_EXCEPT(, )"));
}
+
+private Stream splitTestCases() {
+return Stream.of(
+TestSetSpec.forFunction(BuiltInFunctionDefinitions.SPLIT)
+.onFieldsWithData(
+"123,123,23",
+null,
+",123,123",
+",123,123,",
+123,
+"12345",
+",123,,,123,")
+.andDataTypes(
+DataTypes.STRING().notNull(),
+DataTypes.STRING(),
+DataTypes.STRING().notNull(),
+DataTypes.STRING().notNull(),
+DataTypes.INT().notNull(),
+DataTypes.STRING().notNull(),
+DataTypes.STRING().notNull())
+.testResult(
+$("f0").split(","),
+"SPLIT(f0, ',')",
+new String[] {"123", "123", "23"},
+DataTypes.ARRAY(DataTypes.STRING()).notNull())
+.testResult(
+$("f0").split(null),
+"SPLIT(f0, NULL)",
+null,
+DataTypes.ARRAY(DataTypes.STRING()))
+.testResult(
+$("f0").split(""),
+"SPLIT(f0, '')",
+new String[] {"123,123,23"},
Review Comment:
return input string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
hanyuzheng7 commented on PR #24365:
URL: https://github.com/apache/flink/pull/24365#issuecomment-2115611425
> @MartijnVisser To confirm, you suggest to support an empty delimiter and
in that case split all characters, meaning:
>
> ```
> SPLIT('abcde', '') = ['a', 'b', 'c', 'd', 'e']
> SPLIT('It is', '')=['I', 't', ' ', 'i', 's']
> ```
> @MartijnVisser To confirm, you suggest to support an empty delimiter and
in that case split all characters, meaning:
>
> ```
> SPLIT('abcde', '') = ['a', 'b', 'c', 'd', 'e']
> SPLIT('It is', '')=['I', 't', ' ', 'i', 's']
> ```
> @MartijnVisser To confirm, you suggest to support an empty delimiter and
in that case split all characters, meaning:
>
> ```
> SPLIT('abcde', '') = ['a', 'b', 'c', 'd', 'e']
> SPLIT('It is', '')=['I', 't', ' ', 'i', 's']
> ```
Now if the delimiter is empty, we will return input string, here is a test
case.
https://github.com/apache/flink/pull/24365/files#r1603638142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32706][table] Add built-in SPLIT_STRING function [flink]
hanyuzheng7 commented on code in PR #24365:
URL: https://github.com/apache/flink/pull/24365#discussion_r1603638142
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CollectionFunctionsITCase.java:
##
@@ -1721,4 +1722,83 @@ private Stream arrayExceptTestCases() {
"Invalid input arguments. Expected signatures
are:\n"
+ "ARRAY_EXCEPT(, )"));
}
+
+private Stream splitTestCases() {
+return Stream.of(
+TestSetSpec.forFunction(BuiltInFunctionDefinitions.SPLIT)
+.onFieldsWithData(
+"123,123,23",
+null,
+",123,123",
+",123,123,",
+123,
+"12345",
+",123,,,123,")
+.andDataTypes(
+DataTypes.STRING().notNull(),
+DataTypes.STRING(),
+DataTypes.STRING().notNull(),
+DataTypes.STRING().notNull(),
+DataTypes.INT().notNull(),
+DataTypes.STRING().notNull(),
+DataTypes.STRING().notNull())
+.testResult(
+$("f0").split(","),
+"SPLIT(f0, ',')",
+new String[] {"123", "123", "23"},
+DataTypes.ARRAY(DataTypes.STRING()).notNull())
+.testResult(
+$("f0").split(null),
+"SPLIT(f0, NULL)",
+null,
+DataTypes.ARRAY(DataTypes.STRING()))
+.testResult(
+$("f0").split(""),
+"SPLIT(f0, '')",
+new String[] {"123,123,23"},
Review Comment:
return input string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-20539][table-planner] Fix type mismatch when using ROW in comp… [flink]
snuyanzin commented on PR #23772: URL: https://github.com/apache/flink/pull/23772#issuecomment-2115525355 @xuyangzhong could you please rebase it to be sure it still passes the ci? Once it passes we could merge it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-35331) Download links for binary releases are displayed as source releases on website
[ https://issues.apache.org/jira/browse/FLINK-35331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Ge reassigned FLINK-35331: --- Assignee: Hector Sanchez > Download links for binary releases are displayed as source releases on website > -- > > Key: FLINK-35331 > URL: https://issues.apache.org/jira/browse/FLINK-35331 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Xintong Song >Assignee: Hector Sanchez >Priority: Major > Labels: pull-request-available > > Take Pre-bundled Hadoop as examples. The content for downloading are binary > releases, while the link is displayed as "Pre-bundled Hadoop 2.x.y Source > Release (asc, sha512)". The problem is caused by misusing > `source_release_[url|asc_url|sha512_url]` for binary contents in the > corresponding [yaml > file.|https://github.com/apache/flink-web/blob/asf-site/docs/data/additional_components.yml] > There are many similar cases in the webpage. > And a relevant issues is that, some source releases are displayed as "XXX > Source Release Source Release", due to including "Source Release" in the > `name` field of the corresponding yaml file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35381) LocalRecoveryITCase failure on deleting directory
Ryan Skraba created FLINK-35381:
---
Summary: LocalRecoveryITCase failure on deleting directory
Key: FLINK-35381
URL: https://issues.apache.org/jira/browse/FLINK-35381
Project: Flink
Issue Type: Bug
Reporter: Ryan Skraba
* 1.20 Java 11 / Test (module: tests)
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54856=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11288
F
It looks like some resources in a subdirectory of a JUnit4 {{ClassRule}} temp
directory prevent it from being cleaned up. This was fixed in a different test
in FLINK-33641.
{code}
SEVERE: Caught exception while closing extension context:
org.junit.jupiter.engine.descriptor.MethodExtensionContext@2fc91366
java.io.IOException: Failed to delete temp directory
/tmp/junit7935976901063386613. The following paths could not be deleted (see
suppressed exceptions for details):
tm_taskManager_0/localState/aid_1501e77149be2f931eab0a6c2e818f81/jid_fe61a39afa9873389353abb8bfbfba66/vtx_0a448493b4782967b150582570326227_sti_0,
tm_taskManager_0/localState/aid_1501e77149be2f931eab0a6c2e818f81/jid_fe61a39afa9873389353abb8bfbfba66/vtx_bc764cd8ddf7a0cff126f51c16239658_sti_0/chk_51
at
org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:431)
at
org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:312)
at
org.junit.jupiter.engine.descriptor.AbstractExtensionContext.lambda$static$0(AbstractExtensionContext.java:45)
at
org.junit.platform.engine.support.store.NamespacedHierarchicalStore$EvaluatedValue.close(NamespacedHierarchicalStore.java:333)
at
org.junit.platform.engine.support.store.NamespacedHierarchicalStore$EvaluatedValue.access$800(NamespacedHierarchicalStore.java:317)
at
org.junit.platform.engine.support.store.NamespacedHierarchicalStore.lambda$close$3(NamespacedHierarchicalStore.java:98)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.store.NamespacedHierarchicalStore.lambda$close$4(NamespacedHierarchicalStore.java:98)
at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at
java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258)
at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at
org.junit.platform.engine.support.store.NamespacedHierarchicalStore.close(NamespacedHierarchicalStore.java:98)
at
org.junit.jupiter.engine.descriptor.AbstractExtensionContext.close(AbstractExtensionContext.java:87)
at
org.junit.jupiter.engine.execution.JupiterEngineExecutionContext.close(JupiterEngineExecutionContext.java:53)
at
org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:224)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$cleanUp$1(TestMethodTestDescriptor.java:156)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.cleanUp(TestMethodTestDescriptor.java:156)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.cleanUp(TestMethodTestDescriptor.java:69)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$cleanUp$10(NodeTestTask.java:167)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.cleanUp(NodeTestTask.java:167)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:98)
at
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:202)
at
[jira] [Commented] (FLINK-35358) Breaking change when loading artifacts
[
https://issues.apache.org/jira/browse/FLINK-35358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846991#comment-17846991
]
Ferenc Csaky commented on FLINK-35358:
--
In that test class all directories are generated before every test case, so it
looks like this:
{code}
/var/folders/wp/ccy48gw1255bswh9bx9svxjcgn/T/5ea4975f-13a5-47eb-a61f-4b1166e7f885/_user_dir_with_single_entry_class
{code}
The JAR(s) are under the {{_user_dir_with_single_entry_class}} dir, and even
its parent is generated specifically for the test case, so it is safe to simply
get the parent for testing the recursive file listing. These tests do not use
the actual {{/opt/flink/usrlib}} path at all.
> Breaking change when loading artifacts
> --
>
> Key: FLINK-35358
> URL: https://issues.apache.org/jira/browse/FLINK-35358
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, flink-docker
>Affects Versions: 1.19.0
>Reporter: Rasmus Thygesen
>Priority: Not a Priority
> Labels: pull-request-available
> Fix For: 1.19.1
>
>
> We have been using the following code snippet in our Dockerfiles for running
> a Flink job in application mode
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar
> /opt/flink/usrlib/artifacts/my-job.jar
> USER flink {code}
>
> Which has been working since at least around Flink 1.14, but the 1.19 update
> has broken our Dockerfiles. The fix is to put the jar file a step further out
> so the code snippet becomes
>
> {code:java}
> FROM flink:1.18.1-scala_2.12-java17
> COPY --from=build /app/target/my-job*.jar /opt/flink/usrlib/my-job.jar
> USER flink {code}
>
> We have not spent too much time looking into what the cause is, but we get
> the stack trace
>
> {code:java}
> myjob-jobmanager-1 | org.apache.flink.util.FlinkException: Could not load
> the provided entrypoint class.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:230)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:149)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.lambda$main$0(StandaloneApplicationClusterEntryPoint.java:90)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:89)
> [flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | Caused by:
> org.apache.flink.client.program.ProgramInvocationException: The program's
> entry point class 'my.company.job.MyJob' was not found in the jar file.
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.loadMainClass(PackagedProgram.java:481)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:153)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:65)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:228)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | ... 4 more
> myjob-jobmanager-1 | Caused by: java.lang.ClassNotFoundException:
> my.company.job.MyJob
> myjob-jobmanager-1 | at java.net.URLClassLoader.findClass(Unknown
> Source) ~[?:?]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown Source)
> ~[?:?]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.19.0.jar:1.19.0]
> myjob-jobmanager-1 | at java.lang.ClassLoader.loadClass(Unknown
[jira] [Created] (FLINK-35380) ResumeCheckpointManuallyITCase hanging on tests
Ryan Skraba created FLINK-35380:
---
Summary: ResumeCheckpointManuallyITCase hanging on tests
Key: FLINK-35380
URL: https://issues.apache.org/jira/browse/FLINK-35380
Project: Flink
Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Ryan Skraba
* 1.20 Default (Java 8) / Test (module: tests)
https://github.com/apache/flink/actions/runs/9105407291/job/25031170942#step:10:11841
(This is a slightly different error, waiting in a different place than
FLINK-28319)
{code}
May 16 03:23:58
==
May 16 03:23:58 Process produced no output for 900 seconds.
May 16 03:23:58
==
... snip until stack trace ...
ay 16 03:23:58 at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
May 16 03:23:58 at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
May 16 03:23:58 at
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
May 16 03:23:58 at
org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.runJobAndGetExternalizedCheckpoint(ResumeCheckpointManuallyITCase.java:410)
May 16 03:23:58 at
org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.testExternalizedCheckpoints(ResumeCheckpointManuallyITCase.java:378)
May 16 03:23:58 at
org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.testExternalizedCheckpoints(ResumeCheckpointManuallyITCase.java:318)
May 16 03:23:58 at
org.apache.flink.test.checkpointing.ResumeCheckpointManuallyITCase.testExternalizedFullRocksDBCheckpointsWithLocalRecoveryStandalone(ResumeCheckpointManuallyITCase.java:133)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846975#comment-17846975 ] Lorenzo Affetti commented on FLINK-20628: - Sure thing, thank you [~chalixar] ! > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: RocMarshal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846971#comment-17846971 ] Ahmed Hamdy edited comment on FLINK-20628 at 5/16/24 2:25 PM: -- [~lorenzo.affetti] could you help with this one if you have capacity? was (Author: JIRAUSER280246): [~affo] could you help with this one if you have capacity? > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: RocMarshal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846971#comment-17846971 ] Ahmed Hamdy commented on FLINK-20628: - [~affo] could you help with this one if you have capacity? > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Assignee: RocMarshal >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846968#comment-17846968
]
Martijn Visser edited comment on FLINK-20539 at 5/16/24 2:17 PM:
-
[~qingyue] Should this ticket have been closed already? Or is still open?
Edit: Checked myself: it's in 1.19.0, but not backported to 1.18.*
was (Author: martijnvisser):
[~qingyue] Should this ticket have been closed already? Or is still open?
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28220) Create Table Like support excluding physical columns
[
https://issues.apache.org/jira/browse/FLINK-28220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-28220:
---
Fix Version/s: (was: 1.19.0)
(was: 1.18.2)
> Create Table Like support excluding physical columns
>
>
> Key: FLINK-28220
> URL: https://issues.apache.org/jira/browse/FLINK-28220
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: JustinLee
>Priority: Major
>
> when users want to Create Table A Like B , they can choose to include or
> exclude options, computed columns ,etc. But it's mandatory that table A
> should inherit all physical columns of table B, which may cause inconvenience
> in some scenes , such as table A has its own schema and just want to inherit
> the options of table B.
> In our production case, sometimes we need to consume kafka table as RAW
> format which has an original JSON format , so we would like to Create Table A
> (data string ) with ('format'='raw') Like B, but it encounters an error
> saying table A has more than one physical columns because of inheriting these
> columns from table B.
> so I think it would be more flexible to provide the option to include or
> exclude physical columns when Using Create Table .. Like ..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-20539:
---
Fix Version/s: 1.18.2
1.19.0
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.19.0, 1.18.2
>
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28220) Create Table Like support excluding physical columns
[
https://issues.apache.org/jira/browse/FLINK-28220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-28220:
---
Fix Version/s: 1.18.2
1.19.0
> Create Table Like support excluding physical columns
>
>
> Key: FLINK-28220
> URL: https://issues.apache.org/jira/browse/FLINK-28220
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.15.0
>Reporter: JustinLee
>Priority: Major
> Fix For: 1.19.0, 1.18.2
>
>
> when users want to Create Table A Like B , they can choose to include or
> exclude options, computed columns ,etc. But it's mandatory that table A
> should inherit all physical columns of table B, which may cause inconvenience
> in some scenes , such as table A has its own schema and just want to inherit
> the options of table B.
> In our production case, sometimes we need to consume kafka table as RAW
> format which has an original JSON format , so we would like to Create Table A
> (data string ) with ('format'='raw') Like B, but it encounters an error
> saying table A has more than one physical columns because of inheriting these
> columns from table B.
> so I think it would be more flexible to provide the option to include or
> exclude physical columns when Using Create Table .. Like ..
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20539) Type mismatch when using ROW in computed column
[
https://issues.apache.org/jira/browse/FLINK-20539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846968#comment-17846968
]
Martijn Visser commented on FLINK-20539:
[~qingyue] Should this ticket have been closed already? Or is still open?
> Type mismatch when using ROW in computed column
> ---
>
> Key: FLINK-20539
> URL: https://issues.apache.org/jira/browse/FLINK-20539
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: xuyang
>Priority: Major
> Labels: auto-unassigned, pull-request-available
>
> The following SQL:
> {code}
> env.executeSql(
> "CREATE TABLE Orders (\n"
> + "order_number BIGINT,\n"
> + "priceINT,\n"
> + "first_name STRING,\n"
> + "last_nameSTRING,\n"
> + "buyer_name AS ROW(first_name, last_name)\n"
> + ") WITH (\n"
> + " 'connector' = 'datagen'\n"
> + ")");
> env.executeSql("SELECT * FROM Orders").print();
> {code}
> Fails with:
> {code}
> Exception in thread "main" java.lang.AssertionError: Conversion to relational
> algebra failed to preserve datatypes:
> validated type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" EXPR$0, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT
> NULL buyer_name) NOT NULL
> converted type:
> RecordType(BIGINT order_number, INTEGER price, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" first_name, VARCHAR(2147483647) CHARACTER SET "UTF-16LE"
> last_name, RecordType(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$0,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" EXPR$1) NOT NULL buyer_name) NOT
> NULL
> rel:
> LogicalProject(order_number=[$0], price=[$1], first_name=[$2],
> last_name=[$3], buyer_name=[ROW($2, $3)])
> LogicalTableScan(table=[[default_catalog, default_database, Orders]])
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:467)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:582)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
