[jira] [Updated] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35215: Attachment: image-2024-04-25-14-57-55-231.png > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-25-14-57-55-231.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > [http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35234][hotfix][cdc-common] Fix NullPointerException of org.apache.flink.cdc.common.configuration.ConfigurationUtils#convertToString [flink-cdc]
yuxiqian commented on code in PR #3255:
URL: https://github.com/apache/flink-cdc/pull/3255#discussion_r1578977096
##
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/configuration/ConfigurationUtils.java:
##
@@ -130,7 +130,9 @@ static Duration convertToDuration(Object o) {
}
static String convertToString(Object o) {
-if (o.getClass() == String.class) {
+if (o == null) {
Review Comment:
Maybe we can bring the check to `convertValue` function? Seems other
`convertToXXX` functions also share this problem.
##
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/configuration/ConfigurationUtils.java:
##
@@ -130,7 +130,9 @@ static Duration convertToDuration(Object o) {
}
static String convertToString(Object o) {
-if (o.getClass() == String.class) {
+if (o == null) {
Review Comment:
Maybe we can bring the check to `convertValue` function? Seems other
`convertToXXX` functions also have this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan updated FLINK-35215: Attachment: image-2024-04-25-15-00-32-410.png > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > [http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840678#comment-17840678 ] Rui Fan commented on FLINK-35215: - Hi [~q.xu] [~kkrugler] , flink community has a performance regression detector to detect them, and send the result to Slack channel every day. I noticed these regression there. If you are interested in it, you can can more details from here[1][2]. Sometimes the performance regression detector will also send an alert when some benchmarks fluctuate only occasionally. But according to my observation, serializerKryo and serializerKryoWithoutRegistration really have performance regression.(it's not a big regression.) Let us focus on the java 17 and serializerKryoWithoutRegistration(I saw java 8 and java11 are similar). * After April 19, average performance is lower than before April 19. ** Note: let us focus on the average performance instead of single day. ** The reason for single-day performance regression may be caused by fluctuation. ** In general, the performance will be recovered in a few days if the performance regression is caused by fluctuation. ** All of them (serializerKryoWithoutRegistration and serializerKryo, java 8, java11 and java17) don't recovered for now. * And FLINK-34954 is merged at April 19 * Also, I reverts the FLINK-34954 on my flink fork repo, and try to re-run the benchmark 3 times. ** The performance is recovered after reverting, that's why I'm sure FLINK-34954 causes it. ** I will add the result later. !image-2024-04-25-14-57-55-231.png|width=968,height=365! !image-2024-04-25-15-00-32-410.png|width=974,height=363! [1][https://github.com/apache/flink-benchmarks/blob/master/regression_report_v2.py] [2][https://docs.google.com/document/d/1Bvzvq79Ll5yxd1UtC0YzczgFbZPAgPcN3cI0MjVkIag] > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > [http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [mysql] Fix use latest-offset mode during caused add newly table not … [flink-cdc]
yuxiqian commented on PR #1185: URL: https://github.com/apache/flink-cdc/pull/1185#issuecomment-2076515755 Hi @minchowang, thanks for your contribution! Seems this feature has been covered by adding a new `checkNewlyAddedTableSchema` option. Feel free to reopen this PR if you have any concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] Fix the binlogSplit state incompatibility when upgrading from 2.2.0 [flink-cdc]
yuxiqian commented on PR #1211: URL: https://github.com/apache/flink-cdc/pull/1211#issuecomment-2076517320 Hi @lzshlzsh, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [tdsql] The first draft of TDSQL-CDC [flink-cdc]
yuxiqian commented on PR #1332: URL: https://github.com/apache/flink-cdc/pull/1332#issuecomment-2076525157 Hi @guixin, Thanks for your great work! Also sorry for the delay. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors` package has been moved to `org.apache.flink.cdc.connectors`, and your new connector should be placed under `flink-cdc-connect/flink-cdc-source-connectors` path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34639] Support debezium deserializer in OceanBase source connector [flink-cdc]
PatrickRen merged PR #3124: URL: https://github.com/apache/flink-cdc/pull/3124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
fredia commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1578997828
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -220,12 +283,17 @@ void insertBlockingBuffer(StateRequest request) {
* @param force whether to trigger requests in force.
*/
void triggerIfNeeded(boolean force) {
-// TODO: introduce a timeout mechanism for triggering.
if (!force && stateRequestsBuffer.activeQueueSize() < batchSize) {
+// if the active queue size is 1, it means that the current
request is the oldest one in
+// the active queue.
+if (stateRequestsBuffer.activeQueueSize() == 1) {
+scheduleTimeout(currentTriggerSeq.get());
+}
Review Comment:
Currently, `triggerIfNeeded(false)` is only called in `handleRequest()`, I
added a `schedulingCount==1` condition to avoid "trigger multiple times for one
seq" in the future.
And I moved `scheduleTimeout ` into `StateRequestBuffer`, thanks for the
suggestion.👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34639) Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector
[ https://issues.apache.org/jira/browse/FLINK-34639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840683#comment-17840683 ] Qingsheng Ren commented on FLINK-34639: --- flink-cdc master: 05281e5d6f74d1b9a0d2f9845af25a61bdd49066 > Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector > -- > > Key: FLINK-34639 > URL: https://issues.apache.org/jira/browse/FLINK-34639 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: He Wang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > Support DebeziumDeserializationSchema in OceanBase source connector. > > Related issue: [https://github.com/apache/flink-cdc/issues/2450] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35235) Fix missing dependencies in the uber jar
LvYanquan created FLINK-35235: - Summary: Fix missing dependencies in the uber jar Key: FLINK-35235 URL: https://issues.apache.org/jira/browse/FLINK-35235 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Attachments: image-2024-04-25-15-17-20-987.png, image-2024-04-25-15-17-34-717.png Some class of Kafka were not included in fat jar. !image-2024-04-25-15-17-34-717.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-34639) Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector
[ https://issues.apache.org/jira/browse/FLINK-34639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-34639: - Assignee: He Wang > Flink CDC: Support DebeziumDeserializationSchema in OceanBase source connector > -- > > Key: FLINK-34639 > URL: https://issues.apache.org/jira/browse/FLINK-34639 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: He Wang >Assignee: He Wang >Priority: Major > Labels: pull-request-available > Fix For: cdc-3.1.0 > > > Support DebeziumDeserializationSchema in OceanBase source connector. > > Related issue: [https://github.com/apache/flink-cdc/issues/2450] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
fredia commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1578997828
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -220,12 +283,17 @@ void insertBlockingBuffer(StateRequest request) {
* @param force whether to trigger requests in force.
*/
void triggerIfNeeded(boolean force) {
-// TODO: introduce a timeout mechanism for triggering.
if (!force && stateRequestsBuffer.activeQueueSize() < batchSize) {
+// if the active queue size is 1, it means that the current
request is the oldest one in
+// the active queue.
+if (stateRequestsBuffer.activeQueueSize() == 1) {
+scheduleTimeout(currentTriggerSeq.get());
+}
Review Comment:
Currently, `triggerIfNeeded(false)` is only called in `handleRequest()`, I
added a `schedulingCount==0` condition to avoid "trigger multiple times for one
seq" in the future.
And I moved `scheduleTimeout ` into `StateRequestBuffer`, thanks for the
suggestion.👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Debezium] Fix the server timeZone bug of debeziumDeserialization when using dataSream API（#317） [flink-cdc]
yuxiqian commented on PR #1366: URL: https://github.com/apache/flink-cdc/pull/1366#issuecomment-2076542997 Hi @, thanks for your contribution! Seems this issue has been fixed in #1407, so I'm going to close this PR. Feel free to reopen it if you have other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [docs][pipeline-connector][paimon] fix example bug for `Paimon 'wareh… [flink-cdc]
beryllw opened a new pull request, #3260: URL: https://github.com/apache/flink-cdc/pull/3260 fix example bug for `Paimon 'warehouse' path must be set` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [metric] fix flink cdc metric conflict with flink built in metric [flink-cdc]
yuxiqian commented on PR #1431: URL: https://github.com/apache/flink-cdc/pull/1431#issuecomment-2076545996 Hi @zhmin, thanks for your contribution! Seems this issue has been fixed in #2415, so I'm going to close this PR. Feel free to reopen it if you have other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [docs][pipeline-connector][paimon] fix example bug for `Paimon 'wareh… [flink-cdc]
beryllw commented on PR #3260: URL: https://github.com/apache/flink-cdc/pull/3260#issuecomment-2076549059 When I use paimon sink I encounter a bug.Looks like this is an incorrect attribute setting in Example yaml ``` Exception in thread "main" java.lang.RuntimeException: failed to create or use paimon catalog at org.apache.flink.cdc.connectors.paimon.sink.PaimonDataSinkFactory.createDataSink(PaimonDataSinkFactory.java:72) at org.apache.flink.cdc.composer.flink.FlinkPipelineComposer.createDataSink(FlinkPipelineComposer.java:164) at org.apache.flink.cdc.composer.flink.FlinkPipelineComposer.compose(FlinkPipelineComposer.java:129) at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:71) at org.apache.flink.cdc.cli.CliFrontend.main(CliFrontend.java:71) Caused by: java.lang.NullPointerException: Paimon 'warehouse' path must be set at org.apache.paimon.utils.Preconditions.checkNotNull(Preconditions.java:65) at org.apache.paimon.catalog.CatalogFactory.warehouse(CatalogFactory.java:55) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:83) at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:80) at org.apache.flink.cdc.connectors.paimon.sink.PaimonDataSinkFactory.createDataSink(PaimonDataSinkFactory.java:68) ... 4 more ``` When I correct `metastore` and `warehouse ` to `catalog.properties.xxx`, it's OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] Populate table schemas when scanNewlyAddedTables is enabled in binlog phase [flink-cdc]
yuxiqian commented on PR #1464: URL: https://github.com/apache/flink-cdc/pull/1464#issuecomment-2076550869 Hi @qidian99, thanks for your contribution! Seems this issue has been fixed in latest `master` branch, so I'm going to close this PR. Feel free to reopen this if you have more concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [release] Update version to 3.2-SNAPSHOT [flink-cdc]
PatrickRen opened a new pull request, #3261: URL: https://github.com/apache/flink-cdc/pull/3261 This pull request bumps the version to 3.2-SNAPSHOT for master branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [e2e] optimization for arm64 [flink-cdc]
yuxiqian commented on PR #1560: URL: https://github.com/apache/flink-cdc/pull/1560#issuecomment-2076563412 Hi @deemogsw, thanks for your great contribution! Currently MySQL has upgraded docker image tag to `mysql:8.0`, which should run on Apple Silicon chips natively. As for static port mappings, my concern is 8081 port isn't always available, and there might be some problem if several e2e test cases run simultaneously. Currently TestContainers always maps them to a random free port without explicit `withPortBindings` command, and users could use `docker port ` or `Container.getMappedPort()` to access the mapped port info. I'm going to close this PR since these problems have been addressed in latest `master` branch. Feel free to reopen it if you have other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35167][cdc-connector] Introduce MaxCompute pipeline DataSink [flink-cdc]
kevinwangcs commented on code in PR #3254:
URL: https://github.com/apache/flink-cdc/pull/3254#discussion_r1579029793
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-maxcompute/src/main/java/org/apache/flink/cdc/connectors/maxcompute/sink/MaxComputeEventWriter.java:
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.maxcompute.sink;
+
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.OperationType;
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.connectors.maxcompute.common.Constant;
+import org.apache.flink.cdc.connectors.maxcompute.common.SessionIdentifier;
+import
org.apache.flink.cdc.connectors.maxcompute.coordinator.SessionManageOperator;
+import
org.apache.flink.cdc.connectors.maxcompute.coordinator.message.CommitSessionRequest;
+import
org.apache.flink.cdc.connectors.maxcompute.coordinator.message.CommitSessionResponse;
+import
org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeExecutionOptions;
+import org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeOptions;
+import
org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeWriteOptions;
+import org.apache.flink.cdc.connectors.maxcompute.utils.TypeConvertUtils;
+import
org.apache.flink.cdc.connectors.maxcompute.writer.MaxComputeUpsertWriter;
+import org.apache.flink.cdc.connectors.maxcompute.writer.MaxComputeWriter;
+
+import com.aliyun.odps.tunnel.impl.UpsertRecord;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+/** a {@link SinkWriter} for {@link Event} for MaxCompute. */
+public class MaxComputeEventWriter implements SinkWriter {
Review Comment:
Could you please consider changing "Event" to a generic type to support
various serializer implementations, similar to the StarRocksSink
[EventRecordSerializationSchema](https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-starrocks/src/main/java/org/apache/flink/cdc/connectors/starrocks/sink/EventRecordSerializationSchema.java)
and
[DorisEventSerializer](https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisEventSerializer.java)
implementations in Flink CDC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [mysql] Support the rate limit when Mysql parallel read data output [flink-cdc]
yuxiqian commented on PR #1573: URL: https://github.com/apache/flink-cdc/pull/1573#issuecomment-2076577228 Hi @fuyun2024, thanks for your great contribution! Also sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.debezium` package has been moved to `org.apache.flink.cdc.debezium`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] support table splitting process control when read table snapshot [flink-cdc]
yuxiqian commented on PR #1625: URL: https://github.com/apache/flink-cdc/pull/1625#issuecomment-2076585601 Hi @JustinLeesin , sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. cc @leonardBang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840693#comment-17840693 ] Rui Fan edited comment on FLINK-35215 at 4/25/24 8:00 AM: -- Here is the benchmark result for my test: ||Code||serializerKryo||serializerKryoWithoutRegistration||Comment|| |Case1: current master branch|167.16333 ops/ms|135.03 ops/ms|Lower about 10% than reverts FLINK-34954| |Case2: master branch reverts FLINK-34954 |190.636025667 ops/ms|147.184592333 ops/ms| | |Case3: reverts FLINK-34954 and with [https://github.com/apache/flink/pull/24717]|189.621157333 ops/ms|146.914767 ops/ms|Lower about 0.5% than reverts FLINK-34954 | Note: I choose 3 samples for each cases. h1. Case1: current master branch data from benchmark result WebUI. * serializerKryo (avg: 167.16333 ops/ms) ** [http://flink-speed.xyz/timeline/#/?exe=6&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] ** 164.68 ops/ms (Apr 22) ** 168.80 ops/ms (Apr 23) ** 168.01 ops/ms (Apr 24) * serializerKryoWithoutRegistration (avg: 135.03 ops/ms) ** [http://flink-speed.xyz/timeline/#/?exe=12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50] ** 134.93 ops/ms (Apr 22) ** 134.06 ops/ms (Apr 23) ** 136.10 ops/ms (Apr 24) h1. Case2: master branch reverts FLINK-34954 I trigger it manually, the code branch is : [https://github.com/1996fanrui/flink/tree/revert-kyro-fix] Note: the result of case2 and case3 isn't public, I paste the result from the Flink Community jenkins. * serializerKryo (avg: 190.636025667 ops/ms) ** 194.929708 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/81/artifact/jmh-result.csv/*view*/] ** 186.023176 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/83/artifact/jmh-result.csv/*view*/] ** 190.955193 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/85/artifact/jmh-result.csv/*view*/] * serializerKryoWithoutRegistration (avg: 147.184592333 ops/ms) ** 149.782267 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/81/artifact/jmh-result.csv/*view*/] ** 146.312711 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/83/artifact/jmh-result.csv/*view*/] ** 145.458799 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/85/artifact/jmh-result.csv/*view*/] h1. Case3: reverts FLINK-34954 and with [https://github.com/apache/flink/pull/24717] I trigger it manually, the code branch is : [https://github.com/1996fanrui/flink/tree/revert-kyro-fix-and-return-if-0] * serializerKryo (avg: 189.621157333 ops/ms) ** 182.683889 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/82/artifact/jmh-result.csv/*view*/] ** 191.713318 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/84/artifact/jmh-result.csv/*view*/] ** 194.466265 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/86/artifact/jmh-result.csv/*view*/] * serializerKryoWithoutRegistration (avg: 146.914767 ops/ms) ** 146.514048 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/82/artifact/jmh-result.csv/*view*/] ** 145.825161 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/84/artifact/jmh-result.csv/*view*/] ** 148.405092 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/86/artifact/jmh-result.csv/*view*/] Note: I try to use other solution to fix the bug that FLINK-34954 mentioned, but I'm not sure whether the logic is right/suitable. If the solution is fine, the benchmark result is better than FLINK-34954. Code change: [https://github.com/1996fanrui/flink/commit/ebce171a806a579225897a2d29e4cd28e9164c4f] The change is pretty easy. was (Author: fanrui): Here is the benchmark result for my test: ||Code||serializerKryo||serializerKryoWithoutRegistration||Comment|| |Case1: current master branch|167.16333 ops/ms|135.03 ops/ms|Lower about 10% than reverts FLINK-34954 | |Case2: master branch reverts FLINK-34954 |190.636025667 ops/ms|147.184592333 ops/ms| | |Case3: reverts FLINK-34954 and with https://github.com/apache/flink/pull/24717|189.621157333 ops/ms|146.914767 ops/ms|Lower about 0.5% than reverts FLINK-34954 | Note: I choose 3 samples for each cases. h1. Case1: current master branch data from benchmark result WebUI. * serializerKryo (avg: 167.16333 ops/ms) ** [http://flink-speed.xyz/timeline/#/?exe=6&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] ** 164.68 ops/ms (Apr 22) ** 168.80 ops/ms (Apr 23) ** 168.01 ops/ms (Apr 24) * serializerKryoWithoutRegistration (avg: 135.03 ops/ms) ** [http://flink-speed.xyz/timeline/#/?exe=12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=of
[jira] [Commented] (FLINK-35215) The performance of serializerKryo and serializerKryoWithoutRegistration are regressed
[ https://issues.apache.org/jira/browse/FLINK-35215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840693#comment-17840693 ] Rui Fan commented on FLINK-35215: - Here is the benchmark result for my test: ||Code||serializerKryo||serializerKryoWithoutRegistration||Comment|| |Case1: current master branch|167.16333 ops/ms|135.03 ops/ms|Lower about 10% than reverts FLINK-34954 | |Case2: master branch reverts FLINK-34954 |190.636025667 ops/ms|147.184592333 ops/ms| | |Case3: reverts FLINK-34954 and with https://github.com/apache/flink/pull/24717|189.621157333 ops/ms|146.914767 ops/ms|Lower about 0.5% than reverts FLINK-34954 | Note: I choose 3 samples for each cases. h1. Case1: current master branch data from benchmark result WebUI. * serializerKryo (avg: 167.16333 ops/ms) ** [http://flink-speed.xyz/timeline/#/?exe=6&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] ** 164.68 ops/ms (Apr 22) ** 168.80 ops/ms (Apr 23) ** 168.01 ops/ms (Apr 24) * serializerKryoWithoutRegistration (avg: 135.03 ops/ms) ** [http://flink-speed.xyz/timeline/#/?exe=12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50] ** 134.93 ops/ms (Apr 22) ** 134.06 ops/ms (Apr 23) ** 136.10 ops/ms (Apr 24) h1. Case2: master branch reverts FLINK-34954 I trigger it manually, the code branch is : [https://github.com/1996fanrui/flink/tree/revert-kyro-fix] * serializerKryo (avg: 190.636025667 ops/ms) ** 194.929708 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/81/artifact/jmh-result.csv/*view*/] ** 186.023176 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/83/artifact/jmh-result.csv/*view*/] ** 190.955193 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/85/artifact/jmh-result.csv/*view*/] * serializerKryoWithoutRegistration (avg: 147.184592333 ops/ms) ** 149.782267 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/81/artifact/jmh-result.csv/*view*/] ** 146.312711 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/83/artifact/jmh-result.csv/*view*/] ** 145.458799 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/85/artifact/jmh-result.csv/*view*/] h1. Case3: reverts FLINK-34954 and with https://github.com/apache/flink/pull/24717 I trigger it manually, the code branch is : [https://github.com/1996fanrui/flink/tree/revert-kyro-fix-and-return-if-0] * serializerKryo (avg: 189.621157333 ops/ms) ** 182.683889 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/82/artifact/jmh-result.csv/*view*/] ** 191.713318 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/84/artifact/jmh-result.csv/*view*/] ** 194.466265 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/86/artifact/jmh-result.csv/*view*/] * serializerKryoWithoutRegistration (avg: 146.914767 ops/ms) ** 146.514048 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/82/artifact/jmh-result.csv/*view*/] ** 145.825161 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/84/artifact/jmh-result.csv/*view*/] ** 148.405092 ops/ms link: [http://jenkins.flink-speed.xyz/job/flink-benchmark-request/86/artifact/jmh-result.csv/*view*/] Note: I try to use other solution to fix the bug that FLINK-34954 mentioned, but I'm not sure whether the logic is right/suitable. If the solution is fine, the benchmark result is better than FLINK-34954. Code change: [https://github.com/1996fanrui/flink/commit/ebce171a806a579225897a2d29e4cd28e9164c4f] The change is pretty easy. > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed > - > > Key: FLINK-35215 > URL: https://issues.apache.org/jira/browse/FLINK-35215 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Affects Versions: 1.20.0 >Reporter: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-25-14-57-55-231.png, > image-2024-04-25-15-00-32-410.png > > > The performance of serializerKryo and serializerKryoWithoutRegistration are > regressed[1][2], I checked recent commits, and found FLINK-34954 changed > related logic. > > [1] > [http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryo&extr=on&quarts=on&equid=off&env=3&revs=50] > [2] > http://flink-speed.xyz/timeline/#/?exe=1,6,12&ben=serializerKryoWithoutRegistration&extr=on&quarts=on&equid=off&env=3&revs=50 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [release] Update version to 3.2-SNAPSHOT [flink-cdc]
leonardBang commented on code in PR #3261: URL: https://github.com/apache/flink-cdc/pull/3261#discussion_r1579056868 ## pom.xml: ## @@ -62,7 +62,7 @@ limitations under the License. -3.0-SNAPSHOT +3.2-SNAPSHOT Review Comment: Uh? should we also fix this in release-3.1 branch? ## pom.xml: ## @@ -62,7 +62,7 @@ limitations under the License. -3.0-SNAPSHOT +3.2-SNAPSHOT Review Comment: Uh? should we also fix this in release-3.1 branch? ## pom.xml: ## @@ -62,7 +62,7 @@ limitations under the License. -3.0-SNAPSHOT +3.2-SNAPSHOT Review Comment: Uh? should we also fix this in release-3.1 branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35215][core] Fix the bug when Kryo serialize length is 0 [flink]
1996fanrui commented on code in PR #24717:
URL: https://github.com/apache/flink/pull/24717#discussion_r1579063045
##
flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/NoFetchingInput.java:
##
@@ -114,18 +121,26 @@ public void readBytes(byte[] bytes, int offset, int
count) throws KryoException
throw new IllegalArgumentException("bytes cannot be null.");
}
+if (count == 0) {
+return;
+}
+
try {
int bytesRead = 0;
int c;
-while (bytesRead < count) {
+while (true) {
c = inputStream.read(bytes, offset + bytesRead, count -
bytesRead);
if (c == -1) {
throw new KryoException(new EOFException("No more bytes
left."));
}
bytesRead += c;
+
+if (bytesRead == count) {
Review Comment:
> Not to block this, just curious do you know why
[FLINK-34954](https://issues.apache.org/jira/browse/FLINK-34954) affects
performance?
Hi @reswqa , I'm not very sure the reason. But I guess:
- For non-empty data, let us assume `inputStream.read` will be called once.
- Before FLINK-34954, we only check `bytesRead == count` once.
- After FLINK-34954, we call `bytesRead < count` twice.
- Also, I'm not sure whether FLINK-34954 breaks any JIT optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-35235) Fix missing dependencies in the uber jar
[ https://issues.apache.org/jira/browse/FLINK-35235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35235: --- Labels: pull-request-available (was: ) > Fix missing dependencies in the uber jar > > > Key: FLINK-35235 > URL: https://issues.apache.org/jira/browse/FLINK-35235 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Affects Versions: 3.1.0 >Reporter: LvYanquan >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: image-2024-04-25-15-17-20-987.png, > image-2024-04-25-15-17-34-717.png > > > Some class of Kafka were not included in fat jar. > !image-2024-04-25-15-17-34-717.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [release] Update version to 3.2-SNAPSHOT [flink-cdc]
PatrickRen commented on code in PR #3261: URL: https://github.com/apache/flink-cdc/pull/3261#discussion_r1579068581 ## pom.xml: ## @@ -62,7 +62,7 @@ limitations under the License. -3.0-SNAPSHOT +3.2-SNAPSHOT Review Comment: I did and pushed directly on release-3.1 just now. We never noticed this before... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][docs] Fix the invalid link in 1.18 release note [flink]
reswqa merged PR #24713: URL: https://github.com/apache/flink/pull/24713 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][docs] Fix the invalid link in 1.18 release note [flink]
reswqa commented on PR #24713: URL: https://github.com/apache/flink/pull/24713#issuecomment-2076650925 Congratulations on your first PR merge in the Apache Flink community. 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35215][core] Fix the bug when Kryo serialize length is 0 [flink]
reswqa commented on code in PR #24717:
URL: https://github.com/apache/flink/pull/24717#discussion_r1579092286
##
flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/NoFetchingInput.java:
##
@@ -114,18 +121,26 @@ public void readBytes(byte[] bytes, int offset, int
count) throws KryoException
throw new IllegalArgumentException("bytes cannot be null.");
}
+if (count == 0) {
+return;
+}
+
try {
int bytesRead = 0;
int c;
-while (bytesRead < count) {
+while (true) {
c = inputStream.read(bytes, offset + bytesRead, count -
bytesRead);
if (c == -1) {
throw new KryoException(new EOFException("No more bytes
left."));
}
bytesRead += c;
+
+if (bytesRead == count) {
Review Comment:
Thanks! Yes, I don't know the details also, but it's possible that JIT
optimizations were affected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] 升级Jackson版本 [flink-cdc]
yuxiqian commented on PR #1660: URL: https://github.com/apache/flink-cdc/pull/1660#issuecomment-2076662066 Thanks for @Desperado2's contribution! Debezium version has been bumped to `1.9.8.Final` and Jackson dependencies have been correctly shaded to avoid conflicts. Closing this PR but feel free to reopen it if you have any other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35215][core] Fix the bug when Kryo serialize length is 0 [flink]
1996fanrui commented on code in PR #24717:
URL: https://github.com/apache/flink/pull/24717#discussion_r1579104254
##
flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/NoFetchingInput.java:
##
@@ -65,6 +65,10 @@ public boolean canReadLong() throws KryoException {
*/
@Override
protected int require(int required) throws KryoException {
+if (required == 0) {
+return 0;
+}
Review Comment:
Hi @qinghui-xu , I'm not sure could this change fix the bug that FLINK-34954
mentioned. Would you mind helping double check? Thank you in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [mysql][oracle] Quote primary key when split snapshot [flink-cdc]
yuxiqian commented on PR #1709: URL: https://github.com/apache/flink-cdc/pull/1709#issuecomment-2076672226 Thanks @huhuan1898 for the great work! Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors` package has been moved to `org.apache.flink.cdc.connectors`. Also, noticed that PostgreSQL and SQL Server connectors have similar expressions. Should they be quoted, too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35134][Connectors/Elasticsearch] Drop support for Flink 1.17 [flink-connector-elasticsearch]
boring-cyborg[bot] commented on PR #99: URL: https://github.com/apache/flink-connector-elasticsearch/pull/99#issuecomment-2076677044 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35134][Connectors/Elasticsearch] Drop support for Flink 1.17 [flink-connector-elasticsearch]
reswqa merged PR #99: URL: https://github.com/apache/flink-connector-elasticsearch/pull/99 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Switch to a MySQL image with arm64 support [flink-cdc]
yuxiqian commented on PR #1759: URL: https://github.com/apache/flink-cdc/pull/1759#issuecomment-2076679267 Hi @sap1ens, Thanks for your contribution! Now MySQL testcases have all been upgraded to `mysql:8.0` docker image which provides native arm64 images already. Closing this PR, feel free to reopen it if you have any other concerns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35134) Release flink-connector-elasticsearch vX.X.X for Flink 1.19
[ https://issues.apache.org/jira/browse/FLINK-35134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840708#comment-17840708 ] Weijie Guo commented on FLINK-35134: Drop support for Flink 1.17 via: 44a0a144e2e135641ed3fe1144ad5ddf4cf5c5ff. > Release flink-connector-elasticsearch vX.X.X for Flink 1.19 > --- > > Key: FLINK-35134 > URL: https://issues.apache.org/jira/browse/FLINK-35134 > Project: Flink > Issue Type: Sub-task > Components: Connectors / ElasticSearch >Reporter: Danny Cranmer >Assignee: Hong Liang Teoh >Priority: Major > Labels: pull-request-available > > https://github.com/apache/flink-connector-elasticsearch -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [tidb-cdc]When the task runs for a period of time, there are only Resolved Events [flink-cdc]
yuxiqian commented on PR #1763: URL: https://github.com/apache/flink-cdc/pull/1763#issuecomment-2076681672 Hi @wangxiaojing , sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.tidb` package has been moved to `org.apache.flink.cdc.connectors.tidb`. Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [#1770] fix NumberFormatException when mysql server_id is larger than INT_MAX [flink-cdc]
yuxiqian commented on PR #1771: URL: https://github.com/apache/flink-cdc/pull/1771#issuecomment-2076685111 Hi @legendtkl, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.mysql` package has been moved to `org.apache.flink.cdc.connectors.mysql`. Thanks for your contribution! cc @ruanhang1993 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35158][runtime] Error handling in StateFuture's callback [flink]
jectpro7 commented on code in PR #24698:
URL: https://github.com/apache/flink/pull/24698#discussion_r1579119199
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/asyncprocessing/AbstractAsyncStateStreamOperator.java:
##
@@ -58,17 +60,35 @@ public abstract class AbstractAsyncStateStreamOperator
extends AbstractStre
private RecordContext currentProcessingContext;
+private Environment environment;
+
/** Initialize necessary state components for {@link
AbstractStreamOperator}. */
@Override
public void setup(
StreamTask containingTask,
StreamConfig config,
Output> output) {
super.setup(containingTask, config, output);
-// TODO: properly read config and setup
-final MailboxExecutor mailboxExecutor =
-containingTask.getEnvironment().getMainMailboxExecutor();
-this.asyncExecutionController = new
AsyncExecutionController(mailboxExecutor, null);
+final Environment environment = containingTask.getEnvironment();
+final MailboxExecutor mailboxExecutor =
environment.getMainMailboxExecutor();
+final int inFlightRecordsLimit =
+
environment.getExecutionConfig().getAsyncInflightRecordsLimit();
+final int asyncBufferSize =
environment.getExecutionConfig().getAsyncStateBufferSize();
+final long asyncBufferTimeout =
+environment.getExecutionConfig().getAsyncStateBufferTimeout();
+// TODO: initial state executor and set state executor for aec
+this.asyncExecutionController =
+new AsyncExecutionController(
+mailboxExecutor,
+this::handleStateCallbackException,
+null,
+asyncBufferSize,
+asyncBufferTimeout,
+inFlightRecordsLimit);
+}
+
+private void handleStateCallbackException(String message, Throwable
exception) {
Review Comment:
Sorry my bad, I thought it is AsyncOperator for user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35236) Flink 1.19 Translation error on the Chinese official website
hongxu han created FLINK-35236: -- Summary: Flink 1.19 Translation error on the Chinese official website Key: FLINK-35236 URL: https://issues.apache.org/jira/browse/FLINK-35236 Project: Flink Issue Type: Bug Components: chinese-translation Affects Versions: 1.19.0 Reporter: hongxu han [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35236) Flink 1.19 Translation error on the Chinese official website
[ https://issues.apache.org/jira/browse/FLINK-35236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongxu han updated FLINK-35236: --- Attachment: image-2024-04-25-16-53-49-052.png image-2024-04-25-16-53-34-007.png Description: [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] !image-2024-04-25-16-53-34-007.png! !image-2024-04-25-16-53-49-052.png! 应为，常规输入：既不从广播输入也不从 keyed 输入 was:[https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > Flink 1.19 Translation error on the Chinese official website > > > Key: FLINK-35236 > URL: https://issues.apache.org/jira/browse/FLINK-35236 > Project: Flink > Issue Type: Bug > Components: chinese-translation >Affects Versions: 1.19.0 >Reporter: hongxu han >Priority: Major > Attachments: image-2024-04-25-16-53-34-007.png, > image-2024-04-25-16-53-49-052.png > > > [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > !image-2024-04-25-16-53-34-007.png! > !image-2024-04-25-16-53-49-052.png! > 应为，常规输入：既不从广播输入也不从 keyed 输入 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [oracle] The STRUCT type processing exception [flink-cdc]
yuxiqian commented on PR #1790: URL: https://github.com/apache/flink-cdc/pull/1790#issuecomment-2076698023 Hi @wangxiaojing, sorry for the delay of this PR. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors.oracle` package has been moved to `org.apache.flink.cdc.connectors.oracle`. Thanks for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33506) Make AWS connectors compilable with jdk17
[
https://issues.apache.org/jira/browse/FLINK-33506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840712#comment-17840712
]
Danny Cranmer commented on FLINK-33506:
---
Merged commit
[{{c6e0abb}}|https://github.com/apache/flink-connector-aws/commit/c6e0abb65a0e51b40dd218b890a111886fbf797f]
into apache:main
> Make AWS connectors compilable with jdk17
> -
>
> Key: FLINK-33506
> URL: https://issues.apache.org/jira/browse/FLINK-33506
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / AWS
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Major
> Labels: pull-request-available
>
> Since 1.18 Flink with jdk 17 support is released it would make sense to add
> such support for connectors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33506) Make AWS connectors compilable with jdk17
[ https://issues.apache.org/jira/browse/FLINK-33506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer updated FLINK-33506: -- Fix Version/s: aws-connector-4.4.0 > Make AWS connectors compilable with jdk17 > - > > Key: FLINK-33506 > URL: https://issues.apache.org/jira/browse/FLINK-33506 > Project: Flink > Issue Type: Improvement > Components: Connectors / AWS >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.4.0 > > > Since 1.18 Flink with jdk 17 support is released it would make sense to add > such support for connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [oracle]Error when TIMESTAMP_LTZ type is encoded in string type [flink-cdc]
yuxiqian commented on PR #1800: URL: https://github.com/apache/flink-cdc/pull/1800#issuecomment-2076700872 Thanks @wangxiaojing for the great work, could you please rebase this PR with latest `master` branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-33506) Make AWS connectors compilable with jdk17
[ https://issues.apache.org/jira/browse/FLINK-33506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer resolved FLINK-33506. --- Resolution: Done > Make AWS connectors compilable with jdk17 > - > > Key: FLINK-33506 > URL: https://issues.apache.org/jira/browse/FLINK-33506 > Project: Flink > Issue Type: Improvement > Components: Connectors / AWS >Reporter: Sergey Nuyanzin >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available > Fix For: aws-connector-4.4.0 > > > Since 1.18 Flink with jdk 17 support is released it would make sense to add > such support for connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35236) Flink 1.19 Translation error on the Chinese official website
[ https://issues.apache.org/jira/browse/FLINK-35236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongxu han updated FLINK-35236: --- Priority: Minor (was: Major) > Flink 1.19 Translation error on the Chinese official website > > > Key: FLINK-35236 > URL: https://issues.apache.org/jira/browse/FLINK-35236 > Project: Flink > Issue Type: Bug > Components: chinese-translation >Affects Versions: 1.19.0 >Reporter: hongxu han >Priority: Minor > Attachments: image-2024-04-25-16-53-34-007.png, > image-2024-04-25-16-53-49-052.png > > > [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > !image-2024-04-25-16-53-34-007.png! > !image-2024-04-25-16-53-49-052.png! > 应为，常规输入：既不从广播输入也不从 keyed 输入 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35237) Allow Custom HashFunction in PrePartitionOperator for Flink Sink Customization
zhangdingxin created FLINK-35237:
Summary: Allow Custom HashFunction in PrePartitionOperator for
Flink Sink Customization
Key: FLINK-35237
URL: https://issues.apache.org/jira/browse/FLINK-35237
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: zhangdingxin
The {{PrePartitionOperator}} in its current implementation only supports a
fixed {{HashFunction}}
({{{}org.apache.flink.cdc.runtime.partitioning.PrePartitionOperator.HashFunction{}}}).
This limits the ability of Sink implementations to customize the partitioning
logic for {{{}DataChangeEvent{}}}s. For example, in the case of partitioned
tables, it would be advantageous to allow hashing based on partition keys,
hashing according to table names, or using the database engine's internal
primary key hash functions (such as with MaxCompute DataSink).
When users require such custom partitioning logic, they are compelled to
implement their PartitionOperator, which undermines the utility of
{{{}PrePartitionOperator{}}}.
To address this limitation, it would be highly desirable to enable the
{{PrePartitionOperator}} to support user-specified custom {{{}HashFunction{}}}s
(Function). A possible solution could involve a
mechanism analogous to the {{DataSink}} interface, allowing the specification
of a {{HashFunctionFactory}} class path in the configuration file. This
enhancement would greatly facilitate users in tailoring partition strategies to
meet their specific application needs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35236) Flink 1.19 Translation error on the execution_mode/order-of-processing
[ https://issues.apache.org/jira/browse/FLINK-35236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongxu han updated FLINK-35236: --- Summary: Flink 1.19 Translation error on the execution_mode/order-of-processing (was: Flink 1.19 Translation error on the Chinese official website) > Flink 1.19 Translation error on the execution_mode/order-of-processing > -- > > Key: FLINK-35236 > URL: https://issues.apache.org/jira/browse/FLINK-35236 > Project: Flink > Issue Type: Bug > Components: chinese-translation >Affects Versions: 1.19.0 >Reporter: hongxu han >Priority: Minor > Attachments: image-2024-04-25-16-53-34-007.png, > image-2024-04-25-16-53-49-052.png > > > [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > !image-2024-04-25-16-53-34-007.png! > !image-2024-04-25-16-53-49-052.png! > 应为，常规输入：既不从广播输入也不从 keyed 输入 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35026][runtime][config] Introduce async execution configurations [flink]
Zakelly commented on code in PR #24667:
URL: https://github.com/apache/flink/pull/24667#discussion_r1579142292
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/AsyncExecutionController.java:
##
@@ -94,26 +97,39 @@ public class AsyncExecutionController {
*/
final AtomicInteger inFlightRecordNum;
-public AsyncExecutionController(MailboxExecutor mailboxExecutor,
StateExecutor stateExecutor) {
-this(mailboxExecutor, stateExecutor, DEFAULT_BATCH_SIZE,
DEFAULT_MAX_IN_FLIGHT_RECORD_NUM);
-}
-
public AsyncExecutionController(
MailboxExecutor mailboxExecutor,
StateExecutor stateExecutor,
int batchSize,
+long bufferTimeout,
int maxInFlightRecords) {
this.keyAccountingUnit = new KeyAccountingUnit<>(maxInFlightRecords);
this.mailboxExecutor = mailboxExecutor;
this.stateFutureFactory = new StateFutureFactory<>(this,
mailboxExecutor);
this.stateExecutor = stateExecutor;
this.batchSize = batchSize;
+this.bufferTimeout = bufferTimeout;
this.maxInFlightRecordNum = maxInFlightRecords;
-this.stateRequestsBuffer = new StateRequestBuffer<>();
+
this.inFlightRecordNum = new AtomicInteger(0);
+this.stateRequestsBuffer =
+new StateRequestBuffer<>(
+bufferTimeout,
+() ->
+mailboxExecutor.execute(
+() -> {
+if
(stateRequestsBuffer.currentTriggerSeq.get()
+==
stateRequestsBuffer.scheduledTriggerSeq
Review Comment:
This condition is always true. I suggest you pass the `triggered seq` into
the handler lambda as parameter.
##
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/StateRequestBuffer.java:
##
@@ -53,17 +72,75 @@ public class StateRequestBuffer {
/** The number of state requests in blocking queue. */
int blockingQueueSize;
-public StateRequestBuffer() {
+/** The timeout of {@link #activeQueue} triggering in milliseconds. */
+final long bufferTimeout;
+
+/** The handler to trigger when {@link #activeQueue} size is 1. */
+final Runnable timeoutHandler;
+
+/** The executor service that schedules and calls the triggers of this
task. */
+ScheduledExecutorService scheduledExecutor;
+
+/**
+ * The current scheduled future, when the next scheduling occurs, the
previous one that has not
+ * yet been executed will be canceled.
+ */
+ScheduledFuture currentScheduledFuture;
+
+/**
+ * The current scheduled trigger sequence number, a timeout trigger is
scheduled only if {@code
+ * scheduledTriggerSeq} is less than {@code currentTriggerSeq}.
+ */
+AtomicLong scheduledTriggerSeq;
+
+/**
+ * The current trigger sequence number, used to distinguish different
triggers. Every time a
+ * trigger occurs, {@code currentTriggerSeq} increases by one.
+ */
+AtomicLong currentTriggerSeq;
+
+public StateRequestBuffer(long bufferTimeout, Runnable timeoutHandler) {
this.activeQueue = new LinkedList<>();
this.blockingQueue = new HashMap<>();
this.blockingQueueSize = 0;
+this.bufferTimeout = bufferTimeout;
+this.timeoutHandler = timeoutHandler;
+this.scheduledTriggerSeq = new AtomicLong(-1);
+this.currentTriggerSeq = new AtomicLong(0);
+if (bufferTimeout > 0) {
+this.scheduledExecutor = DELAYER;
+}
+}
+
+void advanceTriggerSeq() {
+currentTriggerSeq.incrementAndGet();
}
void enqueueToActive(StateRequest request) {
if (request.getRequestType() == StateRequestType.SYNC_POINT) {
request.getFuture().complete(null);
} else {
activeQueue.add(request);
+// if the active queue size is 1, it means that the current
request is the oldest one in
Review Comment:
nit. remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
reswqa commented on PR #22026: URL: https://github.com/apache/flink/pull/22026#issuecomment-2076751917 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] make sure run RecordContext#release() in Task thread. [flink]
fredia commented on PR #24705: URL: https://github.com/apache/flink/pull/24705#issuecomment-2076765458 Thanks for the PR, LGTM. How about updating the commit msg to "[hotfix][runtime] Make sure run RecordContext#release() in Task thread." ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-28568) Implements a new lookup join operator (sync mode only) with state to eliminate the non determinism
[
https://issues.apache.org/jira/browse/FLINK-28568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-28568:
---
Description:
{code:java}
##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
machine is running and has a healthy network connection. Anything that
terminates an agent process, starves it for CPU, or blocks its network access
can cause this error. For more information, see:
https://go.microsoft.com/fwlink/?linkid=846610
Agent: AlibabaCI006-agent03
Started: Today at 10:30
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
> Implements a new lookup join operator (sync mode only) with state to
> eliminate the non determinism
> --
>
> Key: FLINK-28568
> URL: https://issues.apache.org/jira/browse/FLINK-28568
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Runtime
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Agent: AlibabaCI006-agent03
> Started: Today at 10:30
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28568) Implements a new lookup join operator (sync mode only) with state to eliminate the non determinism
[
https://issues.apache.org/jira/browse/FLINK-28568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-28568:
---
Description: (was: {code:java}
##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
machine is running and has a healthy network connection. Anything that
terminates an agent process, starves it for CPU, or blocks its network access
can cause this error. For more information, see:
https://go.microsoft.com/fwlink/?linkid=846610
Agent: AlibabaCI006-agent03
Started: Today at 10:30
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483)
> Implements a new lookup join operator (sync mode only) with state to
> eliminate the non determinism
> --
>
> Key: FLINK-28568
> URL: https://issues.apache.org/jira/browse/FLINK-28568
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Runtime
>Reporter: lincoln lee
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35238) AZP is not working since some ci agent unhealthy
Weijie Guo created FLINK-35238: -- Summary: AZP is not working since some ci agent unhealthy Key: FLINK-35238 URL: https://issues.apache.org/jira/browse/FLINK-35238 Project: Flink Issue Type: Bug Components: Build System / CI Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35238) AZP is not working since some ci agent unhealthy
[
https://issues.apache.org/jira/browse/FLINK-35238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840726#comment-17840726
]
Weijie Guo commented on FLINK-35238:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59134&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=0da23115-68bb-5dcd-192c-bd4c8adebde1
> AZP is not working since some ci agent unhealthy
>
>
> Key: FLINK-35238
> URL: https://issues.apache.org/jira/browse/FLINK-35238
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Priority: Blocker
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Agent: AlibabaCI006-agent03
> Started: Today at 10:30
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35238) AZP is not working since some ci agent unhealthy
[
https://issues.apache.org/jira/browse/FLINK-35238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840727#comment-17840727
]
Weijie Guo commented on FLINK-35238:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
> AZP is not working since some ci agent unhealthy
>
>
> Key: FLINK-35238
> URL: https://issues.apache.org/jira/browse/FLINK-35238
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Priority: Blocker
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Agent: AlibabaCI006-agent03
> Started: Today at 10:30
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35238) AZP is not working since some ci agent unhealthy
[
https://issues.apache.org/jira/browse/FLINK-35238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-35238:
---
Description:
{code:java}
##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
machine is running and has a healthy network connection. Anything that
terminates an agent process, starves it for CPU, or blocks its network access
can cause this error. For more information, see:
https://go.microsoft.com/fwlink/?linkid=846610
Agent: AlibabaCI006-agent03
Started: Today at 10:30
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
> AZP is not working since some ci agent unhealthy
>
>
> Key: FLINK-35238
> URL: https://issues.apache.org/jira/browse/FLINK-35238
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Priority: Major
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Agent: AlibabaCI006-agent03
> Started: Today at 10:30
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-35238) AZP is not working since some ci agent unhealthy
[
https://issues.apache.org/jira/browse/FLINK-35238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weijie Guo updated FLINK-35238:
---
Priority: Blocker (was: Major)
> AZP is not working since some ci agent unhealthy
>
>
> Key: FLINK-35238
> URL: https://issues.apache.org/jira/browse/FLINK-35238
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: Weijie Guo
>Priority: Blocker
>
> {code:java}
> ##[error]We stopped hearing from agent AlibabaCI006-agent03. Verify the agent
> machine is running and has a healthy network connection. Anything that
> terminates an agent process, starves it for CPU, or blocks its network access
> can cause this error. For more information, see:
> https://go.microsoft.com/fwlink/?linkid=846610
> Agent: AlibabaCI006-agent03
> Started: Today at 10:30
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59135&view=logs&s=ae4f8708-9994-57d3-c2d7-b892156e7812&j=fc5181b0-e452-5c8f-68de-1097947f6483
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [DRAFT] Declare async state processing [flink]
Zakelly opened a new pull request, #24719: URL: https://github.com/apache/flink/pull/24719 ## this is only a draft, do not merge this ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follow [the conventions for tests defined in our code quality guide](https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#7-testing). *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-35236) Flink 1.19 Translation error on the execution_mode/order-of-processing
[ https://issues.apache.org/jira/browse/FLINK-35236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840731#comment-17840731 ] Weijie Guo commented on FLINK-35236: Wound you mind filing a PR to fix this? > Flink 1.19 Translation error on the execution_mode/order-of-processing > -- > > Key: FLINK-35236 > URL: https://issues.apache.org/jira/browse/FLINK-35236 > Project: Flink > Issue Type: Bug > Components: chinese-translation >Affects Versions: 1.19.0 >Reporter: hongxu han >Priority: Minor > Attachments: image-2024-04-25-16-53-34-007.png, > image-2024-04-25-16-53-49-052.png > > > [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > !image-2024-04-25-16-53-34-007.png! > !image-2024-04-25-16-53-49-052.png! > 应为，常规输入：既不从广播输入也不从 keyed 输入 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-35236) Flink 1.19 Translation error on the execution_mode/order-of-processing
[ https://issues.apache.org/jira/browse/FLINK-35236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840731#comment-17840731 ] Weijie Guo edited comment on FLINK-35236 at 4/25/24 9:40 AM: - Would you mind filing a PR to fix this? was (Author: weijie guo): Wound you mind filing a PR to fix this? > Flink 1.19 Translation error on the execution_mode/order-of-processing > -- > > Key: FLINK-35236 > URL: https://issues.apache.org/jira/browse/FLINK-35236 > Project: Flink > Issue Type: Bug > Components: chinese-translation >Affects Versions: 1.19.0 >Reporter: hongxu han >Priority: Minor > Attachments: image-2024-04-25-16-53-34-007.png, > image-2024-04-25-16-53-49-052.png > > > [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > !image-2024-04-25-16-53-34-007.png! > !image-2024-04-25-16-53-49-052.png! > 应为，常规输入：既不从广播输入也不从 keyed 输入 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [DRAFT] Declare async state processing [flink]
flinkbot commented on PR #24719: URL: https://github.com/apache/flink/pull/24719#issuecomment-2076790077 ## CI report: * 19d9d78565740b99a2cc3f6b1d8301b63da81d5e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35047][state] Support ForStStateBackend and ForStKeyedStateBackend [flink]
masteryhx commented on PR #24682: URL: https://github.com/apache/flink/pull/24682#issuecomment-2076794847 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [release] Update version to 3.2-SNAPSHOT [flink-cdc]
leonardBang merged PR #3261: URL: https://github.com/apache/flink-cdc/pull/3261 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35047][state] Support ForStStateBackend and ForStKeyedStateBackend [flink]
masteryhx commented on PR #24682: URL: https://github.com/apache/flink/pull/24682#issuecomment-2076795175 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] make sure run RecordContext#release() in Task thread. [flink]
fredia merged PR #24705: URL: https://github.com/apache/flink/pull/24705 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [docs]Problem with Case Document Format in Quickstart [flink-cdc]
leonardBang merged PR #3253: URL: https://github.com/apache/flink-cdc/pull/3253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35222][rest] Adding getJobType for AccessExecutionGraph [flink]
1996fanrui commented on PR #24709: URL: https://github.com/apache/flink/pull/24709#issuecomment-2076832608 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
davidradl commented on PR #22026: URL: https://github.com/apache/flink/pull/22026#issuecomment-2076854830 > > > That would be fabulous - thank so much for moving this forward. > > > > > > @davidradl I saw your comment in jira ticket. > > If I remember correctly, I should have tested this. But our code base has gone through too many version iterations, so I'm not sure if it still works correctly. It would be great if you could test it locally. > > Absolutely - could you resolve the conflicts and I will test. @reswqa I have built the fix locally on the latest main and tested with a self signed certificate with `security.ssl.enabled` and separately with `security.ssl.rest.enabled` setting the keystores, truststores and passwords. The SQL client starts with no errors. So as far as I can see, the fix work for us. Can we go to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] expose Enumerator metrics via Source Event [flink-cdc]
zhjwpku commented on PR #853: URL: https://github.com/apache/flink-cdc/pull/853#issuecomment-2076856173 > Hi @zhjwpku, sorry for the long delay of this PR. Could you please help rebase it with latest master branch before we can merge this? A complete refactor had been made since your original commit, from `com.alibaba.ververica.cdc.connectors.mysql.source` to `org.apache.flink.cdc.connectors.mysql.source`. > > Thanks again for your generous contribution! Hi @yuxiqian , I have already forgot what this PR is about, feel free to take it if you think this is a valuable feature ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] expose Enumerator metrics via Source Event [flink-cdc]
zhjwpku closed pull request #853: [mysql] expose Enumerator metrics via Source Event URL: https://github.com/apache/flink-cdc/pull/853 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [mysql] expose Enumerator metrics via Source Event [flink-cdc]
zhjwpku commented on PR #853: URL: https://github.com/apache/flink-cdc/pull/853#issuecomment-2076859269 I will close this PR for now, if anyone think this is a valuable feature, feel free to take it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-35236] [chinese-translation] Modify Translation error on the execution_mode/order-of-processing [flink]
maomao931228 opened a new pull request, #24720: URL: https://github.com/apache/flink/pull/24720 I'm glad to be here for the first time to open a pull request 'Modify Translation error on the execution_mode/order-of-processing'. If there are any mistakes made, please feel free to point them out. https://github.com/apache/flink/assets/152967266/8125a47d-0576-4c9f-8a0f-d8fe84e4eef5";> https://github.com/apache/flink/assets/152967266/8ea0e0ef-4181-49bc-90a7-bdb53405a61f";> result:'既不从广播输入也不从 keyed 输入' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-35236) Flink 1.19 Translation error on the execution_mode/order-of-processing
[ https://issues.apache.org/jira/browse/FLINK-35236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35236: --- Labels: pull-request-available (was: ) > Flink 1.19 Translation error on the execution_mode/order-of-processing > -- > > Key: FLINK-35236 > URL: https://issues.apache.org/jira/browse/FLINK-35236 > Project: Flink > Issue Type: Bug > Components: chinese-translation >Affects Versions: 1.19.0 >Reporter: hongxu han >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-04-25-16-53-34-007.png, > image-2024-04-25-16-53-49-052.png > > > [https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/#order-of-processing] > !image-2024-04-25-16-53-34-007.png! > !image-2024-04-25-16-53-49-052.png! > 应为，常规输入：既不从广播输入也不从 keyed 输入 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-35236] [chinese-translation] Modify Translation error on the execution_mode/order-of-processing [flink]
maomao931228 commented on PR #24720: URL: https://github.com/apache/flink/pull/24720#issuecomment-2076883599 I overlooked the completeness of the submission information during the first submission, resulting in multiple submission records for one change. If necessary, I am willing to correct this error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35236] [chinese-translation] Modify Translation error on the execution_mode/order-of-processing [flink]
flinkbot commented on PR #24720: URL: https://github.com/apache/flink/pull/24720#issuecomment-2076882801 ## CI report: * 3cf05303d57b1f8713d27f2d1cc1919868d0ca31 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [tdsql] The first draft of TDSQL-CDC [flink-cdc]
guixin closed pull request #1332: [tdsql] The first draft of TDSQL-CDC URL: https://github.com/apache/flink-cdc/pull/1332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [tdsql] The first draft of TDSQL-CDC [flink-cdc]
guixin commented on PR #1332: URL: https://github.com/apache/flink-cdc/pull/1332#issuecomment-2076886993 > Hi @guixin, Thanks for your great work! Also sorry for the delay. Could you please rebase it to latest `master` branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that `com.ververica.cdc.connectors` package has been moved to `org.apache.flink.cdc.connectors`, and your new connector should be placed under `flink-cdc-connect/flink-cdc-source-connectors` path. Thanks for the reminder. I closed this pull request firstly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34902) INSERT INTO column mismatch leads to IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-34902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840752#comment-17840752
]
Jeyhun Karimov commented on FLINK-34902:
Hi [~twalthr] sure, I will check the issue
> INSERT INTO column mismatch leads to IndexOutOfBoundsException
> --
>
> Key: FLINK-34902
> URL: https://issues.apache.org/jira/browse/FLINK-34902
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Jeyhun Karimov
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> SQL:
> {code}
> INSERT INTO t (a, b) SELECT 1;
> {code}
>
> Stack trace:
> {code}
> org.apache.flink.table.api.ValidationException: SQL validation failed. Index
> 1 out of bounds for length 1
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:200)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:117)
> at
> Caused by: java.lang.IndexOutOfBoundsException: Index 1 out of bounds for
> length 1
> at
> java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
> at
> java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
> at
> java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
> at java.base/java.util.Objects.checkIndex(Objects.java:374)
> at java.base/java.util.ArrayList.get(ArrayList.java:459)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1(PreValidateReWriter.scala:355)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$reorder$1$adapted(PreValidateReWriter.scala:355)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
davidradl commented on code in PR #22026:
URL: https://github.com/apache/flink/pull/22026#discussion_r1579266186
##
flink-table/flink-sql-gateway-api/src/main/java/org/apache/flink/table/gateway/api/endpoint/SqlGatewayEndpointFactory.java:
##
@@ -50,6 +50,14 @@ interface Context {
/** Gives read-only access to the configuration of the current
session. */
ReadableConfig getFlinkConfiguration();
+/**
+ * Get a map contains all flink configurations.
+ *
+ * @return The copy of flink configurations in the form of map, modify
this map will not
Review Comment:
knit: modify => modifying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35167][cdc-connector] Introduce MaxCompute pipeline DataSink [flink-cdc]
dingxin-tech commented on code in PR #3254:
URL: https://github.com/apache/flink-cdc/pull/3254#discussion_r1579293272
##
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-maxcompute/src/main/java/org/apache/flink/cdc/connectors/maxcompute/sink/MaxComputeEventWriter.java:
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.maxcompute.sink;
+
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.cdc.common.event.CreateTableEvent;
+import org.apache.flink.cdc.common.event.DataChangeEvent;
+import org.apache.flink.cdc.common.event.Event;
+import org.apache.flink.cdc.common.event.OperationType;
+import org.apache.flink.cdc.common.event.SchemaChangeEvent;
+import org.apache.flink.cdc.common.event.TableId;
+import org.apache.flink.cdc.common.schema.Schema;
+import org.apache.flink.cdc.common.utils.SchemaUtils;
+import org.apache.flink.cdc.connectors.maxcompute.common.Constant;
+import org.apache.flink.cdc.connectors.maxcompute.common.SessionIdentifier;
+import
org.apache.flink.cdc.connectors.maxcompute.coordinator.SessionManageOperator;
+import
org.apache.flink.cdc.connectors.maxcompute.coordinator.message.CommitSessionRequest;
+import
org.apache.flink.cdc.connectors.maxcompute.coordinator.message.CommitSessionResponse;
+import
org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeExecutionOptions;
+import org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeOptions;
+import
org.apache.flink.cdc.connectors.maxcompute.options.MaxComputeWriteOptions;
+import org.apache.flink.cdc.connectors.maxcompute.utils.TypeConvertUtils;
+import
org.apache.flink.cdc.connectors.maxcompute.writer.MaxComputeUpsertWriter;
+import org.apache.flink.cdc.connectors.maxcompute.writer.MaxComputeWriter;
+
+import com.aliyun.odps.tunnel.impl.UpsertRecord;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+/** a {@link SinkWriter} for {@link Event} for MaxCompute. */
+public class MaxComputeEventWriter implements SinkWriter {
Review Comment:
I am very willing to provide a sink that supports generic types. However,
the current code framework would require significant changes to implement this
feature. This functionality may appear in the future in
flink-connector-maxcompute rather than here in flink-cdc-connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35196] [Connector / Pulsar] Fix bouncycastle class not found [flink-connector-pulsar]
wenbingshen commented on PR #91: URL: https://github.com/apache/flink-connector-pulsar/pull/91#issuecomment-2076975181 > The pulsar-client-all should be upgrade to the 3.0.4 in the mean time. @syhily sure. I have addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35176][Connector/JDBC] Support property authentication connection for JDBC catalog & dynamic table [flink-connector-jdbc]
RocMarshal commented on PR #116: URL: https://github.com/apache/flink-connector-jdbc/pull/116#issuecomment-2077035421 HI, @caicancai @GOODBOY008 Could you help have a review if you had the free time ? thx a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33463][Connector/JDBC] Support the implementation of dynamic source tables based on the new source [flink-connector-jdbc]
RocMarshal commented on PR #117: URL: https://github.com/apache/flink-connector-jdbc/pull/117#issuecomment-2077036231 HI, @eskabetxe @caicancai Could you help have a review if you had the free time ? thx a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35196] [Connector / Pulsar] Fix bouncycastle class not found [flink-connector-pulsar]
wenbingshen closed pull request #91: [FLINK-35196] [Connector / Pulsar] Fix bouncycastle class not found URL: https://github.com/apache/flink-connector-pulsar/pull/91 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-31223] sql-client.sh fails to start with ssl enabled [flink]
reswqa commented on PR #22026: URL: https://github.com/apache/flink/pull/22026#issuecomment-2077052176 @davidradl Thanks for reviewing this! I think you should be able to review it, right? If so, I will merge it after get your +1 approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34379][table] Fix OutOfMemoryError with large queries [flink]
lsyldliu commented on code in PR #24600:
URL: https://github.com/apache/flink/pull/24600#discussion_r1579356007
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/optimize/program/DynamicPartitionPruningProgramTest.java:
##
@@ -81,6 +87,42 @@ void setup() {
+ ")");
}
+@Test
+void testLargeQueryPlanShouldNotOutOfMemory() {
+// TABLE_OPTIMIZER_DYNAMIC_FILTERING_ENABLED is already enabled
+List strings = new ArrayList<>();
+for (int i = 0; i < 100; i++) {
+util.tableEnv()
+.executeSql(
+"CREATE TABLE IF NOT EXISTS table"
++ i
++ "(att STRING,filename STRING) "
++ "with("
++ " 'connector' = 'values', "
++ " 'runtime-source' = 'NewSource', "
++ " 'bounded' = 'true'"
++ ")");
+strings.add("select att,filename from table" + i);
+}
+
+final String countName = "CNM";
+Table allUnionTable = util.tableEnv().sqlQuery(String.join(" UNION ALL
", strings));
+Table res =
Review Comment:
Can you complete this test pattern using SQL query purely instead of table
API?
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/optimize/program/DynamicPartitionPruningProgramTest.java:
##
@@ -81,6 +87,42 @@ void setup() {
+ ")");
}
+@Test
+void testLargeQueryPlanShouldNotOutOfMemory() {
+// TABLE_OPTIMIZER_DYNAMIC_FILTERING_ENABLED is already enabled
+List strings = new ArrayList<>();
Review Comment:
strings -> subQueries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34379][table] Fix OutOfMemoryError with large queries [flink]
lsyldliu commented on code in PR #24600:
URL: https://github.com/apache/flink/pull/24600#discussion_r1579386652
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/utils/DynamicPartitionPruningUtils.java:
##
@@ -115,7 +117,7 @@ private static class DppDimSideChecker {
private final RelNode relNode;
private boolean hasFilter;
private boolean hasPartitionedScan;
-private final List tables = new ArrayList<>();
+private final Set tables = new HashSet<>();
Review Comment:
I think we can optimize this for loop by the way to reduce the time
complexity. If the `catalogTable` has already been added to the collection
`tables`, we can just exit the loop without having to do subsequent comparison
operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35190][table] Support create materialized table syntax [flink]
lsyldliu commented on code in PR #24707:
URL: https://github.com/apache/flink/pull/24707#discussion_r1579421407
##
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/TestFlinkSqlParserBase.java:
##
@@ -0,0 +1,47 @@
+package org.apache.flink.sql.parser;
+
+import org.apache.flink.sql.parser.impl.FlinkSqlParserImpl;
+
+import org.apache.calcite.sql.parser.SqlParserFixture;
+import org.apache.calcite.sql.parser.SqlParserTest;
+import org.junit.jupiter.api.extension.ConditionEvaluationResult;
+import org.junit.jupiter.api.extension.ExecutionCondition;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.api.extension.ExtensionContext;
+
+import java.lang.reflect.Method;
+import java.util.Optional;
+
+/**
+ * Abstract base class for testing the Flink SQL parser implementation.
+ *
+ * This class extends the {@link SqlParserTest} class and overrides its
fixture method to use the
+ * {@link FlinkSqlParserImpl} factory for parsing SQL statements. It also
includes an extension to
+ * disable all inherited test methods from {@link SqlParserTest} and only
execute specific tests
+ * defined in this class and its subclass.
+ */
+@ExtendWith(TestFlinkSqlParserBase.DisableSqlParserTestExtension.class)
+public abstract class TestFlinkSqlParserBase extends SqlParserTest {
Review Comment:
It would be better to rename to FlinkSqlParserTestBase
##
flink-table/flink-sql-parser/src/test/java/org/apache/flink/sql/parser/MaterializedTableStatementParserTest.java:
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.sql.parser;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.parallel.Execution;
+
+import static org.junit.jupiter.api.parallel.ExecutionMode.CONCURRENT;
+
+/** Sql parser test for materialized related syntax. * */
+@Execution(CONCURRENT)
+public class MaterializedTableStatementParserTest extends
TestFlinkSqlParserBase {
Review Comment:
BTW, FlinkSqlParserImplTest doesn't need to extends TestFlinkSqlParserBase?
##
flink-table/flink-sql-parser/src/main/codegen/includes/parserImpls.ftl:
##
@@ -2189,6 +2267,8 @@ SqlCreate SqlCreateExtended(Span s, boolean replace) :
|
create = SqlCreateTable(s, replace, isTemporary)
|
+create = SqlCreateMaterializedTable(s)
Review Comment:
What is the behavior when create temporary materialized table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-35217) Missing fsync in FileSystemCheckpointStorage
[
https://issues.apache.org/jira/browse/FLINK-35217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840787#comment-17840787
]
Stefan Richter commented on FLINK-35217:
I think you are right, close will only guarantee a flush, i.e. passing all data
to the OS, but not forcing the OS to write to disk.
> Missing fsync in FileSystemCheckpointStorage
>
>
> Key: FLINK-35217
> URL: https://issues.apache.org/jira/browse/FLINK-35217
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Runtime / Checkpointing
>Affects Versions: 1.17.0, 1.18.0, 1.19.0
>Reporter: Marc Aurel Fritz
>Priority: Critical
>
> While running Flink on a system with unstable power supply checkpoints were
> regularly corrupted in the form of "_metadata" files with a file size of 0
> bytes. In all cases the previous checkpoint data had already been deleted,
> causing progress to be lost completely.
> Further investigation revealed that the "FileSystemCheckpointStorage" doesn't
> perform "fsync" when writing a new checkpoint to disk. This means the old
> checkpoint gets removed without making sure that the new one is durably
> persisted on disk. "strace" on the jobmanager's process confirms this
> behavior:
> # The checkpoint chk-60's in-progress metadata is written at "openat"
> # The checkpoint chk-60's in-progress metadata is atomically renamed at
> "rename"
> # The old checkpoint chk-59 is deleted at "unlink"
> For durable persistence an "fsync" call is missing before step 3.
> Full "strace" log:
> {code:java}
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60",
> 0x7fd2ad5fc970) = -1 ENOENT (No such file or directory)
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60",
> 0x7fd2ad5fca00) = -1 ENOENT (No such file or directory)
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc",
> {st_mode=S_IFDIR|0755, st_size=42, ...}) = 0
> [pid 51618] 11:44:30
> mkdir("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60", 0777)
> = 0
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/_metadata",
> 0x7fd2ad5fc860) = -1 ENOENT (No such file or directory)
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/_metadata",
> 0x7fd2ad5fc740) = -1 ENOENT (No such file or directory)
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/._metadata.inprogress.bf9518dc-2100-4524-9e67-e42913c2b8e8",
> 0x7fd2ad5fc7d0) = -1 ENOENT (No such file or directory)
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60",
> {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60",
> {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> [pid 51618] 11:44:30 openat(AT_FDCWD,
> "/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/._metadata.inprogress.bf9518dc-2100-4524-9e67-e42913c2b8e8",
> O_WRONLY|O_CREAT|O_EXCL, 0666) = 168
> [pid 51618] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/._metadata.inprogress.bf9518dc-2100-4524-9e67-e42913c2b8e8",
> {st_mode=S_IFREG|0644, st_size=23378, ...}) = 0
> [pid 51618] 11:44:30
> rename("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/._metadata.inprogress.bf9518dc-2100-4524-9e67-e42913c2b8e8",
> "/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-60/_metadata") = > 0
> [pid 51644] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59/_metadata",
> {st_mode=S_IFREG|0644, st_size=23378, ...}) = 0
> [pid 51644] 11:44:30
> unlink("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59/_metadata")
> = 0
> [pid 51644] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59",
> {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> [pid 51644] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59",
> {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> [pid 51644] 11:44:30 openat(AT_FDCWD,
> "/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59",
> O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 168
> [pid 51644] 11:44:30 newfstatat(168, "", {st_mode=S_IFDIR|0755, st_size=0,
> ...}, AT_EMPTY_PATH) = 0
> [pid 51644] 11:44:30
> stat("/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59",
> {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
> [pid 51644] 11:44:30 openat(AT_FDCWD,
> "/opt/flink/statestore/e1c541c4568515e77df32d82727e20dc/chk-59",
> O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 168
> [pid 51644] 11:44:30 newfstatat(168, "", {st_mode=S_IFDIR|0755, st_size=0,
> ...},
Re: [PR] [FLINK-32080][FLIP-306][checkpoint] Restoration of FileMergingSnapshotManager [flink]
ljz2051 commented on code in PR #24640:
URL: https://github.com/apache/flink/pull/24640#discussion_r1579435004
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java:
##
@@ -575,8 +576,51 @@ private void createManagedDirectory(Path managedPath) {
@Override
public void close() throws IOException {}
+//
+// restore
+//
+
+@Override
+public void restoreStateHandles(
+long checkpointId, SubtaskKey subtaskKey,
Stream stateHandles) {
+
+Set uploadedLogicalFiles;
+synchronized (lock) {
+uploadedLogicalFiles =
+uploadedStates.computeIfAbsent(checkpointId, id -> new
HashSet<>());
+}
+
+stateHandles.forEach(
+fileHandle -> {
+PhysicalFile physicalFile =
+new PhysicalFile(
+null,
+fileHandle.getFilePath(),
+physicalFileDeleter,
Review Comment:
IIUC, In the case of the job failover, need `FileMergingSnapshotManager` to
delete the physical file; but in the case of the job rescaling, need jobManager
to delete the physical file dir.
I will correct this logic. But i meet a problem where the TaskManager side
has no way of knowing if the recovery is a rescaling situation (especially for
the operator state rescaling).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32080][FLIP-306][checkpoint] Restoration of FileMergingSnapshotManager [flink]
ljz2051 commented on code in PR #24640:
URL: https://github.com/apache/flink/pull/24640#discussion_r1579435004
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/filemerging/FileMergingSnapshotManagerBase.java:
##
@@ -575,8 +576,51 @@ private void createManagedDirectory(Path managedPath) {
@Override
public void close() throws IOException {}
+//
+// restore
+//
+
+@Override
+public void restoreStateHandles(
+long checkpointId, SubtaskKey subtaskKey,
Stream stateHandles) {
+
+Set uploadedLogicalFiles;
+synchronized (lock) {
+uploadedLogicalFiles =
+uploadedStates.computeIfAbsent(checkpointId, id -> new
HashSet<>());
+}
+
+stateHandles.forEach(
+fileHandle -> {
+PhysicalFile physicalFile =
+new PhysicalFile(
+null,
+fileHandle.getFilePath(),
+physicalFileDeleter,
Review Comment:
@Zakelly IIUC, In the case of the job failover, need
`FileMergingSnapshotManager` to delete the physical file; but in the case of
the job rescaling, need jobManager to delete the physical file dir.
I will correct this logic. But i meet a problem where the TaskManager side
has no way of knowing if the recovery is a rescaling situation (especially for
the operator state rescaling).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-35239) 1.19 docs show outdated warning
Ufuk Celebi created FLINK-35239: --- Summary: 1.19 docs show outdated warning Key: FLINK-35239 URL: https://issues.apache.org/jira/browse/FLINK-35239 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.19.0 Reporter: Ufuk Celebi Assignee: Ufuk Celebi Fix For: 1.19.0 Attachments: Screenshot 2024-04-25 at 15.01.57.png The docs for 1.19 are currently marked as outdated although it's the currently stable release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-35239] [docs] Unset outdated warning [flink]
uce opened a new pull request, #24721: URL: https://github.com/apache/flink/pull/24721 ## What is the purpose of the change 1.19 is our current stable release. Therefore, we should not show an outdated warning message. It looks like we accidentally changed this with https://github.com/apache/flink/pull/24547 as part of the 1.19 release. ## Brief change log - *Set ShowOutDatedWarning = false* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35022][Connector/DynamoDB] Add TypeInformed DDB Element Converter [flink-connector-aws]
dannycranmer commented on code in PR #136:
URL:
https://github.com/apache/flink-connector-aws/pull/136#discussion_r1579301456
##
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbTypeInformedElementConverter.java:
##
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.BasicArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeinfo.NumericTypeInfo;
+import org.apache.flink.api.common.typeinfo.PrimitiveArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.CompositeType;
+import org.apache.flink.api.connector.sink2.Sink;
+import org.apache.flink.api.connector.sink2.SinkWriter;
+import org.apache.flink.api.java.tuple.Tuple;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.api.java.typeutils.PojoTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.connector.base.sink.writer.ElementConverter;
+import
org.apache.flink.connector.dynamodb.table.converter.ArrayAttributeConverterProvider;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import software.amazon.awssdk.core.SdkBytes;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeConverter;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeConverterProvider;
+import software.amazon.awssdk.enhanced.dynamodb.AttributeValueType;
+import software.amazon.awssdk.enhanced.dynamodb.EnhancedType;
+import software.amazon.awssdk.enhanced.dynamodb.TableSchema;
+import
software.amazon.awssdk.enhanced.dynamodb.internal.mapper.BeanAttributeGetter;
+import software.amazon.awssdk.enhanced.dynamodb.mapper.StaticTableSchema;
+import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
+
+import java.beans.BeanInfo;
+import java.beans.IntrospectionException;
+import java.beans.Introspector;
+import java.beans.PropertyDescriptor;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.function.Function;
+
+/**
+ * A {@link ElementConverter} that converts an element to a {@link
DynamoDbWriteRequest} using
+ * TypeInformation provided.
+ */
+@PublicEvolving
+public class DynamoDbTypeInformedElementConverter
Review Comment:
nit: I am wondering if there is a better name for this? Also, we could make
this the default element converter and promote the field to optional on the
sink?
How about `AutoDynamoDbElementConverter` or
`SimpleDynamoDbElementConverter`? I am not sold on any name yet!
##
flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbTypeInformedElementConverter.java:
##
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.dynamodb.sink;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.api.common.typeinfo.BasicArrayTypeInfo;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache
[jira] [Updated] (FLINK-35239) 1.19 docs show outdated warning
[ https://issues.apache.org/jira/browse/FLINK-35239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-35239: --- Labels: pull-request-available (was: ) > 1.19 docs show outdated warning > --- > > Key: FLINK-35239 > URL: https://issues.apache.org/jira/browse/FLINK-35239 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: Screenshot 2024-04-25 at 15.01.57.png > > > The docs for 1.19 are currently marked as outdated although it's the > currently stable release. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
