Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
github-actions[bot] commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2232038069 This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yeezychao commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2102081668 > Hi @yeezychao, seems MySQL CI job is failing, could you please take a look? https://github.com/apache/flink-cdc/actions/runs/8844257194/job/24756979040?pr=1907 > > (The markdown lint check issue is irrelevant to this PR and should be fixed with a `git rebase master`.) @yuxiqian Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yeezychao commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2102205342 > Hi @yeezychao, seems MySQL CI job is failing, could you please take a look? https://github.com/apache/flink-cdc/actions/runs/8844257194/job/24756979040?pr=1907 > > (The markdown lint check issue is irrelevant to this PR and should be fixed with a `git rebase master`.) @yuxiqian Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yuxiqian commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2102217307 Hi @yeezychao, you may run `mvn spotless:apply` to fix style violations, and use `mvn verify` to ensure following the style rules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yuxiqian commented on code in PR #1907:
URL: https://github.com/apache/flink-cdc/pull/1907#discussion_r1595223016
##
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/table/MySqlTableSourceFactory.java:
##
@@ -189,6 +198,7 @@ public Set> optionalOptions() {
options.add(MySqlSourceOptions.HEARTBEAT_INTERVAL);
options.add(MySqlSourceOptions.SCAN_INCREMENTAL_SNAPSHOT_CHUNK_KEY_COLUMN);
options.add(MySqlSourceOptions.SCAN_INCREMENTAL_SNAPSHOT_BACKFILL_SKIP);
+options.add(CHANGELOG_MODE);
Review Comment:
```suggestion
options.add(MySqlSourceOptions.CHANGELOG_MODE);
```
##
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/table/MySqlTableSourceFactory.java:
##
@@ -47,6 +48,8 @@
import java.util.Set;
import java.util.regex.Pattern;
+import static org.apache.flink.cdc.common.utils.Preconditions.checkArgument;
+import static
org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceOptions.CHANGELOG_MODE;
Review Comment:
Maybe use qualified `MySqlSourceOptions.CHANGELOG_MODE` for consistency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yeezychao commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2105523327 @yuxiqian Turning on upsert mode still fails to filter -u data. I am very confused as to why the same PR application failed to test in cdc 3.2 (flink 1.18) version, but it still works in version 2.2 (flink 1.15). Unfortunately, I have not found the reason yet.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yuxiqian commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2105530310 > @yuxiqian Turning on upsert mode still fails to filter -u data. I am very confused as to why the same PR application failed to test in cdc 3.2 (flink 1.18) version, but it still works in version 2.2 (flink 1.15). Unfortunately, I have not found the reason yet.  Maybe checking the printed log and confirm if MySQL source actually sends any `-U` events to downstream? IIRC Flink will automatically append a `ChangelogNormalize` node to backfill missing update before events if source doesn't provide it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-35313][mysql-cdc] Add upsert changelog mode to avoid UPDATE_BEFORE records … [flink-cdc]
yeezychao commented on PR #1907: URL: https://github.com/apache/flink-cdc/pull/1907#issuecomment-2105588522 > > @yuxiqian Turning on upsert mode still fails to filter -u data. I am very confused as to why the same PR application failed to test in cdc 3.2 (flink 1.18) version, but it still works in version 2.2 (flink 1.15). Unfortunately, I have not found the reason yet.  > > @yeezychao Maybe check the output log and confirm if MySQL source actually sends any `-U` events to downstream? IIRC Flink will automatically append a `ChangelogNormalize` node to backfill missing update before events if source doesn't provide it. You are right！The `ChangelogNormalize` node is indeed added under the Flink 1.18 version，but the Flink 1.15 version haven't.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
