[jira] [Created] (FLINK-36263) Include only changed columns in old for canal format in Kakfa Sink
LvYanquan created FLINK-36263:
-
Summary: Include only changed columns in old for canal format in
Kakfa Sink
Key: FLINK-36263
URL: https://issues.apache.org/jira/browse/FLINK-36263
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Affects Versions: cdc-3.3.0
Reporter: LvYanquan
Fix For: cdc-3.3.0
For Update type events, in Canal's official implementation, the old field only
contains the modified column data.
An example output that only c_int and c_tinyint columns were changed:
{code:java}
{
"id": 0,
...
"type": "UPDATE",
...
"sqlType": {
...
},
"mysqlType": {
...
},
"data": [
{
"c_bigint": "9223372036854775807",
"c_int": "0",
"c_mediumint": "8388607",
"c_smallint": "32767",
"c_tinyint": "0",
"id": "2"
}
],
"old": [
{
"c_int": "2147483647",
"c_tinyint": "127"
}
]
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-36211) Shade kafka related package in Kafka Pipeline connector.
LvYanquan created FLINK-36211:
-
Summary: Shade kafka related package in Kafka Pipeline connector.
Key: FLINK-36211
URL: https://issues.apache.org/jira/browse/FLINK-36211
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cassandra-3.2.0
Reporter: LvYanquan
Fix For: cdc-3.2.0
An issue reported from slack:
I'm trying to create a Flink CDC pipeline from MySql to Kafka (Flink CDC 3.1.1,
Flink 1.18). When I try to submit the yaml file the job starts but then fails
with error:
{{{}Caused by: java.lang.NoSuchMethodError: 'void
org.apache.flink.streaming.connectors.kafka.internals.metrics.KafkaMetricMutableWrapper.({}}}{{{}[org.apache.flink.cdc.connectors.kafka.shaded.org|http://org.apache.flink.cdc.connectors.kafka.shaded.org/]{}}}{{{}.apache.kafka.common.Metric)'{}}}
we should shade {{org.apache.flink.streaming.connectors.kafka to avoid conflict
with flink-connector-kafka.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-36178) Parse CREATE TABLE LIKE to CreateTableEvent
LvYanquan created FLINK-36178: - Summary: Parse CREATE TABLE LIKE to CreateTableEvent Key: FLINK-36178 URL: https://issues.apache.org/jira/browse/FLINK-36178 Project: Flink Issue Type: Improvement Reporter: LvYanquan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36148) Support parsing DDL to new CreateTableEvent in binlog reading phase.
LvYanquan created FLINK-36148: - Summary: Support parsing DDL to new CreateTableEvent in binlog reading phase. Key: FLINK-36148 URL: https://issues.apache.org/jira/browse/FLINK-36148 Project: Flink Issue Type: New Feature Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Currently, we didn't parsing DDL to new CreateTableEvent in binlog reading phase, this will lead to NullPointException when a new table creation DDL was met because a CreateTableEvent was not sent before DataChangeEvent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36115) Allow to scan newly table DDL during incremental reading stage.
LvYanquan created FLINK-36115: - Summary: Allow to scan newly table DDL during incremental reading stage. Key: FLINK-36115 URL: https://issues.apache.org/jira/browse/FLINK-36115 Project: Flink Issue Type: Bug Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Attachments: image-2024-08-21-10-06-17-758.png Currently, MySQL pipeline source will determine all captured tables before building MySqlDataSource. However, this will lead to Ignore of the create table statement for the new table. !image-2024-08-21-10-06-17-758.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36088) Fix NopointException in PaimonDataSink.
LvYanquan created FLINK-36088: - Summary: Fix NopointException in PaimonDataSink. Key: FLINK-36088 URL: https://issues.apache.org/jira/browse/FLINK-36088 Project: Flink Issue Type: Bug Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Fix NopointException in BucketAssignOperator when try to get Schema info. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36087) Fix
LvYanquan created FLINK-36087: - Summary: Fix Key: FLINK-36087 URL: https://issues.apache.org/jira/browse/FLINK-36087 Project: Flink Issue Type: Bug Reporter: LvYanquan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-36082) Fix NotSerializableException for KafkaDataSink
LvYanquan created FLINK-36082:
-
Summary: Fix NotSerializableException for KafkaDataSink
Key: FLINK-36082
URL: https://issues.apache.org/jira/browse/FLINK-36082
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.2.0
Reporter: LvYanquan
Fix For: cdc-3.2.0
When I submitted a job to sync data from MySQL to Kafka using yaml, I met the
following exception:
{code:java}
Caused by: java.io.NotSerializableException:
org.apache.flink.cdc.connectors.kafka.sink.KafkaDataSink
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
~[?:1.8.0_372]
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
~[?:1.8.0_372]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
~[?:1.8.0_372] {code}
and when I used CDC version of 3.1, it didn't happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35938) Avoid commit the same datafile again in Paimon Sink.
LvYanquan created FLINK-35938: - Summary: Avoid commit the same datafile again in Paimon Sink. Key: FLINK-35938 URL: https://issues.apache.org/jira/browse/FLINK-35938 Project: Flink Issue Type: Technical Debt Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Attachments: image-2024-07-31-19-45-14-153.png [Flink will re-commit committables|https://github.com/apache/flink/blob/82b628d4730eef32b2f7a022e3b73cb18f950e6e/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/connector/sink2/GlobalCommitterOperator.java#L148] when job restart from failure. This may cause the same datafile were added twice in current PaimonCommitter. !image-2024-07-31-19-45-14-153.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35893) Add state compatibility for Serializer of TableChangeInfo
LvYanquan created FLINK-35893: - Summary: Add state compatibility for Serializer of TableChangeInfo Key: FLINK-35893 URL: https://issues.apache.org/jira/browse/FLINK-35893 Project: Flink Issue Type: Technical Debt Components: Flink CDC Affects Versions: cdc-3.1.1 Reporter: LvYanquan Fix For: cdc-3.2.0 Version info of SERIALIZER for TableChangeInfo was not included in [TransformSchemaOperator|https://github.com/apache/flink-cdc/blob/ea71b2302ddc5f9b7be65843dbf3f5bed4ca9d8e/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformSchemaOperator.java#L127], which may cause incompatible state in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35891) Support writing dynamic bucket table of Paimon
LvYanquan created FLINK-35891: - Summary: Support writing dynamic bucket table of Paimon Key: FLINK-35891 URL: https://issues.apache.org/jira/browse/FLINK-35891 Project: Flink Issue Type: Technical Debt Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Support writing table in [https://paimon.apache.org/docs/master/primary-key-table/data-distribution/#dynamic-bucket] mode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35888) Add e2e test for paimon DataSink
LvYanquan created FLINK-35888: - Summary: Add e2e test for paimon DataSink Key: FLINK-35888 URL: https://issues.apache.org/jira/browse/FLINK-35888 Project: Flink Issue Type: Technical Debt Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Paimon DataSink was already completed, but not e2e test was added. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35873) Add HashFunctionProvider for Paimon DataSink.
LvYanquan created FLINK-35873: - Summary: Add HashFunctionProvider for Paimon DataSink. Key: FLINK-35873 URL: https://issues.apache.org/jira/browse/FLINK-35873 Project: Flink Issue Type: New Feature Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Paimon does not allow multiple writers to write to the same bucket concurrently, so we need to do this in the PrePartitionOperator using HashFunctionProvider. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35871) Add document for SNAPSHOT startup option mode
LvYanquan created FLINK-35871: - Summary: Add document for SNAPSHOT startup option mode Key: FLINK-35871 URL: https://issues.apache.org/jira/browse/FLINK-35871 Project: Flink Issue Type: Technical Debt Components: Flink CDC Affects Versions: cdc-3.1.1 Reporter: LvYanquan Fix For: cdc-3.2.0 Add document for SNAPSHOT startup option mode as [https://github.com/apache/flink-cdc/pull/2901] merged. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35834) Update Doris Dependency to 1.6.2 to use group commit.
LvYanquan created FLINK-35834: - Summary: Update Doris Dependency to 1.6.2 to use group commit. Key: FLINK-35834 URL: https://issues.apache.org/jira/browse/FLINK-35834 Project: Flink Issue Type: Bug Components: Flink CDC Affects Versions: cdc-3.1.1 Reporter: LvYanquan Fix For: cdc-3.3.0 Doris support streamload group commit to improve performance of writing. Refer to https://github.com/apache/doris-flink-connector/pull/412 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35802) Deadlock may happen after adding new tables
LvYanquan created FLINK-35802: - Summary: Deadlock may happen after adding new tables Key: FLINK-35802 URL: https://issues.apache.org/jira/browse/FLINK-35802 Project: Flink Issue Type: Bug Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Attachments: image-2024-07-10-13-44-49-972.png, image-2024-07-10-13-45-52-450.png, image-2024-07-10-13-47-07-190.png Problem Description: 1.CDC originally consumed the full incremental data of a table, and currently, the snapshot phase has ended, and it is in the binlog consumption phase. 2.Stop the job to add the full incremental data synchronization for a new table. 3.After the full phase of the new table ends, it fails to return to the binlog consumption phase. 4. Checking the thread that consumes the binlog, a deadlock situation is discovered, and the specific thread stack is as follows. 5. The likely cause is that after the Enumerator issues a BinlogSplitUpdateRequestEvent, both the MysqlSplitReader and MySqlBinlogSplitReadTask close the binlogClient connection but fail to acquire the lock. 6. The lock is held by the consumer thread, but the queue is full, waiting for consumers to consume the data out, and yet there are no consumers, thus causing a deadlock. ThreadDump： 1. MysqlSplitReader.pollSplitRecords method !image-2024-07-10-13-44-49-972.png! 2. MySqlStreamingChangeEventSource.execute method !image-2024-07-10-13-45-52-450.png! 3. MySqlBinlogSplitReadTask.handleEvent method !image-2024-07-10-13-47-07-190.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35791) Add database and table infos to Kafka json output.
LvYanquan created FLINK-35791: - Summary: Add database and table infos to Kafka json output. Key: FLINK-35791 URL: https://issues.apache.org/jira/browse/FLINK-35791 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Currently, database and table were not passed to canal/debezium json output format of Kafka sink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35758) [Doc] add scan.startup.timestamp-millis options to MySQL connector docs.
LvYanquan created FLINK-35758: - Summary: [Doc] add scan.startup.timestamp-millis options to MySQL connector docs. Key: FLINK-35758 URL: https://issues.apache.org/jira/browse/FLINK-35758 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.1 Reporter: LvYanquan Fix For: cdc-3.2.0 Startup reading position support to specify timestamp position, but this was not included in connector options. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35718) Cherrypick DBZ-5333 to fix frequently failover by EOFException.
LvYanquan created FLINK-35718: - Summary: Cherrypick DBZ-5333 to fix frequently failover by EOFException. Key: FLINK-35718 URL: https://issues.apache.org/jira/browse/FLINK-35718 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.2.0 This EOFException issue should happen occasionally, and Debezium provided a retry mechanism to avoid frequently failover. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35684) Add missing option for DataSource and DataSink
LvYanquan created FLINK-35684: - Summary: Add missing option for DataSource and DataSink Key: FLINK-35684 URL: https://issues.apache.org/jira/browse/FLINK-35684 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 As https://issues.apache.org/jira/browse/FLINK-35121 add verification for requireOptions and optionalOptions, we found that some DataSource and DataSink miss these options. We need to add those options to pass this verification. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35627) Avoid conflict when pipeline job that used Paimon as sink restart.
LvYanquan created FLINK-35627: - Summary: Avoid conflict when pipeline job that used Paimon as sink restart. Key: FLINK-35627 URL: https://issues.apache.org/jira/browse/FLINK-35627 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.2.0 When use Paimon as pipeline sink, if one job failed and restarted, the following job will continue fail over because there are some conflict files being uncommitted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35589) Support MemorySize type in FlinkCDC ConfigOptions
LvYanquan created FLINK-35589: - Summary: Support MemorySize type in FlinkCDC ConfigOptions Key: FLINK-35589 URL: https://issues.apache.org/jira/browse/FLINK-35589 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 This allow user to set MemorySize config type like Flink. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35586) Detected conflict when using Paimon as pipeline sink with parallelism > 1
LvYanquan created FLINK-35586:
-
Summary: Detected conflict when using Paimon as pipeline sink with
parallelism > 1
Key: FLINK-35586
URL: https://issues.apache.org/jira/browse/FLINK-35586
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: LvYanquan
Fix For: cdc-3.2.0
When submit FlinkCDC pipeline job using yaml like:
{code:java}
source:
type: mysql
name: MySQL Source
hostname: 127.0.0.1
port: 3306
username: root
password: 123456
tables: inventory.t1
sink:
type: paimon
name: Paimon Sink
catalog.properties.metastore: filesystem
catalog.properties.warehouse: /mypath
pipeline:
name: MySQL to Paimon Pipeline
parallelism: 2 {code}
I met the following error message:
{code:java}
Caused by: java.lang.RuntimeException: LSM conflicts detected! Give up
committing. Conflict files are:, bucket 0, level 5, file
data-6bcac56a-2df2-4c85-97f2-2db91f6d8099-0.orc, bucket 0, level 5, file
data-351fd27d-4a65-4354-9ce9-c153ba715569-0.orc {code}
And this will cause the task to constantly restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35444) Paimon Pipeline Connector support changing column names to lowercase for Hive metastore
LvYanquan created FLINK-35444: - Summary: Paimon Pipeline Connector support changing column names to lowercase for Hive metastore Key: FLINK-35444 URL: https://issues.apache.org/jira/browse/FLINK-35444 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Hive metastore Require column names to be lowercase, but field names from upstream tables may not meet the requirements. We can add a parameter configuration in sink to convert all column names to lowercase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35442) Kafka Pipeline Sink may distribute the same key to different partitions after modify parallelism
LvYanquan created FLINK-35442: - Summary: Kafka Pipeline Sink may distribute the same key to different partitions after modify parallelism Key: FLINK-35442 URL: https://issues.apache.org/jira/browse/FLINK-35442 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Currently, Kafka Pipeline Sink partition events by parallelInstanceId % partitions.length，this is because the previous partition operator has already shuffled events by key. However, If the parallelism of the task is modified and restarted, data with the same key before and after the task may be in different partitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35430) ZoneId is not passed to DebeziumJsonSerializationSchema
LvYanquan created FLINK-35430: - Summary: ZoneId is not passed to DebeziumJsonSerializationSchema Key: FLINK-35430 URL: https://issues.apache.org/jira/browse/FLINK-35430 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 ZoneId is used to convert TIMESTAMP_WITH_LOCAL_TIME_ZONE type to specific time zone, but DebeziumJsonSerializationSchema does not used the user-defined zoneid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35391) Bump dependency of Paimon Pipeline connector to 0.8.0
LvYanquan created FLINK-35391: - Summary: Bump dependency of Paimon Pipeline connector to 0.8.0 Key: FLINK-35391 URL: https://issues.apache.org/jira/browse/FLINK-35391 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Apache Paimon has released 0.8 recently, We can update dependencies to use new features. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35325) Paimon connector miss the position of AddColumnEvent
LvYanquan created FLINK-35325: - Summary: Paimon connector miss the position of AddColumnEvent Key: FLINK-35325 URL: https://issues.apache.org/jira/browse/FLINK-35325 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.2.0 Reporter: LvYanquan Fix For: cdc-3.2.0 Currently, new columns are always added in the last position, however some newly add columns had a specific before and after relationship with other column. Source code: [https://github.com/apache/flink-cdc/blob/fa6e7ea51258dcd90f06036196618224156df367/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java#L137] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35235) Fix missing dependencies in the uber jar
LvYanquan created FLINK-35235: - Summary: Fix missing dependencies in the uber jar Key: FLINK-35235 URL: https://issues.apache.org/jira/browse/FLINK-35235 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Attachments: image-2024-04-25-15-17-20-987.png, image-2024-04-25-15-17-34-717.png Some class of Kafka were not included in fat jar. !image-2024-04-25-15-17-34-717.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35213) A potential inconsistent table structure issue
LvYanquan created FLINK-35213: - Summary: A potential inconsistent table structure issue Key: FLINK-35213 URL: https://issues.apache.org/jira/browse/FLINK-35213 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Currently, DataSinkWriterOperator will [request CreateTableEvent|https://github.com/apache/flink-cdc/blob/313726b09690e82aa56fb5b42e89b535d24dadd7/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/sink/DataSinkWriterOperator.java#L149] from SchemaRegistry when restarted. However, If a SchemeChangeEvent is received during this process, SchemaOperator will 1. [request SchemaRegistry|https://github.com/apache/flink-cdc/blob/313726b09690e82aa56fb5b42e89b535d24dadd7/flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java#L252] to update the schema, 2. and then send FlushEvent. As the network situation is quite complex, SchemaRegistry may update the schema first, and then send a CreateTableEvent with the new schema, which is incompatible with DatachangeEvent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34770) Unify download links of legacy connector to fat jar
LvYanquan created FLINK-34770: - Summary: Unify download links of legacy connector to fat jar Key: FLINK-34770 URL: https://issues.apache.org/jira/browse/FLINK-34770 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Some links will lead to download fat jar, but some links will lead to download thin jar, and some are invalid. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34752) "legacy-flink-cdc-sources" Page of TIDB for Flink CDC Chinese Documentation.
LvYanquan created FLINK-34752: - Summary: "legacy-flink-cdc-sources" Page of TIDB for Flink CDC Chinese Documentation. Key: FLINK-34752 URL: https://issues.apache.org/jira/browse/FLINK-34752 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Translate legacy-flink-cdc-sources pages of [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/tidb-cdc.md |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34750) "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation.
LvYanquan created FLINK-34750: - Summary: "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation. Key: FLINK-34750 URL: https://issues.apache.org/jira/browse/FLINK-34750 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate legacy-flink-cdc-sources pages of [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/postgres-cdc.md |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34749) "legacy-flink-cdc-sources" Page of SQLServer for Flink CDC Chinese Documentation.
LvYanquan created FLINK-34749: - Summary: "legacy-flink-cdc-sources" Page of SQLServer for Flink CDC Chinese Documentation. Key: FLINK-34749 URL: https://issues.apache.org/jira/browse/FLINK-34749 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate legacy-flink-cdc-sources pages of [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/sqlserver-cdc.md |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34748) "legacy-flink-cdc-sources" Page of Oracle for Flink CDC Chinese Documentation.
LvYanquan created FLINK-34748: - Summary: "legacy-flink-cdc-sources" Page of Oracle for Flink CDC Chinese Documentation. Key: FLINK-34748 URL: https://issues.apache.org/jira/browse/FLINK-34748 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate legacy-flink-cdc-sources pages of [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/oracle-cdc.md |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34747) "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation.
LvYanquan created FLINK-34747: - Summary: "legacy-flink-cdc-sources" Page of DB2 for Flink CDC Chinese Documentation. Key: FLINK-34747 URL: https://issues.apache.org/jira/browse/FLINK-34747 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate legacy-flink-cdc-sources pages of [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md |https://github.com/apache/flink-cdc/blob/master/docs/content/docs/connectors/legacy-flink-cdc-sources/db2-cdc.md]into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34742) Translate "FAQ" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34742: - Summary: Translate "FAQ" Page for Flink CDC Chinese Documentation Key: FLINK-34742 URL: https://issues.apache.org/jira/browse/FLINK-34742 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/faq/faq.md] page into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34741) "get-started" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34741: - Summary: "get-started" Page for Flink CDC Chinese Documentation Key: FLINK-34741 URL: https://issues.apache.org/jira/browse/FLINK-34741 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate [https://github.com/apache/flink-cdc/tree/master/docs/content/docs/get-started] pages into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34740) "legacy-flink-cdc-sources" Pages for Flink CDC Chinese Documentation
LvYanquan created FLINK-34740: - Summary: "legacy-flink-cdc-sources" Pages for Flink CDC Chinese Documentation Key: FLINK-34740 URL: https://issues.apache.org/jira/browse/FLINK-34740 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate legacy-flink-cdc-sources pages of [https://github.com/apache/flink-cdc/tree/master/docs/content/docs/connectors/legacy-flink-cdc-sources] into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34739) "Connectors" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34739: - Summary: "Connectors" Page for Flink CDC Chinese Documentation Key: FLINK-34739 URL: https://issues.apache.org/jira/browse/FLINK-34739 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Translate pipeline connector pages [https://github.com/apache/flink-cdc/tree/master/docs/content/docs/connectors] into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34738) "Deployment - YARN" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34738: - Summary: "Deployment - YARN" Page for Flink CDC Chinese Documentation Key: FLINK-34738 URL: https://issues.apache.org/jira/browse/FLINK-34738 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Translate [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/yarn.md] into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34737) "Deployment - Kubernetes" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34737: - Summary: "Deployment - Kubernetes" Page for Flink CDC Chinese Documentation Key: FLINK-34737 URL: https://issues.apache.org/jira/browse/FLINK-34737 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Translate [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/kubernetes.md] into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34736) "Deployment - Standalone" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34736: - Summary: "Deployment - Standalone" Page for Flink CDC Chinese Documentation Key: FLINK-34736 URL: https://issues.apache.org/jira/browse/FLINK-34736 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Translate [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/deployment/standalone.md] into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34735) "Developer Guide - Understanding Flink CDC API" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34735: - Summary: "Developer Guide - Understanding Flink CDC API" Page for Flink CDC Chinese Documentation Key: FLINK-34735 URL: https://issues.apache.org/jira/browse/FLINK-34735 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 Translate [https://github.com/apache/flink-cdc/blob/master/docs/content/docs/developer-guide/understand-flink-cdc-api.md] into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34734) Update the titles for Chinese Documents.
LvYanquan created FLINK-34734: - Summary: Update the titles for Chinese Documents. Key: FLINK-34734 URL: https://issues.apache.org/jira/browse/FLINK-34734 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Affects Versions: 3.1.0 Reporter: LvYanquan Fix For: 3.1.0 The titles is used to build directory and document names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34730) "Deployment" Page for Flink CDC Chinese Documentation
LvYanquan created FLINK-34730: - Summary: "Deployment" Page for Flink CDC Chinese Documentation Key: FLINK-34730 URL: https://issues.apache.org/jira/browse/FLINK-34730 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Reporter: LvYanquan Translate "Deployment" Page into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34729) "Core Concept" Pages for Flink CDC Chinese Documentation
LvYanquan created FLINK-34729: - Summary: "Core Concept" Pages for Flink CDC Chinese Documentation Key: FLINK-34729 URL: https://issues.apache.org/jira/browse/FLINK-34729 Project: Flink Issue Type: Sub-task Components: chinese-translation, Documentation, Flink CDC Reporter: LvYanquan Translate "Core Concept" Pages into Chinese. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34691) failed to convert time type when using mysql to doris pipeline.
LvYanquan created FLINK-34691: - Summary: failed to convert time type when using mysql to doris pipeline. Key: FLINK-34691 URL: https://issues.apache.org/jira/browse/FLINK-34691 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: LvYanquan Fix For: pre-apache currently, we will get `reason: Flink doesn't support converting type TIME(0) to Doris type yet.` error. Imported from GitHub Url: https://github.com/lvyanquan/flink-cdc/issues/1 Created by: [lvyanquan|https://github.com/lvyanquan] Labels: Created at: Fri Mar 15 12:54:09 CST 2024 State: open -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34671) Update the content of README.md in FlinkCDC project
LvYanquan created FLINK-34671: - Summary: Update the content of README.md in FlinkCDC project Key: FLINK-34671 URL: https://issues.apache.org/jira/browse/FLINK-34671 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 As we have updated the doc site of FlinkCDC, we should modify the content of README.md to update those links and add some more accurate description. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34648) Avoid RPC time when apply SchemaChangeEvent to external system
LvYanquan created FLINK-34648: - Summary: Avoid RPC time when apply SchemaChangeEvent to external system Key: FLINK-34648 URL: https://issues.apache.org/jira/browse/FLINK-34648 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 When SchemaOperator receive SchemaChangeEvent, it will send request to SchemaRegistry and wait for applying this SchemaChangeEvent to external system synchronously. However, if this process take too long time, it will cause RPC TimeoutException, and During the process of task recovery, there may be other errors like `AddColumnEvent is already existed`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34638) Support default value of table column.
LvYanquan created FLINK-34638: - Summary: Support default value of table column. Key: FLINK-34638 URL: https://issues.apache.org/jira/browse/FLINK-34638 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Support default value of table column of literal constant. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34614) Add e2e test for Flink CDC pipeline job.
LvYanquan created FLINK-34614: - Summary: Add e2e test for Flink CDC pipeline job. Key: FLINK-34614 URL: https://issues.apache.org/jira/browse/FLINK-34614 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Add e2e test for Flink CDC pipeline job. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34613) allow specifing a savepoint file to restart using cli script.
LvYanquan created FLINK-34613: - Summary: allow specifing a savepoint file to restart using cli script. Key: FLINK-34613 URL: https://issues.apache.org/jira/browse/FLINK-34613 Project: Flink Issue Type: Improvement Components: Flink CDC Affects Versions: cdc-3.1.0 Reporter: LvYanquan Fix For: cdc-3.1.0 Allow user to specify a savepoint file with `-s` option to restart like Flink cli [https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#starting-a-job-from-a-savepoint] does. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34595) Fix ClassNotFoundException: com.ververica.cdc.common.utils.StringUtils
LvYanquan created FLINK-34595: - Summary: Fix ClassNotFoundException: com.ververica.cdc.common.utils.StringUtils Key: FLINK-34595 URL: https://issues.apache.org/jira/browse/FLINK-34595 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: LvYanquan Fix For: cdc-3.1.0 In this pr https://github.com/apache/flink-cdc/pull/2986/files#diff-cec13810c47e9465e4f2a72507f655b86f41579768b9924fe024aabc60b31d17R21[,|https://github.com/apache/flink-cdc/pull/2986,] we introduced org.apache.flink.cdc.common.utils.StringUtils class of flink-cdc-common module in flink-connector-mysql-cdc module. However, the sub module flink-sql-connector-mysql-cdc doesn't include flink-cdc-common module when packaging, so we can't find this class in sql jar.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
