[jira] [Updated] (FLINK-35092) Add integrated test for Doris / Starrocks sink pipeline connector
[ https://issues.apache.org/jira/browse/FLINK-35092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiqian YU updated FLINK-35092: -- Summary: Add integrated test for Doris / Starrocks sink pipeline connector (was: Add integrated test for Doris sink pipeline connector) > Add integrated test for Doris / Starrocks sink pipeline connector > - > > Key: FLINK-35092 > URL: https://issues.apache.org/jira/browse/FLINK-35092 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xiqian YU >Priority: Minor > > Currently, no integrated test are being applied to Doris pipeline connector > (there's only one DorisRowConverterTest case for now). Adding ITcases would > improving Doris connector's code quality and reliability. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35120) Add Doris Pipeline connector integration test cases
Xiqian YU created FLINK-35120: - Summary: Add Doris Pipeline connector integration test cases Key: FLINK-35120 URL: https://issues.apache.org/jira/browse/FLINK-35120 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Xiqian YU Currently, Flink CDC Doris pipeline connector has very limited test cases (which only covers row convertion). Adding an ITCase testing its data pipeline and metadata applier should help improving connector's reliability. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35102) Incorret Type mapping for Flink CDC Doris connector
Xiqian YU created FLINK-35102: - Summary: Incorret Type mapping for Flink CDC Doris connector Key: FLINK-35102 URL: https://issues.apache.org/jira/browse/FLINK-35102 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: Xiqian YU According to Flink CDC Doris connector docs, CHAR and VARCHAR are mapped to 3-bytes since Doris uses UTF-8 variable-length encoding internally. |CHAR(n)|CHAR(n*3)|In Doris, strings are stored in UTF-8 encoding, so English characters occupy 1 byte and Chinese characters occupy 3 bytes. The length here is multiplied by 3. The maximum length of CHAR is 255. Once exceeded, it will automatically be converted to VARCHAR type.| |VARCHAR(n)|VARCHAR(n*3)|Same as above. The length here is multiplied by 3. The maximum length of VARCHAR is 65533. Once exceeded, it will automatically be converted to STRING type.| However, currently Doris connector maps `CHAR(n)` to `CHAR(n)` and `VARCHAR(n)` to `VARCHAR(n * 4)`, which is inconsistent with specification in docs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35092) Add integrated test for Doris sink pipeline connector
Xiqian YU created FLINK-35092: - Summary: Add integrated test for Doris sink pipeline connector Key: FLINK-35092 URL: https://issues.apache.org/jira/browse/FLINK-35092 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Xiqian YU Currently, no integrated test are being applied to Doris pipeline connector (there's only one DorisRowConverterTest case for now). Adding ITcases would improving Doris connector's code quality and reliability. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35090) Doris sink fails to create table when database does not exist
[ https://issues.apache.org/jira/browse/FLINK-35090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836386#comment-17836386 ] Xiqian YU commented on FLINK-35090: --- [~renqs] I'm willing to take this ticket. > Doris sink fails to create table when database does not exist > - > > Key: FLINK-35090 > URL: https://issues.apache.org/jira/browse/FLINK-35090 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: Xiqian YU >Priority: Minor > > Currently, Doris sink connector doesn't support creating database > automatically. When user specifies a sink namespace with non-existing > database in YAML config, Doris connector will crash. > Expected behaviour: Doris sink connector should create both database and > table automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35090) Doris sink fails to create table when database does not exist
Xiqian YU created FLINK-35090: - Summary: Doris sink fails to create table when database does not exist Key: FLINK-35090 URL: https://issues.apache.org/jira/browse/FLINK-35090 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: Xiqian YU Currently, Doris sink connector doesn't support creating database automatically. When user specifies a sink namespace with non-existing database in YAML config, Doris connector will crash. Expected behaviour: Doris sink connector should create both database and table automatically. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35079) MongoConnector failed to resume token when current collection removed
[ https://issues.apache.org/jira/browse/FLINK-35079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836040#comment-17836040 ] Xiqian YU commented on FLINK-35079: --- [~renqs] I'm glad to help investigating this. > MongoConnector failed to resume token when current collection removed > - > > Key: FLINK-35079 > URL: https://issues.apache.org/jira/browse/FLINK-35079 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: Xiqian YU >Priority: Major > > When connector tries to create cursor with an expired resuming token during > stream task fetching stage, MongoDB connector will crash with such message: > "error due to Command failed with error 280 (ChangeStreamFatalError): 'cannot > resume stream; the resume token was not found." -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35079) MongoConnector failed to resume token when current collection removed
Xiqian YU created FLINK-35079: - Summary: MongoConnector failed to resume token when current collection removed Key: FLINK-35079 URL: https://issues.apache.org/jira/browse/FLINK-35079 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: Xiqian YU When connector tries to create cursor with an expired resuming token during stream task fetching stage, MongoDB connector will crash with such message: "error due to Command failed with error 280 (ChangeStreamFatalError): 'cannot resume stream; the resume token was not found." -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35072) Doris pipeline sink does not support applying AlterColumnTypeEvent
[ https://issues.apache.org/jira/browse/FLINK-35072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiqian YU updated FLINK-35072: -- Labels: pull-request-available (was: ) > Doris pipeline sink does not support applying AlterColumnTypeEvent > -- > > Key: FLINK-35072 > URL: https://issues.apache.org/jira/browse/FLINK-35072 > Project: Flink > Issue Type: Improvement > Components: Flink CDC >Reporter: Xiqian YU >Priority: Minor > Labels: pull-request-available > > According to [Doris > documentation|https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN/], > altering column types dynamically is supported (via ALTER TABLE ... MODIFY > COLUMN statement) when lossless conversion is available. However, now Doris > pipeline connector has no support to AlterColumnTypeEvent, and raises > RuntimeException all the time. > It would be convenient for users if they can sync compatible type > conversions, and could be easily implemented by extending Doris' > SchemaChangeManager helper class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35077) Add package license check for Flink CDC modules.
Xiqian YU created FLINK-35077: - Summary: Add package license check for Flink CDC modules. Key: FLINK-35077 URL: https://issues.apache.org/jira/browse/FLINK-35077 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Xiqian YU Currently, Flink project has CI scripts checking if dependencies with incompatible licenses are introduced. Flink CDC module heavily relies on external libraries (especially connectors), so running similar checking scripts during every CI would be helpful preventing developers introducing questionable dependencies by accident. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35072) Doris pipeline sink does not support applying AlterColumnTypeEvent
Xiqian YU created FLINK-35072: - Summary: Doris pipeline sink does not support applying AlterColumnTypeEvent Key: FLINK-35072 URL: https://issues.apache.org/jira/browse/FLINK-35072 Project: Flink Issue Type: Improvement Components: Flink CDC Reporter: Xiqian YU According to [Doris documentation|https://doris.apache.org/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-COLUMN/], altering column types dynamically is supported (via ALTER TABLE ... MODIFY COLUMN statement) when lossless conversion is available. However, now Doris pipeline connector has no support to AlterColumnTypeEvent, and raises RuntimeException all the time. It would be convenient for users if they can sync compatible type conversions, and could be easily implemented by extending Doris' SchemaChangeManager helper class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35055) Flink CDC connector release contains dependency with incompatible licenses
[ https://issues.apache.org/jira/browse/FLINK-35055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiqian YU updated FLINK-35055: -- Description: Currently, Flink CDC connector releases both slim and fat jars. Apart from CDC itself, all of its dependencies are packaged into fat jars, including some with incompatible licenses: * Db2 connector: `com.ibm.db2.jcc:db2jcc:db2jcc4` licensed with a non-FOSS license (International Program License Agreement). * MySQL connector: `mysql:mysql-connector-java` licensed with GPLv2 license, which is incompatible with Apache 2.0. * Oracle connector: `com.oracle.ojdbc` licensed with a non-FOSS license (Oracle Free Use Terms and Conditions). * OceanBase connector: `mysql:mysql-connector-java` licensed with GPLv2 license, which is incompatible with Apache 2.0. To fix this problem we may: # Exclude questionable dependencies from released jar; # Add docs to guide user download & place dependencies manually. was: Currently, Flink CDC connector releases both slim and fat jars. Apart from CDC itself, all of its dependencies are packaged into fat jars, including some with incompatible licenses: * Db2 connector: `com.ibm.db2.jcc:db2jcc:db2jcc4` licensed with a non-FOSS license (International Program License Agreement). * MySQL connector: `mysql:mysql-connector-java` licensed with GPLv2 license, which is incompatible with Apache 2.0. * Oracle connector: `com.oracle.ojdbc` licensed with a non-FOSS license (Oracle Free Use Terms and Conditions). To fix this problem we may: # Exclude questionable dependencies from released jar; # Add docs to guide user download & place dependencies manually. > Flink CDC connector release contains dependency with incompatible licenses > -- > > Key: FLINK-35055 > URL: https://issues.apache.org/jira/browse/FLINK-35055 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: Xiqian YU >Assignee: Xiqian YU >Priority: Major > > Currently, Flink CDC connector releases both slim and fat jars. Apart from > CDC itself, all of its dependencies are packaged into fat jars, including > some with incompatible licenses: > * Db2 connector: `com.ibm.db2.jcc:db2jcc:db2jcc4` licensed with a non-FOSS > license (International Program License Agreement). > * MySQL connector: `mysql:mysql-connector-java` licensed with GPLv2 license, > which is incompatible with Apache 2.0. > * Oracle connector: `com.oracle.ojdbc` licensed with a non-FOSS license > (Oracle Free Use Terms and Conditions). > * OceanBase connector: `mysql:mysql-connector-java` licensed with GPLv2 > license, which is incompatible with Apache 2.0. > To fix this problem we may: > # Exclude questionable dependencies from released jar; > # Add docs to guide user download & place dependencies manually. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-35055) Flink CDC connector release contains dependency with incompatible licenses
[ https://issues.apache.org/jira/browse/FLINK-35055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiqian YU updated FLINK-35055: -- Summary: Flink CDC connector release contains dependency with incompatible licenses (was: Flink CDC connector release contains jar with incompatible licenses) > Flink CDC connector release contains dependency with incompatible licenses > -- > > Key: FLINK-35055 > URL: https://issues.apache.org/jira/browse/FLINK-35055 > Project: Flink > Issue Type: Bug > Components: Flink CDC >Reporter: Xiqian YU >Priority: Major > > Currently, Flink CDC connector releases both slim and fat jars. Apart from > CDC itself, all of its dependencies are packaged into fat jars, including > some with incompatible licenses: > * Db2 connector: `com.ibm.db2.jcc:db2jcc:db2jcc4` licensed with a non-FOSS > license (International Program License Agreement). > * MySQL connector: `mysql:mysql-connector-java` licensed with GPLv2 license, > which is incompatible with Apache 2.0. > * Oracle connector: `com.oracle.ojdbc` licensed with a non-FOSS license > (Oracle Free Use Terms and Conditions). > To fix this problem we may: > # Exclude questionable dependencies from released jar; > # Add docs to guide user download & place dependencies manually. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35055) Flink CDC connector release contains jar with incompatible licenses
Xiqian YU created FLINK-35055: - Summary: Flink CDC connector release contains jar with incompatible licenses Key: FLINK-35055 URL: https://issues.apache.org/jira/browse/FLINK-35055 Project: Flink Issue Type: Bug Components: Flink CDC Reporter: Xiqian YU Currently, Flink CDC connector releases both slim and fat jars. Apart from CDC itself, all of its dependencies are packaged into fat jars, including some with incompatible licenses: * Db2 connector: `com.ibm.db2.jcc:db2jcc:db2jcc4` licensed with a non-FOSS license (International Program License Agreement). * MySQL connector: `mysql:mysql-connector-java` licensed with GPLv2 license, which is incompatible with Apache 2.0. * Oracle connector: `com.oracle.ojdbc` licensed with a non-FOSS license (Oracle Free Use Terms and Conditions). To fix this problem we may: # Exclude questionable dependencies from released jar; # Add docs to guide user download & place dependencies manually. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32289) The metadata column type is incorrect in Kafka table connector example
[
https://issues.apache.org/jira/browse/FLINK-32289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17730478#comment-17730478
]
Xiqian YU commented on FLINK-32289:
---
I'm glad to take this ticket.
> The metadata column type is incorrect in Kafka table connector example
> --
>
> Key: FLINK-32289
> URL: https://issues.apache.org/jira/browse/FLINK-32289
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.4, 1.16.2, 1.17.1
>Reporter: Leonard Xu
>Priority: Major
>
> The example[1] defined ts column with TIMESTAMP type
>
> {code:java}
> `ts` TIMESTAMP(3) METADATA FROM 'timestamp'
> {code}
> the correct column type should be TIMESTAMP_LTZ type.
>
> {code:java}
> `ts` TIMESTAMP_LTZ(3) METADATA FROM 'timestamp' {code}
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/kafka/#how-to-create-a-kafka-table
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
