github-actions[bot] commented on code in PR #65325:
URL: https://github.com/apache/doris/pull/65325#discussion_r3540770269
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/SchemaChangeHelper.java:
##########
@@ -136,6 +106,97 @@ public static String columnToDorisType(Column column) {
return pgTypeNameToDorisType(column.typeName(), column.length(),
column.scale().orElse(-1));
}
+ /** Convert a Debezium MySQL Column to a Doris column type string. */
+ public static String mysqlColumnToDorisType(Column column) {
+ Preconditions.checkNotNull(column.typeName());
+ String mysqlTypeName = column.typeName().toUpperCase();
+ String[] typeFields = mysqlTypeName.split(" ");
+ String mysqlType = typeFields[0];
+ boolean unsigned = typeFields.length > 1 &&
"UNSIGNED".equals(typeFields[1]);
+ int length = column.length();
+ int scale = column.scale().orElse(-1);
+ switch (mysqlType) {
+ case "BOOLEAN":
+ case "BOOL":
+ return DorisType.BOOLEAN;
+ case "TINYINT":
+ return unsigned ? DorisType.SMALLINT : DorisType.TINYINT;
+ case "SMALLINT":
+ case "INT2":
+ case "YEAR":
+ return unsigned ? DorisType.INT : DorisType.SMALLINT;
+ case "MEDIUMINT":
+ case "INT":
+ case "INTEGER":
+ case "INT3":
+ case "INT4":
+ return unsigned ? DorisType.BIGINT : DorisType.INT;
+ case "BIGINT":
+ case "INT8":
+ return unsigned ? DorisType.LARGEINT : DorisType.BIGINT;
+ case "FLOAT":
+ case "FLOAT4":
+ return DorisType.FLOAT;
+ case "DOUBLE":
+ case "FLOAT8":
+ case "REAL":
+ return DorisType.DOUBLE;
+ case "DECIMAL":
+ case "DEC":
+ case "FIXED":
+ case "NUMERIC":
+ {
+ int precision = length > 0 ? length : 10;
+ if (unsigned) {
+ precision++;
+ }
+ precision = Math.min(precision, 38);
+ int decimalScale = scale >= 0 ? scale : 0;
+ return String.format("%s(%d, %d)", DorisType.DECIMAL,
precision, decimalScale);
+ }
+ case "DATE":
+ return DorisType.DATE;
+ case "DATETIME":
+ case "TIMESTAMP":
+ {
+ int timeScale = (scale >= 0 && scale <= 6) ? scale : 0;
Review Comment:
`DATETIME(6)` / `TIMESTAMP(6)` added columns lose their fractional precision
here. The custom parser registers these as MySQL dimension data types, and
Debezium stores a single optional dimension such as `DATETIME(6)` in
`Column.length()` while leaving `Column.scale()` empty. Because this branch
only reads `scale`, it emits `DATETIME(0)`. A streaming job that adds
`DATETIME(6)` after startup can then truncate or reject microsecond values even
though the same column would be created with precision 6 in the initial JDBC
table mapping. Please derive the MySQL temporal precision from the parser field
that is actually populated, for example `scale` when present and otherwise
`length` when it is in `[0, 6]`, and add a test for `ADD COLUMN dt DATETIME(6)`
/ `TIMESTAMP(6)`.
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/SchemaChangeHelper.java:
##########
@@ -136,6 +106,97 @@ public static String columnToDorisType(Column column) {
return pgTypeNameToDorisType(column.typeName(), column.length(),
column.scale().orElse(-1));
}
+ /** Convert a Debezium MySQL Column to a Doris column type string. */
+ public static String mysqlColumnToDorisType(Column column) {
+ Preconditions.checkNotNull(column.typeName());
+ String mysqlTypeName = column.typeName().toUpperCase();
+ String[] typeFields = mysqlTypeName.split(" ");
+ String mysqlType = typeFields[0];
+ boolean unsigned = typeFields.length > 1 &&
"UNSIGNED".equals(typeFields[1]);
+ int length = column.length();
+ int scale = column.scale().orElse(-1);
+ switch (mysqlType) {
+ case "BOOLEAN":
+ case "BOOL":
+ return DorisType.BOOLEAN;
+ case "TINYINT":
+ return unsigned ? DorisType.SMALLINT : DorisType.TINYINT;
+ case "SMALLINT":
+ case "INT2":
+ case "YEAR":
+ return unsigned ? DorisType.INT : DorisType.SMALLINT;
+ case "MEDIUMINT":
+ case "INT":
+ case "INTEGER":
+ case "INT3":
+ case "INT4":
+ return unsigned ? DorisType.BIGINT : DorisType.INT;
+ case "BIGINT":
+ case "INT8":
+ return unsigned ? DorisType.LARGEINT : DorisType.BIGINT;
+ case "FLOAT":
+ case "FLOAT4":
+ return DorisType.FLOAT;
+ case "DOUBLE":
+ case "FLOAT8":
+ case "REAL":
+ return DorisType.DOUBLE;
+ case "DECIMAL":
+ case "DEC":
+ case "FIXED":
+ case "NUMERIC":
+ {
+ int precision = length > 0 ? length : 10;
+ if (unsigned) {
+ precision++;
+ }
+ precision = Math.min(precision, 38);
Review Comment:
Capping the MySQL precision to 38 changes the type semantics compared with
initial table creation. The FE MySQL JDBC mapper sends over-precision decimals
through `createDecimalOrStringType()`, which falls back to `STRING` when
precision exceeds Doris' decimal limit. This ADD COLUMN path instead turns
something like `DECIMAL(65,30)` into `DECIMAL(38,30)`, leaving only 8 integer
digits and causing valid source values to overflow after the schema change.
Please match the initial table-creation mapping and fall back to `STRING` when
the MySQL precision is greater than Doris can represent.
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/SchemaChangeHelper.java:
##########
@@ -136,6 +106,97 @@ public static String columnToDorisType(Column column) {
return pgTypeNameToDorisType(column.typeName(), column.length(),
column.scale().orElse(-1));
}
+ /** Convert a Debezium MySQL Column to a Doris column type string. */
+ public static String mysqlColumnToDorisType(Column column) {
+ Preconditions.checkNotNull(column.typeName());
+ String mysqlTypeName = column.typeName().toUpperCase();
+ String[] typeFields = mysqlTypeName.split(" ");
+ String mysqlType = typeFields[0];
+ boolean unsigned = typeFields.length > 1 &&
"UNSIGNED".equals(typeFields[1]);
+ int length = column.length();
+ int scale = column.scale().orElse(-1);
+ switch (mysqlType) {
+ case "BOOLEAN":
+ case "BOOL":
+ return DorisType.BOOLEAN;
+ case "TINYINT":
+ return unsigned ? DorisType.SMALLINT : DorisType.TINYINT;
+ case "SMALLINT":
+ case "INT2":
+ case "YEAR":
+ return unsigned ? DorisType.INT : DorisType.SMALLINT;
+ case "MEDIUMINT":
+ case "INT":
+ case "INTEGER":
+ case "INT3":
+ case "INT4":
+ return unsigned ? DorisType.BIGINT : DorisType.INT;
+ case "BIGINT":
+ case "INT8":
+ return unsigned ? DorisType.LARGEINT : DorisType.BIGINT;
+ case "FLOAT":
+ case "FLOAT4":
+ return DorisType.FLOAT;
+ case "DOUBLE":
+ case "FLOAT8":
+ case "REAL":
+ return DorisType.DOUBLE;
+ case "DECIMAL":
+ case "DEC":
+ case "FIXED":
+ case "NUMERIC":
+ {
+ int precision = length > 0 ? length : 10;
+ if (unsigned) {
+ precision++;
+ }
+ precision = Math.min(precision, 38);
+ int decimalScale = scale >= 0 ? scale : 0;
+ return String.format("%s(%d, %d)", DorisType.DECIMAL,
precision, decimalScale);
+ }
+ case "DATE":
+ return DorisType.DATE;
+ case "DATETIME":
+ case "TIMESTAMP":
+ {
+ int timeScale = (scale >= 0 && scale <= 6) ? scale : 0;
+ return String.format("%s(%d)", DorisType.DATETIME,
timeScale);
+ }
+ case "CHAR":
+ case "NCHAR":
+ return length > 0
+ ? String.format("%s(%d)", DorisType.CHAR, length)
+ : DorisType.STRING;
+ case "VARCHAR":
+ case "NVARCHAR":
+ return length > 0
Review Comment:
This ADD COLUMN mapper should mirror the MySQL table-creation mapper for
string lengths. During initial job creation, `StreamingJobUtils.getColumns()`
multiplies MySQL `VARCHAR` and `CHAR` lengths by 3 before building the Doris
table, but this schema-change path emits the raw MySQL character length. So a
table that starts with `VARCHAR(50)` gets Doris `VARCHAR(150)`, while the same
column added later gets `VARCHAR(50)`, and valid multibyte MySQL values can
fail after the schema change. Please apply the same widening and overflow
handling here as the initial MySQL JDBC mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
