jiangxt2 commented on code in PR #12105:
URL: https://github.com/apache/gravitino/pull/12105#discussion_r3656161658
##########
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTypeConverter.java:
##########
@@ -192,6 +192,11 @@ public DataType toSparkType(Type gravitinoType) {
return DataTypes.createStructType(fields);
} else if (gravitinoType instanceof Types.NullType) {
return DataTypes.NullType;
+ } else if (gravitinoType instanceof Types.ExternalType) {
+ // ExternalType represents types with no Gravitino-native mapping (e.g.
ClickHouse
+ // IPv4/IPv6, Doris LARGEINT/BITMAP, Glue unknown types). Map to
StringType so that
+ // tables containing these columns remain loadable in Spark. Users can
CAST as needed.
+ return DataTypes.StringType;
Review Comment:
I've added a Spark IT in this update —
SparkJdbcMysqlCatalogIT.testExternalTypeEnumColumn() — which verifies the full
path end-to-end using a MySQL ENUM column.
Regarding how we handle ExternalType, here's the conversion flow using
ClickHouse IPv4 as an example:
1. JDBC catalog layer: ClickHouseTypeConverter.toGravitino("IPv4") finds no
native mapping → wraps it as Types.ExternalType.of("IPv4"). The original type
name is preserved via catalogString().
2. Spark connector layer: SparkTypeConverter.toSparkType(ExternalType) →
returns DataTypes.StringType. This follows the same pattern as UUID (which also
has no native Spark type).
3. User experience in Spark: The column appears as STRING. The JDBC driver
returns the value in its string representation (e.g. "192.168.1.1" for IPv4),
so SELECT ip FROM table works directly — no explicit CAST is needed in most
cases.
This approach ensures tables with database-specific types (ClickHouse
IPv4/IPv6, Doris LARGEINT/BITMAP, MySQL ENUM, etc.) remain loadable in Spark
without throwing UnsupportedOperationException, while preserving the original
type name in Gravitino metadata for catalog-level queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
