jiangxt2 opened a new pull request, #12105:
URL: https://github.com/apache/gravitino/pull/12105
### What changes were proposed in this pull request?
Add `ExternalType → DataTypes.StringType` mapping in
`SparkTypeConverter.toSparkType()` base class. This prevents tables with
`ExternalType` columns from throwing `UnsupportedOperationException` during
`loadTable()` in the Spark connector.
Placing the fix in the base class covers all subclasses across Spark
versions 3.3/3.4/3.5 (JDBC, Hive, and Glue connectors) with a single change.
### Why are the changes needed?
`SparkTypeConverter.toSparkType()` handles ~20 Gravitino native types but
has no branch for `Types.ExternalType`. When a JDBC catalog table contains
database-specific types (e.g. ClickHouse `IPv4`/`IPv6`, Doris
`LARGEINT`/`BITMAP`), the Gravitino server maps them to `ExternalType`, which
then falls through to the final `throw` in `toSparkType()` — crashing the
entire table load.
Affected catalogs include ClickHouse (16+ types), Doris (7 types), MySQL,
PostgreSQL, OceanBase, Hologres, and Glue.
The fix follows the existing pattern used for `UUID` (also mapped to
`StringType` because Spark has no native UUID type).
Fix: #12104
### Does this PR introduce _any_ user-facing change?
Yes — tables with `ExternalType` columns are now loadable in Spark. The
columns are presented as `StringType`, and users can `CAST` to the desired type
as needed, e.g. `CAST(ip AS STRING)`.
### How was this patch tested?
- Unit tests:
`TestSparkJdbcTypeConverter.testConvertExternalTypeToSparkString`
(spark-common) + `TestSparkJdbcTypeConverter34` (v3.4), covering
LARGEINT/BITMAP/IPv4
- Docker integration tests:
- `CatalogClickHouseIT.testIPv4ColumnIsExternalType` — verifies ClickHouse
IPv4 columns are exposed as ExternalType in Gravitino metadata
- `CatalogDoris4xIT.testLargeintColumnIsExternalType` — verifies Doris
LARGEINT columns round-trip correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
