dongjoon-hyun opened a new pull request, #59755:
URL: https://github.com/apache/spark/pull/59755
### What changes were proposed in this pull request?
This PR aims to raise a clear error for the geospatial types, `GEOMETRY` and
`GEOGRAPHY`, in Hive SerDe interop like `TIME` (SPARK-57556) because Hive has
no geospatial types.
- `HiveFileFormat.supportDataType` rejects `GeometryType` and
`GeographyType`, including nested ones, so `INSERT OVERWRITE [LOCAL] DIRECTORY
... STORED AS` through the Hive SerDe fails with
`UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` before running a job.
- The Hive ORC `OrcFileFormat.supportDataType` (`spark.sql.orc.impl=hive`)
rejects them like the native ORC data source (SPARK-55262).
- `HiveInspectors.toInspector` throws `UNSUPPORTED_DATATYPE` for them
instead of `scala.MatchError` and `INTERNAL_ERROR`, like
`HiveInspectors.toTypeInfo` already does.
`INSERT OVERWRITE [LOCAL] DIRECTORY ... STORED AS PARQUET` with the default
configurations, `spark.sql.hive.convertMetastoreInsertDir=true` and
`spark.sql.hive.convertMetastoreParquet=true`, is not affected because it uses
the native Parquet data source, which supports the geospatial types.
### Why are the changes needed?
Hive SerDe cannot parse `geometry(<srid>)` and `geography(<srid>)`. So, the
following fails inside a task after the job starts.
```sql
INSERT OVERWRITE LOCAL DIRECTORY '/tmp/geo' STORED AS TEXTFILE
SELECT 1 AS i, ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040',
4326) AS c;
```
```
org.apache.spark.SparkException: Failed inserting overwrite directory
/tmp/geo
...
Caused by: java.lang.IllegalArgumentException: Error: type expected at the
position 4 of 'int:geometry(4326)' but 'geometry' is found.
```
With `spark.sql.orc.impl=hive`, a data source table with geospatial columns
is created while the native ORC data source rejects it. However, writing to it
fails inside a task in the same way. Reading ORC files with a user-specified
geospatial schema also fails inside a task with `ClassCastException`.
```sql
SET spark.sql.orc.impl=hive;
CREATE TABLE t (i INT, c GEOMETRY(4326)) USING ORC;
INSERT INTO t VALUES (1,
ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
```
```
org.apache.spark.SparkException: Job aborted due to stage failure: ...
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:geometry(4326)' but 'geometry' is found.
```
In addition, Hive generic UDFs, UDAFs, and UDTFs fail with
`scala.MatchError` or `INTERNAL_ERROR` for geospatial arguments.
```sql
CREATE TEMPORARY FUNCTION hive_hash AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFHash';
SELECT
hive_hash(ST_GeomFromWKB(X'0101000000000000000000F03F0000000000000040', 4326));
SELECT hive_hash(CAST(NULL AS GEOMETRY(4326)));
```
```
[CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
Caused by: scala.MatchError: GeometryType(4326) (of class
org.apache.spark.sql.types.GeometryType)
[CANNOT_INSTANTIATE_HIVE_FUNCTION] ...
Caused by: org.apache.spark.SparkException: [INTERNAL_ERROR] Hive doesn't
support the constant type [GeometryType(4326)]. SQLSTATE: XX000
```
### Does this PR introduce _any_ user-facing change?
Yes. The above cases fail with the following errors without running a job.
- Hive SerDe writes, wrapped by `Failed inserting overwrite directory` as
before:
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't support
the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
```
- `CREATE TABLE`, writes, and reads with `spark.sql.orc.impl=hive`, like the
native ORC data source:
```
[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support
the column `c` of the type "GEOMETRY(4326)". SQLSTATE: 0A000
```
- Hive generic UDFs, UDAFs, and UDTFs, wrapped by
`CANNOT_INSTANTIATE_HIVE_FUNCTION` as before, like `TIME` and the simple Hive
UDFs:
```
[UNSUPPORTED_DATATYPE] Unsupported data type "GEOMETRY(4326)". SQLSTATE:
0A000
```
### How was this patch tested?
Pass the CIs with the newly added test cases in `HiveInspectorSuite`,
`InsertSuite`, `HiveUDFSuite`, and `HiveOrcQuerySuite`. All of them fail
without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
