VvanFalleaves opened a new pull request, #11767:
URL: https://github.com/apache/gluten/pull/11767
Removed handling for StructType, ArrayType, and MapType in dataType matching.
```
DROP TABLE IF EXISTS hive_support;
CREATE TABLE IF NOT EXISTS hive_support (
int_col INT,
array_col ARRAY<INT>,
map_col MAP<STRING, INT>,
struct_col STRUCT<field1: INT, field2: STRING, field3: BOOLEAN>
)
PARTITIONED BY (part_col INT)
STORED AS PARQUET
TBLPROPERTIES (
'parquet.compression' = 'ZSTD',
'parquet.compression.zstd.level' = '5',
'parquet.enable.dictionary' = 'false'
);
WITH number_seq AS (
SELECT CAST(id AS INT) AS id_int
FROM range(1, 50000000, 1, 100)
)
INSERT OVERWRITE TABLE hive_support
PARTITION (part_col)
SELECT
id_int AS int_col,
ARRAY(
id_int % 10,
id_int % 100,
id_int % 1000
) AS array_col,
MAP(
CONCAT('key_', CAST(id_int % 5 AS STRING)), id_int % 100,
CONCAT('key_', CAST((id_int + 1) % 5 AS STRING)), (id_int + 1) % 100
) AS map_col,
named_struct(
'field1', id_int,
'field2', CONCAT('str_', CAST(id_int AS STRING)),
'field3', id_int % 2 = 0
) AS struct_col,
id_int % 10 AS part_col
FROM number_seq
SORT BY id_int;
```
The following error is reported during the execution:
```
26/03/10 20:50:46 WARN GlutenFallbackReporter: Validation failed for plan:
WriteFiles[QueryId=4], due to: Unsupported native write: Found unsupported
type:ArrayType,MapType,StructType.
```
The` Velox` version downloaded by gluten in the `branch-1.3` is
https://github.com/oap-project/velox/tree/gluten-1.3.0. In this case,
`HiveDataSink::appendData` does not support PARTITIONED BY and complex types.
However, in newer Velox, `HiveDataSink::appendData` method has been updated,
but gluten does not fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
