felipepessoto opened a new pull request, #12074:
URL: https://github.com/apache/gluten/pull/12074
## What changes are proposed in this pull request?
Reading the Change Data Feed (CDF) of a Delta table that contains a VARIANT
column
fails under the Velox native engine when the variant column is **not**
projected:
```
spark.sql("""CREATE TABLE tbl (id INT, data VARIANT) USING delta
TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true')""")
spark.sql("""INSERT INTO tbl
SELECT 1, parse_json('{"status":"active"}') UNION ALL
SELECT 2, parse_json('{"status":"active"}')""")
// Works (variant projected → existing variant validation falls back to
vanilla Spark)
spark.read.format("delta").option("readChangeFeed",
"true").option("startingVersion", 0)
.table("tbl").select("id", "data", "_change_type").show()
// Fails with GlutenNotSupportException: Type VariantType not supported
spark.read.format("delta").option("readChangeFeed",
"true").option("startingVersion", 0)
.table("tbl").select("id", "_change_type").show()
```
```
org.apache.gluten.exception.GlutenNotSupportException: Type VariantType not
supported.
at
org.apache.gluten.expression.ConverterUtils$.getTypeNode(ConverterUtils.scala:248)
at
org.apache.gluten.substrait.rel.LocalFilesNode.buildNamedStruct(LocalFilesNode.java:115)
at
org.apache.gluten.substrait.rel.LocalFilesNode.toProtobuf(LocalFilesNode.java:199)
at org.apache.gluten.backendsapi.velox.VeloxIteratorApi
.$anonfun$genFirstStageIterator$3(VeloxIteratorApi.scala:224)
...
```
## How was this patch tested?
New unit tests. Also tested with Delta unit tests.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Github Copilot CLI v1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
