voonhous commented on code in PR #20159: URL: https://github.com/apache/hudi/pull/20159#discussion_r4152343408
########## website/docs/sql_ddl.md: ########## @@ -333,6 +333,13 @@ decodable in application code if needed. What you do **not** get on Spark 3.x: | Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. | | Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink can read the underlying struct but cannot decode it as a variant value. | +Spark's `PushVariantIntoScan` rule is controlled by `spark.sql.variant.pushVariantIntoScan` (on by +default from Spark 4.1, off by default in Spark 4.0). Hudi reads the rewritten scan on Spark 4.1+. +If a read of a VARIANT column fails with the rule on, set it to `false`: Spark then reads the Review Comment: No change here: on master the 1.2.x issues are fixed (#19783, #20041, #20136), so this page covers only the rule's failure mode and points 1.2.0/1.2.1 users to the release notes. On Spark 4.0, master has an explicit guard (#20033, widened to the whole data schema in #20158) that throws `HoodieNotSupportedException` for any read of a table with a VARIANT column while the rule is on, so "rejects" is accurate. The `Illegal initial character: 0` error is the 1.2.x behavior, which the release notes describe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
