voonhous commented on code in PR #20159: URL: https://github.com/apache/hudi/pull/20159#discussion_r4152253950
########## website/versioned_docs/version-1.2.1/sql_ddl.md: ########## @@ -333,6 +333,16 @@ decodable in application code if needed. What you do **not** get on Spark 3.x: | Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. | | Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink can read the underlying struct but cannot decode it as a variant value. | +:::caution Spark 4.1 and `spark.sql.variant.pushVariantIntoScan` +In Hudi 1.2.0 and 1.2.1, reads of VARIANT columns can return wrong rows or fail while Spark's +`PushVariantIntoScan` rule is on, which is the Spark 4.1 default: on Merge-on-Read tables when +log files are merged, and on any table for a VARIANT column added with +`ALTER TABLE ... ADD COLUMNS`. If a read hits one of these, set Review Comment: Agreed, added the silent MOR null case and the keep-it-off guidance to both versioned sql_ddl pages in 94617b9. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
