hudi-agent commented on code in PR #20159: URL: https://github.com/apache/hudi/pull/20159#discussion_r4144888381
########## website/releases/release-1.2.md: ########## @@ -50,15 +50,30 @@ The Trino-Hudi connector now lives in the Hudi repo (RFC-105, [#18837](https://g ## Known Issues -On Spark 4.1, `spark.sql.variant.pushVariantIntoScan` is on by default. With it on, snapshot reads of Merge-on-Read tables with `VARIANT` columns can go wrong when log files are merged: +These issues affect 1.2.0 and 1.2.1. Spark's `PushVariantIntoScan` rule is controlled by `spark.sql.variant.pushVariantIntoScan`, which is on by default in Spark 4.1 and off by default in Spark 4.0. + +On Spark 4.1 with the rule on, reads of `VARIANT` columns can go wrong. On Merge-on-Read tables, when a snapshot read merges log files: * A top-level `VARIANT` value set to null by an update is read back as a struct of nulls instead of `NULL`, so a filter such as `v IS NULL` misses the row. * A `VARIANT` nested inside a struct can crash the executor JVM. +* A query that extracts several paths from one `VARIANT` can fail with `NegativeArraySizeException` when rows lack some of those paths. -Copy-on-Write tables are not affected. Spark 4.0 is affected only if `spark.sql.variant.pushVariantIntoScan` is turned on, since it is off by default there. A fix is tracked in [#20039](https://github.com/apache/hudi/pull/20039). +On Copy-on-Write and Merge-on-Read tables: -:::tip -Until the fix is released, set `spark.sql.variant.pushVariantIntoScan=false` for Spark 4.1 reads of Merge-on-Read tables with `VARIANT` columns. +* A `VARIANT` column added with `ALTER TABLE ... ADD COLUMNS` cannot be read from files written before it was added. The read fails with `INVALID_VARIANT_SHREDDING_SCHEMA`. + +Reads with the rule off are not affected. The first two are fixed on the 1.2.x branch ([#20039](https://github.com/apache/hudi/pull/20039)); no release carries that fix yet. The other two are tracked in [#20040](https://github.com/apache/hudi/issues/20040) and [#20135](https://github.com/apache/hudi/issues/20135). + +On Spark 4.0 the rule is not supported. With it turned on, every read of a table that has a `VARIANT` column fails with `HoodieSchemaException: Illegal initial character: 0`, so leave it at its default there. Review Comment: 🤖 "no release carries that fix yet" will go stale as soon as a 1.2.2 (or later) release ships. You could name the release the fix is targeted for, or plan to update this line in that release's notes. It might also help to say whether #20039 is on master too, so users on newer versions know if they are affected. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/sql_ddl.md: ########## @@ -333,6 +333,13 @@ decodable in application code if needed. What you do **not** get on Spark 3.x: | Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. | | Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink can read the underlying struct but cannot decode it as a variant value. | +Spark's `PushVariantIntoScan` rule is controlled by `spark.sql.variant.pushVariantIntoScan` (on by +default from Spark 4.1, off by default in Spark 4.0). Hudi reads the rewritten scan on Spark 4.1+. +If a read of a VARIANT column fails with the rule on, set it to `false`: Spark then reads the Review Comment: 🤖 "If a read of a VARIANT column fails with the rule on, set it to `false`" only covers failures. If any known wrong-result case still applies on current master, the text could mention it next to failures. Otherwise, it might help to say plainly that the 1.2.x issues are fixed on master, so readers of the current docs know whether the workaround is still needed. Also, "Hudi rejects a read" on Spark 4.0 reads like a deliberate validation error. The release notes describe it as a `HoodieSchemaException: Illegal initial character: 0`. If master now has an explicit check, the wording is fine. If not, "fails" would match the release notes better. @yihua could you confirm what master does on Spark 4.0 with the rule on? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/versioned_docs/version-1.2.1/sql_ddl.md: ########## @@ -333,6 +333,16 @@ decodable in application code if needed. What you do **not** get on Spark 3.x: | Flink < 2.1 | Throws `UnsupportedOperationException` on VARIANT columns. | | Flink ≥ 2.1 | Surfaces VARIANT as `ROW<metadata BYTES, value BYTES>`. Flink can read the underlying struct but cannot decode it as a variant value. | +:::caution Spark 4.1 and `spark.sql.variant.pushVariantIntoScan` +In Hudi 1.2.0 and 1.2.1, reads of VARIANT columns can return wrong rows or fail while Spark's +`PushVariantIntoScan` rule is on, which is the Spark 4.1 default: on Merge-on-Read tables when +log files are merged, and on any table for a VARIANT column added with +`ALTER TABLE ... ADD COLUMNS`. If a read hits one of these, set Review Comment: 🤖 This caution could say that one of these issues is silent. "If a read hits one of these, set ... false" implies the user will notice. The null-VARIANT case on MOR raises no error: a null is read back as a struct of nulls, so `v IS NULL` just misses the row. The release notes tip covers this ("The first issue raises no error, so keep the rule off for Spark 4.1 reads of Merge-on-Read tables whose VARIANT columns can be null"). It might help to add the same guidance here, since many users will only see this page. The same applies to `version-1.2.0/sql_ddl.md`. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
