voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3828605415
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -977,6 +977,61 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
}
}
+ test("Test Spark 3.x schema-on-read reads of a variant table with a
committed internal schema") {
+ // #18021: hoodie.schema.on.read.enable resolves the schema through the
InternalSchema round
+ // trip, whose sentinel detection restores the VARIANT logical type - the
exact input
+ // HoodieSparkSchemaConverters rejects on Spark 3.x. Verified 2026-08-20:
every leg fails
+ // LOUDLY with the same actionable error as the plain auto-resolve path;
there is no silent
+ // wrong data and no obscure secondary failure. Notably that includes the
documented
+ // struct-DDL compat mode, which works WITHOUT schema-on-read (see the
backward-compat test
+ // below) but breaks once the table carries an internal schema and the
conf is on, because
+ // internal-schema resolution overrides the user's DDL. Real support is
#18285; until then
+ // Spark 3.x compat-mode readers must keep hoodie.schema.on.read.enable
off.
+ assume(HoodieSparkUtils.isSpark3, "This test verifies Spark 3.x behavior
with schema-on-read")
+
+ withTempDir { tmpDir =>
+
HoodieTestUtils.extractZipToDirectory("variant_backward_compat/variant_schema_on_read_cow.zip",
tmpDir.toPath, getClass)
+ val tablePath =
tmpDir.toPath.resolve("variant_schema_on_read_cow").toString
+
+ def assertVariantRejected(leg: String)(f: => Unit): Unit = {
+ val ex = intercept[HoodieSchemaException](f)
+ assert(ex.getCause.getMessage.contains("VARIANT type is only supported
in Spark 4.0+"),
+ s"[$leg] expected the actionable variant rejection, got:
${ex.getCause}")
+ }
+
+ assertVariantRejected("auto-resolve, plain") {
+ spark.read.format("hudi").load(tablePath).collect()
+ }
+ assertVariantRejected("auto-resolve, schema-on-read") {
+ spark.read.format("hudi").option("hoodie.schema.on.read.enable",
"true").load(tablePath).collect()
Review Comment:
Added in d8b0c916: the test now asserts
getTableInternalSchemaFromCommitMetadata is non-empty for the fixture before
any read leg, so the schema-on-read legs cannot pass through the
commit-metadata fallback.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -977,6 +977,61 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
}
}
+ test("Test Spark 3.x schema-on-read reads of a variant table with a
committed internal schema") {
+ // #18021: hoodie.schema.on.read.enable resolves the schema through the
InternalSchema round
+ // trip, whose sentinel detection restores the VARIANT logical type - the
exact input
+ // HoodieSparkSchemaConverters rejects on Spark 3.x. Verified 2026-08-20:
every leg fails
+ // LOUDLY with the same actionable error as the plain auto-resolve path;
there is no silent
+ // wrong data and no obscure secondary failure. Notably that includes the
documented
+ // struct-DDL compat mode, which works WITHOUT schema-on-read (see the
backward-compat test
+ // below) but breaks once the table carries an internal schema and the
conf is on, because
+ // internal-schema resolution overrides the user's DDL. Real support is
#18285; until then
+ // Spark 3.x compat-mode readers must keep hoodie.schema.on.read.enable
off.
+ assume(HoodieSparkUtils.isSpark3, "This test verifies Spark 3.x behavior
with schema-on-read")
+
+ withTempDir { tmpDir =>
+
HoodieTestUtils.extractZipToDirectory("variant_backward_compat/variant_schema_on_read_cow.zip",
tmpDir.toPath, getClass)
+ val tablePath =
tmpDir.toPath.resolve("variant_schema_on_read_cow").toString
+
+ def assertVariantRejected(leg: String)(f: => Unit): Unit = {
+ val ex = intercept[HoodieSchemaException](f)
+ assert(ex.getCause.getMessage.contains("VARIANT type is only supported
in Spark 4.0+"),
+ s"[$leg] expected the actionable variant rejection, got:
${ex.getCause}")
+ }
+
+ assertVariantRejected("auto-resolve, plain") {
+ spark.read.format("hudi").load(tablePath).collect()
+ }
+ assertVariantRejected("auto-resolve, schema-on-read") {
+ spark.read.format("hudi").option("hoodie.schema.on.read.enable",
"true").load(tablePath).collect()
+ }
+ assertVariantRejected("compat struct DDL, schema-on-read") {
Review Comment:
Added in d8b0c916: the struct-DDL table is now created once and read twice -
first with the conf off, asserting both rows (including the post-DDL note
values) come back, then with the conf on, asserting the rejection. The header
comment now points at that leg instead of the variant_cow test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
