voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3838206840
##########
hudi-common/src/test/resources/variant_backward_compat/README.md:
##########
@@ -39,3 +39,38 @@ The test runs on these four arguments:
COW tables generated are the same for both AVRO/SPARK. But for MOR, the log
files metadata are
different. Hence, we only need to generate test files for either 1/2, 3 and 4,
hence, 3 test
resource files.
+
+# variant_schema_on_read_cow.zip
+
+A Spark 4.1-written COW variant table carrying a COMMITTED INTERNAL SCHEMA:
one insert, then a
+schema-on-read DDL, then a second insert. Used by the Spark 3.x schema-on-read
rejection test
+(#18021). Generated by running these exact statements in a Spark 4.1 SQL
session:
+
+```sql
+create table variant_schema_on_read_cow (
+ id int,
+ v variant,
+ ts long
+) using hudi
+location '/path/to/table/parent/variant_schema_on_read_cow'
+tblproperties (
+ primaryKey = 'id',
+ preCombineField = 'ts'
+);
+
+insert into variant_schema_on_read_cow select 1, parse_json('{"key": "v1"}'),
1000;
+
+set hoodie.schema.on.read.enable=true;
+alter table variant_schema_on_read_cow add columns (note string);
+insert into variant_schema_on_read_cow select 2, parse_json('{"key": "v2"}'),
1000, 'n2';
+```
+
+and zipping the table directory:
+
+```shell
+cd /path/to/table/parent/
+zip -r variant_schema_on_read_cow.zip variant_schema_on_read_cow
+```
+
+The rejection test asserts the internal schema is present, so after
regenerating, confirm
+`.hoodie/.schema/` in the zip contains completed schemacommit files.
Review Comment:
Confirmed, and the fixture proves it: its own first insert has the
`.hoodie/.schema/` schemacommit files and no `latest_schema`, so the old check
would have passed a table the assert rejects. README now points at the last
commit's extra metadata, with `strings <instant>_<completion>.commit | grep
latest_schema` as the one-liner (the commit is Avro). 88545658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
