ad1happy2go commented on issue #19654:
URL: https://github.com/apache/hudi/issues/19654#issuecomment-5324608137
Hi @nada-attia, thanks for the report. I tried to reproduce this. I ran a
controlled comparison holding Spark fixed and varying only the Hudi bundle
(0.14.0 and 1.2.0), each with `SET_NULL_FOR_MISSING_COLUMNS=true`:
| Added column | 0.14.0 | 1.2.0 |
|---|---|---|
| top-level **nullable** | ✅ evolves | ✅ evolves |
| nested **nullable** (`address.country`, struct rebuilt) | ✅ evolves | ✅
evolves |
| top-level **non-nullable** | ❌ rejected | ❌ rejected |
| nested **non-nullable** (`address.country`) | ❌ rejected | ❌ rejected |
Two takeaways:
1. A **backward-compatible (nullable) column add still evolves the schema
implicitly in 1.2.0** — no `hoodie.schema.on.read.enable` required, identical
to 0.14.
2. Your `withPhoneAndCountry` transform adds **non-nullable** fields —
`phone` via `.otherwise("000-000-0000")` and nested `address.country` via
`.otherwise("Unknown")`. A non-nullable field with no default can't be
reconciled against already-written records, so the write is rejected in **both
0.14.0 and 1.2.0**. The behavior is the same across versions; only the error
surface changed — 0.14 throws a generic `HoodieUpsertException: Failed upsert
schema compatibility check`, while 1.2 fails earlier with a clearer message
(`READER_FIELD_MISSING_DEFAULT_VALUE` for the top-level field,
`MISSING_UNION_BRANCH` for the nested one). `SET_NULL_FOR_MISSING_COLUMNS=true`
made no difference to new-column adds in any of my runs.
If the new columns are made nullable (default null), implicit evolution
works in 1.2.0 in my testing — including when the nested struct is rebuilt with
`struct(...)`.
To confirm whether you're seeing a true behavior change, could you share:
1. Exact Hudi + Spark versions for both the passing (0.14) and failing (1.2)
runs.
2. Full write configs used (esp. `hoodie.schema.on.read.enable`,
`hoodie.datasource.write.reconcile.schema`, `hoodie.avro.schema.validate`,
operation type).
3. The exact SQL / DataFrame write call that performs the evolution, plus a
sample row before and after.
4. Confirmation the identical test actually passed on 0.14 (and which
0.14.x) — ideally the assertion output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
