ad1happy2go commented on issue #19654: URL: https://github.com/apache/hudi/issues/19654#issuecomment-5343683844
To be precise on framing: this is **not a schema-compatibility regression**. The 1.x behavior — rejecting a newly added **non-nullable field with no default** because existing records can't be reconciled against it — is correct Avro semantics, and I wouldn't weaken that validation. Apologies for the "regression" wording in my earlier comment. What actually changed is a **convenience/tooling capability**: `hoodie.datasource.write.new.columns.nullable` used to let a DataFrame write **auto-relax newly added columns to nullable** (with a `null` default) when the incoming DataFrame typed them as non-nullable — which Spark commonly does (e.g. a column built via `.otherwise(...)`). That option was removed by #10782, so on 1.x the same DataFrame now hits the (correct) compatibility check. Two things worth noting: - Users can **already succeed on 1.x today** by adjusting the incoming DataFrame — make the new columns nullable / give them a default — so there's no data-correctness or compatibility problem, just a lost bit of automatic schema-evolution ergonomics for the "don't specify a schema, infer from the DataFrame" flow. - #19665 restores this as an **opt-in** capability (via `hoodie.write.set.null.for.missing.columns`, with `new.columns.nullable` accepted as an alias), without weakening the default validation. **Verified #19665** on a local 1.3.0-SNAPSHOT build with the exact `generateSampleDf` + `withPhoneAndCountry` repro: - Failing config + `set.null.for.missing.columns=true` → schema evolves (`phone` + nested `address.country`); a record absent from the evolve batch reads back with `phone=null`. - Legacy `new.columns.nullable=true` alias honored. - Without the flag the write still throws — correct default preserved. - Works across no-reconcile, legacy-reconcile, and schema-on-read paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
