nsivabalan commented on PR #19205: URL: https://github.com/apache/hudi/pull/19205#issuecomment-5081738453
One behavior change worth calling out explicitly rather than letting it ride in as a test-assertion update, since it affects tables that never opted into this feature: The nullable meta-column stub fix ([`77f4e91`](https://github.com/apache/hudi/pull/19205/commits/77f4e9148c4a), `HoodieDatasetBulkInsertHelper.scala`) switches the row-writer bulk-insert stubs from `Literal(UTF8String.EMPTY_UTF8)` to `Literal.create(null, StringType)`. The change is required — the columns must be `OPTIONAL` in Parquet for selective / NONE modes to hold nulls — but it also applies to **existing** `hoodie.populate.meta.fields=false` tables, whose meta columns will now materialize as SQL `NULL` instead of `""`. Downstream consumers filtering on `col = ''` would silently stop matching. The three updated assertions in `TestHoodieSparkSqlWriter`, `TestHoodieSparkSqlWriterWithTestFormat` and `TestHoodieDatasetBulkInsertHelper` are the evidence this is observable end-to-end. Flagging so it's a deliberate sign-off rather than a surprise, and so it can go in the release notes. Happy to gate it behind the mode (keep `""` for plain `populate.meta.fields=false`, null only under a selective mode) if reviewers would rather not change existing-table output at all — though that would mean two different null representations for the same logical "unpopulated" state, which seems worse to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
