codope opened a new pull request, #20305:
URL: https://github.com/apache/hudi/pull/20305
### Describe the issue this Pull Request addresses
Closes #20299
Spark CDC `before`/`after` JSON images render DECIMAL, DATE and TIMESTAMP
columns wrongly:
| Path | DECIMAL | DATE | TIMESTAMP |
|---|---|---|---|
| Images built by `InternalRowToJsonStringConverter`: MOR inference,
`op_key_only` / `data_before`, native CDC log files (table version 10) |
`{"zero":false}` (Spark's `Decimal` serialized as a Jackson bean) | `20739`
(epoch days) | `1791865800000000` (epoch micros) |
| Inline Avro CDC log files (table version ≤ 9, `data_before_after` /
`data_before`), rendered with `GenericData.toString` | `[0, 0, 3, -25]` (raw
unscaled bytes) | `20739` | `1791865800000000` |
The decimal value is unrecoverable from either rendering. The same table can
also give different
image strings depending on which path served the record, including different
whitespace.
### Summary and Changelog
CDC images now render DECIMAL, DATE and TIMESTAMP as readable values, and
every CDC path produces
the same image string for the same record. For example:
`{"id":1,"amt":9.99,"d":"2026-10-13","ts":"2026-10-13T04:30:00Z","qty":2}`.
- `InternalRowToJsonStringConverter`:
- `DecimalType`: an exact JSON number (`BigDecimal`, written in plain
notation so scale is kept
and there is no exponent, e.g. `0.00000012`, `1.20`).
- `DateType`: an ISO-8601 date (`2026-10-13`).
- `TimestampType`: an ISO-8601 UTC instant at full precision
(`2026-10-13T17:00:00.123456Z`). It
does not depend on the reader's session time zone.
- `TimestampNTZType`: an ISO-8601 local date-time without an offset
(`2026-10-13T10:00:00`). It
is matched by type name, like the existing VARIANT case, because the
type is absent from some
Spark 3 versions this module compiles against.
- These also apply inside arrays, maps and structs.
- `CDCFileGroupIterator`: images read from inline Avro CDC log files are
deserialized to
`InternalRow` (with the CDC image schema the log is read with) and
rendered by the same converter
as every other path. `nativeCdcImageConverter` is renamed
`cdcImageConverter`, since it now
serves both.
- `HoodieCDCUtils.recordToJson` has no remaining callers and is removed.
- Tests:
- `TestInternalRowToJsonStringConverter`: decimal (including a small value
with scale 8 and a
decimal array), date, timestamp with and without micros, and
timestamp_ntz (Spark ≥ 3.4).
- `TestCDCDataFrameSuite#testCDCImagesWithLogicalTypes`: COW and MOR × all
three supplemental
logging modes × table versions 9 (inline Avro CDC logs) and 10 (native
CDC logs). Each case
asserts the exact before/after image strings, so all 12 combinations
must produce the same
image.
### Impact
User-facing change in the content of the CDC `before`/`after` columns.
**Please call this out in the
release notes:**
- DATE values change from epoch days (`20739`) to ISO strings
(`"2026-10-13"`).
- TIMESTAMP values change from epoch micros to ISO-8601 UTC strings
(`"2026-10-13T04:30:00Z"`).
- DECIMAL values change from `{"zero":false}` or byte arrays to JSON numbers.
- For tables with inline Avro CDC logs (table version ≤ 9), images also
change from the
`GenericData.toString` layout (`{"id": 1, ...}`, with spaces) to the
compact layout the other
paths already use. BINARY and FIXED fields render as base64 strings, as
they already do on the
other paths.
Consumers that parse the old numeric date/timestamp values need to switch to
parsing ISO-8601
strings. `get_json_object` and `from_json` with DATE/TIMESTAMP/DECIMAL
fields now return the
correct values.
### Risk Level
low. The change is limited to how CDC images are serialized to JSON. Data
files and CDC log files
are unchanged.
Verification (JDK 11, `-Pspark3.5`):
- The new tests fail on master: the 2 converter unit tests, and all 12
functional cases. The v9 COW
cases show the raw-byte rendering; the rest show `{"zero":false}`.
- With the fix:
| Suite | Result |
|---|---|
| `TestInternalRowToJsonStringConverter` | 18 run, 4 Spark 4 variant tests
skipped |
| `TestCDCDataFrameSuite` | 49 passed |
| `TestCDCStreamingSuite` | 3 passed |
| `TestCDCForSparkSQL` | 4 passed |
- The #20299 repro on a bundle built from this branch gives the identical
image above for COW and
MOR at table versions 9 and 10.
### Documentation Update
The release notes should mention the new image format. No config is added or
changed.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
