linliu-code opened a new pull request, #668:
URL: https://github.com/apache/hudi-rs/pull/668
**Stacked on #639–#667** — review only the last commit.
## The bug
A decimal column in an Avro log block reads as **entirely NULL**.
```rust
fn resolve_decimal128(v: &Value) -> Option<i128> {
let bytes = match v {
Value::Bytes(b) => b,
Value::Fixed(_, b) => b,
_ => return None, // ← Value::Decimal lands here
};
```
A declared `logicalType: decimal` decodes to `apache_avro::Value::Decimal`,
not `Bytes` or `Fixed`. Confirmed by round-tripping the exact fixture schema
(`fixed(9)`, precision 20, scale 2):
```
DECODED VARIANT = Decimal
```
`None` there is indistinguishable from "this field was genuinely null", so
the column comes back empty rather than failing. Same silent-wrong-answer shape
as the map bug in #664 — and from the same converter.
## Why replace rather than patch
The vendored `avro_to_arrow` (adopted from DataFusion) has now produced two
silent wrong answers. `arrow-avro` handles both correctly out of the box, and
is maintained upstream. Verified against the fixture that has both problem
columns:
```
COL decimal_field type=Decimal128(20, 2) null_count=0/1
COL nullable_map_field type=Map(...) null_count=0/1
```
**No arrow upgrade** — `arrow-avro` is published at 57, matching the
workspace.
## Two shapes reconciled
**Framing.** A Hudi data block frames each datum with a four-byte length and
no Avro framing; `arrow-avro`'s decoder expects a Single Object Encoding
prefix. The writer schema is registered once, and the ten bytes that yields —
marker plus Rabin fingerprint — are written ahead of each body into a buffer
reused across records.
**Timezone spelling.** `arrow-avro` writes a UTC zone as `+00:00`; parquet
writes `UTC`. Same zone, but Arrow compares timezones as strings, so a log
batch refuses to concatenate with the base batch it merges with:
```
It is not possible to concatenate arrays of different data types
(Timestamp(µs, "UTC"), Timestamp(µs, "+00:00"))
```
Decoded batches are relabelled to `UTC`. The values are already UTC
instants, so nothing is converted.
## Also: it is columnar
The replaced path built an `apache_avro::Value` per record — an enum
allocating a `String` per string field and a `Vec` per collection — then walked
that to fill the columns. `arrow-avro` decodes into Arrow buffers directly. On
log-heavy merge-on-read reads that is the hot path.
## Scope
The vendored converter **stays for now**: delete blocks decode from values
that are already parsed, a different shape that this decoder doesn't take.
Moving that too is a follow-up, after which the converter can be deleted.
## Tests
Two previously-ignored harness cases un-ignored and passing
(`harness_all_data_types`, `harness_mixed_column_types`). No new tests: this
swaps an implementation, and the existing coverage — 19 gold fixtures compared
against Spark's own output, plus 58 harness cases — is what proves it.
Full workspace green: 1186 lib + 79 table-read + 39 datafusion + 21 + 12.
**Ignored 5 → 3**, and the remaining three are unrelated (schema promotion, an
HFile expectation, the gold gap list). Clippy clean in the changed files.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
