voonhous opened a new issue, #19361:
URL: https://github.com/apache/hudi/issues/19361
### Task Description
**What needs to be done:**
`AvroDecimalConverter.convert` in `HudiAvroSerializer` builds and parses an
Avro schema for every decimal value it converts:
```java
BigDecimal convert(int precision, int scale, byte[] bytes)
{
Schema schema = new
Schema.Parser().parse(format("{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":%d,\"scale\":%d}",
precision, scale));
return AVRO_DECIMAL_CONVERSION.fromBytes(ByteBuffer.wrap(bytes), schema,
schema.getLogicalType());
}
```
(`hudi-trino-plugin/src/main/java/io/trino/plugin/hudi/util/HudiAvroSerializer.java`)
Call chain: `buildRecordInPage` -> `appendTo` invokes this for every
short-decimal `GenericData.Fixed` cell on the Avro record path (MOR merged
reads). Each decimal cell pays a `String.format` plus a full JSON parse and
`Schema` construction (tens of microseconds) for a conversion that is itself
nanoseconds. On decimal-heavy MOR reads this can dominate deserialization cost.
Fix: cache the `Schema` keyed by `(precision, scale)` (static map), or
precompute it per column. The precision/scale space is tiny and fixed per
column.
Two minor cleanups in the same hot loop, worth folding into the fix:
- `writeRow`: `field.getName().orElse("field" + index)` evaluates the concat
eagerly for every nested-row field even when the name is present; use
`orElseGet`. The name-based `record.get(name)` lookup per field could also use
precomputed positions.
- `buildRecordInPage`: `getFieldFromSchema(columnHandle.getName(),
record.getSchema()).pos()` does a per-record, per-column field lookup;
positions could be cached per schema.
**Why this task is needed:**
Per-cell schema parsing is a large constant-factor overhead on the Trino MOR
read path for decimal columns. Found in a `String.format`-on-hot-path audit;
the rest of the connector only formats on exception paths.
### Task Type
Performance improvement
### Parent
Part of #18780 (hudi-trino artifact migration); the serializer moves into
the `hudi-trino` artifact, so the fix should land on the Hudi side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
