xiangfu0 opened a new pull request, #19073:
URL: https://github.com/apache/pinot/pull/19073
## What
Resolves #19071.
`AvroUtils.getAvroSchemaFromPinotSchema` and
`SegmentProcessorAvroUtils.convertPinotSchemaToAvroSchema` switched on
`fieldSpec.getDataType().getStoredType()`, so the generated Avro schema
described Pinot's **physical storage** rather than its **logical type**:
- `BOOLEAN` was emitted as Avro `int`
- `TIMESTAMP` was emitted as a bare `long` (no logical type)
- `UUID` needed a one-off special case
- `BIG_DECIMAL` fell through as unsupported
## Change
Drive schema generation on the original (logical) `DataType` via a single
shared mapping, `AvroSchemaUtil.toAvroSchema(DataType/FieldSpec)`:
| Pinot type | Avro type | Writer value |
|---|---|---|
| BOOLEAN | `boolean` | `Boolean` (coerced from stored `int` 0/1) |
| TIMESTAMP | `long{logicalType:timestamp-millis}` | `Long` epoch millis |
| BIG_DECIMAL | `bytes{logicalType:big-decimal}` | `BigDecimal` via
`BigDecimalConversion` |
| UUID | `string{logicalType:uuid}` | 16-byte `byte[]` via `UuidConversion` |
INT/LONG/FLOAT/DOUBLE/STRING/JSON/BYTES are unchanged.
`AvroUtils.getAvroSchemaFromPinotSchema` and the
`AvroSchemaUtil.toAvroSchemaJsonObject` JSON view now delegate to this one
mapping, dropping the previous one-off UUID branches.
Coordinate the writer/value side to match:
- Register Avro's `BigDecimalConversion` alongside the existing
`UuidConversion` on the shared data model
(`SegmentProcessorAvroUtils.getAvroDataModel`).
- Make `convertGenericRowToAvroRecord` schema-driven: coerce Pinot's stored
`int` 0/1 to `Boolean` for BOOLEAN fields and wrap `byte[]` as `ByteBuffer` for
BYTES, while leaving values a registered `Conversion` owns (UUID `byte[]`,
`BigDecimal`) untouched. MV values keep a zero-copy view unless their element
type actually needs a transform.
`SegmentProcessorAvroUtils.convertPinotSchemaToAvroSchema` keeps a private
copy of the mapping because `pinot-core` cannot depend on the `pinot-avro-base`
plugin (and vice-versa); a cross-module test pins the two together so they
can't drift.
## Compatibility
- Reverse mapping (`AvroSchemaUtil.valueOf(Schema)`) is deliberately left
one-way, preserving existing Pinot schema **inference** from Avro data.
- None of these schemas flow to ZK, segment metadata, or inter-node wire
protocols — the affected Avro/Parquet outputs are transient intra-process
buffers or self-describing export files, so there is no rolling-upgrade concern.
- **Output-format note for downstream consumers:** the `segment -> Avro` /
`segment -> Parquet` export tools now emit `boolean` (was `int`) for BOOLEAN
and `timestamp-millis` (was a bare `long`) for TIMESTAMP. Both stay
self-describing and decodable.
- The controller recommender's `AvroWriter` is intentionally **not**
touched: its data generator cannot produce `BIG_DECIMAL` values, so wiring the
conversion there would be untested/unreachable. The anonymizer's separate
`getAvroSchemaFromPinotSchema` is likewise out of scope — it has its own
value-generation model and fails loudly rather than silently mis-mapping.
## Tests
- Pinot-schema-to-Avro-schema assertions for every logical type,
single-value and multi-value (`AvroSchemaUtilTest`, `AvroUtilsTest`,
`SegmentProcessorAvroUtilsTest`).
- Cross-module consistency test pinning `SegmentProcessorAvroUtils`'s
mapping to `AvroUtils`'s.
- End-to-end write/read round-trips through the shared data model and the
production `AvroRecordReader`, exercising the same conversions as the
segment-processing writers (`SegmentProcessorAvroUtilsTest`), plus a full
segment-build round-trip (`FileBasedSegmentWriterTest`) and the Avro + Parquet
converters incl. `AvroParquetWriter.withDataModel`
(`PinotSegmentConverterTest`). BigDecimal scale preservation and MV `byte[]`
element-wise equality are asserted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
