xiangfu0 opened a new pull request, #20030:
URL: https://github.com/apache/pinot/pull/20030
## What
Reads Apache Parquet `VARIANT(1)` columns during ingestion. A search of open
apache/pinot PRs and issues found no other work on ingesting the Parquet
Variant logical type.
## Changes
- `ParquetVariantConverter` detects top-level, non-repeated `VARIANT(1)`
groups, both unshredded and shredded. It reassembles shredded values through
parquet-java `VariantConverters` and emits the `PVAR` envelope. Unshredded
values are passed through without re-encoding.
- It hardens the reader against invalid producer data: a key that appears in
both the residual `value` and the shredded `typed_value` fails the row.
Malformed rows fail the current `next()` and the reader recovers on the
following row. Unknown type tags are kept opaque and rejected only when decoded.
- It works around a parquet-java 1.18.1 bug where
`ImmutableMetadata.getOrInsert` throws "Key does not exist in metadata" for
rows such as `{}`. The fix skips shredded object fields that have neither
`value` nor `typed_value` (`testShreddedObjectRowWithoutShreddedKeyInMetadata`).
- `ParquetNativeRecordExtractor`/`Reader`/`RecordReader` wiring. Automatic
reader selection keeps the existing Avro-metadata precedence, so a file with
Avro metadata must opt into the native reader explicitly to ingest Variant.
- An end-to-end test builds a real segment and ingests the same Variant into
a `JSON` column via `variantToJson(...)` and into an `OPEN_STRUCT` column via
`jsonStringToMap(variantToJson(...))`.
## Depends on
3/8 (segment storage) and 2/8 (`variantToJson` and `VariantUtils` in tests).
## About this PR
This is **part 4 of 8** splitting #19101 (end-to-end Parquet VARIANT
support) into layered PRs. #19101 stays open as the umbrella with the full
feature description;
[`docs/design/VARIANT.md`](https://github.com/apache/pinot/blob/xiangfu0/variant-split/01-spi-foundation/docs/design/VARIANT.md)
(added in 1/8) is the design reference.
The base branch is the previous PR in the stack, so the diff below is
exactly this layer. Each layer compiles and passes its own tests on top of the
one below it, and the top of the stack has the same tree as #19101.
> VARIANT is not usable end to end until the whole stack has landed.
Raw-value opacity guards arrive in 5/8 (single-stage) and 6/8 (multi-stage), so
please don't create VARIANT columns on a build that has only part of the stack.
The stack should land within one release.
### Full stack (merge bottom → top)
_filled in once all PRs are open_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
