sdf-jkl opened a new issue, #12078:
URL: https://github.com/apache/arrow-rs/issues/12078
### Is your feature request related to a problem or challenge?
When writing shredded Variant data, I want to separate a selected object
field into its own column without prescribing its internal schema. For example:
```json
{"id": 123, "payload": {"arbitrary": [1, "x"]}, "debug": "..."}
```
A target equivalent to `STRUCT<id BIGINT, payload VARIANT>` would shred `id`
as an integer and store `payload` separately as Variant-encoded bytes. Omitting
`payload` from the target leaves it together with `debug` in the parent
residual value instead.
Currently, `shred_variant` recursively dispatches on each target field's
storage datatype, without recognizing the Variant extension annotation
([source](https://github.com/apache/arrow-rs/blob/23314565b83bc9e71cf723feaa5aa7a81f530c87/parquet-variant-compute/src/shred_variant.rs#L374-L383)).
A Variant-annotated struct therefore does not express an untyped child.
### Describe the solution you'd like
Allow shredding targets to designate an untyped object child, potentially
using a field annotated with `VariantType`. The exact API is open for
discussion.
The result should remain one logical `VariantArray`. In the example,
`typed_value.payload` would contain a binary `value` child and no `typed_value`
child. Its bytes would use the enclosing Variant's metadata, and `payload`
would not also appear in the parent residual. Missing fields and explicit
Variant nulls must remain distinct.
The [Parquet shredding
specification](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#objects)
permits object fields without a `typed_value` child.
### Describe alternatives you've considered
### Additional context
Existing implementations support this layout:
- Spark's [write-shredding
test](https://github.com/apache/spark/blob/e0c6db19f1d4583c25fb252efd586e02d920c623/sql/core/src/test/scala/org/apache/spark/sql/VariantWriteShreddingSuite.scala#L233-L236)
shreds an object with target `a variant, b variant`, expecting separate
value-only children.
- DuckDB 1.5.5 accepts `SHREDDING {data: 'STRUCT(id BIGINT, payload
VARIANT)'}`. A local write/read check confirmed a
`data.typed_value.payload.value` column with no corresponding `typed_value`,
preserving object and array payloads, explicit null, missing payload, and an
unrelated residual field. Its
[writer](https://github.com/duckdb/duckdb/blob/adfe841074da82be48e9d6c1403ad27cde386de8/extension/parquet/writer/variant/convert_variant.cpp#L96-L119)
explicitly omits typed children for Variant fields and list elements.
[#9519](https://github.com/apache/arrow-rs/issues/9519) and its closed,
unmerged [PR #9598](https://github.com/apache/arrow-rs/pull/9598) address
extraction: `variant_get` producing an Arrow struct with Variant-valued
children. This request covers the corresponding write-side behavior in
`shred_variant`, including shared metadata and residual placement. The
extension-type handling may be reusable, but the output contracts differ.
This came up while integrating explicit Parquet shredding in
[datafusion-variant
#145](https://github.com/datafusion-contrib/datafusion-variant/pull/145).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
