sdf-jkl opened a new issue, #11951: URL: https://github.com/apache/arrow-rs/issues/11951
### Is your feature request related to a problem or challenge? `shred_variant` and `ShreddedSchemaBuilder` let callers shred Variant values using an explicit schema. For semi-structured data whose shape is not known in advance, callers must decide which paths and types to extract themselves. A reusable schema-selection API would let applications automatically choose a layout for Parquet output without each implementing its own inference policy. ### Describe the solution you'd like Infer a logical target schema from a bounded sample of Variant values, suitable for passing to `shred_variant`. For example, a sample of events with string `service` and integer `status` fields could produce a target equivalent to `STRUCT(service Utf8, status Int64)`. The inference policy should consider field frequency and type compatibility, with limits on schema width, depth, and inference memory. Define how mixed types are handled—for example, selecting a common type versus leaving that path unshredded—and make the result deterministic for the same sample and options. Empty or all-null samples should have a defined fallback. For Parquet writing, callers would collect a bounded sample, select the layout before creating the writer, then apply it to the sampled and subsequent batches. The layout stays fixed for the file. Later fields and incompatible values remain in residual Variant storage; the sample must not constrain which logical values can be written. The API shape, default thresholds, and whether sampling belongs in a separate writer adapter are open for discussion. Inference should be usable independently of writer integration in #11950. ### Describe alternatives you've considered Use an explicit schema, leave values unshredded, or implement inference downstream. Explicit schemas remain useful when callers know which paths their workloads access; sampling cannot discover that information by itself. ### Additional context - [Existing shredding APIs](https://github.com/apache/arrow-rs/blob/a45acd6dfc871ebb0b62965a5b016d984589a60d/parquet-variant-compute/src/shred_variant.rs). - [DuckDB samples the first row group for Parquet shredding](https://duckdb.org/docs/2.0/sql/data_types/variant#writing-variant-to-parquet). - [Spark's inference implementation](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/InferVariantShreddingSchema.scala) considers field frequency, compatible types, and schema width/depth limits. - #11950 covers explicit writer-level configuration; [datafusion-variant #104](https://github.com/datafusion-contrib/datafusion-variant/issues/104) tracks exposing explicit shredding downstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
