raghuvanshraj commented on issue #21306: URL: https://github.com/apache/datafusion/issues/21306#issuecomment-6630630194
> In my mind what needs to happen here is: > > 1. Variant provides an impl of `PhysicalExprAdapter` to be hooked up to the parquet scan that reads from shredded columns when present in that particular file. > > 2. `variant_get` et al need to be marked to be pushed down into scans. @adriangb Agreed on both. **1. Variant adapter.** This is option 2 in #27089 (mixed shredded / unshredded files). I wrote it there as a new core hook. It is not needed: `PhysicalExprAdapterFactory::create` already receives the table schema and the file schema, and `ListingTableConfig::with_expr_adapter_factory` wires it to the scan. So the adapter is crate work. `datafusion-variant` ships a `VariantPhysicalExprAdapter` that wraps the default adapter, handles columns tagged `arrow.parquet.variant`, and passes everything else through. I will update #27089 accordingly. Two cases the adapter has to cover: - `variant_get(v, ...)`: rewrite to the leaves present in that file. The default struct cast never runs. - `SELECT v` (whole column): produce the table layout by reconstructing from `value`, not by null-filling `typed_value`. **2. Pushdown marking.** This is the capability in #25013 / #26282. Two requirements for whichever lands: one argument may need two leaves (typed leaf plus `value` fallback), and pruning must keep any container that holds a fallback row. If you have a preference between the two PRs, we build the `variant_get` side against it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
