raghuvanshraj commented on issue #27089: URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6918657633
@alamb one question for core, separate from the repro above. Today the fix for this bug lives in `VariantExprAdapterFactory` (datafusion-contrib/datafusion-variant#149), and it has to be installed on each table with `ListingTableConfig::with_expr_adapter_factory`. Any table built outside that path gets the default adapter and the wrong answer again. The url table above is one case (datafusion-contrib/datafusion-variant#177); a user's own `ListingTable` is another. The adapter is really about the type, not the table. Variant is a registered extension type (`arrow.parquet.variant`). Could core let an extension type supply its own `PhysicalExprAdapterFactory`, so any scan that sees a Variant column applies it without the table owner knowing? @adriangb you raised the other shape: fix how the table schema is merged for shredded Variant. I am fine with either. I can open an issue with a proposed shape once we pick one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
