alamb commented on PR #8998: URL: https://github.com/apache/arrow-rs/pull/8998#issuecomment-3656951715
Thank you for this PR @debugmiller > but I was not sure if it was intentional that parquet-variant-compute included from arrow directly. I think this was a convenience rather than anything deliberate I agree that using the existing json reader to read variant is a great(!!) idea and I suspect it will be much faster than the `json_to_variant` kernel as well. I actually think long term we would like to change the implementation of `json_to_variant` to use the arrow JSON reader. Another alternative to rejiggering the dependencies, could be to allow users do provide their own decoders for certain fields. This is probably overkill for just variant, it would also make a nice API for other potential extension types (like some of the geospatial types from @kylebarron and @paleolimbot ) Similar to how [`with_encoder_factory`](https://docs.rs/arrow-json/57.1.0/arrow_json/writer/struct.EncoderOptions.html#method.with_encoder_factory) works for overriding writing of JSON fields, we could add a `with_decoder_factory` / `DecoderFactory` for customizing decoding and then provide a decoder factory implementation in `parquet-variant-compute` Would you be willing to consider this approach? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
