wjones127 commented on issue #7845: URL: https://github.com/apache/arrow-datafusion/issues/7845#issuecomment-1769907740
> We could solve it in two ways: > 1. add Json/Jsonb type to Arrow, then support it in DataFusion natively > 2. only add json functions to DataFusion, treat utf8/binary as json/jsonb A third option (or maybe you meant this by either 1 or 2): have JSON and BSON be [extension types](https://arrow.apache.org/docs/format/Columnar.html#extension-types), so DataFusion could recognize them via field metadata. I think the Arrow community would be open to eventually have [canonical extension types](https://arrow.apache.org/docs/format/CanonicalExtensions.html) for these, given their prevalence in various formats, including Parquet. > If we use binary as BSON, we should consider a pleasure way to convert BSON data to other datatypes. I wonder if `json_extract_path` could return a [sparse union type](https://arrow.apache.org/docs/format/Columnar.html#sparse-union) of `[Utf8, Int64, Float64, Boolean, List, JSON]`. Then this type could be implicitly or explicitly casted to the expected type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
