alamb commented on PR #7403:
URL: https://github.com/apache/arrow-rs/pull/7403#issuecomment-2797228880

   Thank you for this PR @scovich 
   
   > It would be better to leverage a general variant library for variant 
bit-wrangling instead of doing it all manually here.
   
   - 100% agree. I think we'll do this as part of 
https://github.com/apache/arrow-rs/issues/6736. I am currently working on 
breaking up the work into some smaller pieces so we can get the code flowing
   
   > TBD Where/how to expose this functionality through a public API
   
   In my mind this functionality feels like a "computation kernel" (aka 
similarly to the functions in 
https://docs.rs/arrow/latest/arrow/compute/index.html)
   
   
   The signature seems like it would roughly be something like:
   
   ```rust
   /// Covert text stored as JSON in an input `StringArray`, `LargeStringArray` 
or `StringViewArray` into
   /// a single "Variant" array (`StructArray` with an extension type)
   fn json_to_variant(input: &ArrayRef) -> ArrayRef {
    ...
   }
   ```
   
   Since the arrow-json crate is currently for converting `JSON to arrow` it is 
not 100% clear to me that this functionality belongs in the `arrow-json` crate 
at all, espcially as variant is not part of the "core" arrow spec it seems.
   
   > Still TBD how to assemble a bunch of variant metadata+value pairs inside 
an arrow array data that can eventually become a usable arrow array
   
   I think we will sort this out as part of implementing varint in 
https://github.com/apache/arrow-rs/issues/6736. TLDR is via a `StructArray` 
annotated with an extension type I think. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to