phil-opp commented on issue #8684: URL: https://github.com/apache/arrow-rs/issues/8684#issuecomment-3433388494
Thanks for the quick reply! > This crate exists to serve the needs of the arrow integration tests, it isn't meant for usage beyond this. Perhaps some kind of disclaimer that specifies this would be nice? Right now the crate description specifies: > Support for the [Apache Arrow JSON test data format](https://github.com/apache/arrow/blob/master/docs/source/format/Integration.rst#json-test-data-format) The linked document even describes all the data types that are supported in that format. I don't see the limitation to `int8` anywhere and neither do I see a "please don't use this crate or format" note. Given that our use case is integration testing with arrow input as well, the crate seemed like the logical choice. > With regards to binary data, arrow_json does support binary data by hex encoding it. Could you give some more details on this? The [`arrow-json` documentation](https://docs.rs/arrow-json/latest/arrow_json/#binary-data) specifies quite explicitly that binary data is not supported: > ## Binary Data > As per [RFC7159](https://datatracker.ietf.org/doc/html/rfc7159#section-8.1) JSON cannot encode arbitrary binary data. A common approach to workaround this is to use a [binary-to-text encoding](https://en.wikipedia.org/wiki/Binary-to-text_encoding) scheme, such as base64, to encode the input data and then decode it on output. After checking the code is seems to be able to convert binary Arrow arrays to JSON. But not the other way around, i.e. JSON data to binary Arrow arrays: https://github.com/apache/arrow-rs/blob/a7572eb605c0314befa5ef6ffbe9e430127f27bb/arrow-json/src/reader/mod.rs#L746-L748 Or am I missing anything? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
