Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
pitrou commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-4398237390 I've started a discussion in https://lists.apache.org/thread/r1yqc1fzsz2jwpokyysx708bj7cm6o2m -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
pitrou commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-4398094710 Hmm, I think this may warrant further discussion, perhaps on the dev ML. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
clbarnes commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-4213670639 Not to my knowledge - please go ahead! The simplest solution for ensuring only valid data makes it in to arrow buffers without breaking the API (metadata as a bytes->bytes map) is to just try to decode the bytes into a string before any write. But the API is wrong and so should be broken, which would mean on the python side you could expose a str->str map, even if you had to encode/decode these every time you read/write the arrow buffer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
nitrajen commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-4207448867 @clbarnes has this issue been looked at or had a PR? I'd like to take a look at it and try to contribute.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
clbarnes commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-3847858522 I don't know enough about the pyarrow internals to give any direction on how this would be implemented, and don't have enough cython/c++ background to make a lot of sense of what exists currently. It looks like C++ doesn't know or care about the encoding of a string, it just treats it as bytes, so maybe that's the problem. It would be stupid to have to decode the bytes in python every time you read and encode them again every time you write. Either way, this _is_ a bug, because Schema.fbs is the normative description and that requires UTF-8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
v4xsh commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-3847782929 Thanks for the clarification about mutation. I hadn't considered that angle. Before proposing a specific approach, I should probably dig into the codebase to understand: 1. Whether schema metadata is actually mutable after construction in PyArrow 2. Where the Python bytes → C++ conversion currently happens for metadata 3. How other similar validations are handled in the PyArrow bindings Would you be able to point me to the relevant source files where metadata is handled, or should I start by tracing through the Schema/Table construction code? I want to make sure I understand the current flow before proposing changes. If there isn’t a single obvious entry point, I’m happy to start by tracing through the Schema/Table construction paths myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
clbarnes commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-3847630178 Because metadata can be mutated, a construction-time check may not be sufficient. Ideally the internals would only _understand_ the metadata map as `str->str`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [Python] Non-UTF-8 bytes should be disallowed in custom_metadata [arrow]
v4xsh commented on issue #49058: URL: https://github.com/apache/arrow/issues/49058#issuecomment-3847443198 I’d like to work on this issue. From the Arrow spec / `Schema.fbs`, custom metadata keys and values are defined as strings, which implies valid UTF-8. Allowing arbitrary byte sequences in Python can therefore produce schemas that violate the spec and break cross-language interoperability (e.g. Rust, which enforces UTF-8 via `String`). I think it would make sense to validate that metadata keys and values are valid UTF-8 at schema construction time (or at least before serialization), and raise a clear error when invalid bytes are provided. I’m happy to implement this and add tests covering valid UTF-8, invalid byte sequences, and round-trip behavior. If this approach sounds reasonable, I’d be happy to take ownership of the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
