Kasra-G opened a new issue, #12320:
URL: https://github.com/apache/arrow-rs/issues/12320
### Describe the bug
Hi. Not sure if right repo to report this, but encountering some data
corruption with surrogate pairs. Can't quite understand the pattern, but some
surrogate pairs seem to be stored incorrectly
Thanks
### To Reproduce
1. Make an arrow-json `ReaderBuilder` with a schema that has one `Utf8`
field, `s`.
2. Parse this json line:
```json
{"s": "\ud840\udc00"}
```
3. Read back `s`
I was expecting `𠀀` (U+20000), UTF-8 bytes `F0 A0 80 80`.
However, got back `𐀀` (U+10000), UTF-8 bytes `F0 90 80 80`.
### Expected behavior
I expect arrow json to not silently change my unicode
### Additional context
Version: arrow-json 59.3.2
I'm not able to test latest arrow json to see if bug still there, maybe
someone with Claude can check that
I checked a few more and wasn't quite able to determine a pattern.
| Input | Expected | Actual |
|---|---|---|
| `"\ud800\udc00"` | U+10000 | U+10000 |
| `"\ud801\udc37"` | U+10437 | U+10437 |
| `"\ud83d\ude00"` | U+1F600 | U+1F600 |
| `"\ud83d\udca9"` | U+1F4A9 | U+1F4A9 |
| `"\ud840\udc00"` | U+20000 | U+10000 |
| `"\ud87e\udc00"` | U+2F800 | U+1F800 |
| `"\udb40\udc01"` | U+E0001 | U+D0001 |
| `"\udbff\udfff"` | U+10FFFF | U+FFFFF |
It seemed everything above U+1FFFF was wrong but below it seemed to be fine
### Willingness to contribute
I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]