1fanwang opened a new pull request, #12323: URL: https://github.com/apache/arrow-rs/pull/12323
# Which issue does this PR close? - Closes https://github.com/apache/arrow-rs/issues/12320. # Rationale for this change A JSON string that holds a surrogate pair for a character above U+1FFFF, such as \ud840\udc00 for U+20000, was read as a different character (U+10000) without any error. Pairs for characters up to U+1FFFF were read correctly. The decoder built the code point by OR-ing the shifted high surrogate with the low surrogate plus 0x10000. The offset has to be added to the whole value. OR drops it whenever the shifted high surrogate already has that bit set, which is every high surrogate from D840 up, so the result came out 0x10000 too small. # What changes are included in this PR? The tape decoder's surrogate pair function adds the offset instead of OR-ing it in. # Are these changes tested? Yes. A new reader test parses pairs for U+10000, U+1F600, U+20000, U+E0001 and U+10FFFF into a Utf8 column. It fails on main, which returns U+10000 where U+20000 is expected, and passes with this change. Actions is not enabled on my fork, so the informational CI (fork) check has no run to mirror. The [contributing guide](https://github.com/apache/arrow-rs/blob/main/CONTRIBUTING.md#pull-requests-from-forks) says the merge queue runs the required checks. ## Testing Done The reproducer below runs the table from the issue through the public ReaderBuilder, saved as arrow-json/examples/surrogate_repro.rs. <details> <summary>Reproducer source: surrogate_repro.rs</summary> ```rust use std::io::Cursor; use std::sync::Arc; use arrow_array::cast::AsArray; use arrow_json::ReaderBuilder; use arrow_schema::{DataType, Field, Schema}; const CASES: [(&str, u32); 8] = [ (r"\ud800\udc00", 0x10000), (r"\ud801\udc37", 0x10437), (r"\ud83d\ude00", 0x1F600), (r"\ud83d\udca9", 0x1F4A9), (r"\ud840\udc00", 0x20000), (r"\ud87e\udc00", 0x2F800), (r"\udb40\udc01", 0xE0001), (r"\udbff\udfff", 0x10FFFF), ]; fn main() { let schema = Arc::new(Schema::new(vec![Field::new("s", DataType::Utf8, false)])); for (escaped, expected) in CASES { let json = format!(r#"{{"s": "{escaped}"}}"#); let mut reader = ReaderBuilder::new(schema.clone()) .build(Cursor::new(json)) .unwrap(); let batch = reader.next().unwrap().unwrap(); let actual = batch.column(0).as_string::<i32>().value(0).chars().next().unwrap() as u32; let verdict = if actual == expected { "ok" } else { "WRONG" }; println!("{escaped:<14} expected U+{expected:X} got U+{actual:X} {verdict}"); } } ``` </details> On main (https://github.com/apache/arrow-rs/commit/eb62e55752b8376bcedc926e2e37547828ec82f9), from the checkout root: ``` $ cargo run -q -p arrow-json --example surrogate_repro \ud800\udc00 expected U+10000 got U+10000 ok \ud801\udc37 expected U+10437 got U+10437 ok \ud83d\ude00 expected U+1F600 got U+1F600 ok \ud83d\udca9 expected U+1F4A9 got U+1F4A9 ok \ud840\udc00 expected U+20000 got U+10000 WRONG \ud87e\udc00 expected U+2F800 got U+1F800 WRONG \udb40\udc01 expected U+E0001 got U+D0001 WRONG \udbff\udfff expected U+10FFFF got U+FFFFF WRONG ``` With this change: ``` $ cargo run -q -p arrow-json --example surrogate_repro \ud800\udc00 expected U+10000 got U+10000 ok \ud801\udc37 expected U+10437 got U+10437 ok \ud83d\ude00 expected U+1F600 got U+1F600 ok \ud83d\udca9 expected U+1F4A9 got U+1F4A9 ok \ud840\udc00 expected U+20000 got U+20000 ok \ud87e\udc00 expected U+2F800 got U+2F800 ok \udb40\udc01 expected U+E0001 got U+E0001 ok \udbff\udfff expected U+10FFFF got U+10FFFF ok ``` Only macOS arm64 was run. CI covers the rest. # Are there any significant user-facing changes? No. # AI usage disclosure AI disclosure: GitHub Copilot CLI (Claude Sonnet 5.5) wrote this change and its tests, and an automated coordinator reviewed the diff and test output before submission. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
