rustyconover opened a new pull request, #981: URL: https://github.com/apache/arrow-nanoarrow/pull/981
`NANOARROW_VALIDATION_LEVEL_FULL` is documented as the level for untrusted input, but it never checked that string values are valid UTF-8. Arrow C++'s `ValidateFull()` does, so a consumer that relies on FULL to reject bad input accepts strings that Arrow C++ rejects. FULL validation now checks that every non-null `string` and `large_string` value is valid UTF-8 (RFC 3629). Overlong encodings, surrogates (U+D800 to U+DFFF), code points above U+10FFFF, stray continuation bytes and truncated sequences are rejected. Binary types and the levels below FULL are unchanged. `string_view` is left for a follow-up PR, because its out-of-line values can't be read safely until FULL also checks view bounds. **FULL now rejects data it used to accept.** That's the point of the change, but a caller that validated non-UTF-8 bytes in a string array at FULL will now get `EINVAL`. The behaviour follows Arrow C++ (checked against pyarrow 25.0.1): - Each value is checked on its own, so a code point split across two values is invalid even though their concatenation is valid. - Null values are not checked. - Only the values within a slice are checked, and error indices are relative to the slice. Errors name the value, e.g. `[1] Expected valid UTF-8 string value but found invalid sequence at byte 2`. The validator is portable C with no new dependency. It skips ASCII eight bytes at a time. It first validates the whole referenced byte range in one pass and checks that each value starts on a code point boundary (not a continuation byte). That is equivalent to checking each value, but avoids a call per value for short strings. Only if that fails does it check the non-null values one at a time to report the first invalid one. Cost of FULL validation over 10 MB, compared with `main` and with Arrow C++ `ValidateFull()` (pyarrow 25.0.1) on the same machine (Linux aarch64, Neoverse-N1, Release build): | Array | `main` | This PR | Arrow C++ | |---|---|---|---| | string, 1M × 10 B ASCII | 0.8 ms | 3.1 ms (3.3 GB/s) | 7.8 ms | | string, 1M × 10 B ASCII, 10% null | 0.8 ms | 3.0 ms | — | | string, 1M × 10 B two-byte | 0.8 ms | 16.1 ms (0.62 GB/s) | 16.6 ms | | string, 10k × 1000 B ASCII | 0.01 ms | 1.1 ms (9.2 GB/s) | 1.1 ms | | string, 10k × 1000 B mixed ASCII/two-byte | 0.01 ms | 19.1 ms (0.52 GB/s) | 9.0 ms | | large_string, 1M × 10 B ASCII | 0.8 ms | 3.1 ms | — | | binary, 1M × 10 B | 0.8 ms | 0.8 ms | — | Tests: `ArrayViewTestValidateUtf8` covers string and large_string. It checks valid one- to four-byte sequences, each class of invalid sequence above (with and without a null in the array), split code points, invalid bytes in null values, slices, that DEFAULT still accepts invalid UTF-8, and that binary types are not checked. `ctest` passes (Linux aarch64, Debug, 870 tests; `NanoarrowIntegrationTestSchema` hits the known shared-temp-file collision under `ctest -j` fixed by #969 and passes when run alone), including under ASan/UBSan with clang (395 tests). The changed files are formatted with clang-format 16.0.6, the pre-commit version. This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
