rustyconover opened a new pull request, #964: URL: https://github.com/apache/arrow-nanoarrow/pull/964
`ArrowIpcArrayStreamReader` always decodes with `NANOARROW_VALIDATION_LEVEL_FULL`. That's the right default for bytes of unknown origin. But full validation walks every offset of every string, binary and list column and every dictionary index, and a caller that already trusts its producer can't opt out of that pass. The lower-level decoder functions already take a validation level; the stream reader just doesn't expose one. This adds a `validation_level` field to `ArrowIpcArrayStreamReaderOptions`. It applies to both record batches and dictionary batches. Passing `NULL` options keeps the current behaviour (`FULL`). An out-of-range value makes `ArrowIpcArrayStreamReaderInit()` return `EINVAL`. Motivation: I'm moving a WebAssembly build of DuckDB from Arrow C++ to nanoarrow. There, Arrow IPC streams written by Arrow JS in the same page are read into the database. Arrow C++'s reader doesn't do full validation by default, and the extra pass was a large part of an insert slowdown I measured after switching. Reading a 23 MB stream from an in-memory buffer (1M rows of int32, double and string, in 16 batches), Release build, Apple M3, median of 50 runs: | validation level | time | |---|---| | `FULL` (current behaviour) | 0.66 ms | | `DEFAULT` | 0.38 ms | The float64-only fixtures from `dev/benchmarks` don't change, because full validation has nothing extra to check for them. Compatibility: callers that fill in `ArrowIpcArrayStreamReaderOptions` field by field need to set the new field as well. That's the same situation as when `use_shared_buffers` was added in #165. The R and Python bindings pass `NULL`, so they're unaffected. The `EINVAL` check should catch most callers that miss the new field, rather than letting them silently get a random level. Tests: `StreamReaderValidationLevel` points a dictionary index past the end of the dictionary in the existing dictionary fixture, then checks that `FULL` rejects the batch and `DEFAULT` accepts it. `StreamReaderInvalidValidationLevel` covers the `EINVAL` path. The two existing tests that build options by hand now set the field. ctest passes locally (macOS arm64, Debug, extra warnings). Under ASan/UBSan the IPC tests pass, and the only failures are the same six allocation tests that also fail on `main` under ASan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
