rustyconover opened a new pull request, #980: URL: https://github.com/apache/arrow-nanoarrow/pull/980
> **Stacked on #979** (and through it on #968 and #928). Until those merge, the diff here includes their commits. Only the last commit, "Write dictionary deltas from ArrowIpcWriterWriteArrayViewWithDictionaries", is new. It's a draft until they land. With #979, `ArrowIpcWriterWriteArrayViewWithDictionaries()` skips a dictionary that is unchanged since the last batch, but any change still replaces the whole dictionary. A common change is a dictionary that only grows: a producer encodes each batch against a dictionary that it appends new values to. Arrow streams can send just the appended values as a delta (`DictionaryBatch.isDelta`), which nanoarrow's reader supports since #955. When a batch's dictionary is longer than the one last written for its ID and starts with the same values, the writer now writes only the appended values, as a delta. To check this, it copies the first `n` values, where `n` is the length of the saved copy, and compares that with the saved copy. On a match, it copies the appended values and writes them as a delta, then appends them to the prefix copy, which becomes the new saved copy. Anything else is a replacement, as before. Deltas are only written to streams. In a file, a dictionary that grew is a replacement and returns ENOTSUP, matching #928, whose writer doesn't write deltas to files. A dictionary that is written again because a nested dictionary changed is written in full. Cost: the same benchmark as #979, 1000 batches of 2048 rows, `dictionary<int32, string>`, written to a memory buffer (Release, Linux aarch64; best of 3): | | #968 | #979 | this PR | |---|---|---|---| | 10,000 values, growing by 16 a batch | 205 ms, 458.3 MB | 531 ms, 458.3 MB | 346 ms, 9.2 MB | | 10,000 values, unchanged | 126 ms, 258.5 MB | 23 ms, 8.6 MB | 23 ms, 8.6 MB | Tests (`writer_test.cc`): - `WriteArrayViewWithDictionariesDeltas`: dictionaries `[foo, bar]`, `[foo, bar, baz]`, `[foo, bar, baz]`, `[foo, qux]` write a full dictionary, a delta, nothing, and a replacement (`isDelta` 0, 1, 0). Read back with nanoarrow and Arrow C++. - `WriteArrayViewWithDictionariesNullableDelta`: `int64` values `[1, null, 3]` growing to `[1, null, 3, null, 5]`, where the delta starts partway through a byte of the validity bitmap. - `WriteArrayViewWithDictionariesNoDeltaInFile`: in a file, a dictionary that grew returns ENOTSUP. `ctest` passes (Linux aarch64, Debug, 881 tests with `NANOARROW_BUILD_TESTS_WITH_ARROW`), and 406 tests under ASan/UBSan with clang. The writer tests are clean under valgrind. The changed files are formatted with clang-format 16.0.6, the pre-commit version. This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
