rustyconover opened a new pull request, #979: URL: https://github.com/apache/arrow-nanoarrow/pull/979
> **Stacked on #928 and #968.** This branch is based on #968 as rebased onto #928 ([`feat/ipc-writer-write-array-view-with-dictionaries-on-928`](https://github.com/rustyconover/arrow-nanoarrow/tree/feat/ipc-writer-write-array-view-with-dictionaries-on-928)), so until those merge, the diff here includes their commits. Only the last commit, "Skip unchanged dictionaries in ArrowIpcWriterWriteArrayViewWithDictionaries", is new. I'll rebase once they land; it's a draft until then. `ArrowIpcWriterWriteArrayViewWithDictionaries()` (#968) writes every dictionary before every RecordBatch. Its input is borrowed, so it can't do what `ArrowIpcWriterWriteArrayStream()` does (#928): keep the previous array alive and compare dictionaries by buffer address. A producer that writes batches one at a time with the same dictionary therefore sends that dictionary again with every batch. In a file, where a dictionary ID may be written only once, it can't write a second batch with dictionaries at all. The writer now keeps a copy of each dictionary this function writes, keyed by dictionary ID, and compares the next batch's values with it: - First it checks whether the borrowed values are physically identical to the copy (`ArrowArrayViewCompare(..., NANOARROW_COMPARE_IDENTICAL, ...)`). This is the common case, e.g. when the producer builds each batch's dictionary the same way, and it needs no copy. - Otherwise it copies the values into the same canonical form as the saved copy (zero offset, buffers sized to the values, cleared bitmap padding bits, computed null count) and compares the two. A slice, an unknown null count or garbage in padding bits therefore doesn't make an unchanged dictionary look changed. - A dictionary whose values are unchanged isn't written. A changed one is written in full, as before (in a file that returns ENOTSUP, as before). - A dictionary whose values contain a nested dictionary that was written is written again even if its own values are unchanged, since a reader resolves a DictionaryBatch against the nested dictionaries current at that point. - Values that can't be copied (`ArrowArrayAppendStorageFromArrayView()` doesn't support dense unions) are written every time, as before. - `ArrowIpcWriterWriteDictionaryBatch()` drops the saved copy for its ID, so a dictionary written that way is never skipped by mistake, and `ArrowIpcWriterWriteSchema()` drops all of them. `ArrowIpcWriterWriteArrayStream()` is unchanged. The function's documentation now describes this, including the cost: a copy of each dictionary that is written. Cost: 1000 batches of 2048 rows, `dictionary<int32, string>` with 10,000 values, written to a memory buffer (Release, Linux aarch64; best of 3): | | #968 | this PR | |---|---|---| | Dictionary unchanged | 126 ms, 258.5 MB | 23 ms, 8.6 MB | | Dictionary grows by 16 values a batch | 205 ms, 458.3 MB | 531 ms, 458.3 MB | | Unchanged, 100 values | 6.4 ms, 11.0 MB | 5.1 ms, 8.4 MB | A dictionary that changes on every batch costs a copy per batch. The follow-up (rustyconover/arrow-nanoarrow `feat/ipc-writer-dictionary-deltas`) writes a dictionary that only grew as a delta, which brings that row to 346 ms and 9.2 MB. Tests (`writer_test.cc`): - `WriteArrayViewWithDictionariesSkipsUnchanged`: separately built arrays with dictionaries `[foo, bar], [foo, bar], [foo, baz], [foo, baz]` write two DictionaryBatches. Read back with nanoarrow and Arrow C++. - `WriteArrayViewWithDictionariesComparesValues`: an `int64` dictionary `[1, null, 3]`, then the same values as a slice of a longer array with an unknown null count and padding bits set, write one DictionaryBatch. - `WriteArrayViewWithDictionariesNested`: when only a nested dictionary changes, both it and the dictionary that refers to it are written (IDs `1, 0, 1, 0` over three batches). - `WriteArrayViewWithDictionariesAfterWriteDictionaryBatch`: a dictionary written with `ArrowIpcWriterWriteDictionaryBatch()` in between isn't mistaken for the saved copy. - `WriteArrayViewWithDictionariesUncopyableValues`: a dense union dictionary is written for every batch. - `WriteArrayViewWithDictionariesInFile` now writes a second, unchanged batch to the file, checks that a changed dictionary returns ENOTSUP, and reads the file back with Arrow C++. Against #968's `writer.c`, the first three new tests and the updated file test fail. `ctest` passes (Linux aarch64, Debug, 878 tests with `NANOARROW_BUILD_TESTS_WITH_ARROW`), and 403 tests under ASan/UBSan with clang. The writer tests are clean under valgrind. The changed files are formatted with clang-format 16.0.6, the pre-commit version. This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
