rustyconover opened a new pull request, #968: URL: https://github.com/apache/arrow-nanoarrow/pull/968
`ArrowIpcWriterWriteArrayStream()` writes the DictionaryBatch messages each array depends on, but only when it is given a whole `ArrowArrayStream`. A caller that produces batches one at a time and writes them with `ArrowIpcWriterWriteArrayView()` has no public way to get those dictionaries written. It would have to work out the dictionary IDs itself, matching the depth-first order the schema encoder assigns them in, and call `ArrowIpcWriterWriteDictionaryBatch()` for each one. This PR adds `ArrowIpcWriterWriteArrayViewWithDictionaries(writer, view, error)`. It writes a DictionaryBatch for each dictionary-encoded field of the view and then the RecordBatch. That is the same per-array step `WriteArrayStream()` already performs, and `WriteArrayStream()` now calls the new function, so there is one code path. The behavior is unchanged: every dictionary is written in full before every RecordBatch, and file mode has the same limits as `WriteArrayStream()`. The doc comment says so rather than promising anything beyond that. Things worth a look in review: - #928 changes how `WriteArrayStream()` writes dictionaries. It skips a dictionary that is unchanged from the previous batch, and writes nested dictionaries before their parent. Because it does that by keeping the previous `ArrowArray` alive and comparing buffer addresses, the skip can't apply here: this function only gets a borrowed view. The other #928 changes (dependency-first order, file-mode bookkeeping) would apply here too, because the two paths share the helper. Whichever lands second needs a small rebase in `writer.c` and in `MakeDictionaryStructArray`, which I've extended with the same `value1` parameter #928 adds. I can do that rebase. - `in` must not be NULL (a DCHECK, matching `ArrowIpcWriterWriteDictionaryBatch()`). EOS still goes through `ArrowIpcWriterWriteArrayView(writer, NULL, ...)`. Motivation: I'm moving the IPC layer of a DuckDB extension from Arrow C++ to nanoarrow. It writes one batch at a time to a worker process, and DuckDB `ENUM` columns arrive as dictionary arrays. This is the batch-at-a-time version of what `WriteArrayStream()` already does. Together with #967 (`ArrowIpcWriterSetMessageMetadata`), it covers what the extension needed from Arrow C++'s `RecordBatchWriter::WriteRecordBatch()`. Tests: - `WriteArrayViewWithDictionariesMatchesArrayStream` writes a dictionary-encoded batch through the new function and through `WriteArrayStream()`, and checks the bytes are identical. - `WriteArrayViewWithDictionariesPerBatch` writes two batches with different dictionaries one at a time. It checks the message sequence (schema, dictionary, batch, dictionary, batch) and that the reader resolves each batch against its own dictionary. ctest passes locally (macOS arm64, Debug, ZSTD/LZ4, 390 tests). One run hit a single failure in `NanoarrowIntegrationTestBatch`, which doesn't touch the writer. It didn't recur in isolation or in five more full runs. The IPC writer and reader tests are clean under ASan/UBSan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
