rustyconover commented on issue #622: URL: https://github.com/apache/arrow-nanoarrow/issues/622#issuecomment-6220930016
Status update from running archery's dictionary cases against nanoarrow while working on #972. #926 added DictionaryBatch encoding, but two gaps still keep nanoarrow out of IPC integration for every dictionary case, including the existing `dictionary` case: - **The file writer never fills `footer.dictionaries`.** Writing a file whose schema has a dictionary field fails in `ArrowIpcEncoderEncodeFooter` with `Unexpected missing dictionary encoding for field`. That breaks archery's `JSON_TO_ARROW` and `STREAM_TO_FILE` steps. Dictionary batch blocks would also need recording in `footer.dictionary_blocks`. - **The integration driver doesn't emit DictionaryBatches.** It writes batches with `ArrowIpcWriterWriteArrayView`, which writes only the record batch. Even with the footer filled in, readers fail with "Expected dictionary index … < 0". C Data integration is unaffected: C++ ⇄ nanoarrow passes the `dictionary` case, and with #972 the new `dictionary_extension` case as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
