rustyconover opened a new pull request, #924:
URL: https://github.com/apache/arrow-nanoarrow/pull/924
Closes #923.
Adds encoder and decoder support for `Message.custom_metadata` — the
per-message
key/value metadata that is separate from the Schema/Field metadata carried
in the schema.
### Encoder
```c
ArrowErrorCode ArrowIpcEncoderSetMessageMetadata(struct ArrowIpcEncoder*
encoder,
const char* metadata,
struct ArrowError* error);
```
`metadata` is nanoarrow's packed representation (the same as
`ArrowSchema.metadata`, so
`ArrowMetadataBuilder*` can produce it and no new representation is
introduced). It is
copied, applies to the *next* message encoded by
`ArrowIpcEncoderEncodeSchema()` or
`ArrowIpcEncoderEncodeSimpleRecordBatch()`, and is cleared once that message
is encoded.
`NULL` (or metadata with no keys) clears it, and no `custom_metadata` field
is written at
all in that case, so the encoded bytes are unchanged for callers that don't
use this.
`ArrowIpcEncodeMetadata()` now takes the packed `const char* metadata`
instead of a
`const struct ArrowSchema*` so it is reusable from the Message path;
behaviour at the
Schema/Field call sites is unchanged.
### Decoder
```c
// Copy, using the same packed representation
ArrowErrorCode ArrowIpcDecoderGetMessageMetadata(struct ArrowIpcDecoder*
decoder,
struct ArrowBuffer* out,
struct ArrowError* error);
// Zero copy: both borrow from the message header until the next header is
decoded
ArrowErrorCode ArrowIpcDecoderGetMessageMetadataValue(struct
ArrowIpcDecoder* decoder,
struct ArrowStringView
key,
struct
ArrowStringView* value_out,
struct ArrowError*
error);
ArrowErrorCode ArrowIpcDecoderVisitMessageMetadata(struct ArrowIpcDecoder*
decoder,
ArrowIpcMetadataVisitFunction visit,
void* private_data,
struct ArrowError* error);
```
Per @paleolimbot's review comment on the issue, the copying accessor writes
to a
`struct ArrowBuffer*` (what `ArrowMetadataBuilder*` builds), and the
get-by-name and
visit-in-place variants avoid the copy for large metadata. If the message
has no
`custom_metadata`, `out` is left empty so `(const char*)out->data` is `NULL`
and can be
passed straight to `ArrowSchemaSetMetadata()` / `ArrowMetadataReaderInit()`.
A missing key
leaves `value_out` unmodified, matching `ArrowMetadataGetValue()`.
The metadata is recorded by both `ArrowIpcDecoderVerifyHeader()` and
`ArrowIpcDecoderDecodeHeader()` and applies to any message type, since
`Message.custom_metadata` is a field of `Message` rather than of
`RecordBatch`.
### Tests
- `encoder_test.cc`: round trip of message metadata through encoder →
decoder for both
RecordBatch and Schema messages, that Schema-message metadata stays
distinct from the
contained schema's own metadata, that the metadata applies to exactly one
message, that
empty/`NULL`/cleared metadata produces byte-identical output to not
calling the setter,
and that an erroring visitor stops the visit and propagates its error.
- `decoder_test.cc`: interop with Arrow C++ in both directions —
`arrow::ipc::ReadMessage()`
reads back what nanoarrow writes, and nanoarrow reads the metadata written
by
`RecordBatchWriter::WriteRecordBatch(batch, custom_metadata)`.
Verified locally against Arrow C++ 24.0.0
(`-DNANOARROW_BUILD_TESTS_WITH_ARROW=ON`).
### Out of scope
`ArrowIpcArrayStreamReader` / `ArrowIpcWriter` are unchanged:
`ArrowArrayStream` has
nowhere to put per-batch metadata, so as discussed in the issue this first
pass exposes the
feature only at the encoder/decoder layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
