rustyconover commented on PR #965: URL: https://github.com/apache/arrow-nanoarrow/pull/965#issuecomment-6219483999
Thanks, that makes sense: callbacks on the input stream are a better fit than special-casing the buffer stream, and they cover files too. Two questions before I rework it. **How should the struct gain the callbacks?** `ArrowIpcInputStream` is allocated by callers who fill in only `read`, `release` and `private_data`. The R package does this: `r/src/ipc.c` `ArrowMalloc`s the struct and sets `read`/`release` one at a time. If `peek_at`/`read_at` are appended to the struct, those streams (and any out-of-tree ones) will have garbage in the new members, and the reader would call them. Options I can see: 1. Append the members, document that unsupported callbacks must be `NULL`, and update R/Python in this PR. That's simple, but out-of-tree implementations silently get undefined behaviour until they're updated. 2. Keep `ArrowIpcInputStream` as it is and add a separate struct for random access (`peek_at`, `read_at`, `release`, `private_data`). The buffer and file inputs implement both, and a reader init function takes the new one. That's more API, but nothing existing changes. My preference is (2), unless nanoarrow is fine with breaking out-of-tree streams in a release. **Semantics.** For `peek_at`, I'd make the returned pointer valid until the next call on the stream. For `read_at`, the `ArrowBuffer` would be a shared slice when the bytes are already in memory, and a newly read buffer otherwise. Both would advance the stream position only when `offset_from_current` is 0 and the full length is consumed. Does that match what you had in mind, or should offsets be absolute? **Alignment:** agreed. I'd write `ArrowArrayEnsureAligned()` first as its own small core PR: walk the array and copy any buffer that isn't aligned to its type's fixed width. The reader would then call it on arrays that reference a body in place, replacing the 8-byte body-alignment check here. Does that split work for you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
