[
https://issues.apache.org/jira/browse/ARROW-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17658044#comment-17658044
]
Rok Mihevc commented on ARROW-1011:
-----------------------------------
This issue has been migrated to [issue
#16606|https://github.com/apache/arrow/issues/16606] on GitHub. Please see the
[migration documentation|https://github.com/apache/arrow/issues/14542] for
further details.
> [Format] Clarify requirements around buffer padding in validity bitmaps
> -----------------------------------------------------------------------
>
> Key: ARROW-1011
> URL: https://issues.apache.org/jira/browse/ARROW-1011
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Format
> Reporter: Wes McKinney
> Assignee: Wenchen Fan
> Priority: Major
> Fix For: 0.4.0
>
>
> This has come up in https://github.com/apache/arrow/pull/673 and in prior
> discussions.
> The basic summary is that we should not write non-zero padding bytes in IPC
> messages. However: one cannot in general rely on the padding being non-zero
> when the data is in memory (for example: zero-copy slices of Arrow
> arrays/vectors).
> I think it would be good to clarify this point in Layout.md -- namely that
> what gets written to the wire should be deterministic. However, in-memory
> algorithms should not in general expect the padding region to have a
> particular value. As an example, a popcount on a validity bitmap would want
> to exclude padding bytes from the computation. Other elementwise SIMD
> operations are free to use the padding bytes as they wish, with a known
> caveat.
> cc [~cloud_fan]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
